Solving the wave equation numerically constitutes the majority of the computational cost for applications like seismic imaging and full waveform inversion. An alternative approach is to solve the frequency domain Helmholtz equation, which offers a reduction in dimensionality as it can be solved per frequency. However, challenges with the classical Helmholtz solvers such as the need to invert a large stiffness matrix can become computationally intractable for large and 3D models or high frequencies. therefore, a new approach based on physics informed neural networks paradigm have been proposed to solve the Helmholtz equation, but this method still needs further improvements. consequently, in this abstract, we study different activation functions in order to improve the convergence properties of this solution. We compare different activation functions that are regularly used in the literature, in addition to a new variant of ReLU, called swish activation function, and we find the swish offers much improved convergence properties than the other widely used activation functions.