Automatic identification system (AIS) is a maritime navigation safety communication system that operates in the very high frequency (VHF) mobile band and was developed primarily for collision avoidance. Since the height of the antenna cannot be extended indefinitely, communication distance of the AIS is usually within the horizon. However, with the increasing importance of the AIS traffic, this line-of-sight propagation mechanism cannot meet the need to monitor shipping at a longer distance. When AIS signal is propagating in the atmospheric duct, the trapping effect allows it to propagate over-the-horizon. In this work, the parabolic equation method is utilized to calculate the propagation loss of AIS signals in ducting channel and the simulation results show that AIS can perform over-the-horizon propagation. In order to obtain the distribution of atmospheric refractivity profile over the sea surface, a new inversion method based on the AIS signal level is proposed. The particle swarm optimization (PSO) algorithm is selected to minimize the objective function for obtaining the atmospheric refractivity profile in the maritime environment. Numerical simulations are presented to validate this method for refractivity profile estimation, which provides a theoretical basis and effective support for the applications of over-the-horizon communication and radar over-the-horizon detection.