With the continuous expansion of power grid scale, the phenomenon of subsynchronous oscillation caused by series compensated long-distance transmission system occurs frequently. Therefore, in order to analyze the deep reasons for the instability of the series compensated transmission system, based on the open-loop mode resonance theory, an open-loop mode resonance analysis method is proposed to analyze the subsynchronous oscillation caused by the series compensated transmission line. The small signal model of the external series compensated transmission line of permanent magnet synchronous generator (PMSG) farm is established, and the dynamic interaction between the series compensated line and the PMSG farm is studied through the open-loop mode resonance analysis method. The results show that the closed-loop stability of the system decreases when the open-loop subsynchronous oscillation mode dominated by series compensation is close to the open-loop subsynchronous oscillation mode dominated by wind farm in the complex plane. Using residual theory can successfully predict the position of closed-loop mode under the condition of open-loop mode resonance, while changing PMSG parameters or adjusting series compensation can avoid the occurrence of dynamic interaction. The correctness of the above theoretical analysis is verified by an example system in this paper. It is proved that there is a certain risk of instability when the PMSG farm transmits power through the series compensation transmission line, which can provide a reference for the parameter setting of the series compensation transmission system to a certain extent.