In view of the existing signal processing methods can not effectively solve the nonlinear and aliasing problems of low-frequency oscillation signals in power system, an improved variational mode decomposition (VMD) method is introduced into the pattern recognition of low frequency oscillation in this paper. Moreover, sample entropy and fast Fourier transform (FFT) are used to solve the problem of insufficient adaptive ability of VMD. The original signal is decomposed into several mode components by IVMD method. Then, Teager-Kaiser energy operator(TKEO) is applied on the fitting of each component to get the amplitude, frequency and damping of it. By the constructed test signal, the method of this paper is compared with VMD, empirical mode decomposition (EMD), total least squares-estimation of signal parameters via rotational invariance techniques (TLS-ESPRIT), and Prony on the performance of mode parameter identification. Results show that the IVMD method effectively overcomes the shortcomings of EMD, TLS-ESPRIT and Prony in dealing with mode mixing, noise sequence and non-stationary signals. Finally, the feasibility of the method of this paper in extracting the low frequency oscillation mode parameters of power system is verified by the simulation signal identification of the IEEE two-area four-generator power system and the New England 39-bus system.