Converter transformer is an important part of HVDC transmission system. There is no grounding point in the AC side of converter valve, but there is grounding point in the DC side. When one phase grounded fault occurs in the valve side of converter transformer, the fault current changes irregularly with the opening and closing of the converter valves, and the sensitivity of converter transformer protection is usually low. In extreme cases, when the neutral point of Y/Y-connected transformer valve side grounded fault occurs, the protection of converter transformer itself can not trip quickly. Based on the structure and operating conditions of converter transformer, the current characteristics of valve side grounded fault are analyzed, and the characteristics of existing protection are analyzed. Based on the principle of zero sequence differential protection and fast zero sequence overcurrent protection, an optimization scheme of existing converter valve side protection is proposed. It is validated by fault record form a converter substation and simulation waveform from RTDS. The results show that the optimization scheme can effectively improve the sensitivity, operating speed and fault location accuracy of the converter transformer protection under the one phase grounded fault in the valve side.