In recent years, the frequent occurrence of extreme heavy rainfall and drought events has intensified the uncertainty changes of hydrological processes in the basin, making it more difficult for the prediction of medium and long-term runoff in the basin. In order to improve the ability of LSTM (long short-term memory) model in capturing and fitting temporal changes in runoff, taking the Boyang River Basin as the study area, we collected the data of monthly rainfall, evaporation and flow for the simulation. Then, VMD (variational modal decomposition) and correlation testing are used to eliminate the interference of irrelevant frequency components on the regular learning of LSTM model for the purpose of model input optimization. In addition, the effects of different coupling methods between VMD and LSTM model on the accuracy and stability of the model are considered, by which the VMD-LSTM monthly runoff coupled model which takes the advantages of both is finally selected. The simulation results show that the VMD-LSTM coupled model can significantly improve the simulation accuracy but it is lacking in model stability, the VMD-LSTM coupled model based on correlation testing can not only further improve the model accuracy, but also improves the stability of the model. In the comparison of the different coupling methods of the VMD-LSTM coupled model based on the correlation testing, scheme D1 which decomposes both inputs and outputs with VMD and optimizes inputs has the best simulation accuracy and stability, with an NSE value of 0.98 and above for 60 simulations. In addition, when analyzing the interpretability of scheme D1, it is found that historical runoff has a greater impact on the LSTM model than rainfall and evaporation. The results of this study can provide accurate and reliable medium and long-term runoff simulations for the water resources management in the basin.