Establishing a safety monitoring model to analyze and forecast the deformation and displacement of the dam is of great significance for its safe operation. The monitoring model of BP neural networks has the disadvantages of complicated operation and slow convergence, and it is easy to fall into localized optimazition. These drawbacks will lead to inaccurate expression and prediction of the dam operation situation. In order to solve these problems, the ant colony optimization (ACO) algorithm was introduced to BP networks to search for optimal solution of parameters at overall situation, as well as to obtain the dam deformation prediction data through training the sample data by BP neural networks. The engineering case study indicates that the ACO-BP network model is easier to converge in parameter optimization than that of the BP network with less errors and good prediction performance, which can provide a new approach of nonlinear modeling and simulation for dam deformation displacement monitoring and safety prediction.