The application of optimization algorithms with high accuracy is very important for improving the accuracy of the prediction model for landslide displacement; however, the research on the comparison of different optimization algorithms is rarely reported. Here, the Bazimen landslide in the Three Gorges Reservoir area was taken as the example, and the extreme learning machine (ELM) model was used to predict the landslide displacement. Meanwhile, multiple algorithms were used to optimize the parameters in the modelling process to improve the prediction accuracy. In order to improve the prediction accuracy, based on the moving average method, the landslide displacement was decomposed into two phases, which were trend term and periodic term displacements. The trend term displacement was predicted by a polynomial function, and the ELM model that was completed by MATLAB code was used to predict the periodic term displacement. Finally, the trend and periodic displacements were summed up as the predicted total displacement. The results showed that ELM model could accurately predict the cumulative landslide displacement with a step-like curve, the average error of the prediction results was 23.5 mm and the goodness of fit was 0.973. Compared with particle swarm optimization and genetic algorithm, the ant colony optimization (ACO) performed better on computational time and calculation result. Hence, the extreme learning machine model optimized by ant colony algorithm had the best accuracy, with the average error of 10.1 mm and goodness of fit of 0.998. So, this novel model is applicable for the displacement prediction of similar landslides.