Evaluating the excavation stability is important in excavation support design．In this paper, the stability of excavations is studied by combining the upper bound theorem, finite element discretization and mathematical programming method. First, the triangular finite element is used to disperse the soil of excavations. Then according to the upper bound theorem, kinematically admissible velocity fields are established to satisfy the plastic flow constraint conditions of velocity discontinuities for common side, the plastic flow constraint conditions of triangular finite elements and the velocity boundary conditions. The objective function is established based on the internal and external energy balance equation, then the upper bound method mathematical programming model for excavations analysis are established. By solving the mathematical programming model with the help of optimization algorithm, the upper bound solution of the ultimate load and safety factor can be obtained. Finally, the effects of soil shear parameters, the depth of retaining structure and excavation depth on the overall stability of excavations are analyzed.