Aimed at the mortar materials in masonry structure,the paper thought about the influences of 4 kinds of strength grade and 11 kinds of shear-compressing stress state.It used the self-designed simple apparatus to conduct the damage test for mortar prism specimen.Based on the test results, the failure of specimens is classified into 3 types such as shearing, shear-compression couple and compression. Through measuring the shear strength of mortar prism specimen under different normal stresses, the paper obtained the failure criterion of mortar under the shear-compression stress state by non dimensional treatment. The failure criterion was used in the discrete element model of masonry structure,the paper simulated the shear-compression test of brick masonry specimen.The simulated results are in good agreement with the test results. The research means and result can provide the basic data for the numerical simulation of masonry structure based on discrete element method.