Asphalt-aggregate adhesion is a key influencing factor for the water stability of asphalt mixtures. In order to quantitatively analyze the adhesion between asphalt and aggregates, surface free energy theory and improved boiling method were used for the evaluation. First, the surface energy parameters of asphalt, modified asphalt, aged asphalt, granite, limestone and diabase were measured at 25℃ by contact angle measurement test, and the adhesion and exfoliation work of different asphalt-aggregate systems were calculated. The absolute ratio ER between the work of adhesion and the work of exfoliation was used as the evaluation index of adhesion, and the quality loss rate F obtained by improved boiling method as the quantitative index to verify the test results of the surface energy theory. The results show that the surface energy parameters of 90A asphalt mixed with modifiers reduced by 2.5% to 6.2%, the water stability index ER of the modified asphalt-aggregate system increased by 1.9% to 7.8%, so the adhesion increased, of which the exfoliation agent presented the best improvement effect. However, the ER value of aged asphalt decreased by 4.3% to 9.2%, resulting in the decrease of the adhesion. The adhesion of systems with the same asphalt but different aggregates were ranked as follows: asphalt-limestone>asphalt-diabase>asphalt-granite; whereas that of different asphalt with the same aggregate were ranked as follows: 90PA>90HC>90C≈90H>90A. The adhesion grades obtained by the improved boiling method with prolonged boiling time were significantly differentiated, and the results were consistent with those of the surface energy evaluation. The evaluation index ER had a linear negative correlation with the mass loss rate F, which further verified the reliability of the surface energy evaluation system.