Block stone fillers are commonly used for the paving layer of the road. To study the vibration compaction process and effect of the paving layer with block stone fillers, the vibration compaction tests were conducted by discrete element method under two loading conditions of “slow small vibration force” and “fast large vibration force”. The particle displacement, stress response, inter-particle void ratio, and compaction process of block stone fillers under the two loading conditions were compared and analyzed. The results show that increasing the number of compaction passes can significantly increase the particle displacement and reduce the inter-particle void ratio, indicating that the surface settlement rate of the paving layer has a strong linear correlation with the inter-particle void ratio. The effect of compaction energy (such as particle displacement, reduction degree of void ratio, peak dynamic stress, and increment of static soil pressure) decreases rapidly with the increase of the depth. The compaction energy mainly propagates in the vertical direction, so the paving layer can be divided into three zones along the depth, namely, the main compaction zone, the secondary compaction zone, and the weak influence zone. The content of large-sized block stone particles significantly affects the contact structure characteristics, thereby influencing the compaction effect and compaction process. The compaction process of block stone particles is manifested as the combined effects of squeezing compaction, moving compaction, and filling compaction. Under the continuous vibration compaction load, the contact structure characteristics and particle aggregation state of block stone particles will gradually change from “single-point contact” to “multi-point contact”, and from “loose” to “compact”. The research results can provide a preliminary theoretical basis for the determination of maximum paving layer thickness and selection of roller parameters in engineering practice.