Abstract: A six-freedom degree coupled dynamic model was proposed for gear transmission systems on the basis of tribology and gear system dynamics. In the model the wear, time-varying mesh stiffness, eccentricity mass and tooth errors of gears were taken into account to simulate the practical working condition. The nonlinear vibrations of gears were studied by applying the self-adapt and variable step method of numerical simulation. Vibration characteristics were compared between the standard and cracked teeth. Finally, the authors analyzed the effects of the gear tooth root crack on the time domain, frequency domain and time-frequency domain for the dynamic load spectra of gear transmission systems.