Quantum nonlinear dynamics of optomechanical systems in the strong-coupling regime