Dynamics of coupled vibration modes in a quantum non-linear mechanical resonator