Nonlinear dynamics of three-magnon process driven by ferromagnetic resonance in yttrium iron garnet