Quantifying phonon-induced non-Markovianity in color centers in diamond