Cosmic carnage II: the evolution of the galaxy stellar mass function in observations and galaxy formation models