Color-Magnitude Relations of Active and Non-Active Galaxies in the Chandra Deep Fields: High-Redshift Constraints and Stellar-Mass Selection Effects