Daytime radiative cooling has attracted extensive research interest due to its potential impact for energy sustainability. To achieve subambient radiative cooling during the daytime, a white surface that strongly scatters incident solar light is normally desired. However, in many practical applications (e.g., roofing materials and car coatings), colored surfaces are more popular. Because of this, there is a strong desire to develop colorful surfaces for radiative cooling. We summarize the general design criteria of radiative cooling materials with different colors and discuss the limitations in cooling performance. Major efforts on this specific topic are reviewed with some suggested topics for future investigation.