For the Text source and Python Annotation, in the GUI it’s under Font Properties. In the Python API there’s an attribute called Color. You can see how to manipulate it through using the Python Trace functionality in the GUI.
Yeah, that’s why the Box source or Plane source would be clunky. You’d have to manipulate it all the time to get what you want. Maybe a Programmable Filter would be slightly better but these are all really just less clunky workarounds because they’re not putting it in a view location which is really what you want.