You can mostly do this with the Selection Display Inspector (available under the view menu), but it requires that you select points or cells first. Then you can label the selected points/cells with data array values. The labels won’t be sparsified as you zoom, out though, so it’s not exactly what you are requesting.
It may be the case that .vtk (the so-called legacy format) supports string arrays. The XML file formats, do, though.