Categorical color map

Hi ParaView developers,
I have a probem with a categorical color table:
I load two 3D volume datasets which are regular grids (nifti datasets), the first is an MR image and the second contains integer values which indicate region labels. Both are in the same space (as defined by origin, voxel size, etc.). Then I use a contour filter to create an isosurface of the first dataset, and colorize it by the second dataset using a categorical color map. This works, however at the borders of several regions small invalid spots are created, in particular when the labels of neighbouring regions have large differences.
I assume, this is because the integer values of the second dataset are linearly interpolated in the surface vertices, yielding invalid region labels. On the other hands, the transitions between regions which have labels that differ only by one are ok.

A solution would be, that the values of the label dataset are calculated by a nearest neighbour interpolation in the surface vertices. Please could you explain, how this could be achieved?
Thank you very much in advance!