My guess is that the data bounding box is changing over time. Can you check if that is the case?
Changing data bounds would expose the wrinkle that VTK currently uses a heuristic of scaling the implicit spheres and cylinders that the ray tracers draw for point and line primitives by the scene bounding box.
I will push an option to turn off the heuristic as well soon. In the meantime a workaround could be to place a fixed size wireframe bounding box around the scene but oriented out of view.