Like @mwestphal says, ParaView has a built in cache for the animation, but to enable it you first have to go to settings and turn on the Cache Geometry For Animation feature. To be clear, you first have to hit play 
and let ParaView cycle through each of the time steps. As it visits each time step, ParaView will cache the rendered geometry of your data. After ParaView goes through that once, you can then switch between time steps without having to reload the data.
One word of warning, the animation cache does not actually save the data that you are loading from the file directly. Rather, it is saving the geometry at the end of the pipeline that is being rendered. That means if you make a change to your visualization, such as adding a filter, the cache will get cleared and ParaView will load the data from file all over again. So, in your case I would recommend the other option @mwestphal suggested, the Temporal Cache filter.
After you open your data in ParaView, add the Temporal Cache filter to the reader. Set the Cache Size to the number of time steps you have (in your case, 4). Once again, you have to hit play or otherwise let ParaView visit each time step. But once you do so, ParaView will hold the data for the time step in memory.