It is possible to replace the IceT communicator to use something other than MPI. IceT has its own abstraction for a communicator, and you can write one that does not use MPI.
The reason why the one in ParaView/Catalyst uses MPI directly rather than vtkCommunicator is that IceT requires asynchronous sends and receives (i.e. ISend and IRecv) and vtkCommunicator does not support these.
If MochiCommunicator supports asynchronous sends/receives, it should be possible to have the IceT library use it.