Paraview doesn't perform ray tracing (OSPRay) correctly with MPI

[eve-le-guillou]

Hi all.

I can’t seem to install Paraview with MPI and OSPRay properly, though my pipeline executes, my output is incorrect.
I use Paraview 5.10.1.
I have installed OSPRay v2.9.0 (with Embree 3.13.3, Oidn 1.4.3, oneDNN 2.6, RKCommon 1.9.0, OpenVKL 1.2.0 and Snappy latest)
To install OPSRay with MPI, I simply put the compilation variable OSPRAY_MODULE_MPI of OSPRay to ON. This is the only depedency for which I set any MPI related compilation variable.
For Paraview, I set PARAVIEW_ENABLE_RAYTRACING to ON and VTK_ENABLE_OSPRAY to ON, as well as PARAVIEW_USE_MPI to ON.
I use OpenMPI 4.0.3.

I don’t want to use MPI for Offloading computation of OSPRay, I want to use OSPRay while computing an example in MPI.
Here is the pipeline I use:

test_ospray.py (4.8 KB)

The data file can be downloaded here: ttk-data/pegasus.vtu at dev · topology-tool-kit/ttk-data · GitHub
It is quite small (25MB).

I left the rendering options, as this is a rendering problem, but the pipeline is actually very simple:

Pegasus > RedistributeDataSet

I execute the pipeline using:

mpirun -n 8 pvbatch pipeline.py

I also tried with the pvserver:

mpirun -n 8 pvserver

paraview

And connected it to the server with the Paraview client.

Everything computes without error messages, however I obtain the following image:

As can be seen, the Ray tracing is only performed on the domain of each process, independently of each other. For example, the wing of the pegasus should put a shadow on his leg, but that is only done on the part of the leg that is contained by the process containing the wing.
The Raytracing of one process to another is not performed here.
However, when I click on the Pegasus and move it around, the shading is indeed performed correctly as shown here.

When I release the mouse, the error of Raytracing comes back again.

I tried setting the following variables:

export OSPRAY_DEFAULT_DEVICE="mpi"
export OSPRAY_LOAD_MODULES="mpi"

But that won’t change anything.
I tried using the variable OSPRAY_DEVICE, setting it to mpiOffload or mpiDistributed, but that causes Paraview to crash, so I am thinking that this might not be the way and it also seems to be for distributing OSPRay computation and Paraview computation.

I don’t know what else to try. I haven’t been able to find a step by step documentation for this specific case for Paraview.

Thanks for any help.

[mwestphal (Mathieu Westphal (Kitware))]

Can you try with the binary release of ParaView ?
https://www.paraview.org/download/

[eve-le-guillou]

I just did with Paraview 5.11.0 and it works perfectly. Paraview 5.10.1 conflicts with my personal installation.

[Dave_DeMarle (Dave DeMarle (Intel))]

I’m very surprised that it works with 5.11.

Shadow, reflection, refraction and ambient occlusion effects need non-local information to be calculated correctly. Ie a camera ray hits the leg on one rank, then a secondary ray is sent toward the light on that same rank. It should hit the wing and cause a shadow, but the wing is on a different rank so it misses and the leg it gets full luminance.

There are research projects out there, the most recent is TACC’s GraviT ( Gravit - Texas Advanced Computing Center (utexas.edu)) that send either rays or data around the network to solve the problem, but none of them have yet been integrated into ParaView for either OSPRay or OptiX.

Until that happens, my general recommendation is to keep all of these effects off when using ParaView at scale. ParaView with Sandia’s IceT ( IceT – Center for Computing Research (CCR) (sandia.gov)) can (or at least should - we are looking at a couple of reported bugs with them) depth composite primary rays the same way it does for rasterized images.

[eve-le-guillou]

Understood, thank you for answering me.