Declaration of ghost cells for rectilinear grid

(Bastian) #1

I looked at just about any Catalyst example I could find that demonstrates the handling of ghost cells for (though I couldn’t find any for rectilinear grids), and still I can’t get it to function.

Could anyone help me out here, I just can’t see what I am doing wrong.

I have a Fortran code where call createcpdata(gis,gie,gjs,gje,gks,gke,nxb,nyb,nzb,xgrid,ygrid,zgrid) inside FEFortranAdaptor.F90 (1.8 KB)
has the grid defined.

createcpdata is defined in FECxx.cxx (4.3 KB).
This function contains a section where ghost cells are marked:

    // Mark ghost cells
    vtkNew<vtkUnsignedCharArray> ghostCells;
    ghostCells->SetNumberOfTuples(grid->GetNumberOfCells());
    ghostCells->SetName(vtkDataSetAttributes::GhostArrayName());
    ghostCells->Fill(0);
    grid->GetCellData()->AddArray(ghostCells);
    
    vtkIdType cellId = 0;
    for (int k = *ks; k < *ke+1; k++)
    {
      bool zGhosts = ((k == *ks) || (k == *ke)) && (k > 0) && (k < *nz-1);
      for (int j = *js; j < *je+1; j++)
      {
        bool yGhosts = ((j == *js) || (j == *je)) && (j > 0) && (j < *ny-1);
        for (int i = *is; i < *ie+1; i++)
        {
          bool xGhosts = ((i == *is) || (i == *ie)) && (i > 0) && (i < *nx-1);
          if (xGhosts || yGhosts || zGhosts)
          {
            cout << "ghost: (" << i << ", " << j << ", " << k << ")\n";
            ghostCells->SetValue(cellId, ghostCells->GetValue(cellId) | vtkDataSetAttributes::DUPLICATECELL); 
          }
          cellId++;
        } 
      }   
    }

When running in serial, there are no ghost cells and the line ghostCells->SetValue(...) is never reached.
I can do live visualization and have the coprocessor script write .pvtr/.vtr files which I can load in Paraview without any issue.

When running in parallel cells are marked as ghostcells in the above section. I verified that the right cells are marked. (Subdomains overlap by two layers of cells due to one layer of ghost cells.)
When doing live visualization Paraview crashes due to a Segmentation fault.
I can still have a set of .pvtr/.vtr files written and load the .pvtr file in Paraview. However, when slicing the field data I am confronted with a gap shown in the screenshot on the left. One x-normal layer of data is not displayed.
When loading the two .vtr files individually, this gap is not present (right).

Another mystery:
Ghost cells are no longer declared as such in any succeeding time step, i.e. where the .pvtr file read

    <PCellData>
      <PDataArray type="UInt8" Name="vtkGhostType"/>
      <PDataArray type="Float64" Name="pressure"/>
    </PCellData>

the very first time it was produced, it then reads

    <PCellData>
      <PDataArray type="Float64" Name="pressure"/>
    </PCellData>

in all the succeeding time steps.

Strange values when adding cell data to grid
(Bastian) #3

As I wrote above, I declare ghost cells in the function createcpdata_. This information is taken into account the very first time that coprocess() is called. Accordingly, when loading the vtr files that I have my coprocessing script generate into paraview, ghost cells are hidden (left), however and to my surprise the pvtr file (right) is not represented correctly:
Slices of both .vtr files Slice of .pvtr file

When calling the coprocessor In any later time step, ghost cells appear to be ignored entirely. This becomes clear when loading the individual vtr files (left), but that does not appear to bother paraview when loading the corresponding pvtr file (right):
Slices of both .vtr files Slices of both .vtr files

When doing live visualization I generally have the second situation past the first time step and here the Paraview GUI always crashes, stating that subdomains cannot be merged.

If it is possible to have rectilinear grids with ghost cells in Catalyst please let me know.

(Mathieu Westphal (Kitware)) #4

Have you tried reproducing this issue without catalyst in a parallel paraview ?

(Bastian) #5

When running a pvserver on two cores and connecting my local paraview-GUI, I get the same output as in my post above.

(Mathieu Westphal (Kitware)) #6

Could you share a statefile and associated dataset to reproduce ?

(Bastian) #7

Here is an archive containing a Paraview 5.5.2 state file and (p)vtr files produced in the first and second time step. The state file produces the representation included in my posts above.

demo_catalyst_output.zip (21.5 KB)

(Mathieu Westphal (Kitware)) #8

Hi Bastian.

Sorry it took me so long to look into this.
This is caused by a bug in the vtkXMLRectilinearGridReader.

The reader actually repartition your data and do not keep the data the way it was partitioned. However it keep the vtkGhostType array the way it was, resulting in the issue.

This bug should be fixed, but, you actually do not need to provide the ghost cells, as they are computed on the fly when ParaView need them !

So my advice would be to just not generate GhostCells at all !

(Bastian) #9

Dear Mathieu,

Unfortunately this does not work. The Paraview client that is connected to Catalyst then crashes and Catalyst issues the following Warnings/Error messages:

Generic Warning: In /dev/shm/easybuild-build/ParaView/5.5.2/intel-2018a-mpi/ParaView-v5.5.2/ParaViewCore/VTKExtensions/Core/vtkMultiProcessControllerHelper.cxx, line 171
vtkRectilinearGrid cannot be merged

ERROR: In /dev/shm/easybuild-build/ParaView/5.5.2/intel-2018a-mpi/ParaView-v5.5.2/VTK/Parallel/Core/vtkSocketCommunicator.cxx, line 808
vtkSocketCommunicator (0x56199a0): Could not receive tag. 1

ERROR: In /dev/shm/easybuild-build/ParaView/5.5.2/intel-2018a-mpi/ParaView-v5.5.2/VTK/Parallel/Core/vtkMultiProcessController.cxx, line 666
vtkSocketController (0x561a570): Could not receive RMI trigger message.

At the same time Catalyst does not mind writing .pvtr that can indeed be viewed in Paraview and that look fine, but again, Live Visualization is not possible.

(Bastian) #10

Since there appears to be a bug in the vtkRectilinearGrid that prohibits the use of ghost cells, I would instead just use another grid class then.

The Catalyst examples folder is loaded with FEAdaptor.cxx files that use vtkUnstructuredGrid but none that use vtkStructuredGrid. Before going through the trouble of figuring out how to implement coprocessing with vtkStructuredGrid could anyone confirm that this class supports ghost cells?
Are there any sample files out there?

I am reluctant to use vtkUnstructuredGrid before ruling out vtkStructuredGrid.