About warning when going out of memory

Development
1

Im trying to find a acceptable way to warn a user that adding a filter may cause his server node to go out of memory.

Of course, it is out of the question to try to precompute the memory usage before pressing Apply, but I think we can still provide some tools to go in this direction.

I’m imagining a new XML hints to add to certain filters that we may consider impactful in terms of memory (especially filters converting from structured grid to unstructured grid).

Here is the synstax I envision:

<WarnOnCreation>
  <Conditions>
    <InputDataType>
        <DataType value="vtkRectilinearGrid" />
        <DataType value="vtkImageData" />
    </InputDataType>
    <MemoryUsage relative=10 absolute=30>
  </Conditions>
  <Text> Watch out, ParaView may explode </Text>
</WarnOnCreation>

The conditions are AND,
The InputDataType let you specify for which dataset type in input to warn, OR
The memory usage can be used in two ways,
relative, where the provided number is a threshold in %, if the input size / remaining memory is >, then it warn.
absoulte, where the provided number is a threshold in %, if the remaining memory / available memory is <, then it warn.

The text is of course the text shown in the warning, but we could consider having a default for it.

The warning will be similar to the Volume Rendering warnings, where the user can decide to never see that warning again (for this specific filter).

This will not be bullet proof of course, we may warn when it is not needed and we may not warn before running out of memory, but this would still be an improvement imo.

What do you think @cory.quammen @nicolas.vuaille @Francois_Mazen (Feel free to tag whoever) ?

2

Hello @mwestphal, nice proposal!

The memory warning makes sense. However finding a correct “guess” of the size of output based on the input size looks hard or impossible, but I can’t imagine other way to express it at the moment.
So I agree with your proposition, maybe the “absolute” warning should be a global option like always warn the user when available memory is less than 10%.

If we go this way, <MemoryUsage relative=10 absolute=30> is not intuitive to me, especially the absolute tag which sound like an amount of memory instead of a ratio. (First, I read it like “when it remains 30MB of memory”).

An other approach would be to handle the std::bad_alloc exceptions in a clean way, and not crash the application. No sure if this is feasible at the moment, maybe with the next gen of ParaView with asynchronous calls?

3

You are not supposed to use both, only relative or absolute, however the terms are probably not the right ones.

Here are more complete explanation of what I mean in terms of feature.

  • relative=N

Before creating the filter, ParaView will check the size of the input using GetMemorySize (inputSize) as well as checking the remaining ram in the most busy server node (remainingRam). If inputSize/remainingRam > N, ParaView will warn.

  • absolute=N

This term probably not the best one, here is what I mean.
Before creating the filter, ParaView will check the size the remaining ram in the most busy server node (remainingRam) and the size of the complete ram (complateRam). If remainingRam/completeRam > N, ParaView will warn.

This is still relative, but to the RAM only, not to the data. In any case, I’m not sure this mode is actually usefull. RAM usage is really dependant of input size for any filter for this mode to actually make sense.

An other approach would be to handle the std::bad_alloc exceptions in a clean way, and not crash the application. No sure if this is feasible at the moment, maybe with the next gen of ParaView with asynchronous calls?

Just out of the question with the way VTK filters and readers are implemented. Sure, you can catch bad_alloc in the layer SI layer, but then what do you do ? You have no way to recover and resume.
It should be implemented in each filter/reader directly. Not feasible. And even in that case, you end with 99.99% of memory used.

The memory warning makes sense. However finding a correct “guess” of the size of output based on the input size looks hard or impossible.

The only way to do that is to force every single filter to predict how much memory they will need. GetMaxMemoryUsage(size_t input), but of course out of scope.

4

I tried to handle this on windows a while back in an app that loaded photos - so the caveat is that we had a reasonable idea of what the next memory request might be.

When we loaded the program, we just newed a large memory chunk and held it. When we got the std::bad_alloc exception, we put up a warning saying “save your work, you might be about to run out of memory”, and deleted the chunk. It at least gave a chance that the save operation could complete successfully.

I’m doubtful that this would work for ParaView, but it might be applicable in some limited circumstances.

5

Two orthogonal thoughts.

  • I don’t think this is possible with the simple logic here. An example is if you have a structured (rectilinear) grid, and clip it, the output is unstructured. The memory footprint is HUGE. Maybe you took care of that above, but I suspect it will be wickedly complex getting the matrix correct.
  • I think you are missing the biggest issue. The main problem isn’t adding a new file, but rather that the memory inspector shows memory footprint not as work is being done, but the quiescent state after work is done. In other words, moving forward in time tends to grow memory, use that memory, then release that memory, and finally we figure out how much memory is used. The hump in memory use above is an issue.
6

The hint is on an filter by filter basis and let the dev specify on which input type to warn, so the case you mention will be covered.

The hump in memory use above is an issue.

I agree but I dont see how I’m not adressing this issue. The memory inspector cannot show anything during the filter computation anyway.

7

Hmmm… I agree on both counts.

8

The implementation is ongoing with the following syntax:

      <Hints>
        <WarnOnCreate>
          <DataTypeDomain name="input_type">
            <DataType value="vtkImageData" />
            <DataType value="vtkRectilinearGrid" />
            <DataType value="vtkStructuredGrid" />
          </DataTypeDomain>
          <MemoryUsage relative="0.5" />
          <Text title="Potentially running out of memory">
            **Tensor Glyph** filter will create a polydata output and
            you may not have enough memory to create it.
            Do you want to continue?
          </Text>
        </WarnOnCreate>
      </Hints>

I’ve added warnings for the following filters:

  • AppendDataSet
  • Connectivity
  • Delaunay 3D
  • Tensor GLyph
  • Extract Edges
  • Extract Cells by Region
  • Tetrahedralize
  • Shrink
  • Image Data to Point Set
  • Tessallate
  • D3
  • Reflect
  • Redistribute DataSet
  • Convert To Point Cloud
  • Extract Selection
  • Clean to Grid
  • IsoVolume
  • Contour
  • Glyph
  • Linear Cell Extrusion
  • Clip
  • Slice
  • Threshold

Of course, each WarnOnCreate conditions have been adapted to the filter after some testing.

That being said, this may not be complete, so if anyone thinks another filter should warn too, let me know, especially in regards to filter that use lots of temp memory but then release it, as this is a usecase that was hard to evaluate locally (@wascott) .

https://gitlab.kitware.com/paraview/paraview/-/merge_requests/5994