Questions recording test case

I ask the question here, as the topic might be of interest also for others and it is easier to find here than in merge request commentary.

I try to create a test case for a reader plugin. I orientate myself with the GmshIO reader which has a test. From that example it is also clear how a file is loaded. I recorded loading and visualizing data and come to a point, where my screen shows the following content.

How do I continue? Thanks!

BTW: This is stuff, where a youtube-video would be really cool :slight_smile:

What is the question here ?

BTW: This is stuff, where a youtube-video would be really cool

We do have a few webinars ! Please take a look: https://www.paraview.org/webinars/

I found no test case recording and playing in the webinars, did I miss it?

I guess I need to first record loading the plugin, as this is what I need to do when I start paraview.

What I do is:

  • start paraview with --dr
  • start recording a test
  • load the plugin
  • load my file
  • stop recording.

The recording is

<?xml version="1.0" ?>
<pqevents>
  <pqevent object="pqClientMainWindow/PluginManagerDialog/splitter/localGroup/localPlugins" command="setCurrent" arguments="vtkPVInitializerPlugin" />
  <pqevent object="pqClientMainWindow/PluginManagerDialog/splitter/localGroup/localPlugins" command="setCurrent" arguments="CFSReader" />
  <pqevent object="pqClientMainWindow/PluginManagerDialog/splitter/localGroup/localPlugins" command="expand" arguments="5.0" />
  <pqevent object="pqClientMainWindow/PluginManagerDialog/splitter/localGroup/loadSelected_Local" command="activate" arguments="" />
  <pqevent object="pqClientMainWindow/PluginManagerDialog/buttonBox/1QPushButton0" command="activate" arguments="" />
  <pqevent object="pqClientMainWindow/FileOpenDialog" command="filesSelected" arguments="/Users/fwein/project" />
  <pqevent object="pqClientMainWindow/FileOpenDialog" command="filesSelected" arguments="/Users/fwein/project/simp" />
  <pqevent object="pqClientMainWindow/FileOpenDialog" command="filesSelected" arguments="/Users/fwein/project/simp/mech.cfs" />
</pqevents>

But when I run the test, I get

Process started
17:23:06 : 0: Test (0): Event: /localPlugins: setCurrent : vtkPVInitializerPlugin
critical: In unknown, line 0
critical: In event 'object=pqClientMainWindow/PluginManagerDialog/splitter/localGroup/localPlugins' 'command=setCurrent' 'arguments=vtkPVInitializerPlugin':

Couldn't find object  `pqClientMainWindow/PluginManagerDialog/splitter/localGroup/localPlugins`
Found up to           `pqClientMainWindow`
    Available widget: `pqClientMainWindow/_layout`
    Available widget: `pqClientMainWindow/menubar`
    Available widget: `pqClientMainWindow/menubar/qt_menubar_ext_button`
    Available widget: `pqClientMainWindow/menubar/menu_File`
    Available widget: `pqClientMainWindow/menubar/menu_File/QAction0`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionServerConnect`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionServerConnect/pqServerConnectReaction0`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionServerDisconnect`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionServerDisconnect/pqServerDisconnectReaction0`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionFileSaveScreenshot`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionFileSaveScreenshot/pqSaveScreenshotReaction0`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionFileSaveAnimation`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionFileSaveAnimation/pqSaveAnimationReaction0`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionFileSaveData`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionFileSaveData/pqSaveDataReaction0`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionFileLoadServerState`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionFileLoadServerState/pqLoadStateReaction0`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionFileSaveServerState`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionFileSaveServerState/pqSaveStateReaction0`
    Available widget: `pqClientMainWindow/menubar/menu_File/actionFileSaveCatalystState`
    Available widget: .... (and 3869 more!)
    Set PQOBJECTNAMING_MATCH_LIMIT environment var to a +'ve number to limit entries (or 0 for unlimited).

17:23:06 : 0: Done

Oh, on Mac, because the menu is separated from the main window, menu events are not recorded. I recorded opening the plugin-manager on Linux and got:

  <pqevent object="pqClientMainWindow/menubar" command="activate" arguments="menuTools" />
  <pqevent object="pqClientMainWindow/menubar/menuTools" command="activate" arguments="actionManage_Plugins" />

If you add that to the top of your test, it should work.

Thank you, that brought me further!

With hard coded filenames it works!

When I look into GmshIOTest I see for opening the file
“$PARAVIEW_DATA_ROOT/Plugins/GmshIO/Testing/Data/Gmsh/hemisphere.msh”
and for the comparison
$PARAVIEW_DATA_ROOT/Plugins/GmshIO/Testing/Data/Baseline/GmshIOTestA.png

Both variables seem not to to work for me. Shall I define them in my system environment?

Well, I simply tried and it worked!

Even when I set the viewport to 400x400 (by the test) the screenshot was 800x800 - this is probably due to my Retina MacBook? I got an error about different image size. When I scale my screenshot offline to 400x400 I get

Using View API for capture

<DartMeasurement name="ImageError" type="numeric/double">0</DartMeasurement><DartMeasurement name="BaselineImage" type="text/string">Standard</DartMeasurement><DartMeasurement name="WallTime" type="numeric/double">0.00630999</DartMeasurement>

<DartMeasurement name="CPUTime" type="numeric/double">0.006321</DartMeasurement>

debug: In unknown, line 0

debug: Test "CFSReaderTest.xml" is finished. Success = true

I assume <DartMeasurement name="ImageError" type="numeric/double">0</DartMeasurement> just means no error?!

If this is true I just need to find out, how to handle my test data (200KB, cannot easily be smaller) and the reference image (7 KB).

Yes, $PARAVIEW_DATA_ROOT is define by ctest, you can see what it’s doing by running with ctest -V

ParaView handled testing data and baseline images using VTK’s external-data mechanism. You can see the develop docs and the vtk data docs. Are you intending your plugin to be part of the ParaView repo?

Yes, merging it is my great hope :slight_smile:

Not how to test a plugin, there is many test example in the existing plugins, you may want to take a look

eg: https://gitlab.kitware.com/paraview/paraview/-/tree/master/Plugins/GmshIO/Testing

With the hint of the to be added command on macOS the topic “recording a test case” is solved in this context. Thanks.

With respect to the MR of the CFSReader plugin, I guess it is better to go back to the MR-discussion?!
https://gitlab.kitware.com/paraview/paraview/-/merge_requests/5635

As a comment when creating a plugin test case:
In addition to https://gitlab.kitware.com/vtk/vtk/-/blob/master/Documentation/dev/git/data.md on shall not forget in the plugin’s CMakeLists.txt a

if (BUILD_TESTING)
  add_subdirectory(Testing)
endif ()

I could not make the GmshIO example run, as both make GmshIO and ctest -R GmshIO fail. However, it works for GMVReader.

My test was running, when I run the recording manually. As a test case with external data I have an issue.
I have

ExternalData_Expand_Arguments(ParaViewData _
    "DATA{${CMAKE_CURRENT_SOURCE_DIR}/Data/openCFS/Single_Region_Opt.cfs}"

    "DATA{${CMAKE_CURRENT_SOURCE_DIR}/Data/Baseline/Single_Region_Opt_last_PSD.png}"
  )

But that gave me the following error on make

SHA512/106137f13bae250b692fb8f6f221de6821fb66ac01ae6b27b371dde0431d6a3953cea8f09f606e4fd01053dd47a7873f82f070f28444b5613e1ed6700b34f182
-- Linked Plugins/CFSReader/Testing/Data/Baseline/Single_Region_Opt_last_PSD.png.sha512 to ExternalData SHA512/09b9b43589921733ac06af33f71b1230c4118e99f4ba641c3d49accf27af37bf7684b5f247157e03395f0e762f39f78330c9bf083e9bd110e384186a116e29a6
CMake Warning (dev) at /opt/homebrew/Cellar/cmake/3.22.2/share/cmake/Modules/ExternalData.cmake:738 (message):
  Data file referenced by argument

    DATA{/Users/fwein/code/pv_10/paraview_cfsreader/Plugins/CFSReader/Testing/Data/Baseline/CFSReaderTest.png}

  corresponds to source tree path

    Plugins/CFSReader/Testing/Data/Baseline/CFSReaderTest.png

  that does not exist as a file (with or without an extension)!
Call Stack (most recent call first):
  /opt/homebrew/Cellar/cmake/3.22.2/share/cmake/Modules/ExternalData.cmake:542 (_ExternalData_arg)
  /opt/homebrew/Cellar/cmake/3.22.2/share/cmake/Modules/ExternalData.cmake:355 (ExternalData_expand_arguments)
  CMake/ParaViewTesting.cmake:232 (ExternalData_add_test)
  CMake/ParaViewTesting.cmake:289 (_paraview_add_tests)
  Plugins/CFSReader/Testing/CMakeLists.txt:13 (paraview_add_client_tests)

An this Plugins/CFSReader/Testing/Data/Baseline/CFSReaderTest.png instead of Data/Baseline/Single_Region_Opt_last_PSD.png seems to be problem.

When I run the test, I see, that everything works, but I get the error

80: debug: Test "CFSReaderTest.xml" is finished. Success = true

80: <DartMeasurement name="ImageError" type="numeric/double">0</DartMeasurement><DartMeasurement name="BaselineImage" type="text/string">Standard</DartMeasurement><DartMeasurement name="WallTime" type="numeric/double">0.00658298</DartMeasurement>

80: <DartMeasurement name="CPUTime" type="numeric/double">0.006587</DartMeasurement>

80: <DartMeasurement name="ImageNotFound" type="text/string">/Users/fwein/code/pv_10/paraview_cfsreader/Plugins/CFSReader/Testing/Data/Baseline/CFSReaderTest.png</DartMeasurement>

So it is again CFSReaderTest.png instead of Single_Region_Opt_last_PSD.png.

This is my CMakeLists.txt in my plugins’s Testing

set(module_tests
  CFSReaderTest.xml)

if (BUILD_SHARED_LIBS)
  set(_paraview_add_tests_default_test_data_target ParaViewData)

  ExternalData_Expand_Arguments(ParaViewData _
    "DATA{${CMAKE_CURRENT_SOURCE_DIR}/Data/openCFS/Single_Region_Opt.cfs}"

    "DATA{${CMAKE_CURRENT_SOURCE_DIR}/Data/Baseline/Single_Region_Opt_last_PSD.png}"
  )

  paraview_add_client_tests(
    LOAD_PLUGIN   "CFSReader"
    BASELINE_DIR  "${CMAKE_CURRENT_SOURCE_DIR}/Data/Baseline"
    TEST_DATA_TARGET ParaViewData
    TEST_SCRIPTS  ${module_tests})

  paraview_add_client_server_tests(
    LOAD_PLUGIN   "CFSReader"
    BASELINE_DIR  "${CMAKE_CURRENT_SOURCE_DIR}/Data/Baseline"
    TEST_DATA_TARGET ParaViewData
    TEST_SCRIPTS  ${module_tests})

  paraview_add_client_server_render_tests(
    LOAD_PLUGIN   "CFSReader"
    BASELINE_DIR  "${CMAKE_CURRENT_SOURCE_DIR}/Data/Baseline"
    TEST_DATA_TARGET ParaViewData
    TEST_SCRIPTS  ${module_tests})
endif ()
1 Like

Thanks for sharing @Fabian, you may want to reader data.md :
https://gitlab.kitware.com/vtk/vtk/blob/master/Documentation/dev/git/data.md