Best practice to read files in parallel

(sharon) #1

Dear Experts,
I created my own reader plugins which can read raw data from files and convert data into vtk data successfully.
However, in my situation, there will be a large number of files, so i want to use MPI to read data, then all the data can be convert on each separate cpu on server side, finally show them together on the client side screen.

My general ideas is:

  1. change reader to receive a directory
  2. find all the files under this directory
  3. distribute files according to its processID

My questions are: what’s the best practice for my situation?

Thanks and Best Regards,
Sharon

(Kenneth Moreland) #2

Unfortunately, it is not practical for a plugin to change the behavior of File -> Open to open a directory instead of a file (or file series). That would break the behavior of every other reader in ParaView and would lead to problems. However, here are 3 potential solutions to your implementation, which I order from most desirable solution first in my personal opinion (other opinions may vary).

  1. If you have control over how files are written out, require the files to be written out in a file series that the ParaView file browser will group together. For example, if all the files have the same name except a number that increments from 0 to n, then the ParaView file browser will group those together. Users click to open the group, and your reader gets a list of all the files in the group, no introspection of the directory needed. Of course, this is only feasible if all the files have the same extension and you have the ability to dictate how they are written.
  2. You could just let the user open any of the files in the directory in question, and then have the reader find the directory containing the file loaded and load all the files in the directory. All ParaView does is hand your reader a path to the file (or files). The reader can do whatever it wants with that path, so there is nothing stoping it from identifying the path to the containing directory and then loading all files in that directory. The only disadvantage I can think of is that it might be confusing to users who expect to click on the directory and instead have to click on the file.
  3. Since your reader is part of a plugin, your plugin could create its own toolbar with a special “Open” button that opens a directory and creates the reader for that directory. It’s rather awkward to have a second “Open” button, but one advantage is that in addition to creating the reader it can set up state that a typical reader cannot do. For example, it can change the camera if the default orientation is bad or it can change the color map if users prefer a specific one.
(sharon) #3

Thank you Kenneth for your reply!
I will consider your ideas carefully.
Best Regards,
Sharon

(David Thompson) #4

@Kenneth_Moreland Actually, I believe ParaView readers can now accept directories. See the is_directory XML attribute in ParaViewCore/ServerManager/SMApplication/Resources/readers.xml where there are 2-3 occurrences.

1 Like
(Kenneth Moreland) #5

OK, but does the file browser actually allow you to select the directory, or do you have to use some other mechanism to set that property?

(David Thompson) #6

Yes, you when you highlight a directory in the file browser dialog, 2 buttons should become enabled: “Navigate” and “OK”. “Navigate” will browse into the subdirectory while “OK” will attempt to open the directory with a reader.

(Kenneth Moreland) #7

Huh. I didn’t know that. Ignore my previous post then.

(David Thompson) #8

It is new and shiny. :slight_smile: