Does or can Paraview support streaming input?

Development
1

Paraview appears to be a popular flexible visual tool but mostly designed for
polishing and publishing. However, I was still curious if it supported a monitroing
mode and could accept streaming input? That is, I need something simple for monitoring
progress in a variety of numerical codes and could not find anything. I’ve started making
my own which does what I need as I encounter an issue, see mjmdatascope below,
but it may make more sense to just put a streaming interface on something more
sophisticated although “lightweight” was a design goal as it may ( or may not )
ruin on the same machine doing the numerical stuff. In essence this is an oscilloscope lol.

mjmdatascope

Thanks.

2

Welcome to the ParaView Discourse, @mmarchywka!

A straightforward way to get this kind of behavior is to use the Reload Files action (File menu → Reload Files) action in ParaView. This is a one-time update, so you would have to manually invoke it every time you want to check on the latest progress.

3

Thanks. Does this have to go through a regular file or could it use fifo
or socket? I would imagine network file system would be ok but that is a bit
of a limitation. What do people normally do for algorithm development?
Doing everything in batch mode: run, write temp files images, view images, repeat:
is really distracting. I’m doing a simple orbital now looking for spirals etc and
with datascope I can watch it as it runs and in theory could just input changed parameters
on the fly. No publication quality output but it can save movies and stills for later
reading by paraview and of course it could save the original numeric data it
gets from the app.

What is involved in automating the reload or streaming data from a fifo or socket?
Thanks

4

Hi @mmarchywka !

You may have a look at in-situ visualization with Catalyst : Catalyst — Catalyst documentation

Best Regards,

5

Thanks the words sounds right- in-situ and conduit lol. . On a quick read of the
description without looking at API details that sounds like the kind of thing I need
that would be a common benefit, catalyst if you will, to algorigthm dev.
The Conduit docs seem to get to binary formats etc and it looks like
it is based on some coupling between the simulation and viewer.
Is there a communications protocol or over-the-wire format implemented in
any of this? The decoupling I have found is very nice as I can leave the viewer up,
display from multiple sources, and the simulation can react to anyone reading
its output que and wait or stop exporting data if viewr is gone or behind.

I had not implemented a “steering” mode but yes that is the next logical thing
to do .

Paraview appears to be quite “heavy” even i n a base configuration but I guess
for most serious numerical stuff you hvae a lot of hardware sitting around.

6

Hi Mike,

My first comment is that how well things work depends on the nature of the data being streamed. High-frequency (and small bandwidth) data such as one from an experiment is probably not the best fit. Given ParaView’s workflow, lower-frequency data would be the best fit. I would say 20 Hz or less. The data could be individual values or a whole dataset.

The best solution to what you are describing is a combination of Catalyst and ADIOS2 to stream the data to a ParaView client. Check this out:

https://www.kitware.com/catalyst-adios2-a-new-catalyst2-implementation-for-in-transit-analysis/

This has the benefit of linking only a small library to the simulation and streaming the data to the ParaView application.

7

Thanks that looks about right although way more than I need for now.
How does it handle simulation to analysis node communications and issues
like absence of a listener? Generally I would let the simulation continue and drop
data although I’ve started using this to make movies and have the simulation wait for
the display etc. It also sounds like you can re-assemble distributed meshes that way
too which if of course nice to offload from the simulation nodes. How is communication
implemented?

Right now I’m just doing everything on my desktop and lightweight matters.
Sometimes with these elaborate swiss-arm-knife things designed from the
top down you run into simple things that are difficult to do vs bottom up
“make it up as you go along”

8

ADIOS2 is the library responsible for data transport.

See this:

https://adios2.readthedocs.io/en/v2.9.2/engines/engines.html#sst-sustainable-staging-transport

9

Thanks that looks like it has everything thought out for data distribution.
I guess if I scale up I know it is there but right now instead of setting
all that up and have all the overhead I guess I’ll keep using
my mjmdatascope. Even if I end up with a lot of code I can always write an interface
from “datascope” to catalyst/conduit …