As hinted in a previous post, many components of ParaView need to be revisited in order to incorporate the asynchronous features of ParaView Async. The focus of this post is the new Asynchronous API and specifically how it could look like in the Python world. This API targets developers of Python-based applications that utilize ParaView’s ServerManager infrastructure.
First, let’s review the new API. Every call falls into one of the four categories below based on blocking requirements and type/nature of the return value.
-
Local/ClientSide calls, i.e. any call that uses exclusively data that reside on the client. Methods accessing data members of the Proxy and require no communication to the server are included here.
- Examples:
vtkSMProxy::GetGlobalID(),vtkSMProxy::GetProperty(), …
- Examples:
-
Remote calls that invoke a function on the server and return immediately.
-
vtkSMProxy::UpdateVTKObjects()for example, creates (and updates if needed) a remote object.
-
-
Remote calls that return a single value. The result may be needed in subsequent steps so a mechanism to wait for completion is required.
-
vtkClientSession::Connect(url)for example, connects to the server aturl. It returns thesessionIdthat the client uses thereafter to communicate with the server.
-
-
Remote calls that return a stream of values. These can be any status updates, progress events or even rendered images generated at a remote rendering service.
- Example:
vtkClientSession::GetProgressObservable(), returns a stream of progress events that can be consumed by a progress bar.
- Example:
Before delving into the details, let’s see why Python integration of the ServerManager API is not trivial. We start with a simple example:
sphere = sources.SphereSource()
sphere.ThetaResolution = 80
sphere.UpdateVTKObjects()
sphere.UpdatePipeline()
assert "Normals" == sphere.GetPointDataInformation().values()[0].GetName()
The first two calls belong to category 1 so we skip them for the rest of the discussion. UpdateVTKObjects() is also straightforward since it does not update any local state. On the other hand, UpdatePipeline() which updates the pipeline and fetches data information, belongs to category 3. Thus, it needs to act as a blocking call, otherwise the assertion can fail (data may not be ready). Blocking the main thread and delegating the update to a secondary thread is not an option since as mentioned in the previous post, the ServeManager API is not thread-safe. Moreover, the main thread updates local VTK objects and VTK is, to a large extent, thread-unaware.
Our approach is to utilize Python’s Asynchronous I/O library, asyncio, to wrap these methods. For the example this translates as follows:
sphere = sources.SphereSource()
sphere.ThetaResolution = 80
sphere.UpdateVTKObjects()
await sphere.UpdatePipeline()
assert "Normals" == sphere.GetPointDataInformation().values()[0].GetName()
The script will now execute as before up until it reaches the await keyword where it will hand over the control of the (main) thread to Python’s event loop which will be used to pull and update the information of the sphere. Once the information is updated, the execution will continue to the assertion. The use of await resolves both issues mentioned above. Moreover, due to the fact that in Python, awaitables must be awaited explicitly, if we forget to add the await keyword, then we get an error as follows:
RuntimeError: coroutine 'Proxy.UpdatePipeline' was never awaited
For functions that fall into the 4th category we opted to use asynchronous iterators . Here is a mockup snippet:
with image_iterator(view) as Iter:
async for image in Iter:
display(image)
The script will await for the next value while handing the control of the main thread at the end of each iteration. image_iterator is a placeholder for the infrastructure that will convert the stream to a Python iterator.
Currently, the wrapping of C++ asynchronous methods to Python Awaitables is manual. However, the plan is to “teach” the vtkWrappingTools module how to do it during compilation requiring only minor hints from the developer.
Does this look like a reasonable API ? Any thoughts or concerns ?
