# Leveraging MPICH ABI compatibility for distributed binaries
**URL:** https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441
**Category:** Development
**Tags:** proposal
**Created:** [January 29, 2020, 12:58am UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441 "2020-01-29T00:58:34Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@utkarsh.ayachit](https://discourse.paraview.org/u/utkarsh.ayachit)
#### Post date: [January 29, 2020, 12:58am UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/1 "2020-01-29T00:58:34Z")
**Note: this discussion only applies to Linux, and not Windows or macOS**
Anyone who has used ParaView in HPC environments should be very familiar with this MPI specific issue: if want to use the MPI implementation provided by your HPC system (which is most likely tuned for your HPC hardware) you must make sure your application executables are built against the same MPI implementation. For most ParaView HPC site administrators and users, this translates to the aphorism: for HPC, build your own ParaView from source; don’t use ParaView binaries from [paraview.org](http://paraview.org).
The [MPICH ABI compatibility initiative](https://wiki.mpich.org/mpich/index.php/ABI_Compatibility_Initiative) announced in SC13 and now supported by several mainstream MPICH implementations has the potential to make this obsolete. In other words, [paraview.org](http://paraview.org) binaries can potentially work with your HPC provided MPI implementation without the need to compile from source!
ParaView superbuild (as of ParaView 5.8) is already using a compatible version of MPICH (3.3). Thus, technically, the ParaView binaries already support this. The problem is that since the package includes `libmpi.so.12` under the standard library loading search path, the loader loads the libmpi we include in the package rather than the one provided by the platform (one can manually just delete it, but that’s hardly an elegant solution).
**One** possible solution to address this is to follow the pattern we already use for choosing between Mesa GL and system GL. For GL, by default, the ParaView executables use system GL. To use mesa instead, one runs the launcher `paraview-mesa` that sets appropriate environment variables to load the mesa GL libraries instead of your system ones. We could follow the same pattern for MPI: let ParaView executables use system MPI by default, but then provide a new launcher, `paraview-mpi`, that lets one use the MPI implementation packaged with ParaView instead. The advantage of having two launchers is that they can be combined as needed e.g. to launch paraview using mesa and packaged mpi, one can run `paraview-mesa --backend swr paraview-mpi paraview` or `paraview-mpi paraview-mesa --backend swr paraview`. The disadvantage is that the command line gets quite confusing.
**Second** option is we provide a single launcher `paraview-launcher` that can setup paths needed for using packaged Mesa GL or MPI implementation.
However, both these approaches have a serious disadvantage when used for MPI implementation selection. While with GL, it was okay to expect most systems to have a GL implementation, it’s not reasonable to expect the same for MPI. If no compatible MPI implementation is available on the system, the paraview binaries will simply fail to start unless the launcher executable is used.
**Third** option is that we change the package so that by default the standard executables i.e. `paraview`, `pvserver`, `pvpython` etc. are launchers themselves. They setup paths such that the packaged MPI is used. By passing optional command line arguments e.g. `--system-mpi`, they can be made to skip that path setup and use system MPI instead. Similarly it can handle Mesa GL vs System GL, with system GL being the default behavior.
**Fourth** option is that we simply provide separate downloads: one with mpi libraries included in the package, and another without those for those who want to use compatible system MPI implementation (similar to how we do it for Windows).
Thoughts? Any other suggestions?
Thanks!
---
### Author:  [@wascott](https://discourse.paraview.org/u/wascott)
#### Post date: [January 29, 2020, 1:15am UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/2 "2020-01-29T01:15:37Z")
I think that any of the proposed options would work well.
- I slightly like number 3 the best. “pvserver”. Hmmm… what switches do I need? OOhh, that one didn’t work, lets reverse that switch.
- Whatever is done, document well with examples.
- Realize that generally speaking, users will be figuring out these commands once or twice a year, and putting them into scripts. So, length of command doesn’t matter, clarity does.
- This part of the client/server connect is so much simpler than the default\_servers.pvsc, I don’t think it matters.
---
### Author:  [@mwestphal](https://discourse.paraview.org/u/mwestphal)
#### Post date: [January 29, 2020, 3:11am UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/3 "2020-01-29T03:11:44Z")
+1 for number 3 as well. tbh I’m not sure I follow the reasons that made use switch the mesa flag to a dedicated executable in the first place.
The most important is that default behavior stays the same.
---
### Author:  [@utkarsh.ayachit](https://discourse.paraview.org/u/utkarsh.ayachit)
#### Post date: [January 29, 2020, 1:24pm UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/4 "2020-01-29T13:24:41Z")
> [@mwestphal](#):
>
> I’m not sure I follow the reasons that made use switch the mesa flag to a dedicated executable
@ben.boeckel, do you want to comment on that? Personally, I too prefer the `--mesa` command line options to the new `paraview-mesa` executable. If we had continued with the `--mesa` argument, adding another `--system-mpich` (or some such) argument would have been a natural extension.
---
### Author:  [@Kenneth\_Moreland](https://discourse.paraview.org/u/Kenneth_Moreland)
#### Post date: [January 29, 2020, 2:41pm UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/5 "2020-01-29T14:41:35Z")
Rather than have a switch to select use MPI or not, would it make sense to have separate binary distributions for MPI and not-MPI? That’s basically what we do for the Windows distributions.
---
### Author:  [@mwestphal](https://discourse.paraview.org/u/mwestphal)
#### Post date: [January 29, 2020, 3:07pm UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/6 "2020-01-29T15:07:38Z")
That’s option 4 in utkarsh post. It would get a little bit confusing as it would not be a non-mpi and a mpi version, but a mpi-paraview and a mpi-system version.
---
### Author:  [@utkarsh.ayachit](https://discourse.paraview.org/u/utkarsh.ayachit)
#### Post date: [January 29, 2020, 3:38pm UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/7 "2020-01-29T15:38:56Z")
> [@Kenneth\_Moreland](#):
>
> would it make sense to have separate binary distributions for MPI and not-MPI?
While option 4 is indeed a reasonable one, I worry we may end up with too many binary variants. Soon we’ll have binaries with EGL (to support X-less hardware acceleration for rendering). Now we’ll need MPI variants for this version too. Same for OSMesa binaries which, if I am not mistaken, we already distribute.
---
### Author:  [@danlipsa](https://discourse.paraview.org/u/danlipsa)
#### Post date: [January 29, 2020, 4:21pm UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/8 "2020-01-29T16:21:58Z")
+1 for option 3.
Command line for option 2 is quite strange, option 4 can get complicated for users to keep track of what binaries they downloaded and need to launch.
---
### Author:  [@cory.quammen](https://discourse.paraview.org/u/cory.quammen)
#### Post date: [January 29, 2020, 4:39pm UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/9 "2020-01-29T16:39:34Z")
+1 for option 3.
Strong objection to option 4. I would take too many build resources, verifying each binary would be too time consuming, and as Dan said it would be confusing to determine which one to download.
---
### Author:  [@utkarsh.ayachit](https://discourse.paraview.org/u/utkarsh.ayachit)
#### Post date: [January 29, 2020, 6:26pm UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/10 "2020-01-29T18:26:41Z")
Looks like the consensus is option 3. I’ll proceed in that direction. Thanks all!
---
### Author:  [@Andy\_Bauer](https://discourse.paraview.org/u/Andy_Bauer)
#### Post date: [January 29, 2020, 6:51pm UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/11 "2020-01-29T18:51:42Z")
Sounds good to me.
---
### Author:  [@wascott](https://discourse.paraview.org/u/wascott)
#### Post date: [January 30, 2020, 3:04am UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/12 "2020-01-30T03:04:50Z")
Glad I put my thumb on the scale early on! The more I read everyone’s posts, I’m also strongly against 2 and 4. Option 3 wins in my mind still.
---
### Author:  [@ben.boeckel](https://discourse.paraview.org/u/ben.boeckel)
#### Post date: [January 31, 2020, 3:47pm UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/13 "2020-01-31T15:47:32Z")
> [@mwestphal](#):
>
> tbh I’m not sure I follow the reasons that made use switch the mesa flag to a dedicated executable in the first place.
This was done when we removed the forward executables. Calling it `paraview-mesa` rather than `paraview-launcher` could have probably been avoided, but it’s something that seems to have been resolved with the new `launchers` project in the superbuild. We had issues where the `LD_LIBRARY_PATH` we used for ParaView itself interfered with the applications ParaView launched (IIRC, the superbuild’s libfontconfig made PDF viewers crash). Without the forward executable, a separate launcher had to be made anyways and to keep it simple, I made it a dedicated tool rather than a full wrapper.
---
### Author:  [@utkarsh.ayachit](https://discourse.paraview.org/u/utkarsh.ayachit)
#### Post date: [February 26, 2020, 1:01am UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/14 "2020-02-26T01:01:05Z")
Just to close the loop, these changes have now been implemented in `master` and will be included in ParaView 5.9. Below is the output generated using `--help` by pvserver executable from the nightly binaries. The launcher options are list first, followed by the standard executable options (in this case, pvserver options).
```bash
> ./bin/pvserver --help
Launcher options:
--print Print modified environment.
--system-mpi Use MPI implementation available on the system.
--mesa Use Mesa GL for rendering.
--backend Specify mesa backend.
Available backends:
llvmpipe
swr
pvserver options:
--client-host=opt
-ch=opt Tell the data|render server the host name of the client, use with -rc.
--connect-id=opt Set the ID of the server and client to make sure they match. 0 is reserved to imply none specified.
...
```
You should be able to test these out using the latest nightly binaries for Linux.
---
### Author:  [@mwestphal](https://discourse.paraview.org/u/mwestphal)
#### Post date: [February 26, 2020, 2:19am UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/15 "2020-02-26T02:19:25Z")
So using system mpi would be :
`mpirun -np 4 ./bin/pvserver --system-mpi`
?
---
### Author:  [@utkarsh.ayachit](https://discourse.paraview.org/u/utkarsh.ayachit)
#### Post date: [February 26, 2020, 2:43am UTC](https://discourse.paraview.org/t/leveraging-mpich-abi-compatibility-for-distributed-binaries/3441/16 "2020-02-26T02:43:09Z")
yes