Inefficient use of sources on HPC systems

[Ercan (Ercan Umut)]

Hello everybody,

I am trying to use Paraview-Superbuild on a HPC cluster to generate an animation via terminal. When I run my batch file, which looks like this:

#!/bin/bash
#SBATCH -p 
#SBATCH -A 
#SBATCH -J unst
#SBATCH --nodes=1
#SBATCH --ntasks=55
#SBATCH --cpus-per-task=1
#SBATCH -C weka
#SBATCH --time=12:10:00
#SBATCH --output=slurm-%j.out
#SBATCH --error=slurm-%j.err

echo "SLURM_NODELIST $SLURM_NODELIST"
echo "NUMBER OF CORES $SLURM_NTASKS"

module purge
module load comp/gcc/14.1.0

export PATH=/arf/program/cmake-3.31.2/bin:$PATH
export PATH=/arf/program/ParaView/paraview-superbuild/install/bin:$PATH


mpiexec pvbatch --force-offscreen-rendering --verbosity=OFF sample3D.py

sstat -j $SLURM_JOB_ID

I always get the warnings in the picture.

What could be the reason for this and any idea on how to solve it?

[mwestphal (Mathieu Westphal (Kitware))]

I always get the warnings in the picture.

These warning are not coming from ParaView, are they coming from slurm maybe ?

mpiexec pvbatch --force-offscreen-rendering --verbosity=OFF sample3D.py

Please note this is not running distributed but serially, on a single MPI process

export PATH=/arf/program/cmake-3.31.2/bin:$PATH

I don’t see a reason why you would need CMake in your path

[Ercan (Ercan Umut)]

@mwestphal

These warning are not coming from ParaView, are they coming from slurm maybe ?

Exactly, they are from slurm. But I guess they are saying that ParaView does not use all of the sources, right? I guess every CPU is used, let’s say, 5% instead of 90% and this creates a warning. I had an screenshot of memory inspector taken some time ago where I had to open ParaView on OpenOnDemand with GUI and it showed something similar. HPC system bans the user due to inefficient use and that would create a lot of hassle for me

@mwestphal

Please note this is not running distributed but serially, on a single MPI process

How can I make it work distributed? Would it distribute the load all the available CPUs equally? I noticed that task0 takes the main load.

@mwestphal

I don’t see a reason why you would need CMake in your path

I always put install dependencies in my slurm file, as a habit. Would it affect anything?

[mwestphal (Mathieu Westphal (Kitware))]

@Ercan

How can I make it work distributed? Would it distribute the load all the available CPUs equally? I noticed that task0 takes the main load.

mpiexec -np N, where N is the number of cores you want to use, however, in the context of HPC and multi node computation, you may need to generate and use a host file.