Hi Mathieu,
Thanks for your reply!
Our reader is a “directory reader” whose input is a folder name and read all files under this folder, when in multiple process, we will allocate files according to process id and let each process read some of files: [ifileBegin, ifileEnd)
unsigned int nfile = FileNameList.size();
unsigned int ifileBegin = 0;
unsigned int ifileEnd = nfile;
vtkMultiProcessController* controller = vtkMultiProcessController::GetGlobalController();
int numberOfProcesses = controller->GetNumberOfProcesses();
if (numberOfProcesses > 1)
{
int iProcessId = controller->GetLocalProcessId();
int nProcesses = controller->GetNumberOfProcesses();
int nfilePerProcess = nfile / nProcesses;
ifileBegin = iProcessId * nfilePerProcess;
ifileEnd = (iProcessId + 1) * nfilePerProcess;
if (ifileEnd > nfile || iProcessId == nProcesses - 1)
{
ifileEnd = nfile;
}
}
and then use multiple threads to read and create multiblock dataset for all files: if one file is in range [ifileBegin, ifileEnd) then read true data and create true dataset, while creating dummy dataset(the same structure but no true data) if the file is out of range [ifileBegin, ifileEnd).
std::thread::id main_thread_id = std::this_thread::get_id();
int nThread = vtkSMPTools::GetEstimatedNumberOfThreads();
if (numberOfProcesses > 1)
{
int numThreadPerProcess = nThread / numberOfProcesses;
if (numThreadPerProcess < 1) numThreadPerProcess = 1;
vtkSMPTools::Initialize(numThreadPerProcess);
}
vtkSMPTools::For(0, nfile, functor);
after multiple threads reading, all data will be reduce to vector “GlobalDatas”, the multiblock dataset for each file will be assembled into output multiblock dataset, in this way to keep the data structure on all processes has the same structure profile:
for (size_t i = 0; i < nfile; i++)
{
const std::string& file = FileNameList[i];
assert(globalDatas.find(file) != globalDatas.end());
auto processData = globalDatas[file];
if (processData->GetIsDummy())
{
output->AddDummyDataBlock(processData, processData->FileName);
}
else
{
output->AddTrueDataBlock(processData, processData->FileName);
}
processData->Delete();
}
do we still need keep the the same data structure profile for all process?
any suggestions for this reader?