Could Paraview Glance be used as a custom AI tool viewer

Web Support
,
#1

Okay. I have search almost the entire paraview discussion forum and could not get the answer. So this is my problem.

We are developing an AI algorithm that could pick out tumor cells from Dicom images and the algorithm is almost ready. Our workflow includes uploading a Dicom image manually to the python environment and uses TensorFlow as the AI framework to train the POI of the Dicom images. Now the time has come for us to include a user interface to our working model so we could absolutely automate this process.

So the main topic for discussion is the following.

  1. I have seen that paraview glide could paint the point of interest with its modern tool and it is the best I have seen during my search, but can paraview glide mark the region that the AI algorithm specifies? What I understand from my reading is that paraview is a client based solution, which doesn’t have much interaction with a server as such, so is it possible for any algorithm implementation in ParaView glide?

  2. From my reading I have understood that .glide file is the save file that is used for displaying the paraview glide files after marking the point of interest with its tool when it is saved and used at a later time. In that case, can I create my own custom .glide file for displaying my point of interest in the dicom file which I have loaded??

I have many other topics of interest with paraview glide but seems like this thread is becoming quite large. I hope someone could point out if this tool is best suited for me or if my use scenario is a limitation for this tool?

#2

I’m guessing you mean glance not glide?

#3

Yes. Sorry. Do you have a solution for me?

#4

Hi, ParaView Glance can certainly interact with a server of your choosing. In fact, that’s usually preferable to running AI models in the browser depending on how fast the model can run. So yes, Glance can show the ROI results of your algorithm as a paint overlay on an image. Note that the Glance codebase has to be modified to pull results from a server or run a custom algorithm in the browser, as there’s no easy way to generally support this functionality out-of-the-box.

You mentioned DICOM, so I just wanted to also mention another tool being developed that could be of interest to you. ParaView Medical aims to be similar to ParaView Glance, but for medical images (e.g. DICOM).

If you are interested in getting more help for customizing Glance to fit your needs, feel free to direct message me and we can discuss your requirements further. (FYI I’m a core dev for both ParaView Glance and ParaView Medical.)

#6

First question is pretty obvious. What is the best approach of integrating paraview glance or medical to our use case. From my understanding it is done in vue.js and it implements webpack(please correct me if I am wrong).

Is it better to work on the files in the current framework of paraview glance or am I able to exclusively select my needed section from the glance environment and implement in my use case.(by use case I mean my current project)

P.S I am not an expert in vue.js or webpack js. My main domain expertise is react.js. and when I read the files I am totally confused from where I should be starting my work, which file I should edit or how I could change a file without totally disrupting the whole work environment. An answer for this will be a big help as this could help me begin my long journey with paraview.

#7

@Forrest_Li Hi, I’m also working on similar task.

  1. upload input file to paraview glance.
  2. send input file to python backend for model inference, and return the result in format supported by paraview glance (nrrd maybe) to front end.
  3. the front end (paraview glance) get the result in supported format and render.

I check the paraview glance code, this file is for loading file. But I’m not sure how to take over the event that user click event to open a file, and write my code to send this request to backend for model inference and return the result to paraview later. any suggestion?

#8

@MTLathara Hi, I’m trying to do a similar task. I think maybe we could send the file to backend (written in Python code), generate the result (in format that supported by paraview glance) , and return the result to paraview glance for render.

#9

according to this code, when user click LOAD button, method loadFiles (defined here) is called. Now I could send a HTTP request to python backend using axios, and then get the result itk image from the response, and pass it to paraview glance in the loadFiles method.
Now, my question is:

  1. how could I get the file user uploaded in this method?
  2. how to send HTTP request to python backend with this file? I’m thinking write it to temporyfile and send the file path with request.
  3. when I get the response from my python backend, how could I hand the result itk image file to paraview glance?

I found this.fileList might contain some interesting info, could I simply modify it?

#10

Hi,

  1. Check out openFiles. This is the entrypoint for any files that the user opens locally. You can add your upload code there, depending on if you need it uploaded immediately.
  2. You can just make a PUT or POST with the file as FormData. Check out the MDN docs on FormData to see how you can attach a File object, which openFiles gives you.
  3. You will have to load the returned image file. You can either use ReaderFactory or just use the corresponding reader for your filetype. Readers can be found under I/O in our docs.
#11

Thanks. I could handle the 1st and the 2nd step but stuck at the 3rd step.
My python API backend return a url of a nii.gz file. So I add these codes to method openFiles in file src/store/fileLoader.js:

axios.post(url, formdata).then(function (response) {
        console.log(response);
        let result = [{
          url: response.data,
          name: 'myresult.nii.gz',
          withGirderToken: false,
          type: 'remote'
        }];
## this commit is copied from metho `openRemoteFiles`
        commit(
          'addToFileList',
          result.map((rfile) => ({
            type: 'remote',
            name: rfile.name,
            remoteURL: rfile.url,
            withGirderToken: false,
            // Key value pairs to be eventually set on the proxy
            proxyKeys: rfile.proxyKeys,
          }))
        );
      }).catch(function (error) {
        console.log(error);
      })

I could get the response from my python backend API, and it seems the this commit did not read my result file.
Could paravie glance read remote file if I provide a url? Or only girder is supported?

#12

You can do that

#13

Yes, I also check this doc. I could load remote file via url, but I’d like to do this in the paraview glance code, so users could upload their image and I’ll send it to my python inference backend, and return a url to the result, paraview glance finally render the result.
I rewrite the openFiles method:

      const url = 'http://127.0.0.1:8888/myapi';
      const formdata = new FormData();
      formdata.append('image', files[0]);
      axios
        .post(url, formdata)
        .then((response) => {
          console.log(response);
          const result = [
            {
              url: response.data,
              name: files[0].name,
              withGirderToken: false,
              type: 'remote',
            },
          ];
          commit(
            'addToFileList',
            result.map((rfile) => ({
              type: 'remote',
              name: rfile.name,
              remoteURL: rfile.url,
              withGirderToken: false,
              // Key value pairs to be eventually set on the proxy
              proxyKeys: rfile.proxyKeys,
            }))
          );
          return dispatch('readAllFiles');
        })
        .catch((error) => {
          console.log(error);
        });

It works ok. However, my backend takes too much time to result the result (1-2 mins), and the user has to wait for several minutes to click the LOAD button, and finally check the result. Maybe I could add a progress bar.