NA-MIC Project Weeks
Back to Projects List
NVIDIA Holoscan and 3D Slicer
Key Investigators
- Mikael Brudfors (NVIDIA, UK)
- Steve Pieper (Isomics, US)
- Rafael Palomar (NTNU, Norway)
- Jean-Christophe Fillion-Robin (Kitware, USA)
- Shreeraj Jadhav (Kitware, USA)
Project Description
This project aims at integrating NVIDIA Holoscan into 3D Slicer. Holoscan is a hybrid computing platform for real-time streaming data that combines hardware systems for low-latency sensor and network connectivity, optimized libraries for data processing and AI, and core microservices to run surgical video, ultrasound, medical imaging, and other applications anywhere, from embedded to edge to cloud.
Objective
Integrate NVIDIA Holoscan with 3D Slicer. Holoscan is both a hardware platform and an SDK). Holoscan SDK runs on both x86 and arm64 platforms, which means it can be deployed on, e.g., regular laptops, as well as NVIDIA Jetson devices, or even in the cloud!
Approach and Plan
There are various options, not yet sure which one is the best (or if there are ones I have not considered):
- Use OpenIGTLink to stream data from a Holoscan developer kit to 3D Slicer, over network. This would bypass the fact that 3D Slicer does not easily build on ARM (which is the arch of the IGX Orin).
- Create operators in 3D Slicer that can pass data to and from Holoscan SDK. This would give a 3D Slicer user access to the API of Holoscan SDK for efficient AI inference, etc.
- Connect an IGX Orin running Holoscan SDK to the Plus toolkit.
Progress and Next Steps
- Worked with Kitware on building Slicer natively on Holoscan OS/hardware (progress, but not completed)
- Protytpe OpenIGTLink connection streaming slice/volume data through Holoscan
- Made connections for further development
The implementation for (2) will be made available on the Holohub repository soon, and will allow a 3D Slicer user to send imaging data to Holoscan SDK, running on a GPU powered device, do efficient inference and the send the results back to 3D Slicer. Another possibility is to send data acquired on the Holoscan device to 3D Slicer for visualization.