Image Processing Framework on Jetson
Author: Fyodor Serzhenko
Nowadays quite a lot of tasks for image sensors and camera applications are solved with the centralized computing architecture for image processing. Just a minor part of image processing features is implemented on the image sensor itself, so all the rest is done on CPU/GPU/DSP/FPGA which could reside very close to the image sensor. The latest achievements at hardware and software solutions allow us to get enhanced performance of computations and to enlarge the scope of tasks to be solved.
From that point of view, NVIDIA Jetson series is suited exactly for the task of high performance image processing from RAW to YUV. Image sensor or camera module can be connected directly to any Jetson via MIPI SCI-2 (2lane/4lane), USB3 or PCIe interfaces. Jetson could offer high performance computations either on ISP or on GPU. Below we show what could be done on GPU. We believe that raw image processing on GPU can offer more flexibility, better performance, quality and ease of management in comparison with hardware-based ISP for many applications.
What is Image Processing Framework for Jetson?
To get high quality and max performance at image processing tasks on Jetson, we've implemented a GPU-based SDK for raw processing. Now we are expanding that approach by creating an effective framework to control all system components, including hardware and software. For example, it means that image sensor control should be included in the workflow at realtime to become a part of general control algorithm.
Image processing framework components
Image Sensor Control (exposure, gain, awb)
Image Capture (driver, hardware/software interface, latency, zero-copy)
RAW Image Processing Pipeline (full raw to rgb workflow)
Image Enhancement
Image/Video Encoding (JPEG/J2K and H.264/H.265)
Compatibility with third-party libraries for image processing, ML, AI, etc.
Image Display Processing (smooth player, OpenGL/CUDA interoperability)
Image/Video Streaming (including interoperability with FFmpeg and GStreamer)
Image/Video Storage
Additional features for the framework
Image Sensor and Lens Calibration
Quality Control for Image/Video Processing
CPU/GPU/SSD balance load, performance optimization, profiling
Implementation of image sensor control at the workflow brings us additional features which are essential. For example, integrated exposure and gain control will allow to get better quality in the case of varying illumination. Apart from that, calibration data usually depend on exposure/gain and it means that we will be able to utilize correct processing parameters at any moment for any viewing conditions.
In general, standard RAW concept eventually lacks internal camera parameters and full calibration data. We could solve that problem by including image sensor control both in calibration and image processing. We can utilize image sensor abstraction layer to take into account full metadata for each frame.
Such a solution depends on utilized image sensor and task to be solved, so we can configure and optimize the Image Processing Framework for a particular image sensor from SONY, Gpixel, CMOSIS image sensors. These solutions on Jetson have already been implemented by teams of Fastvideo and MRTech.
Integrated Image Sensor Control
Exposure time
AWB
Gain
ROI (region of interest)
Full image sensor control also includes bit depth, FPS (frames per second), raw image format, bit packing, mode of operation, etc.
GPU image processing modules on Jetson for 16/32-bit pipeline
Raw image acquisition from image sensor via MIPI/USB3/PCIe interfaces
Frame unpacking
Raw image linearization
Dark frame subtraction
Flat field correction
Dynamic bad pixel removal
White balance
RAW and RGB histograms as an indication to control image sensor exposure time
Demosaicing with L7, DFPD, MG algorithms
Color correction
Denoising with wavelets
Color space and format conversions
Curves and Levels
Flip/Flop, Rotation to 90/180/270 or to arbitrary angle
Crop and Resize (upscale and downscale)
Undistortion via Remap
Local contrast
Tone mapping
Gamma
Realtime output via OpenGL
Trace module for debugging and bug fixing
Stream-per-thread support for better performance
Additional modules: tile support, image split into separate planes, RGB to Gray transform, defringe, etc.
Time measurements for all SDK modules
Image/Video Encoding modules on GPU
RAW Bayer encoding
JPEG encoding (visually lossless image compression with 8-bit or 12-bit per channel)
JPEG2000 encoding (lossy and lossless image compression with 8-16 bits per channel)
H264 encoder/decoder, streaming, integration with FFmpeg (8-bit per channel)
H265 encoder/decoder, streaming, integration with FFmpeg (8/10-bit per channel)
Is it better or faster than NVIDIA ISP for Jetson?
There are a lot of situations where we can say YES to this question. NVIDIA ISP for Jetson is a great product, it's free, versatile, reliable, and it takes less power/load from Jetson, but we have our own advantages which are also of great importance for our customers:
Processing performance
Image quality
Flexibility in building custom image processing pipeline
Wide range of available image processing modules for camera applications
Image processing with 16/32-bit precision
High-performance codecs: JPEG, JPEG2000 (lossless and lossy)
High-performance 12-bit JPEG encoder
Raw Bayer Codec
Dynamic bad pixel suppression
High quality demosaicing algorithms
Wavelet-based denoiser on GPU for Bayer and RGB images
Flexible output with desired image resolution, bit depth, color/grayscale, rotation, according to ML/AI requirements
We've built that software from the scratch and we've been working in that field more than 10 years, so we have an experience and we can offer reliable solutions and support. Apart from that we are offering custom software design to solve almost any problem in a timely manner.
What are benefits of that approach?
That approach allows us to create embedded image processing solutions on Jetson with high quality, exceptional performance, low latency and full image sensor control. Software-based solution in combination with GPU image processing on NVIDIA Jetson could help our customers to create their imaging products with minimum efforts and maximum quality and performance.
Other blog posts about Jetson hardware and software
Benchmark comparison for Jetson Nano, TX2, Xavier NX and AGX
Jetson Image Processing
Jetson Zero Copy
Jetson Nano Benchmarks on Fastvideo SDK
JPEG2000 performance benchmarks on Jetson TX2
Jetson AGX Xavier performance benchmarks
Remotely operated walking excavator on Jetson
Low latency H.264 streaming on Jetson TX2
JPEG2000 performance benchmarks on Jetson TX2
Performance speedup for Jetson TX2 vs AGX Xavier
Fastvideo SDK vs NVIDIA NPP Library
Original article see at: https://fastcompression.com/blog/jetson-image-processing-framework.htm
Subscribe to our mail list: https://mailchi.mp/fb5491a63dff/fastcompression
