{"id":70622,"date":"2022-08-23T19:16:48","date_gmt":"2022-08-23T11:16:48","guid":{"rendered":"https:\/\/www.seeedstudio.com\/blog\/?p=70622"},"modified":"2022-08-29T13:56:56","modified_gmt":"2022-08-29T05:56:56","slug":"faster-inference-with-tensorrt-on-nvidia-jetson-run-yolov5-at-27-fps-on-jetson-nano","status":"publish","type":"post","link":"https:\/\/www.seeedstudio.com\/blog\/2022\/08\/23\/faster-inference-with-tensorrt-on-nvidia-jetson-run-yolov5-at-27-fps-on-jetson-nano\/","title":{"rendered":"Faster YOLOv5 inference with TensorRT, Run YOLOv5 at 27 FPS on Jetson Nano!"},"content":{"rendered":"\n
\"\"<\/figure><\/div>\n\n\n\n

Why use TensorRT?<\/h2>\n\n\n\n

TensorRT-based applications perform up to 36x faster than CPU-only platforms during inference. It has a low response time of under 7ms and can perform target-specific optimizations. Thus enabling developers to optimize neural network models trained on all major frameworks, such as PyTorch, TensorFlow, ONNX, and Matlab, for faster inference. It can also be integrated with application-specific software development kits such as NVIDIA DeepStream, Riva, Merlin, Maxine, Modulus, Morpheus, and Broadcast Engine. <\/p>\n\n\n\n

How does TensorRT perform optimization?<\/h2>\n\n\n\n
\"\"\/
Source: NVIDIA<\/a><\/figcaption><\/figure><\/div>\n\n\n\n

We have read about how TensorRT can help developers optimize, but now we will look at the six processes of TensorRT that can make it work.<\/p>\n\n\n\n

1. Weight & Activation Precision Calibration<\/h3>\n\n\n\n
\"\"\/
Source: NVIDIA<\/a><\/figcaption><\/figure><\/div>\n\n\n\n

Nearly all deep learning models are trained in FP32 to take advantage of a wider dynamic range. However, these models require a long predicting time, setting back real-time responses. <\/p>\n\n\n\n

In this process, model quantization converts the parameters and activations to FP16 or INT8. This will normally cause lower accuracy and a reduction in latency and model size. But by using KL-divergence<\/a>, TensorRT is able to measure the difference and minimize it, thereby preserving accuracy while maximizing throughput. We can see the difference between FP32 and INT8\/FP16 from the picture above.<\/p>\n\n\n\n

2. Layer & Tensor Fusion<\/h3>\n\n\n\n
\"\"\/
Source: NVIDIA<\/a><\/figcaption><\/figure><\/div>\n\n\n\n

In this process, TensorRT uses layers and tensor fusion to optimize the GPU\u2019s memory and bandwidth by fusing nodes in a kernel vertically or horizontally (sometimes both). This reduces the overhead cost of reading and writing the tensor data for each layer. <\/p>\n\n\n\n

We can see from the picture above that TensorRT recognizes all layers with similar inputs and filter sizes and merges them to form a single layer. It also eliminates concatenation layers, as seen in the picture above (\u201cconcat\u201d).<\/p>\n\n\n\n

Overall, this will result in a smaller, faster, and more efficient graph with fewer layers and kernel launches, which will reduce inference latency. <\/p>\n\n\n\n

3. Kernel Auto-Tuning<\/p>\n\n\n\n

During this process, TensorRT selects the best layers, algorithms, and batch size based on the target GPU platform in order to find the best performance. This ensures that the deployed model is tuned for each deployment platform. <\/p>\n\n\n\n

4. Dynamic Tensor Memory<\/p>\n\n\n\n

For this process, TensorRT minimizes memory footprint and re-uses memory by allocating memory for each tensor only for the duration of its usage, avoiding any memory allocation overhead for faster and more efficient execution. <\/p>\n\n\n\n

5. Multi-Stream Execution<\/p>\n\n\n\n

TensorRT is designed to process multiple input streams in parallel during this process.<\/p>\n\n\n\n

6. Time Fusion<\/p>\n\n\n\n

For the last step before heading to the output stage, TensorRT is able to optimize recurrent neural networks over time steps with dynamically generated kernels. <\/p>\n\n\n\n

What models can be converted to TensorRT<\/h2>\n\n\n\n

TensorRT officially supports the conversion of models such as Caffe, TensorFlow, PyTorch, and ONNX.<\/p>\n\n\n\n

It also provides three ways to convert models:<\/p>\n\n\n\n