{"id":69198,"date":"2022-08-04T21:21:08","date_gmt":"2022-08-04T13:21:08","guid":{"rendered":"https:\/\/www.seeedstudio.com\/blog\/?p=69198"},"modified":"2022-09-09T16:16:56","modified_gmt":"2022-09-09T08:16:56","slug":"yolov7-tensorrt-benchmark-from-cloud-gpus-to-edge-gpus","status":"publish","type":"post","link":"https:\/\/www.seeedstudio.com\/blog\/2022\/08\/04\/yolov7-tensorrt-benchmark-from-cloud-gpus-to-edge-gpus\/","title":{"rendered":"YOLOv7 TensorRT Benchmark from Cloud GPUs to Edge GPUs"},"content":{"rendered":"\n

How does the real-time computer vision inference performance vary from Cloud GPUs to Edge GPUs?  <\/p>\n\n\n\n

Nilvana (an AI software platform company) conducted a series of performance testing of Object Detection, Anomaly Detection, and Instance Segmentation models on different devices, from cloud GPUs A100 to the latest tiny powerhouse Jetson AGX Orin (available now at Seeed!<\/a>). Let\u2019s take a look at the test results! <\/p>\n\n\n\n

<\/div>\n\n\n\n

Object Detection<\/strong><\/h2>\n\n\n\n
<\/div>\n\n\n\n

Nilvana tested YOLOv7<\/a> variants models on A100, RTX 3060, Jetson AGX Orin<\/a>, and Jetson Xavier NX (check out the reComputer J2012<\/a> based on Xavier NX!). Read more about this from the article<\/a> at Nilvana Medium. <\/p>\n\n\n\n

<\/div>\n\n\n\n
\"\"
source: Hello Nilvana<\/figcaption><\/figure><\/div>\n\n\n\n
<\/div>\n\n\n\n

What is Object Detection?<\/strong><\/h3>\n\n\n\n
<\/div>\n\n\n\n

Image classification, object detection, and segmentation are three major tasks in computer vision. Image classification divides an image into a single category, usually corresponding to the most prominent object in the image. However, real world images usually contain more than one object. If the image classification model is used to assign a single label to the image, it can be rough and inaccurate. In such cases, an object detection model is needed to identify multiple objects in a picture and locate different objects (with bounding boxes). Object detection is used in real-world AI such as object tracking, video analytics, security, and autonomous driving. <\/p>\n\n\n\n

<\/div>\n\n\n\n

YOLOv7<\/a><\/strong><\/h3>\n\n\n\n
<\/div>\n\n\n\n

You Only Look Once (YOLO) is a state-of-the-art, real-time object detection system. The current and latest iteration, YOLOv7, infers faster and with great accuracy pushing Object Detection to newer heights. Wanna learn more about YOLO and get started with object detection? Check out our step-by-step tutorial using Roboflow and YOLOv5<\/a> to build the object detection model based on custom datasets and deploy it at NVIDIA Jetson-powered reComputer<\/a>!<\/p>\n\n\n\n

<\/div>\n\n\n\n

Test on DGX A100, RTX 3060, AGX Orin, and Xavier NX<\/strong><\/h3>\n\n\n\n
<\/div>\n\n\n\n

According to the results table, Jetson Xavier NX can run the YOLOv7-tiny model pretty well. AGX Orin can even run the YOLOv7x model at more than 30 FPS!<\/p>\n\n\n\n

<\/div>\n\n\n\n
\"\"<\/figure><\/div>\n\n\n\n
<\/div>\n\n\n\n
<\/div>\n\n\n\n

Anomaly Detection <\/strong><\/h2>\n\n\n\n
<\/div>\n\n\n\n
<\/div>\n\n\n\n
\"\"
source: Hello Nilvana<\/figcaption><\/figure><\/div>\n\n\n\n
<\/div>\n\n\n\n

What is Anomaly Detection?<\/strong><\/h3>\n\n\n\n
<\/div>\n\n\n\n

Anomaly Detection is a step in data mining that identifies rare data points, events, and observations that deviate from a dataset\u2019s normal behavior. Examples are technical glitches or potential opportunities such as a change in consumer behavior.  <\/p>\n\n\n\n

Nilvana has tested out anomaly detection on the RTX 3060 and Jetson AGX Orin on the MVTec AD dataset. Read more about it in Nilvana\u2019s medium article<\/a>! They have tested training and inferencing using PyTorch Lightning by providing about 200 normal images and 20 abnormal images for each defect of the product. Trains the model with only normal images and evaluates the AUROC (Area Under the Receiver Operating Characteristic Curve) and F1 Score by a few abnormal images.<\/p>\n\n\n\n

PaDiM is one of the famous anomaly detection algorithms on the leaderboard, the evaluation shows that PaDim is applicable to both large and small area defects. <\/p>\n\n\n\n

<\/div>\n\n\n\n

Test on RTX 3060 and Jetson AGX Orin<\/strong><\/h3>\n\n\n\n
<\/div>\n\n\n\n

Dataset<\/strong>: MVTec AD[3]<\/p>\n\n\n\n

Class<\/strong>: Bottle<\/p>\n\n\n\n

Training Image<\/strong>: 209 normal images, 20 abnormal images of each defect<\/p>\n\n\n\n

Framework<\/strong>: PyTorch Lightning<\/p>\n\n\n\n

Algorithm<\/strong>: PaDiM<\/p>\n\n\n\n

<\/div>\n\n\n\n
\"\"<\/figure><\/div>\n\n\n\n

The training time and inference FPS on NVIDIA RTX 3060 and Jetson AGX Orin respectively.<\/p>\n\n\n\n

<\/div>\n\n\n\n

We can tell from NVIDIA Jetson AGX Orin can perform perspectively as well as RTX3060 and even A100. To train the model on cloud GPU and deploy it on an edge device is common practice, the above result of Jetson AGX Orin shows that performance on the edge device will be very reliable. We are excited to see more applications on this powerful tiny machine. Seeed is also working on new AGX Orin and upcoming Orin NX-powered edge devices. We released the reServer J5014<\/a>,<\/strong> which is under development now. Let us know your thoughts and expectation on the discord<\/a> edge ai channel! <\/p>\n\n\n\n

<\/div>\n\n\n\n

reServer Jetson: Inference Center for the Edge<\/h2>\n\n\n\n
<\/div>\n\n\n\n

Powered by Jetson AGX Orin<\/a> \/ NX<\/a><\/h3>\n\n\n\n
<\/div>\n\n\n\n

\"reserver\"<\/p>\n\n\n\n