Basic Architecture <\/h3>\n\n\n\nContrastive learning in pre-training<\/h4>\n\n\n\n
CLIP utilizes the Vision Transformer (ViT) architecture to process images, enabling the model to capture global relationships among image features. Meanwhile, for text processing, CLIP employs a transformer-based architecture, tokenizing and processing text through transformer layers to understand semantic relationships within textual descriptions without predicting texts word for word.<\/p>\n\n\n\n
The model is trained using pairs of images and text, aiming to minimize the distance between positive pairs (comprising an image and its corresponding description) and maximize the distance between negative pairs (comprising an image and a randomly chosen description). This contrastive learning objective encourages the model to learn a meaningful representation where related content is close and unrelated content is separated.<\/p>\n\n\n
![\"\"](\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/01\/WeChatWorkScreenshot_2996bae5-d5ec-4468-a7b4-26264f9306d2-1030x723.png\")
Fine-tune for downstream tasks<\/h4>\n\n\n\n
After pre-training, CLIP can be fine-tuned on specific downstream tasks with task-specific datasets. You can easily tailor CLIP’s behavior to specific tasks and domains with prompt engineering. By utilizing a prompt template or even prompt ensembling, which uses multiple prompt templates with more contextual modifications added to match possible situations, the accuracy of similarity matching can be improved.<\/p>\n\n\n\n
Why is CLIP important as a foundation model?<\/h3>\n\n\n\n
Unlike traditional models that require extensive labeled datasets for training, CLIP leverages a unique approach\u2014pretraining on a vast dataset containing image-text pairs from the Internet, enabling it to perform a wide array of tasks without the need for task-specific training data. The model’s ability to connect vision and language makes it adept at tasks such as image classification and even generating textual descriptions for images.<\/p>\n\n\n\n
You only need to define the possible prompts or descriptions the objects of the scene belong to, then CLIP will help you predict the most probable class for the given image or video based on its extensive pertaining.<\/p>\n\n\n
![\"\"](\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2023\/11\/WeChatWorkScreenshot_256a071a-6c26-4f77-acb3-e7224bcc377e-1030x304.png\")
<\/figure><\/div>\n\n\nIn this guide, we are going to show application insights and running performance using CLIP on the Edge. Basically, CLIP can be deployed for:<\/p>\n\n\n\n
Advertisement search engine: <\/strong>classify video advertisement’s category, featured by shooting scenarios or specific main objects.<\/p>\n\n\n\n Content Moderation:<\/strong> identify and flag potentially inappropriate or harmful content in images and videos.<\/p>\n\n\n\n SNS video recommendation: <\/strong>classify the video content and deliver recommendations based on audiences’ interest or viewing history.<\/p>\n\n\n\n Pinpoint important periods<\/strong> of the event in a long video:<\/strong> improve the efficiency for police to find the crime evidence on time.<\/p>\n\n\n\n Automation of the process in<\/strong> QSR<\/strong>: check what ingredients and sauces should be put in the delivered pizza, and provide a production list according to the visual comparison<\/p>\n\n\n\n Interior home design reference<\/strong>: help you discover visually similar interior design ideas based on your provided preferred decorating style images.<\/p>\n\n\n\n We’ll see that the industry and application direction of CLIP is unlimited! So now, let’s dive deep into how to deploy CLIP at the edge to solve your image classification and other project tasks.<\/p>\n\n\n\n Here are some of the interesting demos that we have deployed CLIP models on the reComputer J4012 based on NVIDIA Jetson Orin NX 16GB<\/a> using the Roboflow Inference Server<\/a>. You can take reference of Roboflow’s blog<\/a> for step-by-step guidance, however, just need to change the first Inference server installation part to the following code as adding TensorRT support, since we are using GPU: <\/p>\n\n\n\n Now you can easily classify images into different categories without training your model in advance. <\/p>\n\n\n\n Start the Roboflow Inference Server Docker container on the Jetson Edge device, define the Roboflow API key, and run a demo script to start classifying different images based on the prompts you define. We have used Jetson Orin NX 16GB for this demo and it was able to achieve around 22 FPS performance with TensorRT FP16 precision!<\/p>\n\n\n\n All you need to consider is prompt engineering – to find a more accurate or proper prompt to describe the scene of the image clearly for better classification performance, which means you need to tell CLIP what it should recognize in the image. This whole process of finding the correct prompt could be a long-term trial and error.<\/p>\n\n\n\n If you also want to show the inferencing speed as our demo performed, feel free to add this part easily in the result out print part:<\/p>\n\n\n\n Understanding a video scenario involves breaking down a video into individual frames, applying CLIP independently to each frame for image understanding, and then integrating temporal information across frames. <\/p>\n\n\n\n By fusing features extracted from both individual frames and their temporal context, the model creates a representation that captures the overall content and context of the video. The fused features are then used for scenario classification, predicting the depicted activity or scenario. Fine-tuning a dataset containing video frames and scenario labels may be necessary to adapt CLIP to the specific requirements of the video task. The performance of the model is evaluated using standard metrics, and adjustments are made as needed for optimal results.<\/p>\n\n\n\n In this demo, we use CLIP to identify the video as a scene containing a package. We can classify the scenario type, at what timestamps the package first appears, and how long it’s visible on the scene. It could be a perfect experiment for preventing package delivery from being stolen. Simply follow Roboflow’s blog<\/a> to classify the video step-by-step, also remember to import the supervision library at the beginning since we are going to use supervision to split up our video into frames.<\/p>\n\n\n\n Seeed is an Elite partner for edge AI in the NVIDIA Partner Network<\/u><\/a>. Explore more carrier boards, full system devices, customization services, use cases, and developer tools on Seeed\u2019s NVIDIA Jetson ecosystem<\/u><\/a> page.<\/p>\n\n\n Join the forefront of AI innovation with us! Harness the power of cutting-edge hardware and technology to revolutionize the deployment of machine learning in the real world across industries. Be a part of our mission to provide developers and enterprises with the best ML solutions available. Check out our successful case study catalog<\/a> to discover more edge AI possibilities!<\/p>\n\n\n\n Take the first step and send us an email at edgeai@seeed.cc<\/a> to become a part of this exciting journey! <\/p>\n\n\n\n Download our latest Jetson Catalog<\/a> to find one option that suits you well. If you can\u2019t find the off-the-shelf Jetson hardware solution for your needs, please check out our customization services<\/a>, and submit a new product inquiry to us at odm@seeed.cc for evaluation.<\/p>\n","protected":false},"excerpt":{"rendered":" The traditional method of image classification is quite time and resource-consuming. It requires more than<\/p>\n","protected":false},"author":3606,"featured_media":87404,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_lmt_disableupdate":"","_lmt_disable":"","_price":"","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"","_tribe_ticket_capacity":"0","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"iawp_total_views":0,"footnotes":""},"categories":[1,4393],"tags":[],"class_list":["post-87359","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news","category-tech"],"yoast_head":"\nDeploy CLIP on the Site for Classification<\/h2>\n\n\n\n
```sh\ngit clone https:\/\/github.com\/roboflow\/inference\ncd inference\nsed -i '\/ENV OPENBLAS_CORETYPE=ARMV8\/a ENV ONNXRUNTIME_EXECUTION_PROVIDERS=TensorrtExecutionProvider' docker\/dockerfiles\/Dockerfile.onnx.jetson.5.1.1\ndocker build \\\n -f docker\/dockerfiles\/Dockerfile.onnx.jetson.5.1.1 \\\n -t roboflow\/roboflow-inference-server-jetson-5.1.1:seeed1 .\n```\n\n- Run the container \n\n```sh\ndocker run --privileged --net=host --runtime=nvidia roboflow\/roboflow-inference-server-jetson-5.1.1:seeed1\n```<\/code><\/pre>\n\n\n\nImage-Prompt pair<\/h3>\n\n\n\n
for file in sorted(os.listdir(IMAGE_DIR)):\n image = f\"{IMAGE_DIR}\/{file}\"\n predictions = classify_image(image)\n print(get_highest_prediction(predictions[\"similarity\"]), image)\n print(1\/(predictions['time']))<\/code><\/pre>\n\n\n\n![\"\"\/](\"https:\/\/files.seeedstudio.com\/products\/NVIDIA-Jetson\/CLIP-classify-photo.gif\")
<\/figure>\n\n\n\nVideo theme classification<\/h3>\n\n\n\n
import supervision as sv<\/code><\/pre>\n\n\n\n![\"\"\/](\"https:\/\/files.seeedstudio.com\/products\/NVIDIA-Jetson\/CLIP-classify-video.gif\")
<\/figure>\n\n\n\n
\n\n\n\nSeeed: NVIDIA Jetson Ecosystem Partner<\/h3>\n\n\n
![\"\"](\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2023\/08\/nvidia-elite-partner-badge-rgb-for-screen.jpg\"){kind=icon}
<\/figure><\/div>\n\n\n![\"\"](\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2023\/09\/WeChatWorkScreenshot_0fa3f479-5b1c-4e23-9896-b989d6bde69c-3-1030x599.png\")
<\/figure><\/div>\n\n\n