{"id":86612,"date":"2023-10-25T16:20:15","date_gmt":"2023-10-25T08:20:15","guid":{"rendered":"https:\/\/www.seeedstudio.com\/blog\/?p=86612"},"modified":"2023-10-27T14:29:57","modified_gmt":"2023-10-27T06:29:57","slug":"what-is-minigpt-4-a-deep-dive-in-multimodal-llm-and-deploy-on-jetson-orin","status":"publish","type":"post","link":"https:\/\/www.seeedstudio.com\/blog\/2023\/10\/25\/what-is-minigpt-4-a-deep-dive-in-multimodal-llm-and-deploy-on-jetson-orin\/","title":{"rendered":"What is MiniGPT-4? A Deep Dive and Deploy on Jetson Orin"},"content":{"rendered":"\n

For complex computer vision tasks, we usually require machines not only to interpret complex visual data but also to comprehend the contextual intricacies through language. That’s the power of the vision-language model with multimodal capabilities, enhancing the accuracy and depth of object detection, and moreover, providing huge potential for more intuitive human-machine interaction. MiniGPT-4 is one of the interesting applications that we can dive into the world of multimodal LLM. <\/p>\n\n\n\n

Here we are going to talk about how the miniGPT-4 tech behind these generative AI applications are built, where they can be deployed, and how to release the most power of them on NVIDIA Jetson Orin. To explore more possibilities of generative AI tech, feel free to go through the guidance of Jetson Generative AI Lab<\/a> to have a try!<\/p>\n\n\n

\n
\"\"
Understanding arts in museums from machines in one click could be possible<\/figcaption><\/figure><\/div>\n\n\n

What is MiniGPT-4 <\/h2>\n\n\n\n

MiniGPT-4<\/a> is a lightweight version of the vision-language model quite similar to ChatGPT. It is developed to verify if the sophisticated large language model can enhance the power of multimodal generation capability (We’ll talk about multimodal deep learning in the following parts). <\/p>\n\n\n\n

By aligning a frozen visual encoder which contains the pre-trained ViT and Q-Former with a frozen LLM – Vicuna, using only one projection layer, miniGPT-4 demonstrates many advanced multimodal capabilities similar to GPT-4, such as generating detailed image descriptions and creating websites from hand-drawn sketches, even extending to being able to write poems or giving guidance based on a given image.<\/p>\n\n\n

\n
\"\"
Image resource from https:\/\/minigpt-4.github.io\/<\/figcaption><\/figure><\/div>\n\n\n

To produce a more natural language output for the machine, it’s important to eliminate the interference of noise: fine-tune the model with a detailed image description dataset rather than just using short image captions. All is for the improvement of the model’s generation reliability and better usability.<\/p>\n\n\n\n

MiniGPT-4 Methods by Two Stages<\/h3>\n\n\n\n

1. During the pre-train process, align image-text pairs with huge amounts of data collections<\/strong><\/p>\n\n\n\n