{"id":94730,"date":"2024-05-03T19:47:09","date_gmt":"2024-05-03T19:47:09","guid":{"rendered":"https:\/\/www.seeedstudio.com\/blog\/?p=94730"},"modified":"2024-09-18T08:01:18","modified_gmt":"2024-09-18T08:01:18","slug":"tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai","status":"publish","type":"post","link":"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/","title":{"rendered":"TinyML + Local LLMs: A Trendy Architecture for Efficient and Affordable Edge AI"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"94730\" class=\"elementor elementor-94730\" data-elementor-post-type=\"post\">\n\t\t\t\t<div class=\"elementor-element elementor-element-69b0e00 e-flex e-con-boxed e-con e-parent\" data-id=\"69b0e00\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-2510c99 elementor-widget elementor-widget-heading\" data-id=\"2510c99\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fc154a\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span><h2 class=\"elementor-heading-title elementor-size-default\">Challenge<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-98bcce0 elementor-widget elementor-widget-text-editor\" data-id=\"98bcce0\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fc24aa\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>In recent days, the capabilities of large language models (LLMs) are advancing rapidly, and we are seeing a clear trend in the IoT world where hardware systems invoke these large models to infer more complex data and scenarios.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-b8955e3 e-flex e-con-boxed e-con e-parent\" data-id=\"b8955e3\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-155b9aa elementor-widget elementor-widget-text-editor\" data-id=\"155b9aa\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fc3644\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>Then, the next topic of discussion should be &#8211; how can this be made more cheaper and efficient?<\/p><ul><li>How can it be made cheaper?\u00a0 Frequent call and long-term use of large models is expensive;<\/li><li>Can the waiting time be reduced?\u00a0 The time it takes from sending data to a large model to receiving the inference results can take about 10-40 seconds.<\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-71818de e-flex e-con-boxed e-con e-parent\" data-id=\"71818de\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-43ac9c4 elementor-widget elementor-widget-spacer\" data-id=\"43ac9c4\" data-element_type=\"widget\" data-widget_type=\"spacer.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fc447a\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t<div class=\"elementor-spacer\">\n\t\t\t<div class=\"elementor-spacer-inner\"><\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e3a265f elementor-widget elementor-widget-heading\" data-id=\"e3a265f\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fc4a80\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span><h2 class=\"elementor-heading-title elementor-size-default\">Two Innovative Solutions<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-5786f6c e-flex e-con-boxed e-con e-parent\" data-id=\"5786f6c\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-660335e elementor-widget elementor-widget-heading\" data-id=\"660335e\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fc57da\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span><h3 class=\"elementor-heading-title elementor-size-default\">1. TinyML as a trigger mechanism for activating large models<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-5ded747 e-flex e-con-boxed e-con e-parent\" data-id=\"5ded747\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-66a4992 elementor-widget elementor-widget-text-editor\" data-id=\"66a4992\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fc6892\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<ul>\n<li><strong>Keyframe Filtering:<\/strong> Instead of constantly feeding data to the LLM, a tinyML model can be processed on hardware device to identify key frames or critical data points from the input stream. These key frames might be images, a snippet of audio, or significant fluctuations from a let&#8217;s say &#8211; 3-axis accelerometer. Only these selected data points are forwarded to the large model for in-depth analysis, effectively prioritizing important data and eliminating unnecessary processing.<\/li>\n<li><strong>Reduced Token Usage:<\/strong> By focusing on key frames, the number of tokens sent to the LLM is minimized, leading to significant cost savings. This approach also accelerates the overall response time by concentrating on essential data.<\/li>\n<\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-c9d322f e-flex e-con-boxed e-con e-parent\" data-id=\"c9d322f\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-cfbd8f9 elementor-widget elementor-widget-heading\" data-id=\"cfbd8f9\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fc76be\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span><h4 class=\"elementor-heading-title elementor-size-default\">Practical Demonstration at Nvidia GTC conference<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b10fc8d elementor-widget elementor-widget-text-editor\" data-id=\"b10fc8d\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fc7d43\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>We demonstrated the effectiveness of this approach in a TinyML+<span style=\"text-decoration: underline;\"><a href=\"https:\/\/www.jetson-ai-lab.com\/tutorial_live-llava.html\">LLaVA (Local Language Model-based Video Analytics)<\/a><\/span> demo at GTC. We showcased two device combinations:<\/p><ul><li><strong>Setup 1: a USB camera + Nvidia Jetson Orin AGX<\/strong><br \/>In this standard system, a USB camera was directly connected to an AGX running the LLaVA, continuously processing every captured frame.<\/li><li><strong>Setup 2: <span style=\"text-decoration: underline;\"><a href=\"https:\/\/www.seeedstudio.com\/watcher\">SenseCAP Watcher<\/a><\/span> + <span style=\"text-decoration: underline;\"><a href=\"https:\/\/www.seeedstudio.com\/NVIDIArJetson-AGX-Orintm-64GB-Developer-Kit-p-5641.html\">Nvidia Jetson Orin AGX<\/a><\/span><\/strong><br \/>This system utilized a TinyML vision sensor that only triggered the LLaVA analysis when the sensor detected a person, thus avoiding non-relevant frames like cats.<\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fc6c2ce elementor-widget elementor-widget-image\" data-id=\"fc6c2ce\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fc8d61\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t\t\t\t\t\t\t<img fetchpriority=\"high\" decoding=\"async\" width=\"640\" height=\"437\" src=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-1030x703.png\" class=\"attachment-large size-large wp-image-94747\" alt=\"tinyml+LLM Architeture\" srcset=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-1030x703.png 1030w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-300x205.png 300w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-768x524.png 768w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-1536x1049.png 1536w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-32x22.png 32w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-1024x699.png 1024w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM.png 1790w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-390fd25 e-flex e-con-boxed e-con e-parent\" data-id=\"390fd25\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-ef64b42 elementor-widget elementor-widget-text-editor\" data-id=\"ef64b42\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fc9b83\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>The TinyML configuration demonstrated significant reductions in CPU, RAM, bandwidth, GPU usage, and power consumption compared to the direct LLM call setup.<\/p><p><br \/>Here is the demo video.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d57a505 elementor-widget elementor-widget-video\" data-id=\"d57a505\" data-element_type=\"widget\" data-settings=\"{&quot;youtube_url&quot;:&quot;https:\\\/\\\/www.youtube.com\\\/watch?v=q-QY3FflqGQ&amp;t=9s&quot;,&quot;video_type&quot;:&quot;youtube&quot;,&quot;controls&quot;:&quot;yes&quot;}\" data-widget_type=\"video.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fca88d\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t<div class=\"elementor-wrapper elementor-open-inline\">\n\t\t\t<div class=\"elementor-video\"><\/div>\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c9af962 elementor-widget elementor-widget-spacer\" data-id=\"c9af962\" data-element_type=\"widget\" data-widget_type=\"spacer.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fcae77\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t<div class=\"elementor-spacer\">\n\t\t\t<div class=\"elementor-spacer-inner\"><\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-23d90c8 elementor-widget elementor-widget-heading\" data-id=\"23d90c8\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fcb50e\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span><h3 class=\"elementor-heading-title elementor-size-default\">2. Localization of LLMs on on-premise hardwares<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-563aa1f elementor-widget elementor-widget-text-editor\" data-id=\"563aa1f\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fcbb67\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>Another approach to optimize cost and reduce latency is to run LLMs on local PCs or embedded computers &#8211; which is on-device processing. This approach offers several benefits:<\/p>\n<ul>\n<li><strong>Cost reduction<\/strong>: Eliminates the bandwidth and API usage fees associated with data transfer and calling remote online LLMs in the cloud.<\/li>\n<li><strong>Low latency:<\/strong> The latency of obtaining results from LLMs consists of network delays and inference time. By using local LLMs, network latency is minimized. To further decrease inference time, opting for more powerful computers, such as the Jetson Orin AGX, can help achieve even faster response times, potentially as low as 3 seconds.<\/li>\n<li><strong>Enhanced privacy:<\/strong> Running LLMs locally ensures that your data is not shared with public AI platforms, giving you ownership and control over your data.<\/li>\n<\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-5f43c2a e-flex e-con-boxed e-con e-parent\" data-id=\"5f43c2a\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-2877ed5 elementor-widget elementor-widget-text-editor\" data-id=\"2877ed5\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fcc97d\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>Many large models now support local deployment, such as Llama, LLaVA, and Whisper.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-808172f elementor-widget elementor-widget-image\" data-id=\"808172f\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fcda9e\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"640\" height=\"462\" src=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Generative-AI-at-the-edge-1030x743.png\" class=\"attachment-large size-large wp-image-94757\" alt=\"Generative Al at the Edge\" srcset=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Generative-AI-at-the-edge-1030x743.png 1030w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Generative-AI-at-the-edge-300x216.png 300w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Generative-AI-at-the-edge-768x554.png 768w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Generative-AI-at-the-edge-1536x1108.png 1536w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Generative-AI-at-the-edge-32x23.png 32w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Generative-AI-at-the-edge-1024x739.png 1024w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Generative-AI-at-the-edge.png 1736w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c9049d4 elementor-widget elementor-widget-heading\" data-id=\"c9049d4\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fce0f3\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span><h4 class=\"elementor-heading-title elementor-size-default\">Cost Comparison: GPT-4 Turbo vs Local LLaVA\n<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f5ed30b elementor-widget elementor-widget-text-editor\" data-id=\"f5ed30b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fce705\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>Then when should you opt for an online large model, and when is it better to go with a local setup?<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-1652207 e-flex e-con-boxed e-con e-parent\" data-id=\"1652207\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-54705f1 elementor-widget elementor-widget-text-editor\" data-id=\"54705f1\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fcf4f7\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>We conducted a simple cost comparison between the OpenAI&#8217;s GPT-4 Turbo API and a local LLaVA setup on a <span style=\"text-decoration: underline;\"><a href=\"https:\/\/www.seeedstudio.com\/reComputer-J4012-p-5586.html\">reComputer J4012<\/a><\/span> in this case.<\/p><p>Each system processed an image sized 640px x 480px once per minute. We aimed to determine when the cumulative cost of using the GPT-4 Turbo API would exceed the one-time purchase price of the local LLaVA setup on the reComputer J4012.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4b1996a elementor-widget elementor-widget-text-editor\" data-id=\"4b1996a\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fcfb30\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p><strong>Cost details:<\/strong><\/p><ul><li><strong>OpenAI API:<\/strong> <strong>$0.005<\/strong> per image analysis (640&#215;480 px) from <span style=\"text-decoration: underline;\"><a href=\"https:\/\/openai.com\/pricing#language-models\">OpenAI&#8217;s pricing<\/a><\/span> info.<\/li><li><strong>Local LLaVA: $899<\/strong> for the reComputer J4012 &#8211; NVIDIA Jetson Orin NX 16GB module (one-time cost). Since it operates within a local network, there are no bandwidth costs or API usage fees, making the only expense the purchase of the J4012 device.<\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4ecc6fb elementor-widget elementor-widget-image\" data-id=\"4ecc6fb\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd0eae\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"640\" height=\"259\" src=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/cost-comparison-of-gpt4-vs-local-LLaVA-1030x417.png\" class=\"attachment-large size-large wp-image-94761\" alt=\"cost comparison of gpt4 vs local LLaVA\" srcset=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/cost-comparison-of-gpt4-vs-local-LLaVA-1030x417.png 1030w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/cost-comparison-of-gpt4-vs-local-LLaVA-300x122.png 300w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/cost-comparison-of-gpt4-vs-local-LLaVA-768x311.png 768w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/cost-comparison-of-gpt4-vs-local-LLaVA-32x13.png 32w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/cost-comparison-of-gpt4-vs-local-LLaVA-1024x415.png 1024w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/cost-comparison-of-gpt4-vs-local-LLaVA.png 1352w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-75fd0c2 elementor-widget elementor-widget-text-editor\" data-id=\"75fd0c2\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd1575\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>Based on this rough calculation, <strong>using GPT-4 Turbo for five months would cost around $1080, which already exceeds the price of<\/strong>\u00a0<strong>J4012.<\/strong><\/p><p>Therefore, if you only need LLMs for a short term or infrequent usage, opting for a public LLM is suitable.<\/p><p>However, if you require long-term use of them and are sensitive to costs, installing them on a local computer is a very effective strategy. Higher-resolution images incur higher costs, making the cost advantage of local deployment even more pronounced.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-03a3409 e-flex e-con-boxed e-con e-parent\" data-id=\"03a3409\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-f024745 elementor-widget elementor-widget-text-editor\" data-id=\"f024745\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd23a9\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p><strong>But how does the inference speed of localized LLMs?<\/strong><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9995d54 elementor-widget elementor-widget-text-editor\" data-id=\"9995d54\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd29b1\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>Please refer to the chart below, where we have quantified various popular Llama models and then tested them on the Jetson Orin AGX 64GB to compare their inference speeds.<\/p><p>If we take human reading speed as a benchmark, which is typically between 3-7 tokens per second, then an inference speed exceeding 8 tokens per second would be quite sufficient. It appears that running a large 13B model is feasible on this hardware, and for a 7B model, the performance is even more superior.<\/p><p>However, if you need to run even larger models, such as a 33B model, and have specific requirements for inference speed, it is advisable to opt for a more powerful computer.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-743324e e-flex e-con-boxed e-con e-parent\" data-id=\"743324e\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-89394ec elementor-widget elementor-widget-image\" data-id=\"89394ec\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd40c5\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"640\" height=\"358\" src=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/inference-speed-of-different-llama-1030x576.png\" class=\"attachment-large size-large wp-image-94768\" alt=\"\" srcset=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/inference-speed-of-different-llama-1030x576.png 1030w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/inference-speed-of-different-llama-300x168.png 300w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/inference-speed-of-different-llama-768x429.png 768w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/inference-speed-of-different-llama-1536x859.png 1536w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/inference-speed-of-different-llama-32x18.png 32w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/inference-speed-of-different-llama-1024x573.png 1024w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/inference-speed-of-different-llama.png 1706w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4097fec elementor-widget elementor-widget-spacer\" data-id=\"4097fec\" data-element_type=\"widget\" data-widget_type=\"spacer.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd4695\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t<div class=\"elementor-spacer\">\n\t\t\t<div class=\"elementor-spacer-inner\"><\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3531a9b elementor-widget elementor-widget-heading\" data-id=\"3531a9b\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd4c9b\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span><h2 class=\"elementor-heading-title elementor-size-default\">Exploring Seeed's Products Supporting  TinyML + Local Generative AI Architecture<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ef645a6 elementor-widget elementor-widget-text-editor\" data-id=\"ef645a6\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd52dd\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>Seeed products are divided into two main categories: <strong>AI Sensors and Edge Computers<\/strong>.<\/p><p>These products are supported by the Seeed <span style=\"text-decoration: underline;\"><a href=\"https:\/\/sensecraft.seeed.cc\/\">SenseCraft software suites<\/a><\/span>, enabling sophisticated and integrated solutions.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a7cae13 elementor-widget elementor-widget-heading\" data-id=\"a7cae13\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd58d9\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span><h3 class=\"elementor-heading-title elementor-size-default\">AI Sensors<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-63aab44 elementor-widget elementor-widget-text-editor\" data-id=\"63aab44\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd5f6a\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>The representative products of AI sensors are <strong>SenseCAP Watcher, Grove Vision AI Sensor V2<\/strong>, and <strong>XIAO<\/strong>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-748e916 elementor-widget elementor-widget-text-editor\" data-id=\"748e916\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd6697\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p><span style=\"text-decoration: underline;\"><a target=\"_blank\" href=\"https:\/\/www.seeedstudio.com\/watcher\" rel=\"noopener\">SenseCAP Watcher<\/a>\u00a0<\/span>&#8211; a physical LLM agent for smarter spaces. It natively supports various tinyML and Gen AI models. It can assist in monitoring a designated space, detect any activity that matters to you, and notify you promptly on the SenseCraft APP or your own app.<\/p><p><span data-font-family=\"Inter, sans-serif\">As <\/span><span data-font-family=\"default\">the world&#8217;s first physical LLM agent for smarter space. <\/span><span data-font-family=\"Inter, sans-serif\">SenseCAP Watcher\u00a0is able to:<\/span><\/p><ul><li><span data-font-family=\"inherit\">Monitor a designated space.<\/span><\/li><li><span data-font-family=\"inherit\">Identify and interact with targets you specified.<\/span><\/li><li><span data-font-family=\"inherit\">Spot noteworthy events and give notifications.<\/span><\/li><\/ul><div>\u00a0<\/div><p><b><span data-font-family=\"-apple-system, system-ui, BlinkMacSystemFont, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif !important\">Simply give voice \/ APP commands like \u201ctell me when you see a person,\u201d and the Watcher will notify you when such events occur. <\/span><\/b><span data-font-family=\"-apple-system, system-ui, &quot;system-ui&quot;, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif\">But it is not just about detecting targets, it leverages LLM&#8217;s capabilities to analyze<\/span> <b><span data-font-family=\"-apple-system, system-ui, BlinkMacSystemFont, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif !important\">behaviors<\/span><\/b><b><span data-font-family=\"-apple-system, system-ui, &quot;system-ui&quot;, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif\"> and <\/span><\/b><b><span data-font-family=\"-apple-system, system-ui, BlinkMacSystemFont, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif !important\">states<\/span><\/b><b><span data-font-family=\"-apple-system, system-ui, &quot;system-ui&quot;, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif\">.<\/span><\/b><span data-font-family=\"-apple-system, system-ui, &quot;system-ui&quot;, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif\"> Like identifying a <\/span><b><span data-font-family=\"-apple-system, system-ui, BlinkMacSystemFont, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif !important\">person<\/span><\/b><b><span data-font-family=\"-apple-system, system-ui, &quot;system-ui&quot;, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif\"> + <\/span><\/b><b><span data-font-family=\"-apple-system, system-ui, BlinkMacSystemFont, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif !important\">wearing a red shirt<\/span><\/b><b><span data-font-family=\"-apple-system, system-ui, &quot;system-ui&quot;, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif\">, a\u00a0<\/span><\/b><b><span data-font-family=\"-apple-system, system-ui, BlinkMacSystemFont, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif !important\">dog<\/span><\/b><b><span data-font-family=\"-apple-system, system-ui, &quot;system-ui&quot;, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif\"> is <\/span><\/b><b><span data-font-family=\"-apple-system, system-ui, BlinkMacSystemFont, &quot;PingFang SC&quot;, &quot;Microsoft YaHei&quot;, sans-serif !important\">tearing up tissues.<\/span><\/b><\/p><p><strong>Now this product is LIVE on Kickstarter<\/strong>. Click <strong><span style=\"text-decoration: underline;\"><a href=\"https:\/\/www.kickstarter.com\/projects\/seeed\/sensecap-watcher-open-source-ai-assistant-for-smarter-spaces?ref=9thux2\">here<\/a><\/span><\/strong> to back us, and get the early bird price now.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e3ebb65 elementor-widget elementor-widget-image\" data-id=\"e3ebb65\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd76ff\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t\t\t\t\t\t\t\t<a href=\"https:\/\/forms.gle\/3D6Wj2K5hL9dH5nH7\">\n\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"640\" height=\"140\" src=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/\u90ae\u7bb1-1030x225.jpg\" class=\"attachment-large size-large wp-image-101828\" alt=\"\" srcset=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/\u90ae\u7bb1-1030x225.jpg 1030w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/\u90ae\u7bb1-300x65.jpg 300w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/\u90ae\u7bb1-768x168.jpg 768w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/\u90ae\u7bb1-32x7.jpg 32w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/\u90ae\u7bb1-1024x224.jpg 1024w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/\u90ae\u7bb1.jpg 1420w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/>\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-60d35d4 elementor-widget elementor-widget-text-editor\" data-id=\"60d35d4\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd7db0\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p><span data-font-family=\"-apple-system, system-ui, &quot;system-ui&quot;, &quot;Segoe UI&quot;, Roboto, Oxygen-Sans, Ubuntu, Cantarell, &quot;Helvetica Neue&quot;, sans-serif\"><strong><span style=\"text-decoration: underline;\"><a href=\"https:\/\/www.seeedstudio.com\/Grove-Vision-AI-Module-V2-p-5851.html\">The\u00a0Grove\u00a0Vision\u00a0AI\u00a0Sensor\u00a0V2<\/a><\/span> &#8211;<\/strong> is your BEST choice if you want to add a dedicated sub-processor for vector data to your main controller like Arduino \/ Raspberry Pi. Featuring the <span style=\"text-decoration: underline;\"><a href=\"https:\/\/community.arm.com\/arm-community-blogs\/b\/architectures-and-processors-blog\/posts\/cortex-m55-and-ethos-u55-processors-extending-the-performance-of-arm-ml-portfolio-for-endpoint-devices\">Arm Cortex-M55 &amp; Ethos-U55<\/a><\/span>, it provides a 480x uplift in ML performance over existing Cortex-M based systems.<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8652e65 elementor-widget elementor-widget-image\" data-id=\"8652e65\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd90a8\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t\t\t\t\t\t\t\t<a href=\"https:\/\/www.seeedstudio.com\/Grove-Vision-AI-Module-V2-p-5851.html\">\n\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"640\" height=\"267\" src=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/vision-AI-v2-banner_7-1-1030x430.png\" class=\"attachment-large size-large wp-image-94799\" alt=\"\" srcset=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/vision-AI-v2-banner_7-1-1030x430.png 1030w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/vision-AI-v2-banner_7-1-300x125.png 300w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/vision-AI-v2-banner_7-1-768x321.png 768w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/vision-AI-v2-banner_7-1-1536x641.png 1536w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/vision-AI-v2-banner_7-1-2048x855.png 2048w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/vision-AI-v2-banner_7-1-32x13.png 32w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/vision-AI-v2-banner_7-1-1024x428.png 1024w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/>\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-735c0b7 elementor-widget elementor-widget-text-editor\" data-id=\"735c0b7\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fd9b04\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>Or build your own AI sensors with <strong><span style=\"text-decoration: underline;\"><a href=\"https:\/\/wiki.seeedstudio.com\/tinyml_topic\/\">XIAO &#8211; tinyML MCUs<\/a><\/span>.<\/strong><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-addf394 elementor-widget elementor-widget-image\" data-id=\"addf394\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fdcd44\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"640\" height=\"278\" src=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Xiao-tinyml-mcus-1030x448.png\" class=\"attachment-large size-large wp-image-94772\" alt=\"\" srcset=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Xiao-tinyml-mcus-1030x448.png 1030w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Xiao-tinyml-mcus-300x131.png 300w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Xiao-tinyml-mcus-768x334.png 768w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Xiao-tinyml-mcus-1536x668.png 1536w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Xiao-tinyml-mcus-32x14.png 32w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Xiao-tinyml-mcus-1024x446.png 1024w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/Xiao-tinyml-mcus.png 1820w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cc122dd elementor-widget elementor-widget-heading\" data-id=\"cc122dd\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fdd6d0\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span><h3 class=\"elementor-heading-title elementor-size-default\">Edge Computers<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-61f202b elementor-widget elementor-widget-text-editor\" data-id=\"61f202b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fde035\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>The representative products in this category are the <strong><span style=\"text-decoration: underline;\"><a href=\"https:\/\/www.seeedstudio.com\/tag\/nvidia.html\">reComputer Nvidia Jetson series<\/a><\/span>.<\/strong> These are capable of running local Generative AI (Gen AI), ranging from the entry-level 40 TOPS J3011 to the high-performance 275 TOPS J50.<\/p><p>Click <span style=\"text-decoration: underline;\"><a href=\"https:\/\/www.seeedstudio.com\/blog\/edge-ai-generative-ai\/\">he<\/a><\/span><span style=\"text-decoration: underline;\"><a href=\"https:\/\/www.seeedstudio.com\/blog\/edge-ai-generative-ai\/\">re<\/a><\/span> to learn more about this product line for Gen AI.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-121d3de elementor-widget elementor-widget-image\" data-id=\"121d3de\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fdfa08\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t\t\t\t\t\t\t\t<a href=\"https:\/\/www.seeedstudio.com\/blog\/edge-ai-generative-ai\/\">\n\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"640\" height=\"422\" src=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-1030x679.png\" class=\"attachment-large size-large wp-image-94773\" alt=\"\" srcset=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-1030x679.png 1030w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-300x198.png 300w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-768x507.png 768w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-1536x1013.png 1536w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-32x21.png 32w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-1024x676.png 1024w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer.png 1810w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/>\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3c26685 elementor-widget elementor-widget-image\" data-id=\"3c26685\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fe0afe\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"640\" height=\"419\" src=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-selection-guide-1030x675.png\" class=\"attachment-large size-large wp-image-94774\" alt=\"\" srcset=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-selection-guide-1030x675.png 1030w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-selection-guide-300x196.png 300w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-selection-guide-768x503.png 768w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-selection-guide-1536x1006.png 1536w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-selection-guide-32x21.png 32w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-selection-guide-1024x671.png 1024w, https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/recomputer-selection-guide.png 1808w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-439e77a elementor-widget elementor-widget-spacer\" data-id=\"439e77a\" data-element_type=\"widget\" data-widget_type=\"spacer.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fe121a\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t<div class=\"elementor-spacer\">\n\t\t\t<div class=\"elementor-spacer-inner\"><\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-459c5e6 elementor-widget elementor-widget-text-editor\" data-id=\"459c5e6\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span id=\"scroll6a237e1fe188a\"  class=\"scrollMagicControl\" type=\"hidden\" effect = {} wpmp_enable_desktop=\"yes\" wpmp_enable_tablet=\"yes\" wpmp_enable_mobile=\"yes\" wpmp_trigger_hook=\"0.5\" wpmp_reverse=\"yes\" wpmp_class_CSS =\"custom\" split-text = {} value=\"scrollmagic\"><\/span>\t\t\t\t<p>\u00a0I hope the insights and information shared have been enlightening and inspiring.<\/p><p>If you have any thoughts, experiences, or questions you&#8217;d like to share, please don&#8217;t hesitate to comment below.\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Challenge In recent days, the capabilities of large language models (LLMs) are advancing rapidly, and<\/p>\n","protected":false},"author":35,"featured_media":94747,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_lmt_disableupdate":"","_lmt_disable":"","_price":"","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"","_tribe_ticket_capacity":"0","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"iawp_total_views":0,"footnotes":""},"categories":[4391,4393],"tags":[4726,1608,1824,4869,3171],"class_list":["post-94730","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-build","category-tech","tag-llm","tag-machine-learning","tag-nvidia-jetson","tag-sensecap-watcher","tag-tinyml"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v24.0 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>TinyML + Local LLMs: A Trendy Architecture for Efficient and Affordable Edge AI - Latest News from Seeed Studio<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"TinyML + Local LLMs: A Trendy Architecture for Efficient and Affordable Edge AI - Latest News from Seeed Studio\" \/>\n<meta property=\"og:description\" content=\"Challenge In recent days, the capabilities of large language models (LLMs) are advancing rapidly, and\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/\" \/>\n<meta property=\"og:site_name\" content=\"Latest News from Seeed Studio\" \/>\n<meta property=\"article:published_time\" content=\"2024-05-03T19:47:09+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-09-18T08:01:18+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1790\" \/>\n\t<meta property=\"og:image:height\" content=\"1222\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Shuyang Zhou\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Shuyang Zhou\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/\",\"url\":\"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/\",\"name\":\"TinyML + Local LLMs: A Trendy Architecture for Efficient and Affordable Edge AI - Latest News from Seeed Studio\",\"isPartOf\":{\"@id\":\"https:\/\/www.seeedstudio.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM.png\",\"datePublished\":\"2024-05-03T19:47:09+00:00\",\"dateModified\":\"2024-09-18T08:01:18+00:00\",\"author\":{\"@id\":\"https:\/\/www.seeedstudio.com\/blog\/#\/schema\/person\/5589526d3156c5c8151bed987b96f555\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/#primaryimage\",\"url\":\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM.png\",\"contentUrl\":\"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM.png\",\"width\":1790,\"height\":1222,\"caption\":\"tinyml+LLM Architeture\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.seeedstudio.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"TinyML + Local LLMs: A Trendy Architecture for Efficient and Affordable Edge AI\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.seeedstudio.com\/blog\/#website\",\"url\":\"https:\/\/www.seeedstudio.com\/blog\/\",\"name\":\"Latest News from Seeed Studio\",\"description\":\"Emerging IoT, AI and Autonomous Applications on the Edge\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.seeedstudio.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.seeedstudio.com\/blog\/#\/schema\/person\/5589526d3156c5c8151bed987b96f555\",\"name\":\"Shuyang Zhou\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.seeedstudio.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/bcfb71df0d77f0fb207da674aafaa392?s=96&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/bcfb71df0d77f0fb207da674aafaa392?s=96&r=g\",\"caption\":\"Shuyang Zhou\"},\"url\":\"https:\/\/www.seeedstudio.com\/blog\/author\/shuyang\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"TinyML + Local LLMs: A Trendy Architecture for Efficient and Affordable Edge AI - Latest News from Seeed Studio","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/","og_locale":"en_US","og_type":"article","og_title":"TinyML + Local LLMs: A Trendy Architecture for Efficient and Affordable Edge AI - Latest News from Seeed Studio","og_description":"Challenge In recent days, the capabilities of large language models (LLMs) are advancing rapidly, and","og_url":"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/","og_site_name":"Latest News from Seeed Studio","article_published_time":"2024-05-03T19:47:09+00:00","article_modified_time":"2024-09-18T08:01:18+00:00","og_image":[{"width":1790,"height":1222,"url":"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM.png","type":"image\/png"}],"author":"Shuyang Zhou","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Shuyang Zhou","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/","url":"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/","name":"TinyML + Local LLMs: A Trendy Architecture for Efficient and Affordable Edge AI - Latest News from Seeed Studio","isPartOf":{"@id":"https:\/\/www.seeedstudio.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/#primaryimage"},"image":{"@id":"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/#primaryimage"},"thumbnailUrl":"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM.png","datePublished":"2024-05-03T19:47:09+00:00","dateModified":"2024-09-18T08:01:18+00:00","author":{"@id":"https:\/\/www.seeedstudio.com\/blog\/#\/schema\/person\/5589526d3156c5c8151bed987b96f555"},"breadcrumb":{"@id":"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/#primaryimage","url":"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM.png","contentUrl":"https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM.png","width":1790,"height":1222,"caption":"tinyml+LLM Architeture"},{"@type":"BreadcrumbList","@id":"https:\/\/www.seeedstudio.com\/blog\/2024\/05\/03\/tinyml-local-llms-a-trendy-architecture-for-efficient-and-affordable-edge-ai\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.seeedstudio.com\/blog\/"},{"@type":"ListItem","position":2,"name":"TinyML + Local LLMs: A Trendy Architecture for Efficient and Affordable Edge AI"}]},{"@type":"WebSite","@id":"https:\/\/www.seeedstudio.com\/blog\/#website","url":"https:\/\/www.seeedstudio.com\/blog\/","name":"Latest News from Seeed Studio","description":"Emerging IoT, AI and Autonomous Applications on the Edge","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.seeedstudio.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.seeedstudio.com\/blog\/#\/schema\/person\/5589526d3156c5c8151bed987b96f555","name":"Shuyang Zhou","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.seeedstudio.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/bcfb71df0d77f0fb207da674aafaa392?s=96&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/bcfb71df0d77f0fb207da674aafaa392?s=96&r=g","caption":"Shuyang Zhou"},"url":"https:\/\/www.seeedstudio.com\/blog\/author\/shuyang\/"}]}},"modified_by":"Shuyang Zhou","views":19161,"featured_image_urls":{"full":["https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM.png",1790,1222,false],"thumbnail":["https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-80x80.png",80,80,true],"medium":["https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-300x205.png",300,205,true],"medium_large":["https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-768x524.png",640,437,true],"large":["https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-1030x703.png",640,437,true],"1536x1536":["https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-1536x1049.png",1536,1049,true],"2048x2048":["https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM.png",1790,1222,false],"visody_icon":["https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-32x22.png",32,22,true],"magazine-7-slider-full":["https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-1536x1020.png",1536,1020,true],"magazine-7-slider-center":["https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-936x897.png",936,897,true],"magazine-7-featured":["https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-1024x699.png",1024,699,true],"magazine-7-medium":["https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-720x380.png",720,380,true],"magazine-7-medium-square":["https:\/\/www.seeedstudio.com\/blog\/wp-content\/uploads\/2024\/05\/tinymlLLM-675x450.png",675,450,true]},"author_info":{"display_name":"Shuyang Zhou","author_link":"https:\/\/www.seeedstudio.com\/blog\/author\/shuyang\/"},"category_info":"<a href=\"https:\/\/www.seeedstudio.com\/blog\/category\/build\/\" rel=\"category tag\">Build<\/a> <a href=\"https:\/\/www.seeedstudio.com\/blog\/category\/tech\/\" rel=\"category tag\">Tech<\/a>","tag_info":"Tech","comment_count":"0","_links":{"self":[{"href":"https:\/\/www.seeedstudio.com\/blog\/wp-json\/wp\/v2\/posts\/94730","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.seeedstudio.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.seeedstudio.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.seeedstudio.com\/blog\/wp-json\/wp\/v2\/users\/35"}],"replies":[{"embeddable":true,"href":"https:\/\/www.seeedstudio.com\/blog\/wp-json\/wp\/v2\/comments?post=94730"}],"version-history":[{"count":77,"href":"https:\/\/www.seeedstudio.com\/blog\/wp-json\/wp\/v2\/posts\/94730\/revisions"}],"predecessor-version":[{"id":101840,"href":"https:\/\/www.seeedstudio.com\/blog\/wp-json\/wp\/v2\/posts\/94730\/revisions\/101840"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.seeedstudio.com\/blog\/wp-json\/wp\/v2\/media\/94747"}],"wp:attachment":[{"href":"https:\/\/www.seeedstudio.com\/blog\/wp-json\/wp\/v2\/media?parent=94730"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.seeedstudio.com\/blog\/wp-json\/wp\/v2\/categories?post=94730"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.seeedstudio.com\/blog\/wp-json\/wp\/v2\/tags?post=94730"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}