{"id":87800,"date":"2023-11-28T17:42:18","date_gmt":"2023-11-28T17:42:18","guid":{"rendered":"https:\/\/www.seeedstudio.com\/blog\/?p=87800"},"modified":"2024-05-27T08:37:33","modified_gmt":"2024-05-27T08:37:33","slug":"empowering-travel-safety-with-xiao-esp32s3-sense-round-display-for-xiao-and-tinyml-ai-driven-keychain-detection-for-instant-alerts-and-location-request","status":"publish","type":"post","link":"https:\/\/www.seeedstudio.com\/blog\/2023\/11\/28\/empowering-travel-safety-with-xiao-esp32s3-sense-round-display-for-xiao-and-tinyml-ai-driven-keychain-detection-for-instant-alerts-and-location-request\/","title":{"rendered":"Empowering Travel Safety with XIAO ESP32S3 Sense, Round Display for XIAO, and TinyML: AI-Driven Keychain Detection for Instant Alerts and Location Request"},"content":{"rendered":"\n

Updated on Feb 28th, 2024<\/strong><\/p>\n\n\n\n

Introducing an AI-powered BLE Travel Emergency Assistant utilizing XIAO ESP32S3 Sense, Round Display for XIAO, and TinyML. Detect keychains for emergency alerts, notifying contacts via WhatsApp\/SMS, with added functionality for requesting location info from the Maps API.<\/p>\n\n\n\n

Seeed Hardware<\/strong>: Seeed Studio XIAO ESP32S3 Sense<\/a>, Round Display for XIAO <\/a><\/p>\n\n\n\n

Software:<\/strong> Edge Impulse, Android Studio, MIT App Inventor 2, Twilio API for WhatsApp, Twilio SMS Messaging API, Google Maps, Autodesk Fusion 360, Ultimaker Cura, Microsoft Visual Studio 2017<\/p>\n\n\n\n

Industry:<\/strong> Healthcare, Hospitality, Travel and Tourism, Transportation, Emergency Services, Insurance, Security<\/p>\n\n\n\n

The Background<\/h2>\n\n\n\n

Travelers with disabilities are at greater risk of becoming victims of crimes such as violence, theft, verbal abuse, and neglect. They are at a higher risk, at least four to ten times, of being a victim of crime. To prevent these crimes, it is necessary to provide them with state-of-the-art devices. However, smartphones may not be suitable since they are time-consuming and arduous in a time of crisis for people with mobility impairments. Therefore, budget-friendly and accessible AIoT assistive devices should provide various features in a broader spectrum, considering people with disabilities and special needs.<\/p>\n\n\n

\n
\"\"<\/figure><\/div>\n\n\n

The Challenge<\/h2>\n\n\n\n

To address this problem, Kutluhan Aktar<\/a> created different keychains denoting emergencies and used a XIAO ESP32S3 Sense to capture and save images to an SD card. With Edge Impulse, he built an object detection model to identify the customized keychains and notify emergency contacts via WhatsApp\/SMS. The results are displayed on a Round Display for XIAO through BLE, and emergency contacts can obtain location information from Maps API.<\/p>\n\n\n\n

The Solution<\/h2>\n\n\n\n

Step 1: Designing and printing a smartwatch-inspired case<\/h3>\n\n\n\n

Focused on creating a budget-friendly assistive device, Kutluhan designed a versatile case that easily attaches to mobility aids for seamless communication with emergency contacts via WhatsApp or SMS. The main case, split into two parts for compatibility with different batteries, features snap-fit joints. To enhance functionality and prevent dust exposure, a modular part for XIAO ESP32S3 Sense and Round Display for XIAO was added. Dents in both parts allow flexible positioning of the built-in camera while scanning or collecting samples. The main case is adorned with the Seeed Studio logo, Twilio logo, and a traveling airplane icon, emphasizing the device’s capabilities.<\/p>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/figure>\n\n\n\n

Step 2: Developing a GPS, GPRS, and BLE-enabled Android application w\/ the MIT APP Inventor<\/h3>\n\n\n\n

To ensure the assistive device functions without Wi-Fi, Kutluhan created an Android app using MIT App Inventor<\/a>. Serving as the device’s interface, the app utilizes cellular network connections for data transfer via GPRS, obtains GPS-based location data, and communicates with XIAO ESP32S3 through BLE for model detection and data collection commands.<\/p>\n\n\n\n

Step 3: Creating an account to utilize Twilio’s WhatsApp & SMS APIs<\/h3>\n\n\n\n

In order to notify the user’s emergency contacts about the latest detected keychain via the object recognition model through WhatsApp and SMS, Kutluhan integrated Twilio’s WhatsApp and SMS APIs. Twilio <\/a>facilitates communication with a Twilio-verified phone through WhatsApp using a free webhook for trial accounts. Additionally, Twilio offers a trial text messaging service for international SMS transfer from a virtual to a verified phone number. Supporting official helper libraries in various programming languages, including PHP, Twilio ensures reliable API enforcement.<\/p>\n\n\n\n

Step 4: Developing a web application to communicate w\/ the Android app and process requests from WhatsApp<\/h3>\n\n\n\n

Needing to relay model detection results from XIAO ESP32S3 to inform emergency contacts, Kutluhan developed a basic web application utilizing Twilio’s WhatsApp and SMS APIs. The web app acts as a proxy between the Android application and emergency contacts, allowing the user to update contact information with a single HTTP GET request. Using stored data, the web app automatically notifies contacts via WhatsApp or SMS based on model detection results. Twilio’s APIs enable the web app to receive commands through WhatsApp for sending location inspections generated by Google Maps to the primary emergency contact. To meet Twilio’s requirements, he hosted the web app on an SSL-enabled server, but an HTTP tunneling tool like ngrok can also set up a public URL for the webhook.<\/p>\n\n\n\n

Step 5: Setting up XIAO ESP32S3 Sense and Round Display on Arduino IDE<\/h3>\n\n\n\n

Utilizing the XIAO ESP32S3 Sense expansion board’s file-handling capability, Kutluhan opted to directly capture and save images from the built-in OV2640 camera to the microSD card. XIAO ESP32S3 communicates with the Android application via BLE to receive user commands and transmit model detection results.<\/p>\n\n\n\n

Step 6: Capturing customized keychain (token) images w\/ the built-in OV2640 camera<\/h3>\n\n\n\n

After configuring XIAO ESP32S3 with the round display and installing necessary libraries, Kutluhan programmed it to capture raw image buffers, convert them to JPG files, and save samples directly to the SD card on the expansion board.<\/p>\n\n\n\n

To streamline the creation of a valid dataset for the object detection model, he used the Android application to send user commands over BLE to XIAO ESP32S3. This allowed capturing and saving samples to the SD card with the specified emergency class, eliminating the need for additional components and preventing an increase in the main case dimensions.<\/p>\n\n\n\n

Step 7: Building an object detection (FOMO) model with Edge Impulse<\/h3>\n\n\n\n

After capturing images of customized keychains and storing them on the SD card, Kutluhan focused on developing an object detection (FOMO) model to identify keychains individually for immediate notification to emergency contacts via WhatsApp or SMS.<\/p>\n\n\n\n

Opting for Edge Impulse<\/a>, he leveraged its extensive microcontroller and development board support for model deployment. Edge Impulse’s advanced machine learning algorithm (FOMO) enabled the creation of more accessible and faster object detection models on edge devices like XIAO ESP32S3.<\/p>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/figure>\n\n\n\n

Step 8: Setting up the Edge Impulse FOMO model on XIAO ESP32S3 Sense<\/h3>\n\n\n\n

Having constructed, trained, and deployed the object detection model as an Arduino library on Edge Impulse, Kutluhan uploaded the generated library to XIAO ESP32S3. This allowed the model to run directly, ensuring the creation of an accessible assistive device with minimal latency, memory usage, and power consumption. Edge Impulse’s optimization and formatting of signal processing, configuration, and learning blocks into a single package simplified the import process for running inferences.<\/p>\n\n\n\n

Step 9: Running the model and informing emergency contacts via WhatsApp & SMS<\/h3>\n\n\n\n

Following the successful setup of the Edge Impulse object detection (FOMO) model on XIAO ESP32S3:<\/p>\n\n\n\n

    \n
  1. Users can detect emergency-class keychains using the assistive device through BLE connection with the Android application.<\/li>\n\n\n\n
  2. XIAO ESP32S3 runs an inference every 30 seconds, notifying users upon detecting an emergency class through the XIAO round display.<\/li>\n\n\n\n
  3. Post-detection, XIAO ESP32S3 transfers model results to the Android app via BLE.<\/li>\n\n\n\n
  4. The Android app displays and transfers model results, current location, and date to the web app through GPRS.<\/li>\n\n\n\n
  5. The web app generates a Google Maps URL from decoded location information.<\/li>\n\n\n\n
  6. Using Twilio’s APIs, the web app notifies emergency contacts via WhatsApp or SMS based on the detected emergency class.<\/li>\n<\/ol>\n\n\n
    \n
    \"\"<\/figure><\/div>\n\n\n

    The Conclusion<\/h2>\n\n\n\n

    In summary, integrating object detection models trained on customized keychains into assistive devices not only deters potential crimes against individuals with disabilities but also empowers those with mobility impairments in emergency situations. The system offers versatile notifications for various mobility aids, ensuring prompt communication of emergencies to contacts with precise location data. Additionally, it provides the ability to generate a travel itinerary of previously visited destinations upon request, enhancing its overall utility.<\/p>\n\n\n\n

    \n