Use SSCMACore library output model info

This wiki provides a step-by-step guide on how to configure the model output on SenseCraft AI for the XIAO ESP32S3 Sense board using the SSCMACore (Seeed SenseCraft Model Assistant Core) library. By following these instructions, you will be able to set up your XIAO ESP32S3 Sense to work with a pre-trained model and utilize the SSCMACore library to process the model’s output.

Prerequisites

Before proceeding, ensure that you have the following:

• XIAO ESP32S3 Sense
• USB-C data cable for connecting the XIAO ESP32S3 Sense to your computer
• Arduino IDE with the Seeed_Arduino_SSCMACore library installed

XIAO ESP32S3 Sense
Get One Now 🖱️

Step 1. Ensure a Model is Loaded on the XIAO ESP32S3 Sense

Before proceeding, make sure that your XIAO ESP32S3 Sense board has a trained model loaded. If you haven’t loaded a model yet, refer to the SenseCraft AI documentation on how to train and deploy models to your device.

• Using a model for XIAO ESP32S3 Sense

If you want to use your own trained model, you can refer to the following two Wikis.

• Type of training - Classification
• Type of training - Object Detection

Step 2. Set Up the SSCMA Library

Download the Seeed_Arduino_SSCMACore library from the GitHub repository.

Download the Library

Add the downloaded library to your Arduino environment by following these steps:

• Open the Arduino IDE.
• Go to Sketch -> Include Library -> Add .ZIP Library.
• Navigate to the downloaded Seeed_Arduino_SSCMACore library and select it.
• Click Open to add the library to your Arduino environment.

Step 3. Choose and Configure the Example Code

The SSCMA library provides two example codes that demonstrate how to work with the model output on the XIAO ESP32S3 Sense board. Choose one of the following examples based on your requirements:

Example 1: Inference

1. Open the inference.ino example code.

cpp

#include <SSCMA_Micro_Core.h>

#include <Arduino.h>
#include <esp_camera.h>


SET_LOOP_TASK_STACK_SIZE(40 * 1024);


SSCMAMicroCore instance;
SSCMAMicroCore::VideoCapture capture;


void setup() {

    // Init serial port
    Serial.begin(115200);

    // Init video capture
    MA_RETURN_IF_UNEXPECTED(capture.begin(SSCMAMicroCore::VideoCapture::DefaultCameraConfigXIAOS3));

    // Init SSCMA Micro Core
    MA_RETURN_IF_UNEXPECTED(instance.begin(SSCMAMicroCore::Config::DefaultConfig));

    Serial.println("Init done");

}

void loop() {

    auto frame = capture.getManagedFrame();

    MA_RETURN_IF_UNEXPECTED(instance.invoke(frame));

    for (const auto& box : instance.getBoxes()) {
        Serial.printf("Box: x=%f, y=%f, w=%f, h=%f, score=%f, target=%d\n", box.x, box.y, box.w, box.h, box.score, box.target);
    }

    for (const auto& cls : instance.getClasses()) {
        Serial.printf("Class: target=%d, score=%f\n", cls.target, cls.score);
    }

    for (const auto& point : instance.getPoints()) {
        Serial.printf("Point: x=%f, y=%f, z=%f, score=%f, target=%d\n", point.x, point.y, point.z, point.score, point.target);
    }

    for (const auto& kp : instance.getKeypoints()) {
        Serial.printf("Keypoints: box: x=%f, y=%f, w=%f, h=%f, score=%f, target=%d\n", kp.box.x, kp.box.y, kp.box.w, kp.box.h, kp.box.score, kp.box.target);
        for (const auto& point : kp.points) {
            Serial.printf("Keypoint: x=%f, y=%f, z=%f, score=%f, target=%d\n", point.x, point.y, point.z, point.score, point.target);
        }
    }

    auto perf = instance.getPerf();
    Serial.printf("Perf: preprocess=%dms, inference=%dms, postprocess=%dms\n", perf.preprocess, perf.inference, perf.postprocess);

}

This example code demonstrates how to perform inference using the SSCMA library and retrieve the model’s output, including bounding boxes, classes, points, and keypoints. The code initializes the video capture, SSCMA Micro Core, and performs inference on each frame captured by the camera. The output of the model, such as bounding boxes, classes, points, and keypoints, is printed to the serial monitor.

Note

The loop() function is where the inference process takes place. It starts by capturing a frame using capture.getManagedFrame() and invoking the inference on the captured frame using instance.invoke(frame).

After inference, the code retrieves and prints various outputs from the model:

• Bounding boxes: The instance.getBoxes() function returns a vector of bounding boxes. Each bounding box contains information such as coordinates (x, y, w, h), score, and target class.
• Classes: The instance.getClasses() function returns a vector of detected classes. Each class contains information about the target class and its corresponding score.
• Points: The instance.getPoints() function returns a vector of detected points. Each point contains information such as coordinates (x, y, z), score, and target class.
• Keypoints: The instance.getKeypoints() function returns a vector of detected keypoints. Each keypoint contains a bounding box and a vector of associated points. The code prints the bounding box information and iterates over the points, printing their coordinates, score, and target class.

Finally, the code retrieves the performance metrics using instance.getPerf() and prints the preprocessing, inference, and postprocessing times in milliseconds.

Example 2: Inference with Callbacks

1. Open the inference_cb.ino example code.

cpp

#include <SSCMA_Micro_Core.h>

#include <Arduino.h>
#include <esp_camera.h>


SET_LOOP_TASK_STACK_SIZE(40 * 1024);


SSCMAMicroCore instance;
SSCMAMicroCore::VideoCapture capture;


void setup() {

    // Init serial port
    Serial.begin(115200);

    // Init video capture
    MA_RETURN_IF_UNEXPECTED(capture.begin(SSCMAMicroCore::VideoCapture::DefaultCameraConfigXIAOS3));

    // Init SSCMA Micro Core
    MA_RETURN_IF_UNEXPECTED(instance.begin(SSCMAMicroCore::Config::DefaultConfig));

    instance.registerPerfCallback(SSCMAMicroCore::DefaultPerfCallback);
    instance.registerBoxesCallback(SSCMAMicroCore::DefaultBoxesCallback);
    instance.registerClassesCallback(SSCMAMicroCore::DefaultClassesCallback);
    instance.registerPointsCallback(SSCMAMicroCore::DefaultPointsCallback);
    instance.registerKeypointsCallback(SSCMAMicroCore::DefaultKeypointsCallback);

    Serial.println("Init done");

}

void loop() {

    auto frame = capture.getManagedFrame();

    MA_RETURN_IF_UNEXPECTED(instance.invoke(frame));

}

This example code demonstrates how to perform inference using the SSCMA library and register callback functions to handle the model’s output. The code initializes the video capture, SSCMA Micro Core, and registers callback functions for performance metrics, bounding boxes, classes, points, and keypoints. During the inference process, the registered callback functions are invoked, allowing you to customize the handling of the model’s output.

Note

The code defines several callback functions:

• perfCb: This function is called when performance metrics are available. It receives an instance of SSCMAMicroCore::PerfMetrics and prints the preprocessing, inference, and postprocessing times.
• boxCb: This function is called for each detected bounding box. It receives an instance of SSCMAMicroCore::Box and prints the box coordinates, score, and target class.
• classCb: This function is called for each detected class. It receives an instance of SSCMAMicroCore::Class and prints the target class and score.
• pointCb: This function is called for each detected point. It receives an instance of SSCMAMicroCore::Point3D and prints the point coordinates, score, and target class.
• keypointsCb: This function is called for each detected keypoint. It receives an instance of SSCMAMicroCore::Keypoints, which contains a bounding box and a vector of associated points. The function prints the bounding box information and iterates over the points, printing their coordinates, score, and target class.

In the setup() function, the serial communication is initialized with a baud rate of 115200. The video capture is then initialized using the capture.begin() function with the default camera configuration for XIAO ESP32S3 Sense. The SSCMA Micro Core is initialized using the instance.begin() function with the default configuration.

After initialization, the code registers the callback functions using the appropriate methods provided by the SSCMAMicroCore instance:

• instance.setPerfCallback(perfCb): Registers the perfCb function as the callback for performance metrics.
• instance.setBoxCallback(boxCb): Registers the boxCb function as the callback for bounding boxes.
• instance.setClassCallback(classCb): Registers the classCb function as the callback for detected classes.
• instance.setPointCallback(pointCb): Registers the pointCb function as the callback for detected points.
• instance.setKeypointsCallback(keypointsCb): Registers the keypointsCb function as the callback for detected keypoints.

The loop() function captures a frame using capture.getManagedFrame() and invokes the inference on the captured frame using instance.invoke(frame). During the inference process, the registered callback functions will be invoked whenever the corresponding output data is available.

You can modify the callback functions to perform specific actions or further process the received data based on your application’s requirements.

Step 4. Upload and Run the Code

Connect your XIAO ESP32S3 Sense board to your computer using the USB-C data cable. Open the selected example code (inference.ino or inference_cb.ino) in the Arduino IDE.

Select the appropriate board and port in the Arduino IDE:

• Go to Tools -> Board and select “XIAO ESP32S3 Sense”.
• Go to Tools -> Port and select the port to which your XIAO ESP32S3 Sense is connected.
• Go to Tools -> PSRAM -> OPI PSRAM. Be sure to turn on PSRAM!

Click the “Upload” button in the Arduino IDE to compile and upload the code to your XIAO ESP32S3 Sense board. Once the upload is complete, open the Serial Monitor in the Arduino IDE to view the output of the model.

Conclusion

By following this step-by-step guide, you should now be able to configure the model output on SenseCraft AI for your XIAO ESP32S3 Sense board using the SSCMA library. Depending on the example code you chose, you can either retrieve the model’s output directly or use callback functions to customize the handling of the output.

Feel free to explore and modify the example codes to suit your specific requirements. The SSCMA library provides a powerful set of tools and functions to work with computer vision and machine learning models on the XIAO ESP32S3 Sense board.

If you encounter any issues or have further questions, please refer to the SenseCraft AI documentation or seek assistance from the Seeed Studio community forums.

Happy coding and exploring the world of computer vision and machine learning with your XIAO ESP32S3 Sense board!

Tech Support & Product Discussion

Thank you for choosing our products! We are here to provide you with different support to ensure that your experience with our products is as smooth as possible. We offer several communication channels to cater to different preferences and needs.