CV / MobileSAM

MobileSAM

Prompt-based MobileSAM image segmentation with RKNN encoder and decoder models.

31 downloads

Precision

RKNN

Choose the device you're using, the set up guide and documentation will update accordingly.

Platform

Inference Engine

Getting Started

Deploy

sudo docker run --rm --privileged --net=host \
  -e PYTHONUNBUFFERED=1 -e RKNN_LOG_LEVEL=0 \
  --device /dev/dri/renderD129:/dev/dri/renderD129 \
  -v /proc/device-tree/compatible:/proc/device-tree/compatible:ro \
  ghcr.io/seeed-projects/recomputer-rk-cv/rk3576-mobilesam:latest \
  python web_service.py --platform rk3576 --model_dir /app/model --camera_id -1 --host 0.0.0.0 --port 8000

REST API

Use the REST API to run inference. Copy the commands below.

Curl

curl http://localhost:8080/v1/chat/completions -d '{
  "model": "mobilesam-rknn",
  "messages": [{"role": "user", "content": "Hello"}]
}'

Python

import requests

resp = requests.post(
    "http://localhost:8080/v1/chat/completions",
    json={"model": "mobilesam-rknn", "messages": [{"role": "user", "content": "Hello"}]},
)
print(resp.json())

Model Details

Quick Start

1. Install Docker

Run the following commands on the development board to install Docker:

bash

# Download installation script
curl -fsSL https://get.docker.com -o get-docker.sh
# Install using Aliyun mirror source
sudo sh get-docker.sh --mirror Aliyun
# Start Docker and enable auto-start on boot
sudo systemctl enable docker
sudo systemctl start docker

2. Run the Project (One command, dual-mode preview)

This deployment packages Mobile Segment Anything from reComputer-RK-CV. The image encoder and fixed two-prompt decoder both run as RKNN models.

Model Information

Component	File
Image encoder	`model/mobilesam_encoder.rknn`
Prompt decoder	`model/mobilesam_decoder.rknn`
Input preprocessing	`448 x 448`
Prompts	Box or foreground/background points
Output	Mask, selected-mask index, quality score, and pixel count

Step A: Pull Images

bash

sudo docker pull ghcr.io/seeed-projects/recomputer-rk-cv/rk3588-mobilesam:latest
sudo docker pull ghcr.io/seeed-projects/recomputer-rk-cv/rk3576-mobilesam:latest

Step B: Run with One Click

For RK3576:

bash

sudo docker run --rm --privileged --net=host \
  -e PYTHONUNBUFFERED=1 -e RKNN_LOG_LEVEL=0 \
  --device /dev/dri/renderD129:/dev/dri/renderD129 \
  -v /proc/device-tree/compatible:/proc/device-tree/compatible:ro \
  ghcr.io/seeed-projects/recomputer-rk-cv/rk3576-mobilesam:latest \
  python web_service.py --platform rk3576 --model_dir /app/model \
  --camera_id -1 --host 0.0.0.0 --port 8000

For RK3588:

bash

sudo docker run --rm --privileged --net=host \
  -e PYTHONUNBUFFERED=1 -e RKNN_LOG_LEVEL=0 \
  --device /dev/dri/renderD129:/dev/dri/renderD129 \
  -v /proc/device-tree/compatible:/proc/device-tree/compatible:ro \
  ghcr.io/seeed-projects/recomputer-rk-cv/rk3588-mobilesam:latest \
  python web_service.py --platform rk3588 --model_dir /app/model \
  --camera_id -1 --host 0.0.0.0 --port 8000

Open http://<BOARD_IP>:8000 to upload an image and draw a box or add foreground/background points. The current prompt is shared by image, camera, local-video, and uploaded-video inference until it is changed.

🔌 API Documentation

1. Prompted Segmentation Interface (Predict)

Endpoint: POST /api/models/mobilesam/predict

Request Parameters (Multipart/Form-Data):

file: Image file to segment.
point_coords: JSON array containing exactly two source-image coordinates.
point_labels: JSON array containing exactly two prompt labels. Use 0 for a negative point, 1 for a positive point, 2 for the top-left box corner, and 3 for the bottom-right box corner.

Usage Examples:

bash

curl -X POST "http://<BOARD_IP>:8000/api/models/mobilesam/predict" \
  -F "file=@picture.jpg" \
  -F 'point_coords=[[190,70],[460,280]]' \
  -F 'point_labels=[2,3]'

The runtime scales prompt coordinates to the encoder input, runs the encoder and decoder, selects the requested or best-scoring mask, and restores it to the source image. Health, configuration, MJPEG, and asynchronous-video interfaces are also available.

2. System Configuration Interface (Config)

The response contains iou_scores, selected_mask, mask_pixels, and the effective prompt. GET /api/config returns the persistent stream prompt; POST /api/config accepts point_coords and point_labels, or both as null to restore full-image mode. Generic threshold and topk values do not change mask selection.

Uploaded videos use the persistent prompt active when analysis starts. The standard /api/video/* endpoints provide upload, analysis, progress, list, and download operations.

The encoder and decoder are pipeline stages, not model-size alternatives. Both files must remain compatible when replacing either model. The encoder preserves aspect ratio, resizes the long side to 448, and pads to 448 x 448; the fixed decoder contract requires exactly two prompt coordinates and labels.

3. Command Line Arguments

Argument	Default	Description
`--platform`	Required	`rk3576` or `rk3588`.
`--model_dir`	`model`	Encoder, decoder, and warm-up image directory.
`--camera_id`	`-1`	Camera index; `-1` enables uploads only.
`--video`, `--video_path`	None	Looping local video; overrides the camera.
`--host` / `--port`	`0.0.0.0` / `8000`	Service address and port.

The Web UI supports a dragged box, positive point, negative point, and a full-image prompt. Updating the prompt affects subsequent stream and uploaded video frames without restarting the container.

Real-time Video Stream Interface (Video Feed)

Get the latest annotated MJPEG stream for browser preview:

Endpoint: GET /api/video_feed
Example Usage: <img src="http://<BOARD_IP>:8000/api/video_feed">

🛠️ Developer Guide (Production Recommendations)

Build Local Images

bash

docker build -f docker/rk3576/mobilesam.dockerfile \
  -t rk3576-mobilesam:local src/rk3576_mobilesam

docker build -f docker/rk3588/mobilesam.dockerfile \
  -t rk3588-mobilesam:local src/rk3588_mobilesam

Inputs and Outputs

Input: image plus box or point prompts. Output: segmentation mask, mask score, and overlay.