CV / YOLO-World

YOLO-World

Open-vocabulary YOLO-World detection with RKNN detector and CLIP text encoder.

24 downloads

Precision

INT8 detector / FP16 text encoder

Choose the device you're using, the set up guide and documentation will update accordingly.

Platform

Inference Engine

Getting Started

Deploy

sudo docker run --rm --privileged -p 8000:8000 \
  -v /dev/dri/renderD129:/dev/dri/renderD129 \
  -v /proc/device-tree/compatible:/proc/device-tree/compatible:ro \
  ghcr.io/seeed-projects/recomputer-rk-cv/rk3576-yolo_world:latest \
  python3 web_detection.py --platform rk3576 \
  --model_path /app/model/yolo_world_v2s_i8.rknn \
  --text_model /app/model/clip_text_fp16.rknn \
  --text_features /app/model/coco_text_outp.npy \
  --vocab_path /app/model/clip_vocab.txt --class_path /app/model/detect_classes.txt \
  --camera_id -1 --host 0.0.0.0 --port 8000

REST API

Use the REST API to run inference. Copy the commands below.

Curl

curl http://localhost:8080/v1/chat/completions -d '{
  "model": "yolo-world-rknn",
  "messages": [{"role": "user", "content": "Hello"}]
}'

Python

import requests

resp = requests.post(
    "http://localhost:8080/v1/chat/completions",
    json={"model": "yolo-world-rknn", "messages": [{"role": "user", "content": "Hello"}]},
)
print(resp.json())

Model Details

Quick Start

1. Install Docker

Run the following commands on the development board to install Docker:

bash

# Download installation script
curl -fsSL https://get.docker.com -o get-docker.sh
# Install using Aliyun mirror source
sudo sh get-docker.sh --mirror Aliyun
# Start Docker and enable auto-start on boot
sudo systemctl enable docker
sudo systemctl start docker

2. Run the Project (One command, dual-mode preview)

This project provides open-vocabulary detection from reComputer-RK-CV. Both the INT8 YOLO-World detector and FP16 CLIP text encoder run on the NPU, allowing detection categories to change without rebuilding the detector.

The bundled CLIP model is primarily trained for English. Short English noun phrases such as person, red bus, and traffic light are recommended.

Model Information

Component	File
Detector	`/app/model/yolo_world_v2s_i8.rknn`
Text encoder	`/app/model/clip_text_fp16.rknn`
Initial COCO features	`/app/model/coco_text_outp.npy`
BPE vocabulary	`/app/model/clip_vocab.txt`
Default labels	`/app/model/detect_classes.txt`

Step A: Pull Images

bash

sudo docker pull ghcr.io/seeed-projects/recomputer-rk-cv/rk3588-yolo_world:latest
sudo docker pull ghcr.io/seeed-projects/recomputer-rk-cv/rk3576-yolo_world:latest

Step B: Run with One Click

Use the image matching the board and set --platform to the same value.

For RK3576:

bash

sudo docker run --rm --privileged -p 8000:8000 \
  -v /dev/dri/renderD129:/dev/dri/renderD129 \
  -v /proc/device-tree/compatible:/proc/device-tree/compatible:ro \
  ghcr.io/seeed-projects/recomputer-rk-cv/rk3576-yolo_world:latest \
  python3 web_detection.py --platform rk3576 \
  --model_path /app/model/yolo_world_v2s_i8.rknn \
  --text_model /app/model/clip_text_fp16.rknn \
  --text_features /app/model/coco_text_outp.npy \
  --vocab_path /app/model/clip_vocab.txt \
  --class_path /app/model/detect_classes.txt \
  --video_path /app/video/test.mp4 --host 0.0.0.0 --port 8000

For RK3588:

bash

sudo docker run --rm --privileged -p 8000:8000 \
  -v /dev/dri/renderD129:/dev/dri/renderD129 \
  -v /proc/device-tree/compatible:/proc/device-tree/compatible:ro \
  ghcr.io/seeed-projects/recomputer-rk-cv/rk3588-yolo_world:latest \
  python3 web_detection.py --platform rk3588 \
  --model_path /app/model/yolo_world_v2s_i8.rknn \
  --text_model /app/model/clip_text_fp16.rknn \
  --text_features /app/model/coco_text_outp.npy \
  --vocab_path /app/model/clip_vocab.txt \
  --class_path /app/model/detect_classes.txt \
  --video_path /app/video/test.mp4 --host 0.0.0.0 --port 8000

Open http://<BOARD_IP>:8000. Use --prompts "person|red bus|bicycle" to set startup prompts. Between 1 and 80 non-empty pipe-separated prompts are supported, and each prompt is truncated to 20 CLIP tokens.

🔌 API Documentation

1. Prompt Configuration Interface

Endpoint: POST /api/prompts

Request Parameters (JSON):

text: One to 80 non-empty prompt phrases separated by |.

Usage Example:

bash

curl -X POST "http://<BOARD_IP>:8000/api/prompts" \
  -H "Content-Type: application/json" \
  -d '{"text":"person|red bus|traffic light"}'

The request replaces the active prompts used by the browser preview and video stream. Prompt embeddings are cached in memory. GET /api/prompts returns the current prompt list.

2. Model Inference Interface (Predict)

Endpoint: POST /api/models/yolo_world/predict

Request Parameters (Multipart/Form-Data):

file: Image file to analyze.
text: Optional request-only prompt list separated by |.
conf: Optional confidence threshold; defaults to the service value.
iou: Optional NMS IoU threshold; defaults to the service value.

Usage Example:

bash

curl -X POST "http://<BOARD_IP>:8000/api/models/yolo_world/predict" \
  -F "file=@bus.jpg" -F "text=person|bus|red vehicle" \
  -F "conf=0.25" -F "iou=0.45"

The text field changes prompts only for that prediction; /api/prompts changes the active stream prompts. Dynamic embeddings fill the first rows of the detector's fixed 80-row text input, and padded rows are ignored during post-processing.

3. System Configuration Interface (Config)

GET /api/config: Returns the active confidence and NMS thresholds.
POST /api/config: Updates the thresholds used by preview and video processing.

Health, MJPEG, video upload, asynchronous analysis, status, list, and download endpoints are also available. The offline BPE vocabulary means the runtime does not download tokenizer assets from Hugging Face.

4. Command Line Arguments

Argument	Default	Description
`--platform`	Required	Target NPU, `rk3576` or `rk3588`.
`--model_path`	Required	Quantized YOLO-World detector.
`--text_model`	`model/clip_text_fp16.rknn`	CLIP text encoder for dynamic prompts.
`--text_features`	`model/coco_text_outp.npy`	Initial `(1,80,512)` COCO features.
`--vocab_path`	`model/clip_vocab.txt`	Offline CLIP BPE vocabulary.
`--class_path`	Built-in COCO labels	Labels matching the initial feature rows.
`--prompts`	COCO 80 classes	Initial pipe-separated prompt phrases.
`--video_path`	None	Looping local MP4; overrides the camera.
`--camera_id`	`1`	Camera index; `-1` enables analysis-only mode.
`--host` / `--port`	`0.0.0.0` / `8000`	Service address and port.

Real-time Video Stream Interface (Video Feed)

Get the latest annotated MJPEG stream for browser preview:

Endpoint: GET /api/video_feed
Example Usage: <img src="http://<BOARD_IP>:8000/api/video_feed">

🛠️ Developer Guide (Production Recommendations)

Model Replacement Constraints

The detector and text encoder are a matched pipeline. Replacements must retain the detector's 80-row text-feature input, the CLIP embedding width, compatible tokenization, and the output layout expected by the post-processing code.

Build Local Images

bash

docker build -f docker/rk3576/yolo_world.dockerfile \
  -t rk3576-yolo-world:local src/rk3576_yolo_world

docker build -f docker/rk3588/yolo_world.dockerfile \
  -t rk3588-yolo-world:local src/rk3588_yolo_world

Inputs and Outputs

Input: image, video frame, or camera frame plus up to 80 prompts. Output: prompt-matched boxes and confidence scores.