CLIP ViT-B/32

Quick Start

1. Install Docker

Run the following commands on the development board to install Docker:

# Download installation script
curl -fsSL https://get.docker.com -o get-docker.sh
# Install using Aliyun mirror source
sudo sh get-docker.sh --mirror Aliyun
# Start Docker and enable auto-start on boot
sudo systemctl enable docker
sudo systemctl start docker

2. Run the Project (One command, dual-mode preview)

This project supports zero-shot image classification and natural-language image retrieval through a Web browser. Both the image and text encoders run as RKNN models.

Step A: Pull Images

sudo docker pull ghcr.io/Seeed-Projects/recomputer-rk-cv/rk3588-clip:latest
sudo docker pull ghcr.io/Seeed-Projects/recomputer-rk-cv/rk3576-clip:latest

Step B: Run with One Click

For RK3588:

sudo docker run --rm --privileged --net=host \
    -e PYTHONUNBUFFERED=1 \
    -e RKNN_LOG_LEVEL=0 \
    -v /proc/device-tree/compatible:/proc/device-tree/compatible:ro \
    ghcr.io/seeed-projects/recomputer-rk-cv/rk3588-clip:latest

Access via: http://<Board_IP>:8000

For RK3576:

sudo docker run --rm --privileged --net=host \
    -e PYTHONUNBUFFERED=1 \
    -e RKNN_LOG_LEVEL=0 \
    -v /proc/device-tree/compatible:/proc/device-tree/compatible:ro \
    ghcr.io/seeed-projects/recomputer-rk-cv/rk3576-clip:latest

Access via: http://<Board_IP>:8000

3. Command-Line Inference

Classify an image with candidate prompts:

python inference.py --platform rk3576 --model_dir model \
    --file samples/dog_224x224.jpg \
    --prompts 'a photo of a dog|a photo of a cat'

Rank multiple images with a natural-language query:

python inference.py --platform rk3576 --model_dir model \
    --files image1.jpg image2.jpg --query 'a red car' --topk 2

Use --platform rk3588 when running the RK3588 source directory.

🔌 API Documentation

The service provides separate REST interfaces for zero-shot classification and text-to-image retrieval. Interactive OpenAPI documentation is available at http://<Board_IP>:8000/docs.

1. Zero-Shot Image Classification

Endpoint: POST /api/models/clip/predict

Request Parameters (Multipart/Form-Data):

• file: Required image file. Images are preprocessed to 224 x 224.
• prompts: Required candidate labels as a JSON array, newline-separated text, or |-separated text. No more than 32 labels are accepted.
• topk: Optional number of returned labels, from 1 to 32. The default is 5.

Usage Example:

curl -X POST "http://127.0.0.1:8000/api/models/clip/predict" \
    -F 'file=@samples/dog_224x224.jpg' \
    -F $'prompts=a photo of a dog\na photo of a cat' \
    -F 'topk=2'

Response Format (JSON):

{
  "success": true,
  "model": "clip",
  "platform": "rk3576",
  "inference_time_ms": 58.2,
  "result": {
    "mode": "classification",
    "image": {"width": 224, "height": 224},
    "candidate_count": 2,
    "predictions": [
      {"text": "a photo of a dog", "score": 0.97}
    ]
  }
}

2. Natural-Language Image Retrieval

Endpoint: POST /api/models/clip/retrieve

Request Parameters (Multipart/Form-Data):

• files: Required repeated field containing 1–32 images.
• text: Required natural-language query.
• topk: Optional number of returned matches, from 1 to 32.

Usage Example:

curl -X POST "http://127.0.0.1:8000/api/models/clip/retrieve" \
    -F 'files=@image1.jpg' \
    -F 'files=@image2.jpg' \
    -F 'text=a red car' \
    -F 'topk=2'

The converted text encoder has a fixed 20-token input, so prompts and queries should remain concise. Similarity is calculated with normalized image and text features.

3. System Configuration Interface

Get Current Configuration

• Endpoint: GET /api/config
• Response: {"topk": 5}

Update Configuration

curl -X POST "http://127.0.0.1:8000/api/config" \
    -H 'Content-Type: application/json' \
    -d '{"topk": 3}'

Use GET /api/health to inspect the active platform, loaded RKNN files, and runtime capabilities.

🛠️ Developer Guide (Production Recommendations)

Code Description

• web_service.py:
  • Web API: Provides the browser interface, prediction API, retrieval API, configuration API, and health status.
  • Request Validation: Limits uploads to 64 MB and retrieval to 32 images.
• task_runtime.py:
  • Image Processing: Resizes input to 224 x 224 and normalizes image features.
  • Text Processing: Tokenizes prompts with clip_vocab.h and applies the converted model's fixed 20-token input.
  • Ranking: Computes normalized cosine similarity for classification and retrieval.
• inference.py: Provides one-shot command-line classification and retrieval.

Model Files

Modifying Models

1. Place compatible image and text encoder .rknn files in the model/ directory and preserve the filenames above.
2. Keep the two encoders and vocabulary from the same conversion bundle.
3. Confirm the replacement maintains the 224 x 224 image input and fixed 20-token text contract before deployment.

The upstream project currently supplies one model bundle per platform. The two encoders are pipeline components rather than alternative quantization sizes; this release has no model-variant parameter.

Build Local Images

Run from the root of reComputer-RK-CV:

docker build -f docker/rk3576/clip.dockerfile \
    -t rk3576-clip:local src/rk3576_clip

docker build -f docker/rk3588/clip.dockerfile \
    -t rk3588-clip:local src/rk3588_clip