Zipformer

Quick Start

1. Install Docker

Run the following commands on the development board to install Docker:

# Download installation script
curl -fsSL https://get.docker.com -o get-docker.sh
# Install using Aliyun mirror source
sudo sh get-docker.sh --mirror Aliyun
# Start Docker and enable auto-start on boot
sudo systemctl enable docker
sudo systemctl start docker

2. Run the Project (One command, dual-mode preview)

This project provides Chinese-English speech recognition. The Web interface supports complete transcription of uploaded files and streaming browser microphone transcription with partial results.

Step A: Pull Images

sudo docker pull ghcr.io/Seeed-Projects/recomputer-rk-cv/rk3588-zipformer:latest
sudo docker pull ghcr.io/Seeed-Projects/recomputer-rk-cv/rk3576-zipformer:latest

Step B: Run with One Click

For RK3588:

sudo docker run --rm --privileged --net=host \
    -e PYTHONUNBUFFERED=1 \
    -e RKNN_LOG_LEVEL=0 \
    -v /proc/device-tree/compatible:/proc/device-tree/compatible:ro \
    ghcr.io/seeed-projects/recomputer-rk-cv/rk3588-zipformer:latest

Access via: http://<Board_IP>:8000

For RK3576:

sudo docker run --rm --privileged --net=host \
    -e PYTHONUNBUFFERED=1 \
    -e RKNN_LOG_LEVEL=0 \
    -v /proc/device-tree/compatible:/proc/device-tree/compatible:ro \
    ghcr.io/seeed-projects/recomputer-rk-cv/rk3576-zipformer:latest

Access via: http://<Board_IP>:8000

The browser sends microphone PCM over WebSocket, so host /dev/snd mapping is not required.

3. Command-Line Inference

python inference.py --platform rk3576 --model_dir model \
    --file samples/test.wav

Use --platform rk3588 when running the RK3588 source directory.

🔌 API Documentation

The service provides a REST interface for complete file transcription and a WebSocket interface for streaming PCM. OpenAPI documents the REST routes at http://<Board_IP>:8000/docs.

1. File Transcription Interface

Endpoint: POST /api/models/zipformer/predict

Request Parameters (Multipart/Form-Data):

• file: Required audio file. The service decodes the file and resamples it to 16 kHz before inference. The upload limit is 64 MB.

Usage Example:

curl -X POST "http://127.0.0.1:8000/api/models/zipformer/predict" \
    -F 'file=@samples/test.wav'

Response Format (JSON):

{
  "success": true,
  "model": "zipformer",
  "platform": "rk3576",
  "inference_time_ms": 164.3,
  "result": {
    "text": "example transcript",
    "tokens": [12, 34],
    "timestamps": [3, 8],
    "audio_duration_seconds": 2.4,
    "processed": true
  }
}

2. Streaming Transcription Interface

Endpoint: WS /api/models/zipformer/stream

Send mono 16 kHz signed 16-bit little-endian PCM as binary WebSocket messages. When capture is complete, send:

{"action":"stop"}

Send {"action":"reset"} to discard the current streaming session and begin again. Server messages use the following types:

• ready: The session is ready and reports sample_rate: 16000 and format: pcm_s16le.
• partial: Contains an updated result after enough audio is processed.
• final: Contains the completed result after the stop action.
• error: Reports invalid PCM framing or a runtime failure.

3. System Status and Configuration

• Health Endpoint: GET /api/health
• Configuration Endpoint: GET /api/config
• Current Configuration Response: {} because Zipformer has no hot-swappable runtime settings.

Sending non-empty data to POST /api/config is rejected. The encoder, decoder, joiner, and vocabulary are loaded together when the service starts.

🛠️ Developer Guide (Production Recommendations)

Code Description

• web_service.py:
  • REST API: Decodes uploaded audio and returns a complete transcript.
  • WebSocket API: Maintains streaming sessions and emits ready, partial, final, and error messages.
• task_runtime.py:
  • Audio Processing: Creates Kaldi-compatible filterbank features for mono 16 kHz audio.
  • Streaming State: Preserves encoder caches, token history, and timestamps across chunks.
  • RKNN Inference: Runs the encoder, decoder, and joiner pipeline.
• inference.py: Provides one-shot command-line file transcription.

Model Files

Modifying Models

1. Place the converted encoder, decoder, and joiner models in model/ using the filenames above.
2. Keep all three RKNN files aligned with the same vocabulary and streaming cache contract.
3. Verify both full-file inference and binary PCM WebSocket streaming.

These files are stages of one Zipformer pipeline rather than alternative model sizes. The upstream project currently provides one bundle per platform and no separate quantization selector.

Build Local Images

Run from the root of reComputer-RK-CV:

docker build -f docker/rk3576/zipformer.dockerfile \
    -t rk3576-zipformer:local src/rk3576_zipformer

docker build -f docker/rk3588/zipformer.dockerfile \
    -t rk3588-zipformer:local src/rk3588_zipformer