Wav2Vec2

Quick Start

1. Install Docker

Run the following commands on the development board to install Docker:

# Download installation script
curl -fsSL https://get.docker.com -o get-docker.sh
# Install using Aliyun mirror source
sudo sh get-docker.sh --mirror Aliyun
# Start Docker and enable auto-start on boot
sudo systemctl enable docker
sudo systemctl start docker

2. Run the Project (One command, dual-mode preview)

This project provides English automatic speech recognition based on facebook/wav2vec2-base-960h. Use the Web interface to upload an audio file or record audio with the browser microphone.

Step A: Pull Images

sudo docker pull ghcr.io/Seeed-Projects/recomputer-rk-cv/rk3588-wav2vec2:latest
sudo docker pull ghcr.io/Seeed-Projects/recomputer-rk-cv/rk3576-wav2vec2:latest

Step B: Run with One Click

For RK3588:

sudo docker run --rm --privileged --net=host \
    -e PYTHONUNBUFFERED=1 \
    -e RKNN_LOG_LEVEL=0 \
    -v /proc/device-tree/compatible:/proc/device-tree/compatible:ro \
    ghcr.io/seeed-projects/recomputer-rk-cv/rk3588-wav2vec2:latest

Access via: http://<Board_IP>:8000

For RK3576:

sudo docker run --rm --privileged --net=host \
    -e PYTHONUNBUFFERED=1 \
    -e RKNN_LOG_LEVEL=0 \
    -v /proc/device-tree/compatible:/proc/device-tree/compatible:ro \
    ghcr.io/seeed-projects/recomputer-rk-cv/rk3576-wav2vec2:latest

Access via: http://<Board_IP>:8000

Browser microphone permission often requires HTTPS or a trusted/localhost context. File upload remains available without microphone permission, and the container does not require host /dev/snd mapping.

3. Command-Line Inference

python inference.py --platform rk3576 --model_dir model \
    --file samples/test.wav --beam_width 8

Use --platform rk3588 when running the RK3588 source directory.

🔌 API Documentation

The project provides synchronous transcription for uploaded audio. Audio is converted to mono 16 kHz, silence-trimmed, level-normalized, and checked for clipping before inference.

1. Speech Recognition Interface

Endpoint: POST /api/models/wav2vec2/predict

Request Parameters (Multipart/Form-Data):

• file: Required WAV or other libsndfile-supported audio file. The upload limit is 64 MB.
• beam_width: Optional CTC decoding width from 1 to 32. The default is 8; 1 selects greedy decoding.

Usage Example:

curl -X POST "http://127.0.0.1:8000/api/models/wav2vec2/predict" \
    -F 'file=@samples/test.wav' \
    -F 'beam_width=8'

Response Format (JSON):

{
  "success": true,
  "model": "wav2vec2",
  "platform": "rk3576",
  "inference_time_ms": 812.4,
  "result": {
    "text": "EXAMPLE TRANSCRIPT",
    "frames": 999,
    "sample_rate": 16000,
    "original_duration_seconds": 4.2,
    "processed_duration_seconds": 4.0,
    "segments": 1,
    "beam_width": 8,
    "truncated": false,
    "warnings": []
  }
}

The RKNN model has a fixed 20-second input. Longer recordings are split into overlapping segments and merged instead of being truncated. A larger beam_width increases CPU-side decoding work and may improve the transcript; it does not change the RKNN model.

2. System Configuration Interface

Get Current Configuration

• Endpoint: GET /api/config
• Response: {"beam_width": 8}

Update Configuration

curl -X POST "http://127.0.0.1:8000/api/config" \
    -H 'Content-Type: application/json' \
    -d '{"beam_width": 1}'

Use GET /api/health to inspect the active platform, model readiness, loaded RKNN file, and runtime capabilities. OpenAPI documentation is at /docs.

🛠️ Developer Guide (Production Recommendations)

Code Description

• web_service.py:
  • Web API: Handles browser audio uploads, microphone recordings, configuration, and synchronous transcription.
  • Validation: Rejects empty files, files over 64 MB, and invalid beam widths.
• task_runtime.py:
  • Audio Processing: Decodes audio, converts it to mono 16 kHz, trims silence, normalizes levels, and creates overlapping 20-second segments.
  • RKNN Inference: Executes wav2vec2.rknn for each segment.
  • CTC Decoder: Supports greedy and beam-search decoding and merges segment transcripts.
• inference.py: Provides one-shot command-line transcription.

Model Files

Modifying Models

1. Place the converted model at model/wav2vec2.rknn.
2. Preserve the fixed 20-second, mono 16 kHz input contract and output token ordering expected by the included CTC vocabulary.
3. Test both short input and overlapping long-audio segmentation before publishing the replacement.

The upstream project currently provides one fixed-shape RKNN model per platform. beam_width changes the decoder rather than selecting another size or quantization.

Build Local Images

Run from the root of reComputer-RK-CV:

docker build -f docker/rk3576/wav2vec2.dockerfile \
    -t rk3576-wav2vec2:local src/rk3576_wav2vec2

docker build -f docker/rk3588/wav2vec2.dockerfile \
    -t rk3588-wav2vec2:local src/rk3588_wav2vec2