Building a Private Home Assistant Voice Assistant on the RK3576

Building a Private Home Assistant Voice Assistant on the RK3576

Home Assistant Assist can be more than a cloud-connected voice interface. With a Rockchip RK3576 board, it is possible to run the complete voice pipeline locally: speech recognition, wake-word detection, language-model reasoning, and speech synthesis.

This project packages that pipeline as a Docker Compose stack. It combines Whisper, Piper, openWakeWord, and Qwen, with the main AI workloads accelerated by the RK3576 NPU.

1. Quick Start

Requirements

• A Rockchip RK3576 board running Linux ARM64
• Docker Engine and the Docker Compose plugin
• Access to the RK3576 device nodes, including /dev/rknpu and /dev/dma_heap
• A Home Assistant instance on the same network, or enough resources to run Home Assistant on the board

Clone the project on the RK3576 board:

git clone https://github.com/Hanzo-Huang/rk3576-home-assistant-voice.git
cd rk3576-home-assistant-voice

Start the voice services:

sudo docker compose up -d --pull always

If Home Assistant should run on the same board, use the optional profile:

sudo docker compose --profile homeassistant up -d --pull always

Check the running containers and logs:

sudo docker compose ps
sudo docker compose logs -f

2. Architecture

The stack separates each stage of the voice interaction into a service. Home Assistant communicates with the speech services through the Wyoming protocol and with the local language model through an OpenAI-compatible API.

User
  │ voice input
  ▼
Home Assistant Assist
  │ audio
  ▼
openWakeWord ── activated ──▶ Whisper STT
                                  │ transcript
                                  ▼
                         Qwen 2.5 Local LLM
                                  │ response text
                                  ▼
                              Piper TTS
                                  │ audio reply
                                  ▼
                         Home Assistant Assist
                                  │
                                  ▼
                                User

Whisper STT, Qwen 2.5, and Piper TTS run with RK3576 NPU acceleration.

The services are exposed on these ports:

The default language model is Qwen2.5-1.5B-Instruct using W4A16 quantization. The RKLLM service makes it available to Home Assistant as a local conversation agent, so voice requests can be processed without sending conversation data to a remote provider.

3. Performance

The tested configuration uses:

• Whisper for speech-to-text
• Qwen2.5-1.5B-Instruct for conversation
• Piper Amy Medium for speech synthesis

With the models accelerated by the RK3576 NPU, the measured latency was:

The end-to-end response time depends on the length of the spoken request, the generated answer, and Home Assistant’s own processing time. Even so, the RK3576 provides a practical foundation for a responsive, private voice assistant at the edge.

The 1.5B model uses approximately 1.5 GB of RAM. A tested 3B W4A16 model uses approximately 2.5 GB of RAM, making memory capacity an important consideration when choosing a larger model.

4. Setup Guide

Add the Wyoming services

In Home Assistant, open Settings → Devices & services, select Add integration, and search for Wyoming Protocol.

Add the following services using the RK3576 board’s IP address as the host:

Create an Assist pipeline

Open Settings → Voice assistants and create or edit an Assist pipeline. Select:

1. The Wyoming Whisper service for speech-to-text.
2. The Wyoming Piper service for text-to-speech.
3. The Wyoming openWakeWord service for wake-word detection.

Configure the local conversation agent

To use Qwen as the conversation agent, install the Local LLM integration through HACS. Configure it with the following values:

Backend: OpenAI Compatible Conversations API
API hostname: RK3576_BOARD_IP
API port: 8001
API path: /v1
API key: sk-local
Model name: rkllm-model

The API key is only a placeholder for this local server. After adding the integration, return to the Assist pipeline and select the new local conversation agent.

Choose a different LLM

The LLM is provided as a Docker image, so users can change the model without modifying the Home Assistant configuration. The default image is:

image: ghcr.io/hanzo-huang/rkllm-docker/qwen2.5-1.5b-instruct:w4a16-rk3576

To use another model, select an RK3576-compatible image from the rkllm-docker repository and replace the llm.image value in docker-compose.yml:

llm:
  image: <rkllm-docker-model-image>

Then restart the stack:

sudo docker compose up -d

The available models may have different RAM requirements, response quality, and latency. Check the model documentation before choosing a larger model for the board.

Customize the wake word

The default wake-word model is ok_nabu. Change the openwakeword command in docker-compose.yml to preload another supported model:

command:
  - --uri
  - tcp://0.0.0.0:10400
  - --preload-model
  - ok_nabu

Change the Whisper language

The Whisper image includes English and Chinese vocabularies. English is the default; Chinese can be selected by adding --language zh to the Whisper service command in docker-compose.yml.

5. Improvements: Adding More Models

The current stack uses compact models to balance quality, memory usage, and latency. A natural next step is to support more models across every stage of the voice pipeline: language models, speech-to-text models, and text-to-speech voices.

Add more LLM models

The LLM image can be changed in docker-compose.yml. For example, replace the default image:

image: ghcr.io/hanzo-huang/rkllm-docker/qwen2.5-1.5b-instruct:w4a16-rk3576

with another image from the rkllm-docker repository, then restart the stack:

sudo docker compose up -d

Add more STT models

Whisper is currently used for speech recognition. The Whisper service can be extended to support additional model sizes, languages, and RKNN-converted encoder and decoder pairs. Users could choose a smaller model for lower latency or a larger model for improved transcription accuracy.

STT model support should include:

• English, Chinese, and additional language vocabularies.
• Multiple Whisper model sizes.
• Model-specific configuration in docker-compose.yml.
• Benchmarks for transcription accuracy, memory usage, and real-time factor.

Add more TTS models and voices

Piper currently provides the Amy Medium voice. More Piper voices can be packaged in the TTS image, allowing users to select different languages, accents, and voice styles. RKNN-compatible decoder models can also be added to use the RK3576 NPU for more voice configurations.

TTS model support could include:

• Multiple Piper voices and languages.
• Voice selection through Compose configuration or Home Assistant.
• Different quality and speed profiles.
• Streaming and shorter audio chunks for faster playback.

Other improvements

Future improvements could also include:

• Providing a simple model-selection variable for each service.
• Benchmarking the complete pipeline across model combinations.
• Adding streaming LLM responses to reduce perceived response time.
• Supporting custom wake-word models.
• Documenting recommended model combinations for different RAM capacities.

Larger models may improve conversation quality, but they also require more RAM and can increase latency. The best model is therefore a balance between the capabilities of the board and the experience desired in the home.

Conclusion

This project demonstrates that a modern Home Assistant voice assistant does not need to depend on the cloud. By combining open-source voice models, Docker, the Wyoming protocol, and RK3576 NPU acceleration, the entire interaction can run locally on a compact edge device.

The result is a voice interface that is private, customizable, and tightly integrated with Home Assistant—while leaving room to experiment with faster, larger, and more capable models in the future.