reComputer RK3576 Module Dev Kit Application

Last updated: 2026-10-18

Application

Run Local LLM and VLM with RK1820

The RK1820 can be used not only for model performance testing, but also to deploy local LLMs (large language models) and VLMs (vision-language models) on the reComputer RK3576 through rkllm3-server, and to build local AI applications through its OpenAI-compatible API.

This example will introduce:

• Use Qwen3.5-0.8B to build a local LLM chat service

• Use Qwen2.5-VL-3B_prune to build a local image understanding service

rkllm3-server is based on RKNPU3 and provides LLM inference through an OpenAI-compatible Chat API. It can be integrated into Python applications, web UIs, and other AI agent frameworks.

Prerequisites

Before starting this tutorial, please complete the installation of RK1820 and the RKNN3 Runtime configuration, and confirm:

PlainCopy

rknn-smi info -l

The RK1820 should be recognized normally.

Run a Local LLM

This example uses Qwen3.5-0.8B.

The model directory contains:

Qwen2.5-VL-3B_prune/
├── Qwen2.5-VL-3B_prune.embed.bin
├── Qwen2.5-VL-3B_prune.tokenizer.gguf
├── llm_Qwen2.5-VL-3B_prune.rknn
├── llm_Qwen2.5-VL-3B_prune.weight
├── vision_Qwen2.5-VL-3B_prune.rknn
└── vision_Qwen2.5-VL-3B_prune.weight

Enter the model directory:

cd ~/Downloads/Qwen3.5-0.8B

Start the Local LLM Server

Execute:

rkllm3-server \
-m Qwen3.5-0.8B.rknn \
--weight Qwen3.5-0.8B.weight \
--vocab Qwen3.5-0.8B.tokenizer.gguf \
--embed Qwen3.5-0.8B.embed.bin \
--host 0.0.0.0 \
--port 8080 \
-c 1024 \
--n_predict 256 \
--core-mask 0xff \
--repeat-penalty 1.1 \
--presence-penalty 1.0 \
--frequency-penalty 1.0 \
--top-k 1 \
--top-p 0.8 \
--temp 0.8

The server supports:

Rockchip's official rkllm3-server loads the local LLM from the model, weight, vocab, and embed files. It also lets you configure the context length, maximum output tokens, and NPU core mask.

After the model is loaded successfully, you can see something like:

rknn3_model_init success!
main: model loaded
server is listening on http://0.0.0.0:8080

At this point, the local LLM API has started.

Chat with the Local LLM

Open another terminal.

Install the Python OpenAI SDK:

pip3 install openai

Create a test program:

nano llm_chat.py

Write:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8080/v1",
    api_key="sk-no-key-required"
)

response = client.chat.completions.create(
    model="",
    messages=[
        {
            "role": "user",
            "content": "Explain edge AI in three sentences."
        }
    ],
    stream=True,
    extra_body={
        "n_keep": 0,
        "cache_prompt": False,
        "id_slot": 0,
        "n_predict": 256
    }
)

for chunk in response:
    content = chunk.choices[0].delta.content

    if content:
        print(content, end="", flush=True)

print()

Run:

python3 llm_chat.py

The model will generate replies locally through the RK1820.

The official Rockchip example also uses client.chat.completions.create() to access the local RKLLM3 server and supports parameters such as streaming, KV cache, and maximum output tokens.

Qwen3.5-0.8B Performance

Measured on the reComputer RK3576 Dev Kit with RK1820 AI Accelerator:

The test results show that Qwen3.5-0.8B can reach approximately:

Generate speed: approximately 92.89 tokens/s

It can meet the needs of real-time interactive applications such as local chat assistants, text generation, and instruction understanding.

Run a Local VLM

In addition to text-only LLMs, the RK1820 can also run VLMs (Vision-Language Models).

This example uses:

Qwen2.5-VL-3B_prune

This model can process:

Image + Prompt
      ↓
Vision Encoder
      ↓
Multimodal LLM
      ↓
Text Response

That is to say, applications such as image description, scene understanding, and visual question answering can be implemented.

Prepare Qwen2.5-VL-3B_prune

The model directory contains:

Qwen2.5-VL-3B_prune/
├── Qwen2.5-VL-3B_prune.embed.bin
├── Qwen2.5-VL-3B_prune.tokenizer.gguf
├── llm_Qwen2.5-VL-3B_prune.rknn
├── llm_Qwen2.5-VL-3B_prune.weight
├── vision_Qwen2.5-VL-3B_prune.rknn
└── vision_Qwen2.5-VL-3B_prune.weight

The runtime of Qwen2.5-VL includes independent Vision Model and LLM Model.

Start the VLM Server

If an LLM Server is currently running, first use:

Ctrl+C

to stop the previous service.

Enter the model directory:

cd ~/Downloads/Qwen2.5-VL-3B_prune

Execute:

rkllm3-server \
-m llm_Qwen2.5-VL-3B_prune.rknn \
--weight llm_Qwen2.5-VL-3B_prune.weight \
--model2 vision_Qwen2.5-VL-3B_prune.rknn \
--weight2 vision_Qwen2.5-VL-3B_prune.weight \
--vocab Qwen2.5-VL-3B_prune.tokenizer.gguf \
--embed Qwen2.5-VL-3B_prune.embed.bin \
--host 0.0.0.0 \
--port 8080 \
-c 768 \
--n_predict 256 \
--core-mask 0xff \
--core-mask2 0xff \
--repeat-penalty 1.1 \
--presence-penalty 1.0 \
--frequency-penalty 1.0 \
--top-k 1 \
--top-p 0.8 \
--temp 0.8 \
--img-start "<|vision_start|>" \
--img-end "<|vision_end|>" \
--img-content "<|image_pad|>" \
--img-width 392 \
--img-height 392

The key parameters are:

-m         → LLM Model
--model2   → Vision Model
--core-mask  → NPU Core used by LLM
--core-mask2 → NPU Core used by Vision

The official Rockchip Qwen2.5-VL server example also uses separate LLM and vision models, and runs the vision-language model with a 392 脳 392 input image.

Test Image Understanding

This example uses the image:

/home/seeed/Downloads/people.png

Create:

nano vlm_test.py

Write:

import base64
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8080/v1",
    api_key="sk-no-key-required"
)

image_path = "/home/seeed/Downloads/people.png"

with open(image_path, "rb") as image_file:
    base64_image = base64.b64encode(
        image_file.read()
    ).decode("utf-8")

response = client.chat.completions.create(
    model="Qwen2.5-VL",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url":
                        f"data:image/png;base64,{base64_image}"
                    }
                },
                {
                    "type": "text",
                    "text":
                    "Describe this image in detail."
                }
            ]
        }
    ],
    stream=True,
    extra_body={
        "n_keep": 0,
        "cache_prompt": False,
        "id_slot": 0,
        "n_predict": 256
    }
)

for chunk in response:
    content = chunk.choices[0].delta.content

    if content:
        print(content, end="", flush=True)

print()

Execute:

python3 vlm_test.py

The official RKLLM3 multimodal example also converts the local image to Base64 and submits it to the local model in the OpenAI-compatible image_url format.

VLM Output Example

Input:

Describe this image in detail.

Qwen2.5-VL-3B_prune can recognize people, positional relationships, street environments, buildings, trees, and lighting in images.

For example:

The image depicts a group of people walking down
an urban street on what appears to be a sunny day...

Qwen2.5-VL-3B_prune Performance

The end-to-end VLM results from this test are summarized below.

TTFT (Time To First Token)

1.42 s

This is the time from submitting the image and prompt until the model starts generating the first token.

For a VLM, this time includes image processing, Vision Encoder, Multimodal Prefill, and the time needed to start text generation, so it is closer to the real application experience than testing a pure LLM alone.

This test sets n_predict=256, so the model output is truncated after reaching 256 Streaming Chunks. In actual applications, you can adjust the maximum output Token count according to the Context Length and application requirements.

Build Local AI Applications

Through rkllm3-server, the model capability of the RK1820 can be provided to upper-layer applications through a standard HTTP API.

Therefore, you can further build:

Local Chat Assistant

Image Understanding

Visual Question Answering

Document / Image Analysis

Private Edge AI Assistant

A typical application architecture is as follows:

Web UI / Python / Application
              │
              │ OpenAI-compatible API
              ▼
        rkllm3-server
              │
              ▼
          RKNN3 Runtime
              │
         PCIe Gen2 x1
              │
              ▼
            RK1820
        ┌─────┴─────┐
        │           │
       LLM         VLM

In this way, upper-layer applications can call the local AI models running on the RK1820 through the API without directly operating the RKNN3 Runtime.

Monitor RK1820

When running LLM or VLM applications, you can use:

sudo rknn-smi info -w

to view in real time:

NPU Utilization
Memory Usage
Temperature
Health Status

to observe the RK1820 status during model operation.

Summary

Through the RK1820, the reComputer RK3576 can run local LLM and VLM, and through rkllm3-server provide the model capability as a local API to upper-layer applications.

Actual verification has been completed in this run: