← Models

LLM / Gemma 4

Gemma 4 E2B IT

Published RKLLM images of Gemma 4 E2B IT with an OpenAI-compatible API. Choose a quantization (W4A16-G128 / W8A8) after selecting the target device.

19 downloads
Size
5.54GB

Choose the device you're using, the set up guide and documentation will update accordingly.

Getting Started

Deploy
sudo docker run --rm -d \
  --name recomputer-rk-llm \
  --privileged \
  -p 8001:8001 \
  -v /dev:/dev \
  -e INTERACTIVE_CHAT=true \
  -e LOG_LEVEL=warning \
  ghcr.io/seeed-projects/recomputer-rk-llm/llm/gemma-4-e2b-it:rk3576-w4a16-g128

Model Details

LLM deployment on reComputer RK

This shared guide applies to every LLM listed in the catalog. Select the model family and quantization in the model page, then use the corresponding published image tag.

Published image format

text
ghcr.io/seeed-projects/recomputer-rk-llm/llm/<model-id>:<platform>-<quantization>

Published LLM images include the converted .rkllm model, the RKLLM runtime, and an OpenAI-compatible server. The available tags are listed in each model family’s catalog records. W4A16/W4A16-G128 images are generally RK3576-only; RK3588 records use W8A8.

Requirements

  • reComputer RK3576 or RK3588 running a 64-bit Linux image
  • Docker and Docker Buildx installed
  • NPU access through /dev
  • Port 8001 available on the board

The examples use --privileged and -v /dev:/dev for NPU access. The local API has no authentication or TLS; keep it on a trusted network.

Run a published LLM

Replace <model-id> and <platform>-<quantization> with the values shown on the selected model page.

RK3576

bash
sudo docker run --rm -it \
  --name recomputer-rk-llm \
  --privileged \
  -p 8001:8001 \
  -v /dev:/dev \
  -e INTERACTIVE_CHAT=true \
  -e LOG_LEVEL=warning \
  ghcr.io/seeed-projects/recomputer-rk-llm/llm/<model-id>:rk3576-<quantization>

RK3588

bash
sudo docker run --rm -it \
  --name recomputer-rk-llm \
  --privileged \
  -p 8001:8001 \
  -v /dev:/dev \
  -e INTERACTIVE_CHAT=true \
  -e LOG_LEVEL=warning \
  ghcr.io/seeed-projects/recomputer-rk-llm/llm/<model-id>:rk3588-w8a8

The attached command enables terminal interactive chat. For background mode, change -it to -d and remove INTERACTIVE_CHAT=true.

Health and documentation

bash
curl http://localhost:8001/health
curl http://localhost:8001/docs

The service is ready when the health endpoint responds successfully.

OpenAI-compatible API

Non-streaming request:

bash
curl http://localhost:8001/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "rkllm-model",
    "messages": [{"role": "user", "content": "Explain edge AI in one sentence."}],
    "max_tokens": 128,
    "stream": false
  }'

Streaming request:

bash
curl -N http://localhost:8001/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "rkllm-model",
    "messages": [{"role": "user", "content": "Give me three uses for an edge AI board."}],
    "max_tokens": 128,
    "stream": true
  }'

Python client:

python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8001/v1", api_key="dummy-key")
response = client.chat.completions.create(
    model="rkllm-model",
    messages=[{"role": "user", "content": "Hello from reComputer RK."}],
    max_tokens=128,
)
print(response.choices[0].message.content)

Ollama-compatible API

bash
curl http://localhost:8001/api/chat \
  -H 'Content-Type: application/json' \
  -d '{"model":"rkllm-model","messages":[{"role":"user","content":"Hello"}],"stream":false}'

Stop and troubleshoot

bash
sudo docker logs recomputer-rk-llm
sudo docker stop recomputer-rk-llm

If platform detection fails, set TARGET_PLATFORM=rk3576 or TARGET_PLATFORM=rk3588. If port 8001 is occupied, change the host-side port mapping and use the new API URL.

Inputs and Outputs

Input: text prompt or chat messages. Output: generated text through an OpenAI-compatible API.