How to Run CNN Models on RK3588

Convert YOLO11n on an Ubuntu PC, deploy it to RK3588 over SSH/SCP, and verify C++ and Python inference on its built-in NPU.

Quick start: Run CNN models on RK3588

This guide uses YOLO11n to cover NPU driver check → Runtime and Toolkit Lite2 installation → PC model conversion → C++ build and deployment → on-board inference. The model runs on the RK3588 built-in NPU through SSH/SCP or a local board terminal.

#Before you begin

ItemConfiguration and tested environment
PCUbuntu 20.04.6 LTS，x86_64，Python 3.11
BoardreComputer RK3588, aarch64
Board OSDebian 12 / Armbian 26.08.0-trunk
Board kernel6.1.115-vendor-seeed-rk3588
NPU driverTested with 0.9.8
Tool versionsRKNN-Toolkit2 / Toolkit Lite2 2.3.2
RuntimeThis guide installs librknnrt.so 2.3.2
ExampleYOLO11n, 640 × 640, INT8 quantization
Conversion / build targetrk3588 / rk3588 + aarch64

Where to run commands: “On PC” means the Ubuntu development computer; “On board” means an SSH session or local board terminal. Replace BOARD_IP and the example username rk3588 with the actual address and account.

If SSH is not enabled on the board, run this locally on the board first:

bash
sudo apt update
sudo apt install openssh-server
sudo systemctl enable --now ssh
hostname -I

Then connect from the PC. On first connection, verify the host fingerprint and enter your account password when prompted:

bash
ssh rk3588@BOARD_IP

#1. Fixing a missing NPU driver

#1.1 Kernel driver versus user-space software

The RK3588 NPU driver is RKNPU in the kernel, controlled by CONFIG_ROCKCHIP_RKNPU. Neither the Runtime .so nor the Toolkit Lite2 Python wheel can replace the kernel driver.

#1.2 Check the driver, device node, and load

On board:

bash
uname -r
sudo dmesg | grep -i rknpu
sudo cat /sys/kernel/debug/rknpu/version
sudo cat /sys/kernel/debug/rknpu/load
ls -l /sys/class/drm/renderD*/device/driver

Excerpt from the actual test output:

text
RKNPU driver: v0.9.8
NPU load:  Core0:  0%, Core1:  0%, Core2:  0%,

/sys/class/drm/renderD130/device/driver -> .../bus/platform/drivers/RKNPU

The NPU node here was renderD130; the number may change. Identify it by the device/driver link to RKNPU.

If debugfs is not mounted, run:

bash
mountpoint -q /sys/kernel/debug || sudo mount -t debugfs debugfs /sys/kernel/debug

Retry after mounting, and assess the driver using boot logs and the driver binding as well.

Check the kernel configuration on the board:

bash
grep '^CONFIG_ROCKCHIP_RKNPU=' /boot/config-$(uname -r)

Actual output:

text
CONFIG_ROCKCHIP_RKNPU=y

=y means the driver is built into the kernel, so rknpu being absent from lsmod does not mean the driver is missing. The official RKNN SDK V2.3.2 quick-start guide recommends RKNPU driver 0.9.2 or later; inference was verified here with 0.9.8.

#1.3 If the driver is missing or needs updating

This guide uses the Seeed/Armbian kernel for reComputer RK3588. If the driver checks pass, go straight to Section 2 without reinstalling the kernel.

On the same board model with the Seeed package source configured, first inspect the kernel and device-tree packages:

bash
sudo apt update
apt-cache policy linux-image-vendor-seeed-rk3588 linux-dtb-vendor-seeed-rk3588

After confirming both packages come from the board's matching package source and their candidate versions agree, install/update the matching kernel and device tree:

bash
sudo apt install --reinstall linux-image-vendor-seeed-rk3588 linux-dtb-vendor-seeed-rk3588
sudo reboot

Rebooting disconnects SSH; after reconnecting, verify the driver as in Section 1.2.

If the packages are unavailable, use the vendor image or BSP for this board model. For a custom kernel, enable CONFIG_ROCKCHIP_RKNPU=y and use this board's NPU device-tree configuration.

#2. Install and verify Runtime and Toolkit Lite2

#2.1 What the three components do

ComponentInstalled onMain purpose
RKNN-Toolkit2Isolated Python environment on Ubuntu PCConvert and quantize ONNX into .rknn
RKNN Runtime / librknnrt.soRK3588 boardC/C++ API for loading models and running NPU inference
RKNN-Toolkit-Lite2Python environment on RK3588 boardPython inference API; requires board Runtime and driver

The PC uses from rknn.api import RKNN; board Python uses from rknnlite.api import RKNNLite. Lite2 does not convert models; the C++ example calls Runtime directly.

This guide runs inference locally on the board; rknn_server is not needed.

#2.2 Get fixed revisions of the official repositories on the PC

Sections 3 and 4 reuse these directories; clone them once in a new working directory.

On PC:

bash
sudo apt update
sudo apt install git wget cmake make gcc g++ openssh-client
mkdir -p ~/RKNN2_Project
cd ~/RKNN2_Project

git clone https://github.com/airockchip/rknn-toolkit2.git
git -C rknn-toolkit2 checkout 59a913d172e7f5ff03c9076e2ec7b1b1288ffd08

git clone https://github.com/airockchip/rknn_model_zoo.git
git -C rknn_model_zoo checkout bad6c7334531becaf90a561988519b7bec34d0ab

These are the commits used in this test; Model Zoo corresponds to v2.3.2. Pin the code and packages so future repository updates do not change command paths.

#2.3 Install the board Runtime

On PC:

bash
ssh rk3588@BOARD_IP 'mkdir -p ~/RKNN2_Project/packages'

scp ~/RKNN2_Project/rknn-toolkit2/rknpu2/runtime/Linux/librknn_api/aarch64/librknnrt.so \
  rk3588@BOARD_IP:~/RKNN2_Project/packages/

On board: Back up any existing library before installing the new one.

bash
sudo apt update
sudo apt install binutils

if [ -e /usr/lib/librknnrt.so ]; then
  sudo cp -a /usr/lib/librknnrt.so "/usr/lib/librknnrt.so.bak-$(date +%Y%m%d-%H%M%S)"
fi
sudo install -m 0644 ~/RKNN2_Project/packages/librknnrt.so /usr/lib/librknnrt.so
sudo ldconfig
strings /usr/lib/librknnrt.so | grep 'librknnrt version'

The version string in this package is:

text
librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27)

Lite2 needs /usr/lib/librknnrt.so; setting LD_LIBRARY_PATH only for the Demo directory does not replace this step.

#2.4 Install Toolkit Lite2 on the board

The board runs Debian 12 with Python 3.11, so use the cp311 aarch64 wheel.

On PC:

bash
scp ~/RKNN2_Project/rknn-toolkit2/rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl \
  rk3588@BOARD_IP:~/RKNN2_Project/packages/

On board:

bash
sudo apt install python3.11-venv
python3 -m venv ~/venvs/rknn-lite2
source ~/venvs/rknn-lite2/bin/activate

python -m pip install \
  ~/RKNN2_Project/packages/rknn_toolkit_lite2-2.3.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
python -m pip check
python -c "from rknnlite.api import RKNNLite; print('RKNN Toolkit Lite2 import OK')"

Actual results of the two checks:

text
No broken requirements found.
RKNN Toolkit Lite2 import OK

If you see ensurepip is not available, install python3.11-venv and recreate the virtual environment. Section 4 verifies actual inference.

#3. Configure Toolkit on Ubuntu PC and convert YOLO11n

#3.1 Create a Python 3.11 environment on the PC

On PC: These commands install Miniforge on a new PC. If Conda is already installed, use it to create the same environment.

bash
mkdir -p ~/Downloads
cd ~/Downloads
wget -c https://github.com/conda-forge/miniforge/releases/download/25.3.0-1/Miniforge3-25.3.0-1-Linux-x86_64.sh
bash Miniforge3-25.3.0-1-Linux-x86_64.sh -b -p "$HOME/miniforge3"
source ~/miniforge3/etc/profile.d/conda.sh
conda create -n rknn2 python=3.11 -y
conda activate rknn2

In each new PC terminal, run source ~/miniforge3/etc/profile.d/conda.sh and conda activate rknn2 before the subsequent Python commands.

#3.2 Install Toolkit2 2.3.2

On PC:

bash
cd ~/RKNN2_Project/rknn-toolkit2/rknn-toolkit2
python -m pip install \
  -r packages/x86_64/requirements_cp311-2.3.2.txt \
  packages/x86_64/rknn_toolkit2-2.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl \
  'onnx==1.16.1' 'protobuf==4.25.4'

python -m pip check
python -m pip show rknn-toolkit2
python -c "from rknn.api import RKNN; print('RKNN Toolkit2 import OK')"

The tested setup used Python 3.11.14, Toolkit2 2.3.2, ONNX 1.16.1, and protobuf 4.25.4; pip check returned No broken requirements found.. Pin dependencies in an isolated environment to avoid conflicts with other projects.

#3.3 Download the officially adapted YOLO11n

On PC:

bash
cd ~/RKNN2_Project/rknn_model_zoo/examples/yolo11/model
bash download_model.sh
ls -lh yolo11n.onnx

The downloaded yolo11n.onnx was 10,527,859 bytes.

Use the Rockchip-adapted ONNX whose output layout matches the example C++ post-processing. Export custom-trained models according to the YOLO11 export instructions.

#3.4 Convert to an RK3588 INT8 model

On PC:

bash
conda activate rknn2
cd ~/RKNN2_Project/rknn_model_zoo
export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"
ls datasets/COCO/coco_subset_20.txt
ls datasets/COCO/subset

cd examples/yolo11/python
python convert.py ../model/yolo11n.onnx rk3588 i8 ../model/yolo11n_rk3588.rknn
ls -lh ../model/yolo11n_rk3588.rknn

The four arguments are the ONNX path, target platform, quantization type, and output path. This page must use rk3588.

Run from examples/yolo11/python so the script finds the calibration set datasets/COCO/coco_subset_20.txt. This set is for example verification; use data representing real scenarios to calibrate production models.

Excerpt from the actual conversion output:

text
I rknn-toolkit2 version: 2.3.2
...
--> Building model
...
I rknn building ...
I rknn building done.
done
--> Export rknn model
done

A successful conversion creates the .rknn file. The notice about input/output types changing to int8 is a quantization reminder; use the matching C++ example to handle input/output.

#4. Build, deploy with SCP, and run

#4.1 Prepare the ARM64 cross-compiler

On PC:

bash
cd ~/RKNN2_Project
wget -c https://dn.odroid.com/compiler/gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu.tar
tar -xf gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu.tar

export GCC_COMPILER="$HOME/RKNN2_Project/gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu"
"${GCC_COMPILER}-gcc" --version

The tested compiler was Linaro GCC 6.3.1 20170404. GCC_COMPILER is a prefix without the trailing -gcc; the build script appends it.

#4.2 Build the RK3588 YOLO11 C++ example

On PC:

bash
cd ~/RKNN2_Project/rknn_model_zoo
bash build-linux.sh -t rk3588 -a aarch64 -b Release -d yolo11

Convert the model before building so the script copies .rknn into the deployment directory.

The deployment directory should look like this; other bundled example programs may remain:

text
install/rk3588_linux_aarch64/rknn_yolo11_demo/
├── rknn_yolo11_demo
├── lib/
│   ├── librknnrt.so
│   └── librga.so
└── model/
    ├── yolo11n_rk3588.rknn
    ├── bus.jpg
    └── coco_80_labels_list.txt

#4.3 Deploy the entire directory with SCP

On PC:

bash
ssh rk3588@BOARD_IP 'mkdir -p ~/RKNN2_Project'
cd ~/RKNN2_Project/rknn_model_zoo
scp -r install/rk3588_linux_aarch64/rknn_yolo11_demo \
  rk3588@BOARD_IP:~/RKNN2_Project/

Copy the complete directory, including lib/ and model/; the bundled Runtime version is 2.3.2.

#4.4 Run object detection on the board

On board:

bash
cd ~/RKNN2_Project/rknn_yolo11_demo
sudo env LD_LIBRARY_PATH="$PWD/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \
  ./rknn_yolo11_demo model/yolo11n_rk3588.rknn model/bus.jpg

Excerpt from the actual RK3588 output:

text
model input num: 1, output num: 9
model is NHWC input fmt
model input height=640, width=640, channel=3
...
rknn_run
bus @ (91 135 552 435) 0.948
person @ (109 236 223 536) 0.898
person @ (212 240 285 509) 0.843
person @ (477 230 559 521) 0.827
person @ (79 359 116 515) 0.448
write_image path: out.png width=640 height=640 channel=3 ...

Detection is complete when the process exits normally and creates out.png; coordinates and confidence may vary with version and quantization.

On PC, retrieve the image:

bash
mkdir -p ~/RKNN2_Project/results
scp rk3588@BOARD_IP:~/RKNN2_Project/rknn_yolo11_demo/out.png \
  ~/RKNN2_Project/results/yolo11n-rk3588-out.png

Open the result image on the PC to inspect detection boxes, or open out.png in the board's graphical interface.

#4.5 Verify that Lite2 runs the same model

Use the same model and an all-zero input to check the Lite2 inference path; this step does not post-process an image for detection.

On board:

bash
cd ~/RKNN2_Project/rknn_yolo11_demo
sudo "$HOME/venvs/rknn-lite2/bin/python" - <<'PYCODE'
import numpy as np
from rknnlite.api import RKNNLite

rknn = RKNNLite()
try:
    ret = rknn.load_rknn('model/yolo11n_rk3588.rknn')
    if ret != 0:
        raise RuntimeError(f'load_rknn failed: {ret}')
    ret = rknn.init_runtime()
    if ret != 0:
        raise RuntimeError(f'init_runtime failed: {ret}')
    outputs = rknn.inference(inputs=[np.zeros((1, 640, 640, 3), dtype=np.uint8)])
    if outputs is None:
        raise RuntimeError('inference failed')
    print('Lite2 inference OK; output count:', len(outputs))
finally:
    rknn.release()
PYCODE

Actual result:

text
Lite2 inference OK; output count: 9

Use the virtual environment Python by absolute path so sudo python3 does not accidentally call the system Python without Lite2.

#4.6 Troubleshooting and acceptance checks

SymptomWhat to check
NPU driver not foundRecheck the RKNPU binding and boot log in Section 1.2; a built-in driver need not appear in lsmod
Lite2 imports but initialization failsCheck /usr/lib/librknnrt.so, the NPU driver, and access permissions
Can not find dynamic libraryInstall the system Runtime as in Section 2.3; the Demo's lib/ alone is insufficient
pip check reports dependency conflictsUse the isolated PC environment and dependencies in Section 3.2, not another project's old ONNX/protobuf
Calibration images or py_utils not foundCheck the working directory, repository calibration dataset, and PYTHONPATH
Model platform mismatchConvert again for rk3588 and run yolo11n_rk3588.rknn
Model or label file missingCopy the entire deployment directory and enter rknn_yolo11_demo first
Dynamic library missingCheck lib/librknnrt.so and pass LD_LIBRARY_PATH as in Section 4.4

After completion, the NPU driver should be queryable, Runtime/Lite2 usable, PC conversion successful, the C++ program able to create a detection image, and Lite2 able to return nine output tensors.

#4.7 References