On-device AI
The NPU-equipped KTM boards run AI models locally — no cloud, no per-call cost, no data leaving the device. This section is the practical path from a trained model to inference running on the board.
There are two toolchains, for two kinds of model:
| You want to run | Use | Works on |
|---|---|---|
| Vision / detection / classification (YOLO, ResNet, etc.) | RKNN | RK3588, RK3576, RK3568, RK3566 |
| A local large language model (chat, summarization) | RKLLM | RK3588, RK3576 only |
What your board can actually do
AI capability is set by the chip, not the board. Match your workload to the NPU:
| SoC | NPU | Realistic workload | Boards |
|---|---|---|---|
| RK3588 | 6 TOPS | Multi-camera analytics, several models at once, local LLMs | KTM-S88CD |
| RK3576 | 6 TOPS | Same AI tier as RK3588, local LLMs | KTM-S76 |
| RK3568 | ~1 TOPS | Single-stream detection, lightweight CNNs | KTM-S58, KTM-S68, KTM-MS68 |
| RK3566 | ~0.8 TOPS | Presence detection, one light model | KTM-LT66, KTM-MS66, KTM-S68B |
| RK3399 / RK3288 / T982 | — | No NPU — CPU/GPU inference only | KTM-S39-II, S88, S38, MS98 |
The honest read: 1 TOPS is a feature, 6 TOPS is a platform. A KTM-S68 can run a face match or people counter alongside your signage app. A KTM-S88CD or S76 can be the AI product itself — multiple camera streams, or a chatbot running entirely on the device.
The shape of every RKNN/RKLLM project
Both toolchains split the work the same way, and it matters because it decides what runs where:
- Convert on a PC — an x86 Linux machine turns your trained model (PyTorch,
ONNX, TensorFlow, or a Hugging Face LLM) into a
.rknnor.rkllmfile. This is where quantization happens. The board is not involved. - Deploy on the board — copy the converted file to the KTM board and run it through the NPU, via a Python or C/C++ API.
You do not convert models on the board. You do not run PyTorch on the board. The PC does the heavy conversion once; the board runs the compact result many times.
Start here
- Vision model → RKNN deployment
- Local LLM (RK3588/RK3576 only) → RKLLM deployment
Both connect back to the SoC reference for chip capabilities and the board pages for which camera and display interfaces you have to work with.
Toolchain versions and supported-model lists change frequently. Where this wiki
gives a version, treat it as a starting point and check Rockchip's current
rknn-toolkit2 and rknn-llm releases for the latest.