Skip to main content
Android mainboards · Touch displays · Firmware — since 2013
LUMA AI StudioStart a project

RKLLM — local language models on the NPU

RKLLM runs large language models directly on the board's NPU. A chatbot, a summarizer, or a voice assistant that works with no cloud, no API cost, and no data leaving the device. For products where privacy or offline operation is the point, this is a strong differentiator.

Only two chips

RKLLM requires 6 TOPS and runs on RK3588 and RK3576 only — in our range, KTM-S88CD and KTM-S76. The 1 TOPS RK3568/RK3566 boards cannot run RKLLM; use RKNN for lighter AI on those.

What it can realistically run​

Small, quantized instruction models — the 1.5B–3B class — run interactively. Qwen2.5-1.5B, DeepSeek-R1-Distill-Qwen-1.5B and similar are the sweet spot. Larger models load but slow down; on edge hardware, smaller-and-quantized beats larger-and-sluggish for a responsive product.

Be honest with your product expectations: this is not a datacenter GPU. It is enough for structured, bounded language tasks — summarize this text, answer from this context, format this into speech — which is exactly what most on-device assistants need.

The workflow (same shape as RKNN)​

  1. Convert on an x86 Linux PC with RKLLM-Toolkit — turns a Hugging Face model into a quantized .rkllm file.
  2. Deploy on the board with RKLLM Runtime (C/C++ API) — loads the .rkllm and runs inference through the NPU.

Conversion is PC-only, exactly like RKNN.

Step 1 — Convert on a PC​

Recommended: x86 Ubuntu 20.04, Python managed with miniforge. Install the RKLLM-Toolkit wheel from the rknn-llm release, then:

convert_llm.py — runs on the PC
from rkllm.api import RKLLM

llm = RKLLM()

# Point at a downloaded Hugging Face model directory
llm.load_huggingface(model='./Qwen2.5-1.5B-Instruct')

# Quantize. w8a8 and w4a16 are the supported types; w8a8 is the common start.
llm.build(
    do_quantization=True,
    optimization_level=1,
    quantized_dtype='w8a8',
    target_platform='rk3588',    # or rk3576
)

llm.export_rkllm('Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm')

The filename convention tells you what you built: quantization type (W8A8) and target chip (RK3588). Keep it — it prevents deploying an RK3576 model to an RK3588 by accident.

Step 2 — Check the board's NPU driver​

RKLLM needs a newer NPU driver than plain RKNN — v0.9.6 or above.

cat /sys/kernel/debug/rknpu/version

If it is below 0.9.6, update the RKNPU2 userspace package and driver before going further. failed to open rknpu module at runtime almost always means the driver is missing or too old.

Step 3 — Deploy and run on the board​

Copy the .rkllm to the board, then run it through the RKLLM Runtime. Rockchip's rknn-llm repo ships a llm_demo you can build and use directly:

# on the board
scp user@pc:Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm .

# run the demo runtime (args: model, max_new_tokens, max_context)
./llm_demo Qwen2.5-1.5B-Instruct_W8A8_RK3588.rkllm 512 512

For a real product you embed the RKLLM Runtime C API in your own application: load the model once, then feed prompts and stream tokens back through a callback. The repo's demo source is the reference implementation to build from.

Where this fits an Itemptechnology product​

A local LLM on the board turns a signage or kiosk device into something that can speak, summarize and answer on its own — offline, private, no running cost.

Concrete pairings with the rest of the platform:

  • A kiosk that answers customer questions from a fixed knowledge base, with no network dependency and no per-query cloud bill.
  • A signage player that generates a spoken daily briefing on-device — weather, schedule, notices — combining a local LLM with text-to-speech.
  • A device that summarizes or reformats content locally before displaying it, keeping source data on the device.

For anything voice-facing, pair the LLM output with TTS. For structured knowledge answering, a small model plus a retrieval step over your own documents beats a larger model trying to hold everything in weights.

rknn-llm moves fast — supported models, quantization types and driver requirements change between releases. Verify against the current airockchip/rknn-llm repo before building.