Can RK3588 Run Local LLMs? Quantized Inference on a 6-TOPS NPU, Tested
“Can an edge box run an LLM?” has moved from tech curiosity to real procurement demand: privacy-sensitive industries want data to stay on-device, and disconnected sites need local intelligence. Where does RK3588 — one of the strongest general-purpose edge SoCs — actually land?
Test results: real quantized-model throughput
- 1.5-3B models (Q4): 5-10 tokens/s on CPU, improvable further with vendor NPU toolchains — fine for low-frequency text tasks like ticket summaries and inspection reports;
- 0.5B-class models: interactive response times, enough for offline Q&A and device voice assistants;
- 7B Q4: roughly 2-3 tokens/s — usable for overnight batch, not for conversation.
Memory is the first gate
A 7B Q4 model needs ~4 GB for weights alone, so 8 GB units run comfortably while 4 GB units should stop at 3B quantized models. That’s exactly why our RK3588 edge line offers 4-8 GB options — both the mini PC and the 01B EDGE-RK3588 board can be configured accordingly.
Position it as an inference endpoint, not a training box
RK3588’s role is “inference endpoint”: train in the cloud or on workstations, quantize, then push models to the edge for continuous inference. The productive architecture is “small model high-frequency inference on NPU + large model fallback in cloud”. For vision workloads, YOLO-class models run at 30fps+ on the NPU with far better economics than LLMs.
Learn more: RK3588 Mini PC, 01B EDGE-RK3588 board. NPU toolchains and SDKs are in the Hugsun Documentation Center; contact us for deployment programs.
