What Is an NPU Actually Good For? Testing CPU vs GPU vs NPU for Frigate

What Is an NPU Actually Good For? Testing CPU vs GPU vs NPU for Frigate

●3 ●5
calendar_today • schedule6 min read
— Originally published at codewithromi.com

NPUs are suddenly everywhere. Intel, AMD, and Qualcomm are all shipping them in consumer hardware, and the marketing copy makes them sound like a game-changer for AI workloads. But if you run a homelab and care about practical use cases, the real question is: what can an NPU actually do for you today?

One of the most concrete answers comes from Frigate, the self-hosted NVR that uses object detection to identify people, cars, and animals in your camera feeds. It supports multiple inference backends, which makes it a great real-world benchmark for comparing CPU, GPU, and NPU performance on the same workload.

💡 Why This Matters for Homelabbers

Most homelab AI conversations center on GPUs. A discrete GPU is great for running local LLMs or doing batch inference, but it draws significant power and takes up a PCIe slot. If your goal is always-on, low-latency inference for something like security cameras, that power draw adds up fast.

An NPU is purpose-built for sustained, efficient inference. It is not trying to compete with a GPU on raw throughput for training or large model inference. It is designed to run smaller, optimized models continuously without hammering your power budget. For Frigate, which is doing object detection on every camera frame all day long, that efficiency story is genuinely relevant.

🧰 Prerequisites

  • A machine with an Intel Core Ultra processor (Meteor Lake, Lunar Lake, Arrow Lake or Panther Lake) for NPU access
  • Frigate 0.17 or newer, ideally in Docker. NPU support landed in 0.17, so earlier releases will not offer it at all
  • OpenVINO detector configured in your Frigate config
  • A host kernel with the Intel NPU driver (intel_vpu) and NPU firmware at least as new as the linux-npu-driver release bundled in your Frigate image. Frigate ships the NPU userspace inside the container, so do not mount host NPU libraries into it. Check the firmware build date with sudo dmesg | grep -i vpu
  • Both /dev/dri and /dev/accel passed through to the container. The NPU lives at /dev/accel/accel0, and missing it is the most common reason the NPU never shows up

The MINISFORUM MS-03 used in the source video runs an Intel Core Ultra 9 386H (Panther Lake) with an integrated GPU and a 50 TOPS NPU. One catch: Intel added Panther Lake NPU support in linux-npu-driver 1.28.0, but the official Frigate images bundle older NPU userspace (1.17.0 in 0.17.2 and 1.19.0 in 0.18.0), so Techno Tim ran a rebuilt Frigate 0.17.2 image with a newer Intel stack. On Meteor Lake, Lunar Lake and Arrow Lake, start with the official image. Mini PCs like this are a compelling homelab option because they pack modern silicon into a compact, low-power form factor.

🔧 Configuring Frigate with OpenVINO

Frigate supports OpenVINO as a detector type, which is Intel's inference runtime. It can target the CPU, the integrated GPU (via iGPU), or the NPU depending on the device string you pass in your config.

Step 1: In your frigate.yml, define a detector block using the OpenVINO type. The device value controls which processor handles inference. Common options are:

  • CPU: runs on the main processor cores
  • GPU: targets the integrated Intel GPU
  • NPU: targets the dedicated NPU block
  • AUTO: lets OpenVINO pick based on availability and load
detectors:
  openvino:
    type: openvino
    device: NPU
    model_path: /config/model_cache/yolov9t.xml

# note: the global model block sits at the TOP level, not nested under the detector
model:
  model_type: yolo-generic
  width: 640
  height: 640
  input_tensor: nchw
  input_dtype: float
  labelmap_path: /labelmap/coco-80.txt

Frigate does not ship a YOLOv9 model. The Frigate docs include a Docker build command that exports YOLOv9 to ONNX in sizes t, s, m, c or e. Techno Tim's test used YOLOv9-tiny converted to a static 640x640 OpenVINO IR, which is an .xml file plus a matching .bin. Put both files in config/model_cache and point model_path at the .xml.

Step 2: Pass the device nodes through. This is the step people get wrong: /dev/dri alone gives you the iGPU but not the NPU. You need /dev/accel as well, plus the video and render groups.

services:
  frigate:
    image: ghcr.io/blakeblackshear/frigate:0.18.0
    devices:
      - /dev/dri/card0:/dev/dri/card0
      - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/accel:/dev/accel      # the NPU; without this it is invisible
    group_add:
      - "44"    # video (GIDs vary by distro: check with getent group video)
      - "993"   # render (check with getent group render)
    volumes:
      - /path/to/config:/config
      - /path/to/media:/media/frigate
    ports:
      - "8971:8971"    # authenticated UI and API. Port 5000 is unauthenticated, so do not publish it
      - "8554:8554"

Step 3: Restart Frigate and check the logs. You should see OpenVINO initialize and report the active device. If the NPU is not detected, the most common cause is a missing driver or an unsupported kernel version. Intel's documentation covers the driver installation steps for each supported OS.

docker compose logs -f frigate | grep -i openvino

📊 CPU vs GPU vs NPU: What the Testing Shows

Based on the Techno Tim video linked below, running the same Frigate object detection workload across all three backends on the same Intel Core Ultra hardware produced some clear patterns worth understanding.

CPU: Works out of the box with no extra configuration. Inference is functional but burns CPU cycles that could be going to other processes. On a machine doing double duty as a NAS or VM host, this matters.

iGPU: Faster inference than CPU and offloads the work from your main cores. A solid middle ground that works well on most modern Intel platforms with proper drivers.

NPU: The NPU trades raw peak throughput for consistent, power-efficient inference. It is not necessarily faster than the iGPU on a frames-per-second basis, but it can free the iGPU up for other tasks like video transcoding, and it does the work at a lower power cost. For an always-on Frigate instance running 24/7, that matters more than peak speed.

The practical takeaway is that the NPU is not a replacement for a GPU in general-purpose AI workloads. It is a purpose-fit accelerator for sustained, low-complexity inference, exactly what Frigate does all day long. Frigate's own documentation makes the same split explicit: if you have both, run object detection on the NPU and leave the iGPU free for enrichments like semantic search and face recognition, plus video transcoding.

🛡️ Security Notes

Passing hardware device nodes into Docker containers does expand the attack surface slightly. A few things worth keeping in mind:

  • Only expose the specific device nodes Frigate needs, not broad access to all of /dev
  • Run the Frigate container without unnecessary capabilities
  • Keep Frigate and the host OS updated, especially OpenVINO and Intel driver packages
  • Put Frigate on a dedicated VLAN or network segment away from your cameras if possible

🔧 Troubleshooting

NPU device not found: First check that /dev/accel/accel0 exists on the host and that you actually passed /dev/accel into the container. The NPU firmware is loaded by the host kernel and is not part of the Frigate image, so a kernel without the Intel NPU driver will never expose the device.

Confirm which device is actually in use: At startup Frigate logs Loading OpenVINO model ... on device NPU, and it does not quietly move object detection from the NPU to the CPU. If the NPU is missing or its firmware is too old, the detector fails to load and the error appears in docker compose logs. Firmware older than the bundled driver can fail with MAPPED_INFERENCE_VERSION is NOT compatible with the ELF.

High latency on first detection: OpenVINO compiles the model for the selected device every time the detector starts, and the first inference includes that compilation. On the NPU, Intel's driver caches the compiled model by default, so later starts are usually quicker as long as that cache survives. Give it a few minutes to warm up before drawing conclusions about performance.

Model compatibility: Not all models work on the NPU. NPU compilation requires static input shapes, which is why the config above pins 640x640 rather than leaving the size dynamic. Stick to models Frigate ships or lists as tested for NPU use.

✅ Wrap Up

NPUs are not hype for every workload, but for always-on inference tasks like Frigate object detection, they are a genuinely useful tool. The key insight is matching the right accelerator to the right job. CPU is flexible, iGPU is fast, and the NPU is efficient for sustained, repetitive inference.

If you are running a mini PC with a modern Intel Core Ultra chip and you have Frigate deployed, it is worth spending an afternoon switching the OpenVINO device string and watching your system metrics. You might free up meaningful resources without spending anything extra.

Full credit for the hands-on testing goes to Techno Tim. Watch the original video for the full walkthrough and real metric comparisons: What Is an NPU Actually Good For? on YouTube.



Originally published at codewithromi.com.

🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.

More Posts

How AI Is Changing Web Development, and What Developers Still Need to Understand

Daniel Decker - Nov 10

Comparison: Universal Import vs. Plaid/Yodlee

Pocket Portfolio - Mar 12

MCP Is the USB-C of AI. So Why Are You Plugging Everything In?

Ken W. Algerverified - Jun 10

Make for Small Business: What It Is, What It Costs, and When to Use It

floxolab - Oct 23

The Rowling-Galbraith Mystery: Testing Stylometry with Burrows’ Delta

nevmenandr - Oct 10
chevron_left
182 Points • 8 Badges
3Posts
1Comments
2Connections
Homelab, self-hosted infrastructure & cybersecurity. I test things in isolated VMs and publish what ... Show more

Related Jobs

Commenters (This Week)

1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!