Skip to main content

Intelai

TOPS Is Not a Workload Number

Edge AI runs out of memory and memory bandwidth before it runs out of compute. Most hardware requirements measure the thing that is not binding.

By Amyn PorbanderwalaApril 1, 20254 min read

Contents
  1. The arithmetic that decides what fits
  2. Bandwidth is the real limit
  3. What to put in a requirement
  4. Where NPUs fit, and where they do not
  5. Buy the workload, not the benchmark

Every edge AI device ships with a TOPS number on the box. Almost none of them ship with the two numbers that decide whether a model will actually run: memory capacity and memory bandwidth.

TOPS is a peak throughput figure. It is measured under conditions your workload will not reproduce, at a precision you may not use, often with sparsity assumptions that do not hold. It is useful for comparing chips in the same family. It is nearly useless for predicting whether a model fits.

The arithmetic that decides what fits

A model's weights have to be in memory. The size is parameters times bytes per parameter.

A 70 billion parameter model at 16-bit precision needs about 140 gigabytes just for the weights. At 8-bit, about 70 gigabytes. At 4-bit, about 35 gigabytes, before you add the key-value cache that grows with context length and batch size.

That arithmetic rules out most edge devices for large models, regardless of TOPS. A device with 32 gigabytes of memory cannot hold a 70 billion parameter model at 4-bit with room for the cache. It does not matter how fast the matrix multiply is if the weights are on a disk.

Smaller models change the math. A 7 or 8 billion parameter model at 4-bit is roughly 4 gigabytes, which fits comfortably on a phone or a ruggedized tablet. That is the real reason small language models matter at the edge. It is a capacity fact, not a quality preference.

Bandwidth is the real limit

Generating a token requires reading the model's weights, or the active portion of them, once per token. The limit on tokens per second is roughly the memory bandwidth divided by the bytes read per token.

On a device with 100 gigabytes per second of bandwidth and a 4 gigabyte model, the ceiling is about 25 tokens per second, before overhead. That is a reasonable interactive rate. A device with the same model and 25 gigabytes per second tops out near 6 tokens per second, which feels broken to a user.

Notice what did not appear in that calculation: TOPS. Inference for a single user is memory-bound, not compute-bound. Compute matters when you batch many requests, which is a datacenter pattern, not a tactical one.

This is why a datacenter accelerator and an edge accelerator are not interchangeable, and why a laptop with a large unified memory pool can run a model that a smaller device with a higher TOPS rating cannot.

What to put in a requirement

A hardware requirement for edge AI should describe the workload, not the chip.

  • Memory capacity, in gigabytes, with the model size and context length named. The requirement is "runs this model at this context," not "has at least this much RAM."
  • Memory bandwidth, in gigabytes per second. This sets the token rate ceiling.
  • Sustained performance, not peak. A device that throttles after two minutes is a device with a two-minute workload.
  • Power and thermal envelope. Joules per inference, and the ambient temperature range. A fan is a liability in the field.
  • Update path. How a new model version reaches the device, and how the old one is rolled back.
  • Quantization support. Which precisions the runtime actually accelerates, not which ones it accepts.

Two of those, memory bandwidth and sustained performance, are the ones vendors omit and buyers forget to ask for.

Where NPUs fit, and where they do not

A neural processing unit is a fixed-function accelerator. It is efficient at the operations it was designed for, usually quantized matrix math, and it does that work at low power. For a phone, a laptop, or a sensor, that is exactly the right trade.

The limits are real. An NPU's supported operations are a subset. A model architecture that does not map onto them falls back to the CPU or fails. Memory is often shared with the rest of the system, which caps the model size. Tooling is vendor-specific, which means the model you can run is partly a function of the compiler you have.

None of that makes NPUs a bad choice. It makes them a choice that has to be evaluated against a specific model and a specific runtime, not against a spec sheet.

Buy the workload, not the benchmark

The useful question for any AI hardware purchase is a sentence: run this model, at this context length, at this rate, in this environment, for this long.

If a vendor cannot answer that sentence with a demonstration, the TOPS number is decoration. The devices that win will be the ones whose memory and bandwidth match the models people actually deploy, and no marketing department has ever put that on a slide.