Decision models on phones, Jetson and laptops

Decision models are the easiest kind of model to run at the edge: the useful ones start well under 1 GB, and they answer in one forward pass instead of generating text. On a Mac or a laptop with a small GPU, almost everything in the table below fits. On a Jetson, d1-3B is the one with published timings. On a plain CPU, start with OpenDecider-nano. On a phone, nothing here documents a native runtime yet, so a phone is best used as a client to a small server on your own network. Check your exact machine in the hardware analyzer before you download.

Why small decision models suit edge hardware

A chat model spends most of its time generating: every word of an answer is another decode step, paced by memory bandwidth. A decision model reads your input, scores the options you gave it, and stops. There is no long answer to generate, so the cost is dominated by processing the input once. That changes what matters on small hardware:

• Size decides whether it fits at all. These models are small enough that a phone-class or laptop-class memory budget is realistic, which is not true of chat models of similar quality.
• Input length decides the latency. A long ticket or a big image costs more than the question itself. Liquid AI's d1-3B card shows this directly: on the same device, a 384 px image takes roughly two to four times as long as a single short question, and a 3.4k-token state far longer still.
• Batching is cheap. Several questions about the same input can often share one pass; the d1-3B card times three questions in one pass at well under three times one question.

The models

Seven small decision models, smallest download first. The table renders from each model's record, so the sizes, runtimes and latencies match the model pages:

Every latency is the publisher's own figure on the publisher's own device and setup, so two rows are not a like-for-like race: one is a single question on a laptop, another a batched call on a datacenter GPU. Practical deployment is derived from the smallest published file plus 0.8 GB of framework overhead, with the same tier thresholds as every model page.

Three rules for reading the latency column:

1. It is the publisher's own number, on the publisher's own device and setup. Nobody has reproduced most of these yet. Each row says who measured it.
2. The conditions differ, so this is not a race. One row is a single question on a datacenter GPU, another a batched call where the time is divided by fifty, another a warm single question on a Jetson. Compare a model's rows with each other, not across models.
3. "Not published" means not published. Tev1 0.8B and Bosun 0.6B are small and fast in principle, but their authors publish no timings, and we do not fill the gap with an estimate.

There is deliberately no ranking column. Accuracy depends on your questions, and the evaluation guide shows how to compare candidates on 200 of your own rows.

By device

Phone

No model in this table documents a native phone runtime. Two routes come close, and both are experiments rather than recipes:

• The browser. OpenDecider-nano publishes ONNX builds that run in Chrome with WebGPU. Its card times them on a Mac, not on a phone, so treat a phone as untested.
• llama.cpp on Android. llama.cpp documents an Android build, and Bosun 0.6B ships a Q4_K_M GGUF that is the smallest download in the table. Bosun's own server uses a llama.cpp path for GGUF, but nobody publishes a phone setup or timing for it.

For anything you rely on, run the model on a laptop, a mini PC or a Jetson on your network, and let the phone send requests to it. Every server in the server comparison listens over HTTP, and the phone never needs to hold the model.

Apple Silicon laptop or mini PC

The best-documented edge platform in this list. OpenDecider's cards test nano and small on a Mac mini M4 with 16 GB and on an M4 Max, and ship a 4-bit MLX build of small for Macs. d1-3B's card times it on an M5 Pro through PyTorch's MPS backend. Tev1 0.8B runs through Ollama like any other Ollama model. With 16 GB of unified memory, everything in the table fits with room to spare. Serve with opendecider serve for OpenDecider, Ollama for Tev1, and a small Python process for d1.

NVIDIA Jetson

d1-3B is the one model here with published Jetson timings: Liquid AI measured it, in collaboration with NVIDIA, on a Jetson Orin Nano, an AGX Orin 64 GB and an AGX Thor, warm, one question at a time. It runs through Transformers in Python, so the work is getting a PyTorch build for your JetPack version. The model code itself is the same as on a desktop. Its BF16 download is the largest in the table, so check how much memory your module leaves free before you choose the smallest board. For a long-running service on a bigger Jetson, SGLang lists Jetson Orin among its supported NVIDIA platforms.

A small GPU (4–8 GB)

Every text-only model in the table fits a 4 GB card in its smallest build, and OpenDecider's card says nano should fit any NVIDIA GPU with 4 GB, though it has not tested that. d1-3B needs a card that holds its BF16 file plus working memory, so plan on 8 GB. THX-01's authors report their latency on "one GPU" without naming it, which is the number to re-measure first. Use the analyzer with your exact card: it judges fit from the publisher's file where there is one.

CPU only

OpenDecider-nano is the model to start with: its card tests it on a plain CPU and reports sub-second answers, and its live demo runs on a basic CPU. The d1 code falls back to the CPU in 32-bit floats, which works but is the slowest way to run it, and Liquid AI publishes no CPU timing. For a GGUF on CPU, llama.cpp's CPU builds are the usual route. If you already run Ollama, Tev1 0.8B works there on a CPU-only machine too. Expect hundreds of milliseconds per short question rather than tens, and measure with your real input lengths before you put it in a loop.

Measure on your own device

Published latencies are a starting point. Before you deploy, time the model on the device you will use, with your own inputs:

# latency.py: median latency of one decision request, after a warm-up
import json, statistics, time, urllib.request

URL = "http://localhost:8000/v1/systemone"   # your server's port
BODY = {"state": "I was charged twice for my order.", "questions": {
    "refund": {"type": "noul", "instructions": "Is a refund being requested?"}}}

def call():
    req = urllib.request.Request(URL, json.dumps(BODY).encode(), {"Content-Type": "application/json"})
    t = time.perf_counter(); urllib.request.urlopen(req).read(); return (time.perf_counter() - t) * 1000

call()                                            # warm-up: loads weights, compiles kernels
runs = [call() for _ in range(20)]
print(f"median {statistics.median(runs):.1f} ms   p90 {sorted(runs)[17]:.1f} ms")

Add a "model" field if your server needs one, and use a state as long as your real ones: the input length moves the number more than anything else. The benchmarking guide covers what to record alongside the result.

Sources, checked 2026-10-14: d1-3B card: edge timings, multi-question and image rows · Liquid AI: d1 on NVIDIA Jetson · OpenDecider-nano card: "Will it fit?" (Mac, CPU, browser, 4 GB GPU) · OpenDecider-small card: GGUF and MLX builds · Bosun v3.1 0.6B card: GGUF builds · Hanno Labs: jev-compatible-server Transformers and llama.cpp paths · THX-01 card: latency on one GPU · llama.cpp README: Android build, CPU backends · SGLang README: supported platforms incl. Jetson Orin.

Frequently asked questions

Keep going

• Decision models (Jev-style System One): run them locally→
• Run Jev-style decision models locally with Ollama (Nimble, Tev1)→
• Open Jev reproductions beyond Ollama: Winnow, Decider, JevK5, AutoJev, Laya→
• Decision-model confidence, thresholds and calibration — use the probabilities safely→
• Decision models on images: classify receipts, screenshots and forms locally with Clef→
• How to evaluate a decision model on your own data→
• Ollama /v1/systemone errors and fixes→
• Ollama /v1/systemone image errors→
• Decision request builder→