Written by Jakub Rusinowski · Last updated 2026-07-12 · Hardware figures computed by our VRAM engine
Evaluating an on-prem AI vendor comes down to six dimensions: where your data actually flows, what you may run and keep if you part ways, whether the deployment model is genuinely self-hosted, what the hardware sizing really requires, how updates and support work behind your firewall, and what the total cost looks like after year one. The 30 questions below turn those into an RFP-ready checklist — with the red flags that should end a conversation early.
The on-prem AI market inherited a bad habit from enterprise software: calling things "private" or "sovereign" that are neither. Some "on-premise" products are a thin gateway that still calls a cloud API for the actual inference. Some "your model" licenses evaporate the day the subscription ends. And some vendor hardware quotes are sized for the demo, not for your document volumes and concurrency.
None of this requires bad faith to go wrong — it requires only that the buyer didn't ask. This checklist is the asking. It's organized into six dimensions; inside each, the questions are ordered so that a disqualifying answer surfaces as early as possible. Use it as an RFP appendix, a demo script, or a scorecard — a simple 0/1/2 per question works better in practice than elaborate weighting.
Jakub Rusinowski, the founder of LLM Configurator, runs vendor-neutral evaluations and workshops for teams selecting on-prem AI: sizing sanity checks, deployment architecture review, and the questions specific to your compliance situation. No vendor relationships, no resale margin.
The single most important dimension, because it's the one "on-premise" marketing most often fudges.
| # | Question | Disqualifying answer |
|---|---|---|
| 1 | Draw the full data path for one prompt: which processes, which machines, which networks? | Any hop to vendor-operated infrastructure during inference |
| 2 | Does ANY feature (OCR, embeddings, "advanced reasoning", telemetry) call an external service? | "Only for some features" without a hard off switch |
| 3 | Can the system run with outbound network access fully blocked? | "It needs to phone home for licensing" with no offline option |
| 4 | What does the vendor's telemetry contain, and can we inspect it before it leaves? | Opaque or non-disableable telemetry |
| 5 | Where do logs live, who can read them, and what's the retention default? | Vendor-accessible logs of prompt content |
A genuinely self-hosted product can demonstrate question 3 live: pull the network cable and run the demo again. Vendors who can't should be reclassified as private-cloud offerings and evaluated under your cloud-processor rules instead — a legitimate category, but a different checklist.
| # | Question | Disqualifying answer |
|---|---|---|
| 6 | Which base models does the product use, under which licenses (Apache 2.0, MIT, Llama Community, proprietary)? | "Proprietary" with no escrow or continuity terms |
| 7 | If we fine-tune on our data, who owns the resulting weights? | Vendor owns or co-owns your fine-tunes |
| 8 | What exactly stops working when the contract ends? | Inference itself stops (kill-switch licensing) |
| 9 | Can we swap the underlying model for an open-weight alternative ourselves? | Hard-locked to vendor-supplied models |
| 10 | Are model weights stored on our hardware in a standard format (safetensors, GGUF)? | Encrypted blobs only the vendor's runtime can load |
The pattern to protect against is the rented brain: you buy servers, the vendor's license expires, and your $100k of hardware serves nothing. If the product builds on open-weight models — most do — the honest version of the answer to question 8 is "our tooling and support end; the models keep running."
| # | Question | Disqualifying answer |
|---|---|---|
| 11 | Air-gapped install: is it supported, documented, and actually tested? | "Possible in principle" |
| 12 | What are the exact update mechanics behind a firewall (offline bundles, checksums, rollback)? | Updates require vendor remote access |
| 13 | Does support require screen-sharing or remote sessions into our environment? | Mandatory vendor remote access |
| 14 | Which identity providers does it integrate with (SAML/OIDC/LDAP), and is SSO in the base tier? | SSO sold as a premium add-on on a security product |
| 15 | What's the minimum privileged footprint (root? cluster-admin? database superuser)? | Vague or maximal privilege requirements |
Question 14 deserves emphasis: the "SSO tax" is a known industry anti-pattern, and on a product whose entire pitch is security, charging extra for single sign-on tells you how the vendor thinks about security versus revenue.
| # | Question | Disqualifying answer |
|---|---|---|
| 16 | What model size and quantization does the quoted hardware actually serve, at what context length? | Quotes that omit quantization or context length |
| 17 | What's the concurrency assumption — how many simultaneous users before latency degrades? | "Depends" without a load-test offer |
| 18 | Can we reproduce the sizing math independently? | Sizing presented as proprietary secret |
| 19 | Does the quote include headroom for RAG (embedding models, vector DB, reranking)? | LLM-only sizing for a RAG product |
| 20 | Will it run on hardware we already own or can buy openly? | Locked to vendor-supplied appliances |
This is the dimension where this site can help directly: VRAM requirements for any open model at any quantization are computable, not negotiable. Check any vendor's claimed model-per-GPU math against our compatibility checker — if their numbers and the physics disagree, ask why. A vendor who quotes "Llama 70B on a 24 GB card" without mentioning aggressive quantization is either confused or hoping you are.
| # | Question | Disqualifying answer |
|---|---|---|
| 21 | What does the vendor need from us for routine support, and what's the SLA behind an air gap? | No offline support path |
| 22 | How are security patches for CVEs in the stack (inference server, web UI, dependencies) delivered and how fast? | No stated patch SLA |
| 23 | What observability do we get (token throughput, latency, per-user audit logs) and in what format? | Metrics visible only in the vendor's cloud dashboard |
| 24 | Who has performed a third-party security assessment of the product, and can we read it? | None, or NDA-walled summary only |
| 25 | What's the vendor's release cadence for new open models, and what happened the last three times a major model shipped? | No track record of keeping current |
Question 25 is the quiet one. The open-model landscape turns over every few months; a vendor whose product still headlines a 2024-era model has already shown you their maintenance velocity.
| # | Question | Disqualifying answer |
|---|---|---|
| 26 | Full three-year TCO: licenses, support tiers, per-seat vs per-GPU vs per-token components? | Per-token pricing on your own hardware |
| 27 | What triggers repricing — users, GPUs, documents, model swaps? | Unbounded usage-based repricing |
| 28 | Exit: what do we keep (configs, fine-tunes, embeddings, audit history) and in what format? | Proprietary export formats or no export |
| 29 | Reference customers at our scale, in our industry, reachable without vendor chaperones? | None after two years in market |
| 30 | If the vendor is acquired or folds, what's the continuity plan (escrow, license survival)? | Silence |
Question 26 hides the most common surprise: some on-prem products meter tokens *on your own GPUs*. That model can still be worth it for the tooling — but it must be priced against the alternative of serving open models yourself with vLLM or Ollama for the cost of an engineer-day of setup. Our TCO comparison gives you the baseline number to negotiate against.
Score each question 0 (disqualifying/unanswered), 1 (acceptable with caveats), or 2 (clean answer). Three practical thresholds from running this process with teams:
And keep the null hypothesis honest: for internal-assistant and RAG workloads, the DIY baseline — open-weight model, vLLM or Ollama, standard SSO gateway — is an engineer-week, not a moonshot. A vendor has to beat that baseline on integration depth, governance tooling, or support, not merely match it. If the checklist leaves you unsure which side of build-vs-buy you're on, that's usually the answer.
Jakub Rusinowski, the founder of LLM Configurator, runs corporate workshops and lectures on deploying local LLMs — hardware sizing, model selection, compliance-friendly architectures, and hands-on setup for your team. Direct, vendor-neutral, practitioner-level.