This page is practitioner guidance on deployment mechanics, not legal advice — involve your counsel or DPO for decisions about your specific obligations.
Financial services AI deployment has to satisfy two obligations that most AI tooling ignores: everything must be reconstructable years later for supervision and records rules, and the model itself becomes a model under your existing model risk management framework. Self-hosting solves the first cleanly, because the complete prompt, retrieval, and completion record lands in systems you already retain from. It does not exempt you from the second — an internal assistant still needs an owner, documented limitations, and validation proportionate to its use.
Why this is hard
Supervision and records rules assume you can reproduce a communication. A vendor's 30-day log retention cannot do that.
Model risk governance was written for scorecards and pricing models, and nobody has told you where an LLM sits in it.
Customer data in a third-party inference log is an examination finding waiting to be written up.
What the deployment gives you
A record you can actually reconstruct
Prompt, retrieved context, model version, and completion written to your own archive with the user identity attached. Retention follows your existing schedule rather than a vendor's default, which is the difference between an answerable examination request and an awkward one.
Customer data stays inside the bank
Account details, transaction narratives, and KYC material never reach an outside processor. The Safeguards Rule analysis stops at your own perimeter, and there is no subprocessor chain to enumerate under examination.
Model risk management that fits SR 11-7
A pinned model version with a named owner, documented limitations, and validation proportionate to use. Self-hosting is what makes "pinned version" true — a hosted endpoint can change underneath you between validations.
Supervision built into the workflow
Review queues, retention, and surveillance hooks on AI-assisted communications, so an assistant that drafts client-facing text lands inside the supervisory system rather than beside it.
The audit trail is the architecture
In most sectors logging is an operational concern bolted on afterwards. Here it is the reason the design looks the way it does: the gateway exists so that a complete, immutable record is written before the model runs and after it answers, into storage governed by your retention rules.
Pin the model version and record it with every completion; "which model answered this" is an examination question you will be asked.
Send AI-assisted client communications through the same surveillance and review path as any other outbound communication.
Set retention from your records schedule, not from a tool default — prompts and completions are business records once they inform a decision.
Document the intended use and the known limitations before deployment; this is the artefact model risk review will ask for first.
Keep a human decision-maker on anything credit, suitability, or fair-lending adjacent. Explainability obligations do not soften because the tool is new.
What you have to satisfy
SR 11-7 / OCC 2011-12 — Model risk management: Supervisory guidance on model development, validation, and governance. An LLM used in a business process is a model under this framework and needs an owner and documented limitations.
SEC Rule 17a-4 — Records preservation: Requires records be preserved in a non-rewriteable, non-erasable format for defined periods — which a third-party inference log with short retention cannot satisfy.
FINRA Rule 3110 — Supervision: Firms must supervise business communications, including electronic ones. AI-assisted client-facing text belongs inside that supervisory system, not outside it.
16 CFR Part 314 — GLBA Safeguards Rule: Requires a written security programme covering service providers handling customer information — an obligation that shrinks considerably when there is no external processor.
Models that fit this deployment
Model
VRAM (Q4)
Runs on
Context
License
Qwen 3 32B Desk tier — one 24 GB card — Research summarisation, document extraction, and grounded Q&A over internal policy. ollama pull qwen3:32b
20 GB
24 GB GPU (RTX 3090/4090) Mac: 32 GB unified
128,000
Apache 2.0
Llama 3.3 70B Instruct Institution tier — 48 GB class — Longer documents and more nuanced analysis; single pro card, pinned version. ollama pull llama3.3:70b
43.1 GB
2×24 GB GPUs or 48 GB card Mac: 64 GB unified
128K
Llama Community
GPT-oss 120B Enterprise tier — 2×48 GB — Frontier-adjacent quality where the alternative is sending material to an external API. ollama pull gpt-oss:120b
We help banks, asset managers, and insurers deploy AI on their own infrastructure — audit and retention design that satisfies records rules, model risk documentation, hardware sizing, and the architecture write-up your second line will want. Vendor-neutral and practitioner-level.
Does an internal LLM assistant fall under model risk management?
Generally yes, and treating it as out of scope is the common early mistake. SR 11-7 defines a model broadly — a quantitative method that processes input into output used in decision-making — and an assistant that summarises research or drafts client material is informing decisions. What varies is the intensity of the treatment: a low-materiality drafting tool warrants documentation, an owner, stated limitations, and periodic review, not the full validation programme a pricing model gets. Agree the tiering with model risk before deployment rather than after.
How do you satisfy records rules for AI-generated content?
By capturing the whole interaction into the archive you already use. Records rules require preservation in a non-rewriteable format for defined periods, so the prompt, the retrieved context, the model version, and the completion all need to land in retention-governed storage with the user identity attached. This is exactly where hosted APIs make life hard: their retention is set for their operational needs, not your regulatory ones, and it is often measured in days. Self-hosting puts the record in your archive by default.
Can we use AI for anything customer-facing?
Yes, with the supervision you would apply to any other customer communication, and with real caution around credit, suitability, and fair-lending adjacent decisions. The safe pattern is assistive: the model drafts, a qualified person reviews and takes responsibility, and the whole exchange is retained and surveilled. Fully automated customer-facing decisioning brings explainability and fair-lending obligations that a general-purpose language model is poorly placed to discharge, so keep the human decision-maker where consequences attach.
What does DORA change for EU financial entities?
The Digital Operational Resilience Act tightens governance around ICT third-party risk, with register-of-information obligations and stricter requirements for critical providers. Practically, adding a hosted AI provider to a critical process adds an ICT third party to manage, document, test, and potentially exit. Self-hosted inference on your own infrastructure keeps the dependency internal, which shortens the third-party register and removes an exit-strategy problem. The underlying resilience and testing obligations still apply to the system you built.
Is on-premise inference cost-effective at bank scale?
At sustained volume, comfortably. Document extraction, research summarisation, and communication drafting across a large institution produce steady, predictable token demand — the profile where a fixed hardware cost amortised over three years beats a metered bill. The economics turn on utilisation, so model the real daily volume rather than the peak, and remember that batch workloads run overnight on hardware you have already paid for. Our cost calculator takes your volumes and current API prices directly.