FINANCIAL SERVICES

Financial Services AI: Every Prompt Recorded, Nothing in a Vendor Log

An examiner will ask what the model was told, what it answered, and who saw it. Three years later.

Written by Jakub Rusinowski · Last updated 2026-08-01 · Hardware figures computed by our VRAM engine

This page is practitioner guidance on deployment mechanics, not legal advice — involve your counsel or DPO for decisions about your specific obligations.

Financial services AI deployment has to satisfy two obligations that most AI tooling ignores: everything must be reconstructable years later for supervision and records rules, and the model itself becomes a model under your existing model risk management framework. Self-hosting solves the first cleanly, because the complete prompt, retrieval, and completion record lands in systems you already retain from. It does not exempt you from the second — an internal assistant still needs an owner, documented limitations, and validation proportionate to its use.

Why this is hard

What the deployment gives you

A record you can actually reconstruct

Prompt, retrieved context, model version, and completion written to your own archive with the user identity attached. Retention follows your existing schedule rather than a vendor's default, which is the difference between an answerable examination request and an awkward one.

Customer data stays inside the bank

Account details, transaction narratives, and KYC material never reach an outside processor. The Safeguards Rule analysis stops at your own perimeter, and there is no subprocessor chain to enumerate under examination.

Model risk management that fits SR 11-7

A pinned model version with a named owner, documented limitations, and validation proportionate to use. Self-hosting is what makes "pinned version" true — a hosted endpoint can change underneath you between validations.

Supervision built into the workflow

Review queues, retention, and surveillance hooks on AI-assisted communications, so an assistant that drafts client-facing text lands inside the supervisory system rather than beside it.

The audit trail is the architecture

In most sectors logging is an operational concern bolted on afterwards. Here it is the reason the design looks the way it does: the gateway exists so that a complete, immutable record is written before the model runs and after it answers, into storage governed by your retention rules.

The audit trail is the architectureRegulated perimeter — retained systems. 1. Analyst / advisor: Authenticated to the firm directory; role and desk attached to every request. 2. Gateway & recorder: Writes prompt, context, and model version to the archive before inference runs. 3. Inference server: Pinned open-weight model on firm GPUs. No egress, no vendor telemetry. 4. WORM archive: Immutable, retention-governed storage — the same one your comms already use.REGULATED PERIMETER — RETAINED SYSTEMSAnalyst / advisorAuthenticated to the firmdirectory; role and deskattached to every request.Gateway & recorderWrites prompt, context, andmodel version to thearchive before inferenceruns.Inference serverPinned open-weight model onfirm GPUs. No egress, novendor telemetry.WORM archiveImmutable,retention-governed storage— the same one your commsalready use.

What you have to satisfy

Models that fit this deployment

模型VRAM (Q4)可运行于上下文许可
Qwen 3 32B
Desk tier — one 24 GB card — Research summarisation, document extraction, and grounded Q&A over internal policy.
ollama pull qwen3:32b		20.6 GB24 GB GPU (RTX 3090/4090)
Mac: 32 GB 统一内存		125KApache 2.0
Llama 3.3 70B Instruct
Institution tier — 48 GB class — Longer documents and more nuanced analysis; single pro card, pinned version.
ollama pull llama3.3:70b		43.1 GB2×24 GB GPUs or 48 GB card
Mac: 64 GB 统一内存		128KLlama Community
GPT-OSS 120B
Enterprise tier — 2×48 GB — Frontier-adjacent quality where the alternative is sending material to an external API.
ollama pull gpt-oss:120b		71.3 GB2×48 GB GPUs / big unified memory
Mac: 96 GB 统一内存		128KApache-2.0

Deploying AI that has to survive an examination?

We help banks, asset managers, and insurers deploy AI on their own infrastructure — audit and retention design that satisfies records rules, model risk documentation, hardware sizing, and the architecture write-up your second line will want. Vendor-neutral and practitioner-level.

Ask about a workshop →

常见问题

Does an internal LLM assistant fall under model risk management?
Generally yes, and treating it as out of scope is the common early mistake. SR 11-7 defines a model broadly — a quantitative method that processes input into output used in decision-making — and an assistant that summarises research or drafts client material is informing decisions. What varies is the intensity of the treatment: a low-materiality drafting tool warrants documentation, an owner, stated limitations, and periodic review, not the full validation programme a pricing model gets. Agree the tiering with model risk before deployment rather than after.
How do you satisfy records rules for AI-generated content?
By capturing the whole interaction into the archive you already use. Records rules require preservation in a non-rewriteable format for defined periods, so the prompt, the retrieved context, the model version, and the completion all need to land in retention-governed storage with the user identity attached. This is exactly where hosted APIs make life hard: their retention is set for their operational needs, not your regulatory ones, and it is often measured in days. Self-hosting puts the record in your archive by default.
Can we use AI for anything customer-facing?
Yes, with the supervision you would apply to any other customer communication, and with real caution around credit, suitability, and fair-lending adjacent decisions. The safe pattern is assistive: the model drafts, a qualified person reviews and takes responsibility, and the whole exchange is retained and surveilled. Fully automated customer-facing decisioning brings explainability and fair-lending obligations that a general-purpose language model is poorly placed to discharge, so keep the human decision-maker where consequences attach.
What does DORA change for EU financial entities?
The Digital Operational Resilience Act tightens governance around ICT third-party risk, with register-of-information obligations and stricter requirements for critical providers. Practically, adding a hosted AI provider to a critical process adds an ICT third party to manage, document, test, and potentially exit. Self-hosted inference on your own infrastructure keeps the dependency internal, which shortens the third-party register and removes an exit-strategy problem. The underlying resilience and testing obligations still apply to the system you built.
Is on-premise inference cost-effective at bank scale?
At sustained volume, comfortably. Document extraction, research summarisation, and communication drafting across a large institution produce steady, predictable token demand — the profile where a fixed hardware cost amortised over three years beats a metered bill. The economics turn on utilisation, so model the real daily volume rather than the peak, and remember that batch workloads run overnight on hardware you have already paid for. Our cost calculator takes your volumes and current API prices directly.

Go deeper