# Osolix · AI Training & Refinement Playbook — v1

**Status:** Wave-B Foundation evidence · 2026-Q2
**Owner:** Head of AI Development
**Audit dimensions:** #9 AI integration · #18 Observability + AI governance

This document is the operating playbook for keeping the 33 production AI
agents accurate, fast, and bounded — across both engines (Online +
Offline) and across every tenant.

## 1 · Training cadence

| Cadence | What runs | Owner |
|---|---|---|
| Per request | Live latency + reaction telemetry → AiFeedbackSignal table | Atlas |
| Nightly (02:00 UTC) | NightlyAgentKnowledgeService re-runs every agent over today's data; refreshes per-agent snapshot | NightlyAgents service |
| Weekly | Snapshot review: each agent's accept-rate / reject-rate / latency-p95 reviewed in the AI Governance dashboard | AI lead |
| Monthly | Per-tenant top 10 false-positive findings reviewed; rule-engine updated | AI lead + customer success |
| Quarterly | Re-attestation of every agent's pass-criteria; re-issue or update certificate | Head of AI Development |

## 2 · Quality bar per agent

Every agent must hit:
* **Latency p95 < 500ms** for Offline path; **< 2s** for Online path.
* **Accept rate ≥ 70%** in the customer base (i.e., users follow the recommendation).
* **False-positive rate ≤ 10%** (recommendation reversed by user within 5 minutes).
* **Reasoning text mandatory** for every recommendation < 9/10 confidence.
* **`<AiSourceBadge>` on every output** — Online (gold) or Offline (silver) + confidence pill.

## 3 · Online → Offline grounding contract

When the tenant's `AiMode = Online`, the chat engine ALWAYS runs the
deterministic Offline path first; the Online (Anthropic) path then
rewrites the deterministic answer for tone + clarity. The Online path
NEVER invents numbers or asset facts — those come from the Offline path.

This is the patent-pending Hybrid AI Resolver pattern (see
`docs/35-patent-hybrid-ai-disclosure.md`).

## 4 · Offline-only agents

Some agents are deliberately Offline-only:
* CFE Fraud Lens — every signal must be deterministic + auditable.
* Tenant query filter validator — same.
* RBAC test matrix runner — same.
* Repair-vs-NBV Advisor — pure deterministic compare.

Offline-only is a feature, not a limitation: it lets the agent run in
air-gapped tenants (gov + defence + healthcare) without ANY external
LLM round-trip.

## 5 · Failure-mode handling

| Failure | Handling |
|---|---|
| Anthropic 503 | Fall back to Offline answer; mark output `<AiSourceBadge>` = Offline |
| Anthropic rate-limit | Queue with exponential backoff; banner the user |
| Anthropic prompt-injection detected | Reject the input; log + alert CISO |
| Atlas response > 5s | Return Offline immediately; spawn Online in background; reconcile next page-load |
| Offline rule-set produces no answer | Surface "I don't know how to answer that yet" + `Show me suggestion menu` button |
| AI agent disagrees with deterministic source | Always defer to deterministic; alert AI Governance dashboard |

## 6 · Per-tenant override matrix

Tenant admins can per-agent:
* **Disable** the agent entirely (some industries don't want any AI).
* **Switch to Offline-only** (e.g. air-gapped tenant).
* **Override the system prompt** (per-tenant brand voice tuning).
* **Set per-agent confidence threshold** (e.g. only show recommendations ≥ 85% confidence).

All overrides flow through `AgentSetting` (agent id + tenant + system prompt + enabled flag).

## 7 · Customer-facing data-handling guarantees

* No PII ever embedded in an LLM prompt — only IDs + scrubbed summaries.
* Anthropic retains prompt + response 30 days per their DPA (we surface this in the consent banner).
* Offline-only mode produces no Anthropic call → tenant data NEVER leaves the customer's region.
* Per-tenant audit log captures every AI agent call with: agent id, latency, source, confidence, reaction.

## 8 · Open items
* Build the per-tenant Online ↔ Offline switch UI (currently DB-flag only).
* Wire per-agent token-cost telemetry → AI Governance dashboard.
* Add the customer-facing AI consent banner ("agree to share scrubbed summaries with Anthropic").