# Architecture

## Phase 5 Desktop Persistence and Reliability

The desktop uses SQLite only for non-secret read-cache and offline mutation metadata. Tokens are
never persisted in Zustand, Web Storage, or SQLite. Session restore remains cookie-only: the
backend's httpOnly `rt` cookie is rotated by `/auth/refresh`, while the short-lived access token
stays in module memory. A native Windows credential-manager bridge is available for future
desktop secrets and has no plaintext fallback.

- `tauri-plugin-sql` (registered in `desktop/src-tauri/src/lib.rs`) is the only SQL access path;
  `cache_entries` stores last-known-good list responses and `offline_mutation_queue` stores
  request metadata/payloads for safe non-message writes. No token-bearing data is accepted by
  the queue.
- `keyring` with the `windows-native` backend exposes `secure_credential_set/get/delete` Tauri
  commands backed by Windows Credential Manager. `secureCredentialStore.ts` calls those commands
  directly and deliberately fails closed when the native bridge is unavailable; it never falls
  back to localStorage, sessionStorage, a file, or SQLite.
- Cache-then-network, always-refresh, no TTL: `loadWithCache()` always attempts the network
  fetch. A cached fallback is used only when the failure is classified offline (`ApiError`
  status `0`, i.e. `backend_unavailable` or `request_timeout`); any real HTTP error, or an
  offline failure with no prior cached row, rethrows unmasked so existing error UI is unaffected.
- Only the most recent, unfiltered/default list view per entity is cached (one row per entity
  type in a single generic `cache_entries` table) — not per-filter-combination result sets.
- API-mode only. `loadWithCache` checks `isApiMode` and bypasses the database entirely in mock
  mode, so synthetic mock data is never persisted to disk.
- This module never imports `getAccessToken`/`setAccessToken`/`authApiService` — the access
  token remains module-memory-only exactly as before (see "Native API Authentication &
  Readiness Closure" below); nothing token-related is ever serialized into `cache.db`.
- A concurrency-safe global `isShowingCachedData` flag (derived from an internal
  `staleEntities: Set<CacheEntityType>` via `markEntityCacheState()`) drives a global
  `OfflineBanner` rendered in `AuthenticatedLayout.tsx`, so multiple simultaneous `loadX()`
  calls (e.g. Inbox loading both conversations and customers) cannot clobber each other's
  cache-fallback state.
- Offline replay is opt-in and currently covers customer profile PATCH updates. Each row has a
  unique idempotency key and expected version; transient network/5xx/429 failures use bounded
  exponential retry, while HTTP 409 version conflicts are terminal and retained for resolution.
  Outbound message paths are rejected, so this slice never starts customer-facing sends.
- Remaining out of scope: draft persistence and conversation message caching (only the
  conversation list itself is cached).

## Phase 4C Media And Interactive Acceptance Notes

- Inbound WhatsApp button and list replies persist reply metadata and correlate
  back to the source outbound message, provider attempt, and BroadcastRecipient
  when the context provider message ID resolves inside the same organization and
  WhatsApp connection.
- Production media storage uses a private S3-compatible object-storage adapter
  with tenant-scoped opaque keys, safe API download responses, and no public
  object URLs. Local MinIO smoke evidence is development evidence only, not live
  production storage activation.
- Phase 4C native fake-provider acceptance is complete (2026-06-20). The full
  native acceptance script proved: outbound image send, inbound image/document/
  audio/video ingestion, quarantine/recovery, duplicate webhook deduplication,
  quick-reply source-message correlation, BroadcastRecipient correlation,
  broadcast pause/resume/cancel lifecycle, media-header broadcast dispatch,
  mark-as-read provider call, restart/logout persistence, and browser-storage
  security checks through the Tauri UI.
- Full desktop monitoring states for media-header broadcast campaigns remain
  partially wired; the monitoring page shows status but not full media-asset
  preview. This is a UI polish gap, not an acceptance blocker.
- Live Meta connection, live media activation, and Phase 4D start remain blocked
  until explicit operator authorization.

## Phase 4E Facebook Messenger Architecture

Phase 4E adds Facebook Page Messenger (inbound + outbound) alongside the existing WhatsApp
Cloud and Instagram scaffolding, reusing the same webhook, identity, conversation, message,
and provider-attempt infrastructure.

### Backend components

```text
Meta webhook POST /webhooks/meta/facebook_page
        |
        v
WebhookProcessingService._handle_facebook_page_delivery()
        |
        +-- FacebookPageConnection lookup by page id
        +-- CustomerIdentity: identity_type = "facebook_page", normalized_value = sender PSID
        +-- Conversation: channel = "facebook_page", facebook_page_connection_id populated
        +-- Message: direction = "inbound", provider = "facebook_page"

CRM reply POST /api/v1/conversations/{id}/messages
        |
        v
MessageService.send_message()
        |
        +-- channel == "facebook_page" && facebook_page_connection_id != null
        +-- settings.facebook_page_send_enabled must be true
        +-- Message provider = "facebook_page", simulated = false
        +-- _attempt_facebook_page_provider_send()
            +-- FacebookPageConnectionService.get_by_org_or_raise()
            +-- decrypt encrypted_page_access_token
            +-- resolve PSID from CustomerIdentity (identity_type = "facebook_page")
            +-- FacebookPageMessagingProvider.send_text()
            +-- FacebookGraphApiClient.send_text_message()
            +-- POST https://graph.facebook.com/v22.0/{page_id}/messages
            +-- record MessageProviderAttempt (success/failure)
```

### Key patterns

- Separate live-send gate: `META_FACEBOOK_PAGE_LIVE_MODE` + `META_FACEBOOK_PAGE_ALLOW_LIVE_SEND`,
  independent from WhatsApp and Instagram gates.
- Encrypted token: `FacebookPageConnection.encrypted_page_access_token` uses the same AES-256-GCM
  credential encryption as `WhatsAppConnection`.
- No PSID exposure to frontend: the frontend sends `{conversation_id, text}`; the backend resolves
  the PSID server-side from `CustomerIdentity`.
- Idempotent webhook: `MetaWebhookDelivery.payload_hash` deduplicates identical payloads;
  `MetaWebhookEvent.event_key` deduplicates by event id.
- `MessageProviderAttempt` records every outbound send attempt with HTTP status, provider message id,
  error summary, and timing.

### Frontend components

- `desktop/src/types/domain.ts`: `Conversation.facebookPageConnectionId?: string`
- `[redacted].ts`: maps `facebook_page_connection_id` from API
  response into frontend model.
- `desktop/src/pages/InboxPage.tsx`: `ChannelBadge()` renders Facebook Page badge for
  `channel === "facebook_page"`; composer uses channel-aware `isChannelSendable()` instead of a
  global `LIVE_SEND_ENABLED=false` constant.

### Validation

- Live inbound: external Messenger message → webhook → backend processed with HTTP 200.
- Live outbound: CRM reply "Facebook CRM round-trip test" → Meta Graph API HTTP 200 OK → external
  recipient confirmed receipt.

## Target Architecture

```text
Windows desktop app
Tauri 2 + React + TypeScript
        |
        | HTTPS and WebSocket
        v
FastAPI backend
        |
        +-- PostgreSQL
        +-- Redis when needed
        +-- Object storage when needed
        +-- Meta WhatsApp Cloud API
```

## Desktop

The desktop app is the first implementation target. It uses:

- Tauri 2
- React
- TypeScript
- Vite
- React Router
- Reusable accessible components
- Strict TypeScript

The desktop app must eventually produce a Windows installer such as .exe or .msi.

## Backend

The backend is planned for a later phase and should use:

- Python
- FastAPI
- Pydantic
- SQLAlchemy
- Alembic
- PostgreSQL
- WebSockets
- Pytest

Every tenant-owned table must include organization_id.

## WhatsApp Integration

WhatsApp integration will use the official Meta WhatsApp Cloud API in a later phase. Privileged Meta communication must pass through the backend.

Never embed these in the desktop executable:

- Meta access token
- Meta app secret
- Database password
- Webhook secret
- Private signing keys

### Phase 4A Provider Pipeline

Phase 4A adds the backend provider baseline while preserving the desktop/backend
security boundary:

- `WhatsAppConnection` stores organization-scoped WABA/phone identifiers and an
  encrypted Meta access token. Plaintext tokens are decrypted just in time for
  provider calls and are never returned to the desktop.
- `MetaGraphApiClient` is versioned by `META_GRAPH_API_VERSION` and is wrapped
  by `MetaWhatsAppCloudProvider`; mock and Meta providers share the
  `MessagingProvider` abstraction.
- `MessageService.send_message()` routes `whatsapp_cloud` conversations through
  the provider only when `settings.provider_send_enabled` is true.
- `WHATSAPP_PROVIDER_TRANSPORT=fake` selects a deterministic in-process fake
  HTTP transport in development only. Production startup rejects this setting.
- Dev-only acceptance endpoints can reset fake provider state, select fake
  scenarios, create a cloud conversation fixture, and create a low-permission
  viewer user. These endpoints require development mode and backend
  permissions.
- Webhook ingestion verifies `X-Hub-Signature-256` over raw request bytes before
  processing inbound messages or status events.

Native Phase 4A acceptance drives the real Tauri UI against the real FastAPI
endpoints while only the backend provider client receives deterministic fake
Meta responses.

## Phase 1 Architecture

Phase 1 is frontend-only plus the Tauri shell:

- Mock phone validation
- Mock OTP send success state
- Mock QR login state
- No network calls
- No database
- No local persistence beyond component state

## Phase 2 Stage 2A Architecture

Stage 2A remains frontend-only and native-desktop-first:

- React Router guards protect `/app/*` routes.
- The existing login modal creates an in-memory mock session after OTP verification or simulated QR scan.
- Zustand holds the mock session, sidebar state, notification state, and global search state.
- Typed mock services live under `desktop/src/services/` and return promises with short simulated delays.
- Mock data lives under `desktop/src/data/`; UI components call services/state rather than importing raw arrays directly.
- The authenticated shell lives beside the Phase 1 shell and includes full CRM navigation, top header, notification center, global search, sidebar collapse, and logout.
- No backend, database, SQLite cache, Redis, or Meta integration has been added.

Current frontend service boundaries:

```text
authService
notificationService
searchService
```

Future stages should add services for broadcasts, tickets, team, settings, and analytics before implementing full module workflows.

## Phase 2 Stage 2B Architecture

Stage 2B is still frontend-only and mock-service-driven. It adds the first coherent CRM vertical slice without changing the Tauri/backend boundary:

- `desktop/src/types/domain.ts` now models customers, tags, team members, conversations, assignments, messages, dashboard metrics, activity, filters, and toast state.
- `desktop/src/data/mockData.ts` now follows the minimal-fixture policy: only enough customers, conversations, messages, team members, tags, products, orders, and catalogues to cover required fields and UI states, plus deterministic fixture factories for pagination or load-testing needs.
- Zustand owns the shared in-memory data graph and exposes mutations for message sending, retries, conversation status, assignment, customer create/edit, notes, tags, archive/restore, and filters.
- Dashboard, Inbox, Customers, and Customer Detail pages consume store actions rather than importing mock arrays directly.
- Global search is generated from the current customer and conversation state so newly created or renamed customers are searchable immediately.
- UI feedback uses an in-memory toast host only; there is no persistence or outbound integration.

Current Stage 2B frontend service boundaries:

```text
dashboardService
conversationService
messageService
customerService
```

No backend, database, SQLite cache, Redis, real OTP, real JWT, or Meta WhatsApp integration has been added.

## Phase 2 Stage 2C Architecture

Stage 2C remains frontend-only and mock-service-driven. It adds the commerce slice without changing the Tauri/backend boundary:

- `desktop/src/types/domain.ts` models orders, order line items, order timelines, payment status, fulfillment status, products, product variants, product availability, catalogues, and commerce filters.
- `orderService`, `productService`, and `catalogueService` expose promise-based mock operations with short simulated delays.
- Zustand owns orders, products, catalogues, selected records, filters, loading states, errors, and success toasts alongside the existing dashboard, inbox, and customer state.
- Dashboard order metrics and recent orders derive from current in-memory order state.
- Customer detail reads real Stage 2C mock orders instead of placeholder order summaries.
- Global search records are generated from current customers, conversations, orders, products, and catalogues.
- Product/catalogue mutations keep membership state consistent in both modules; product rename/price changes update order item display.
- No backend, database, SQLite cache, Redis, real payment flow, Meta WhatsApp integration, or cloud service has been added.

Current Stage 2C frontend service boundaries:

```text
orderService
productService
catalogueService
```

## Phase 2 Stage 2D Architecture

Stage 2D remains frontend-only and mock-service-driven. It adds the communication and analytics slice:

- `broadcastService`, `templateService`, `segmentService`, `ticketService`, `analyticsService` expose promise-based mock operations.
- Zustand state expands to cover broadcasts, templates, segments, tickets, and analytics with 60+ actions.
- Dashboard, Inbox, CustomerDetail, and OrderDetail are cross-integrated with Stage 2D modules.
- Global search includes broadcasts, tickets, templates, and segments.

## Phase 2 Stage 2E Architecture

Stage 2E remains frontend-only and mock-service-driven. It adds team management, roles, settings, and audit:

- `teamService`, `roleService`, `invitationService`, `organizationSettingsService`, `userPreferencesService`, `auditService` expose promise-based mock operations.
- `desktop/src/utils/permissions.ts` exports `createPermissionChecker(role)`, `PERMISSION_KEYS`, `PERMISSION_GROUPS`, `HIGH_RISK_PERMISSIONS`, `formatPermissionLabel`.
- `desktop/src/hooks/usePermissions.ts` exports `usePermissions()` — a memoized hook that returns `can(key: PermissionKeyLiteral) => boolean` from the current user's role.
- `[redacted].tsx` renders an access-denied screen with optional `requiredPermission` label and a disclaimer.
- Route-level permission guards are applied to BroadcastsPage, TicketsPage, AnalyticsPage, TeamPage, RolesPage, SettingsPage, and AuditPage using `const can = usePermissions(); if (!can("x.view")) return <AccessDenied ... />`.
- `appendAuditEvent` is called after significant business mutations (role create/edit, member invite, settings save, order status, product stock, ticket assign/resolve, broadcast send/cancel).

> **IMPORTANT:** Client-side permissions are for interface modelling only and are not a security boundary. The future backend must enforce all authorization. `can()` on the desktop controls which UI elements render — it cannot prevent a determined user from calling store actions directly.

Current Stage 2E frontend service boundaries:

```text
teamService
roleService
invitationService
organizationSettingsService
userPreferencesService
auditService
```

## Phase 2 Stage 2F Architecture

Stage 2F remains frontend-only and mock-service-driven. It hardens the permission layer and settings without changing the Tauri/backend boundary:

- `PERMISSION_KEYS` expanded from 54 to 92 granular keys covering all module-level actions.
- All 5 mock role permission arrays (owner/admin/manager/agent/viewer) updated to include new keys.
- `usePermissions()` now returns `{ can, cannot, canAny, canAll, role, permissions }` instead of a single function.
- `PermissionGate` component added — renders `children` when `can(permission)` is true, otherwise renders `fallback` (default: null).
- `DisabledAction` component added — renders a visually disabled button with `title` tooltip and `aria-disabled` when the user lacks a permission.
- `AuthNavItem.permission` field added to `appConfig.ts`; `AuthenticatedSidebar` filters nav items using `can(item.permission)`.
- `SettingsPage` guards each section with `can("settings.manage_X")` and shows `AccessDenied` for inaccessible sections; nav links replaced with `<button>` elements that check `isDirtyRef` before navigating.
- `UnsavedChangesDialog` added inside `SettingsPage` with Stay / Discard / Save and continue actions.
- `appendAuditEvent` calls added to all previously missing store mutations.

> **IMPORTANT:** Client-side permissions are for interface modelling only and are not a security boundary. The future backend must enforce all authorization. `can()` on the desktop controls which UI elements render — it cannot prevent a determined user from calling store actions directly.

Current Stage 2F frontend service boundaries (no new services; hardening of existing permission and settings infrastructure):

```text
permissions.ts (utils) — createPermissionChecker, PERMISSION_GROUPS, HIGH_RISK_PERMISSIONS
usePermissions.ts (hook) — { can, cannot, canAny, canAll, role, permissions }
PermissionGate / DisabledAction (components)
UnsavedChangesDialog (inline component in SettingsPage)
```

## Future Backend Model Notes

The following domain models will require backend implementation when Phase 3 begins:

```text
organizations          — multi-tenant root; every table scoped by organization_id
users                  — unified user record across org memberships
organization_members   — links users to organizations with a role
roles                  — per-organization role definitions
permissions            — role-permission join table
sessions               — JWT sessions with refresh tokens
audit_logs             — immutable append-only event ledger (server-enforced)
invitations            — pending email invitations with accept tokens (server-side expiry)
```

Audit integrity requires server-side append-only enforcement. Client-side `appendAuditEvent` is a design placeholder only.

## Phase 3A Backend Architecture

Phase 3A implements the FastAPI backend under `backend/`. The desktop remains the primary UI; the backend is the real security boundary.

### Stack

```text
Python 3.12 + FastAPI
SQLAlchemy 2 async + asyncpg
Alembic — database migrations
PyJWT — HS256 access tokens
pwdlib[argon2] — Argon2 password hashing
pydantic-settings — env-based configuration
Ruff — linter + formatter
mypy strict — type checking
pytest-asyncio (auto mode) — async test runner
```

### Token Architecture

```text
Access token:
  - HS256 JWT, 15 min expiry
  - Returned in response body
  - Caller keeps in application memory ONLY (never localStorage/sessionStorage)
  - Cleared on application restart, but automatically re-minted from the
    refresh cookie on startup restore — no re-login required (see below)

Refresh token:
  - Opaque random bytes (secrets.token_urlsafe(48))
  - Only sha256 hex digest persisted in refresh_tokens table, alongside the
    organization_id it was issued for (added 2026-07-21 — see fix note below)
  - Raw token sent as httpOnly `rt` cookie
  - Cookie persistence is caller-controlled at login time via `remember_me`
    (added 2026-07-21): `true` (default) issues a 30-day persistent cookie;
    `false` issues a session cookie (no Max-Age/Expires) that the OS clears
    when the browser/WebView process fully exits. Rotation via `/refresh`
    preserves whichever policy was chosen at the original login — it does not
    silently upgrade a session-only login into a persistent one.
  - Token rotation: old revoked, new issued with same family_id
  - Reuse of revoked token → entire family revoked immediately (theft detection)
  - `POST /api/v1/auth/logout-all` revokes every active refresh token for the
    user (all devices/sessions), desktop-wired via Settings → Security →
    "Sign out all other sessions" (2026-07-21)
```

**Fix note (2026-07-21):** `POST /api/v1/auth/refresh` previously required a valid
`Authorization: Bearer` access token in addition to the `rt` cookie (via a
`CurrentPrincipal` dependency), which meant it could only rotate a token while
already logged in — it could never actually restore a session once the access
token was gone (app restart, or natural 15-minute expiry), contradicting the
"Startup restore" contract documented below. Fixed: the endpoint now
authenticates purely from the `rt` cookie; `RefreshToken` gained an
`organization_id` column so a cookie-only restore can resolve the exact
organization the token was issued for without needing a principal (a user can
belong to more than one organization).

### Principal

Every protected endpoint receives a `Principal` dataclass:

```python
@dataclass
class Principal:
    user_id: uuid.UUID
    organization_id: uuid.UUID
    role_type: str
    permissions: list[str]
    jti: str

    def require(self, permission: str) -> None: ...     # raises HTTP 403
    def require_any(self, *permissions: str) -> None: ...
```

Endpoints call `principal.require("permission.key")` before any DB work.

### Desktop Integration

```text
VITE_AUTH_MODE=mock  — in-memory mock session (default)
VITE_AUTH_MODE=api   — calls FastAPI /api/v1/auth/*; token in memory

desktop/src/services/api/client.ts         — fetch client, token in module var
desktop/src/services/api/authApiService.ts — API-backed auth operations
desktop/src/services/authService.ts        — dispatches mock vs API
```

### Audit Integrity

`AuditEvent` has no `updated_at` column and no update or delete API routes. Server-side enforcement (Phase 3A) replaces the client-side `appendAuditEvent` placeholder from Stage 2E/2F.

## Phase 3C Backend Architecture

Phase 3C adds persistent commerce data — Orders, Products, Catalogues, and Inventory — without changing the Tauri/desktop application boundary.

### Commerce Models

```text
orders                 — order_number (ORD-YYYY-NNNNNN), status/payment/fulfillment with ClassVar
                         transition maps, Decimal pricing, optimistic version lock
order_items            — snapshot of product_name/sku/unit_price at time of order creation
order_number_counters  — per-org counter; SELECT ... FOR UPDATE prevents duplicate numbers
products               — sku (uppercase normalized), track_inventory, stock_quantity, availability
product_variants       — per-product variants with independent stock and pricing
inventory_movements    — append-only audit log for every stock change (delta, reason, before/after)
catalogues             — named product lists with status (active/archived), position-ordered membership
catalogue_products     — membership join: position, is_featured; unique (catalogue_id, product_id)
```

### Key Patterns

```text
Order number:   SELECT ... FOR UPDATE on order_number_counters; format ORD-{year}-{seq:06}
Snapshot:       OrderItem captures product_name, sku, unit_price at order time (not a FK-live price)
Decimal:        Numeric(19, 4) for all monetary columns; avoid float rounding in financial data
Transitions:    ClassVar dicts (STATUS_TRANSITIONS, PAYMENT_TRANSITIONS, FULFILLMENT_TRANSITIONS)
                tested directly on the class — SQLAlchemy metaclass prevents uninitialised ORM use
Identity map:   CatalogueService add_product uses SELECT products.id (not get_by_org_or_raise)
                to avoid loading Product into identity map before membership insert; expire_all()
                after commit before _reload() clears stale cached relationships
```

### Desktop Commerce Integration

```text
VITE_DATA_MODE=mock  — in-memory Zustand mock state (default)
VITE_DATA_MODE=api   — calls FastAPI commerce endpoints

[redacted].ts      — list, getById, create, update, setStatus,
                                                   setPaymentStatus, setFulfillmentStatus
[redacted].ts    — list, getById, create, update, adjustStock,
                                                   setAvailability, archive, restore, delete
[redacted].ts  — list, getById, create, update, archive,
                                                   restore, addProduct, removeProduct, reorderProducts

useAppStore.ts isApiMode branches cover all 20 commerce actions.
```

## Phase 3D Backend Architecture

Phase 3D adds persistent Tickets, Broadcast Templates, Customer Segments, Broadcasts, and Analytics
without changing the Tauri/desktop application boundary. Broadcast delivery remains simulated and
deterministic — no Meta WhatsApp Cloud API calls are made (that is Phase 4 scope).

### Communication & Analytics Models

```text
tickets                 — TKT-YYYY-NNNNNN generation via SELECT FOR UPDATE on ticket_number_counters;
                         status transitions ([redacted]);
                         version lock; linked to customer/conversation/order
ticket_notes            — internal agent notes (body, author_user_id)
ticket_timeline         — append-only event log per ticket (event, detail, actor)
broadcast_templates     — reusable message templates with category ([redacted]),
                         language, variables (JSONB), archive/restore, duplicate
customer_segments       — JSONB rules with SEGMENT_ALLOWED_FIELDS and SEGMENT_ALLOWED_OPERATORS
                         allowlists (server-enforced validation); estimated_count via rule evaluation
broadcasts              — campaign records with audience_type ([redacted]),
                         simulated deterministic delivery (94% delivered, 61% read of recipient_count)
broadcast_recipients    — per-recipient delivery status (sent/delivered/read/failed timestamps)
broadcast_timeline      — append-only campaign event log
```

### Key Patterns

```text
Ticket number:    SELECT ... FOR UPDATE on ticket_number_counters; format TKT-{year}-{seq:06}
Segment rules:    JSONB with server-side allowlist validation (SEGMENT_ALLOWED_FIELDS,
                  SEGMENT_ALLOWED_OPERATORS) — client cannot inject arbitrary fields/operators
Broadcast send:  Deterministic simulation only — recipient_count, sent_count, delivered_count,
                  read_count computed server-side; no Meta API calls (Phase 4 scope)
_reload() helper: Re-fetch via repo.get_by_org_or_raise() after commit with selectinload to
                  pre-populate timeline/notes relationships — avoids MissingGreenlet in SQLAlchemy 2 async
expire_on_commit=False on both real and test sessions means explicit expire_all() is NOT used
                  (it was harmful: expiring freshly-loaded attributes triggered sync lazy-loads)
```

### Desktop Stage 2D Integration

```text
VITE_DATA_MODE=mock  — in-memory Zustand mock state (default)
VITE_DATA_MODE=api   — calls FastAPI Stage 2D endpoints

[redacted].ts     — list, getById, create, update, assign,
                                                    setStatus, addNote
[redacted].ts   — list, getById, create, update, archive,
                                                    restore, duplicate
[redacted].ts   — list, getById, create, update, delete
[redacted].ts  — list, getById, create, update, startSend,
                                                    completeSend, schedule, pause, resume, cancel, delete
[redacted].ts — getOverview (maps OverviewMetrics → analytics state)

useAppStore.ts isApiMode branches cover all Stage 2D actions (tickets, templates, segments,
broadcasts, analytics) following the same early-return pattern established in Phase 3B/3C.
```

## Native API Authentication & Readiness Closure

Closes the Phase 3A desktop login-UI gap so the native desktop can establish a
real backend session through its own UI, plus the `/ready` endpoint and the
seed-script type-check gate. No Phase 4 functionality is added.

### Mode boundary

A single runtime selector is the only place the UI reads the auth mode.
Components never read `import.meta.env` directly.

```text
desktop/src/services/api/authMode.ts
  isApiAuthMode / isMockAuthMode

VITE_AUTH_MODE=mock (default) → phone OTP / QR mock login (no network)
VITE_AUTH_MODE=api            → email + password login against FastAPI
```

### API-mode login + session lifecycle

```text
LoginModal (email form)
  → store.loginWithEmail
  → authService.loginWithEmail (dispatches to authApiService)
  → POST /api/v1/auth/login   (access token in body, rt cookie set)
  → setAccessToken (module memory only)
  → GET /api/v1/auth/me       (user/org/role/permissions)
  → AuthSession{ method:"email", permissions:[...] }
  → route guard opens /app/*
```

Startup restore (API mode only, runs once):

```text
store.restoreSession (concurrent-guarded, no refresh loop)
  → authService.tryRestoreSession
  → POST /api/v1/auth/refresh (uses httpOnly rt cookie)
  → success → /auth/me → load app
  → failure → null/false → Login screen (never surfaced as authError)
```

Logout:

```text
store.logout
  → authService.logout → POST /api/v1/auth/logout (revoke session, clear cookie)
  → clearAccessToken (memory) — unconditional, even if backend call fails
  → clearSensitiveData() — purge all API-backed records from active UI state
  → session undefined → Login screen
```

Key files:

```text
desktop/src/services/api/authMode.ts       single mode boundary
desktop/src/services/api/authErrors.ts     classifyAuthError → 7 stable reason codes
desktop/src/services/api/authApiService.ts loginWithEmail, refresh, logout, getMe
desktop/src/services/api/client.ts         fetch client; access token in module memory; bounded timeout;
                                           BASE_URL normalized (strips trailing / and /api/v1) so the
                                           /api/v1 prefix — already on every service path — appears once
desktop/src/services/authService.ts        mock vs API dispatch; tryRestoreSession
desktop/src/components/LoginModal/...      email form (API) + phone/QR (mock)
desktop/src/state/useAppStore.ts           loginWithEmail, restoreSession, clearSensitiveData, logout
desktop/src/App.tsx                        one-shot startup restore + UI hold
```

### Token storage policy (enforced)

Access token: module memory only — never `localStorage`, `sessionStorage`,
`IndexedDB`, URL parameters, or logs. Refresh token: httpOnly `rt` cookie only;
only the sha256 digest is persisted server-side. See `docs/SECURITY.md`.

### Error mapping

`classifyAuthError` maps `ApiError` (and network `TypeError`) onto stable codes
(`invalid_credentials`, `authentication_required`, `session_expired`,
`account_inactive`, `organization_access_denied`, `backend_unavailable`,
`request_timeout`, `unknown_auth_error`). `authErrorMessage` resolves each to a
concrete sentence with no trace, status code, or credential detail. The store's
`runLogin` catch uses this mapping for both the form and the restore path.

### Liveness vs readiness (backend)

```text
GET /health → process liveness (always 200 when responding)
GET /ready  → readiness; bounded SELECT 1 (2s); 200 {ready,ok} / 503 {not_ready,unavailable}
```

`database_reachable` is a module-scope dependency override target so the 503
path is tested without a live database. No credentials or traces are exposed.

### Seed-script type-check gate

`mypy app scripts` now type-checks `scripts/seed.py` alongside the application.
The seed uses concrete `uuid.UUID` types, `TypedDict` collections for order
specs, and narrowly documented `Decimal` casts (SQLAlchemy `Numeric` is
statically `object`). The prior runtime seed failure (model constructor field
mismatch) is now caught at type-check time without weakening mypy globally.

## Phase F — WhatsApp Flows (F1-F2)

Flows ride the same `/{phone_number_id}/messages` endpoint and the same
`interactive` message type WhatsApp already uses for button/list replies, so
this feature is additive to the existing provider architecture rather than a
parallel system (unlike the forthcoming Instagram DM channel, which is a real
architectural extension — see the SaaS Commercialization research log entry
for that comparison).

### Outbound send (F1)

`FlowSendComponents` (flow id, flow token, header/body/footer text, CTA text,
`flow_action` of `"navigate"` or `"data_exchange"`, initial screen, action
payload) is a new dataclass in `app/providers/base.py`, alongside a new
`send_flow(...)` method on the `MessagingProvider` Protocol. All three
concrete providers implement it:

```text
MetaGraphApiClient.send_flow_message()     builds interactive.flow payload on
                                            the existing _messages_url()
MetaWhatsAppCloudProvider.send_flow()      same 200/error-extraction pattern
                                            as send_template/send_media
MockWhatsAppProvider.send_flow()           wamid.mock.flow.{uuid} deterministic
```

`runtime_fake.py` gained a `flow_send_success` scenario for native desktop
acceptance testing; the fake `/messages` handler already accepted any payload
shape, so this only needed a distinct `wamid.fake.vendale.flow.*` id prefix
for evidence readability, not new branching logic.

### Authoring/publishing (F2)

`WhatsAppFlowDefinition` (`app/models/whatsapp_flow.py`) mirrors
`WhatsAppProviderTemplate`'s shape: an org-scoped local record wrapping a
synced/published external Meta object, with a `status` lifecycle
(`draft` → `published` → `deprecated`) enforced via a `STATUS_TRANSITIONS`
ClassVar dict, matching the pattern already used by `Broadcast`/`Ticket`.

`WhatsAppFlowService` (`app/services/whatsapp_flow_service.py`):

```text
create_draft()   local-only; no Meta call
update_draft()   draft-only (422 once published); bumps version
publish()        create_flow (if no provider_flow_id yet) → update_flow_json
                 → publish_flow; requires connection.status in
                 {"verified", "active"}, matching the existing template-sync gate
deprecate()      calls Meta's deprecate endpoint only when a provider_flow_id
                 exists (a never-published draft has nothing to deprecate remotely)
```

New Meta Flow management client methods on `MetaGraphApiClient` /
`MetaWhatsAppCloudProvider`, using the same DI-transport pattern as every
other provider client method in this codebase:

```text
create_flow(waba_id, name, categories)      POST {waba_id}/flows
update_flow_json(flow_id, flow_json)        POST {flow_id}/assets (multipart FLOW_JSON)
get_flow / get_flow_status(flow_id)         GET  {flow_id}
publish_flow(flow_id)                       POST {flow_id}/publish
deprecate_flow(flow_id)                     POST {flow_id}/deprecate
```

See `docs/RESEARCH_LOG.md` (2026-07-31) for the official endpoint research.
No API routes exist yet for this service — it is service-layer only in this
slice; routes are deferred to F5 alongside the desktop UI that will call them.
No live Meta Flow credentials or Flow-enabled WABA have been used; everything
is verified against `FakeHttpTransport` (unit/integration tests) and
`RuntimeFakeHttpTransport` (native desktop acceptance) only.

### Encrypted data-exchange endpoint (F3)

`app/security/flow_encryption.py` is a direct protocol port (not a code
merge) of Meta's official `WhatsApp/WhatsApp-Flows-Tools` Node.js reference
(`[redacted].js`, MIT licensed):

```text
decrypt_flow_request()   RSA-OAEP(SHA-256) unwraps the client's AES-128 key,
                          then AES-128-GCM decrypts the request body. Any
                          failure (wrong key, tampered ciphertext, wrong
                          keypair) raises FlowEndpointError(421, ...) — Meta's
                          documented signal telling the client to refresh its
                          cached public key.
encrypt_flow_response()  Re-encrypts the response with the SAME AES key but a
                          bitwise-flipped IV (Meta's documented scheme, not
                          an accidental quirk) and returns raw base64 TEXT —
                          not a JSON envelope; this is the entire HTTP body.
generate_flow_keypair()  RSA-2048, SPKI public key / PKCS8-encrypted private
                          key (PKCS8 substituted for the reference's PKCS1
                          since it's what `cryptography`'s
                          BestAvailableEncryption targets — only the public
                          key format and RSA-OAEP/AES-128-GCM wire scheme are
                          Meta-client-facing and must match exactly).
```

`WhatsAppFlowKeyPair` (`app/models/whatsapp_flow.py`) stores one RSA keypair
per connection. The private key PEM and its passphrase are AES-256-GCM
encrypted via the existing `credential_encryption.py` helper, reusing
`WHATSAPP_CREDENTIAL_ENCRYPTION_KEY` (same secret class as the connection's
access token) with distinct `field_name` bindings
(`"flow_private_key"` / `"flow_private_key_passphrase"`) — not a new master
key, since only genuinely distinct product surfaces (WhatsApp vs. the AI
gateway) get their own encryption key in this codebase.
`WhatsAppFlowKeyPairService.generate_for_connection()` rotates in place
(same row, new keys) rather than creating a new row per rotation.

`app/api/whatsapp_flow_endpoint.py` is a new, separate router (not folded
into `webhooks.py`) because its response contract is fundamentally
different: success returns raw base64 text, not JSON, and its error status
codes (421/427/432) are Meta's own Flow-specific codes rather than this
codebase's usual 400/403/404 conventions. Route:
`POST /webhooks/meta/whatsapp/flows/{connection_id}/data-exchange` — public,
outside the JWT chain, identified by connection UUID in the path because
Meta's Flow client sends no bearer token and no organization context, only
the encrypted payload (which cannot be parsed until the connection's private
key is already known). Signature verification reuses
`verify_webhook_signature` unmodified — the same `X-Hub-Signature-256`
HMAC-SHA256 scheme Meta uses for both the WhatsApp webhook and this endpoint.

`[redacted].py::handle_flow_data_exchange`
ports `flow.js`'s `getNextScreen()` action dispatch (`ping` health check,
client error acknowledgement, `INIT`, `data_exchange`) but deliberately does
not implement per-Flow custom screen logic — `WhatsAppFlowDefinition.flow_json`
can define arbitrary org-authored screens, and mapping "handle this specific
screen" to real business logic is deferred to F4/F5 once inbound `nfm_reply`
correlation and the desktop authoring UI exist to define what that even means
for a given org's Flow. An unhandled action raises `UnhandledFlowActionError`
(mapped to a plain HTTP 500) — distinct from `FlowEndpointError`, since an
unimplemented screen is this codebase's own gap, not a signal about the
client's encryption key or flow_token.

No live Meta Flow-enabled WABA or real Flow client was used to verify this;
the full round trip was verified against a simulated client built from the
same cryptographic primitives (`tests/unit/test_flow_encryption.py`,
`[redacted].py`), including negative cases.

### Inbound Flow response + desktop send/render (F4-F5)

`webhook_processing_service.py`'s inbound-message parser handles
`interactive.type == "nfm_reply"` (Meta's WhatsApp Flow completion webhook
shape) as its own branch, separate from `button_reply`/`list_reply`, because
its payload is structurally different: `nfm_reply.response_json` is a JSON
*string* (Flow-defined, arbitrary shape) rather than a flat `title`/`id`
object. The string is parsed defensively — a malformed `response_json` still
persists the message with an empty payload rather than dropping the webhook.
No new correlation logic was needed: the existing generic
`_resolve_interactive_correlation()` already keys off
`message_type == "interactive"` and `context_provider_message_id` regardless
of interactive subtype.

`MessageService.send_flow_message()` mirrors `send_media_message()`'s exact
shape (require `inbox.reply`, require an active WhatsApp Cloud connection in
provider mode, create the outbound `Message` row, then attempt the real
provider send via `_attempt_provider_flow_send()`) and additionally requires
the target `WhatsAppFlowDefinition` to already be `status="published"` with a
`provider_flow_id` — a Flow cannot be sent until it exists on Meta's side.
Exposed as `POST /conversations/{id}/messages/flow`.

`ConversationRead.whatsapp_connection_id` was added (purely additive) so the
desktop can resolve which connection's Flows are sendable for the currently
open conversation, without which the composer's Flow picker would have no
way to filter to the right connection's published Flows.

Desktop: `InboxPage.tsx`'s composer gained a "Send a Flow…" `<select>`
alongside Attach/Suggest reply, populated from
`whatsappApiService.listFlows(connectionId)` filtered to `status ===
"published"`, and gated by the exact same `LIVE_SEND_ENABLED` compile-time
constant that already gates every other outbound send — launching a Flow is
still an outbound WhatsApp send, not a separate bypass path.
`FlowResponseSummary` renders a completed Flow's `response_json` fields
generically (key/value pairs, `flow_token` excluded), since Vendale cannot
know an org's authored screen field names ahead of time; it reuses the
existing interactive-message bubble rather than introducing a new
message-type renderer. Settings' `WhatsAppConnectionCard` gained a "Flows"
section (generate/rotate keypair, create draft, publish, deprecate) visible
only for `status === "active"` connections, following the same inline-styled
component shape as the adjacent template-sync section.

## Phase I — Instagram DM Channel (I1-I2)

Instagram is a real architectural extension, not a small addition: every
hardcoded `"whatsapp_cloud"` string, every `phone_number_id`/`to_phone`/
`waba_id` parameter baked into the `MessagingProvider` Protocol, and the
hard FK from `Conversation.whatsapp_connection_id`/`Message.provider_connection_id`
to `whatsapp_connections.id` block a second channel without changes. Per the
locked architecture decision, Instagram gets a **parallel** connection model
and **separate** Protocol — not a generalized multi-channel table/Protocol —
so the live WhatsApp integration paying customers depend on is touched by
zero lines of this work.

### Connection model and provider

`InstagramConnection` (`app/models/instagram.py`) is a sibling to
`WhatsAppConnection`, with its own `INSTAGRAM_CONNECTION_STATUS_TRANSITIONS`
dict (not imported from `whatsapp.py`, so the two lifecycles can diverge
independently later) and Instagram-shaped fields: `ig_user_id` (analogous to
`phone_number_id` — identifies "us") and `ig_username` in place of
`waba_id`/`phone_number_id`/`phone_number`.

```text
InstagramGraphApiClient        reuses the existing HttpTransport Protocol
(instagram_client.py)          and FakeHttpTransport test double from
                                meta_client.py unmodified — only URL builders
                                and payload shapes differ:
                                POST https://graph.instagram.com/{v}/{IG_ID}/messages
                                {"recipient": {"id": IGSID}, "message": {"text": ...}}

InstagramMessagingProviderProtocol   a genuinely separate Protocol (not
(instagram_provider.py)              MessagingProvider) shaped around
                                      Instagram's actual parameters — IGSID
                                      recipient, no phone number, no WABA, no
                                      template system (IG DM has no
                                      marketing-template exception to the
                                      24-hour customer-service window)

MockInstagramProvider          deterministic ig_mid.mock.* fake, mirroring
(instagram_mock.py)            MockWhatsAppProvider's contract exactly

get_instagram_provider()       a NEW separate factory function
(instagram_factory.py)         (not an overload of the existing
                                get_provider(), which returns a different
                                Protocol type)

instagram_runtime_fake.py      development-only deterministic transport for
                                native desktop acceptance testing, mirroring
                                runtime_fake.py's pattern (send_success/
                                [redacted]
                                scenarios)
```

`InstagramConnectionService` (`[redacted].py`)
mirrors `WhatsAppConnectionService`'s full shape (CRUD, `set_access_token`,
`get_plaintext_token`, `set_status`, `verify`) with one deliberate deviation:
`activate()` replaces WhatsApp's `subscribe()` step, because Instagram Login
app webhook subscription is configured once at the app level in the Meta
developer console — there is no per-connection `POST .../subscribed_apps`
call to make for Instagram, unlike WhatsApp's per-WABA subscription.

Token encryption reuses `WHATSAPP_CREDENTIAL_ENCRYPTION_KEY` — the same
secret class as the WhatsApp connection token — with a distinct
`field_name="ig_access_token"` AAD binding via the existing
`credential_encryption.py` helper, rather than a new master key. This is
consistent with the project's established rule that only genuinely distinct
product surfaces (e.g. the Axon AI gateway, which has its own
`AI_CREDENTIAL_ENCRYPTION_KEY`) get their own encryption key; Instagram and
WhatsApp connection tokens are the same kind of secret (a per-connection
Meta bearer token), just for different Meta products.

Config gates are deliberately **separate** from WhatsApp's:
`META_INSTAGRAM_LIVE_MODE` / `META_INSTAGRAM_ALLOW_LIVE_SEND` /
`INSTAGRAM_PROVIDER_TRANSPORT` / `INSTAGRAM_GRAPH_API_VERSION`, with the same
production fail-closed rejection of `INSTAGRAM_PROVIDER_TRANSPORT=fake` that
WhatsApp's transport setting already has. Instagram requires its own Meta
App Review approval and go-live timeline, independent of WhatsApp's
already-approved status — sharing a gate would incorrectly couple the two
channels' live-activation readiness.

`instagram_connection.view`/`.manage` permission keys were added to the
catalogue and flow through owner/admin/manager exactly like
`whatsapp_connection.*` (no special-casing needed in `MANAGER_KEYS`'s
exclusion-prefix logic, since it excludes by prefix and Instagram's keys
don't match any excluded prefix).

REST routes (`GET/POST /instagram/connections`,
`GET/PATCH/DELETE /instagram/connections/{id}`, `.../token`, `.../status`,
`.../verify`, `.../activate`) mirror WhatsApp's connection routes 1:1 except
for the `/subscribe` → `/activate` naming change described above.

No live Instagram Business account or live credentials have been used for
any of this; everything is verified against `FakeHttpTransport` (unit/
integration tests) or the `MockInstagramProvider`/`instagram_mock` channel
only. See `docs/RESEARCH_LOG.md` (2026-07-31) for the official Instagram
Messaging API research this was built against.

### Parallel columns, webhook dispatch, and send path (I3-I5)

Three parallel nullable FK columns were added, each alongside an existing
WhatsApp-shaped column that is left completely untouched:
`Conversation.instagram_connection_id` (next to `whatsapp_connection_id`),
`Message.instagram_connection_id` (next to `provider_connection_id`), and
`MessageProviderAttempt.instagram_connection_id` (next to `connection_id`,
which is FK'd to `whatsapp_connections.id` only — a third parallel column
the plan's I3 bullet list didn't explicitly itemize but which the same
"parallel FK, not shared FK" principle required once I5's send-attempt audit
log needed somewhere to point). `CONVERSATION_CHANNEL` gained `"instagram_dm"`.
None of these migrations added an index — `whatsapp_connection_id` and
`provider_connection_id` aren't indexed either (a precedent from the
`c2e7f3b5a1d9` migration), and the first draft of these migrations was
caught by `alembic check` for adding indexes SQLAlchemy's own model
definitions didn't produce.

`WebhookProcessingService.handle_delivery()` dispatches on
`payload.get("object") == "instagram"` at the very top, before any of the
existing WhatsApp-specific envelope parsing runs. Instagram's envelope is
structurally different — `entry[].messaging[]` arrays of
`{sender, recipient, timestamp, message}`, not WhatsApp's
`entry[].changes[].value.{messages[],metadata}`, and there is no
`phone_number_id` analogue inside the envelope (the receiving IG account's
ID is `entry[].id` itself) — so sibling methods
(`_process_instagram_entries()`, `_handle_instagram_inbound_message()`,
`_resolve_instagram_connection()`, `_resolve_or_create_instagram_customer()`,
`[redacted]()`) exist rather than reshaping
Instagram's payload to fit the WhatsApp-shaped parser. One genuine wire-format
difference required its own helper: Instagram webhook timestamps are Unix
epoch **milliseconds**, not WhatsApp's epoch seconds, so `_parse_unix_ts_ms()`
exists rather than reusing `_parse_unix_ts()`.

Everything else about inbound processing is reused with zero changes:
`_resolve_or_create_customer()`'s `CustomerIdentity` pattern works
unmodified with `identity_type="instagram_dm"` (the unique constraint on
`(organization_id, identity_type, normalized_value)` already supports a
second identity type), and the same
`emit_safely(..., trigger_type="conversation_message_received")` call fires
from the Instagram path exactly as it already does from the WhatsApp path —
no automation-engine changes were needed for a second channel.

New `GET/POST /webhooks/meta/instagram` routes
(`app/api/webhooks.py::meta_instagram_verify`/`meta_instagram_event`) reuse
`verify_webhook_signature` and the `hub.challenge` handshake with **zero
modification** — Meta's HMAC scheme and one-verify-token-per-app model are
identical for both products, so `META_WEBHOOK_VERIFY_TOKEN` and
`META_APP_SECRET` are shared, not duplicated.

`MessageService.send_message()` gained `is_instagram_dm`, a sibling
condition to the existing `is_whatsapp_cloud` (not a rename or in-place
modification), gated by the new `settings.instagram_provider_send_enabled`.
A new sibling method `_attempt_instagram_provider_send()` mirrors
`_attempt_provider_send()`'s exact shape but resolves the recipient's
Instagram-scoped ID (IGSID) via the customer's `instagram_dm`
`CustomerIdentity` row (Instagram has no phone number to look up) rather
than `Customer.normalized_phone`.

A dedicated test
(`[redacted]` in
`[redacted].py`) sends one WhatsApp
webhook and one Instagram webhook through the same `WebhookProcessingService`
instance in the same test and asserts both produce fully correct, mutually
exclusive `Message`/`Conversation` rows — the concrete verification of the
plan's stated goal that the parallel design produces zero WhatsApp
regressions, not just an assumption.

## Phase I closure — AI draft verification and desktop UI (I6-I7)

`AiService._conversation_context()`/`generate_draft()` and the
`generate_ai_draft` automation action were verified, not rebuilt: both read
`Message`/`Conversation` rows keyed purely by `conversation_id`/
`organization_id`/`direction`/`body` (or, for the automation action,
`event.entity_type`/`event.entity_id`), with zero reference to `channel`,
`provider`, or any WhatsApp-specific field. Three new tests in
`[redacted].py` prove this
concretely against a real `instagram_dm` conversation — direct
`generate_draft()` call, the automation action triggered end-to-end through
a real `AutomationRule` + `emit()` + `AutomationRuntime.run_once()`, and the
internal-note-exclusion safety property — rather than resting on the code
read alone. Zero changes were made to `ai_service.py` or
`automation_runtime.py`.

Desktop: `instagramApiService.ts` mirrors `whatsappApiService.ts`'s shape,
with `verify`+`activate` replacing WhatsApp's `verify`+`subscribe` (no
per-connection subscribe call exists for Instagram — see the I1-I2
connection-service note above). Settings gained an `InstagramSettings`
component alongside the existing `WhatsAppSettings`, deliberately smaller
since Instagram has no templates/Flows sub-features. `Conversation.channel`
was newly exposed to the desktop domain type (it had not been surfaced at
all before this) so Inbox rows could render a `ChannelBadge` — a pure
rendering addition; the inbox itself needed no structural change since
`Conversation.channel` already carried this information end-to-end. No
Instagram icon exists in the installed `lucide-react` version (removed for
trademark reasons); `Camera` is used instead for both the Settings section
and the Inbox badge.

This closes Phase I (Instagram DM Channel) entirely: 725 backend tests, 549
desktop tests, ruff/mypy/tsc all clean, production build succeeds. No live
Instagram Business account or live credentials were used anywhere in this
phase.

## Platform Admin V3.3c-1: Auth Security Telemetry

Adds a second, deliberately separate append-only ledger,
`AuthSecurityEvent`, alongside the existing `PlatformAuditEvent` (platform
administrative actions) and `AuditEvent` (tenant business events). The three
ledgers answer different questions and are queried independently:
`AuditEvent` for "what changed in this workspace," `PlatformAuditEvent` for
"what did a platform administrator do," and `AuthSecurityEvent` for "who
authenticated, and when." Security Center (Platform Admin) combines the
latter two for its overview; nothing merges the three tables.

Event vocabulary is fixed and enforced at the service layer
(`AUTH_SECURITY_EVENT_TYPES` in `app/models/auth_security_event.py`):
`auth.login.succeeded`, `auth.login.failed`, `auth.session.created`,
`auth.session.revoked`. `auth.mfa.*` and `auth.account.locked` are reserved
for V3.3c-2, when real MFA and account-lockout behavior exist to emit them.

Recording is fail-open by design: `AuthSecurityEventService.record()` wraps
its insert in `self.db.begin_nested()` (a SAVEPOINT), the exact pattern
already established in `automation_runtime.py` for per-rule action
isolation, and swallows any exception. A telemetry write failure must never
break login, logout, or token refresh.

`RefreshToken` gained `ip_address`, `user_agent`, `last_seen_at`, and
`revoked_at`. A token rotation (every `/auth/refresh` call) carries
`ip_address`/`user_agent` forward from the row it replaces rather than
re-capturing them, because rotation continues the same logical session; it
does not emit a new `auth.session.created` event for the same reason. Only
real login, registration, and (once built) MFA re-authentication start a
new session.

`AuthSecurityEvent.target_is_admin` is stamped once at write time from the
resolved or attempted account's `is_superuser` flag. This lets Security
Center's admin-scoped rollups avoid a join back to `users` (whose
`is_superuser` value may have changed since), and keeps the scope decision
explicit: the ledger itself records every user's login/session activity
system-wide, but Security Center's overview deliberately surfaces only the
administrator-scoped subset, matching that console's existing scope (see
`docs/ROADMAP.md`'s V3.3c-1 entry for the full reasoning).

The "repeated failed authentication" rule is a fixed, named threshold over
real counted rows (`AUTH_REPEATED_FAILURE_THRESHOLD` /
`AUTH_REPEATED_FAILURE_WINDOW_MINUTES` in `app/core/config.py`), grouped by
`attempted_email` within a rolling window. It is presented in both the API
response and the desktop UI as a threshold being exceeded, never as anomaly
detection or a risk score.

Retention: `retention_expires_at` is computed at insert time
(`AUTH_TELEMETRY_RETENTION_DAYS`, default 90). `AuthTelemetryRetentionService`
and `[redacted].py` mirror
`media_retention_service.py`'s lease/dry-run shape exactly, simplified
because deleting an expired auth-telemetry row needs no storage-backend call
and no audit trail of its own. The worker is not yet wired to a systemd unit
or cron schedule; that activation step is a deliberate, separate follow-up.

`User.mfa_enabled` is a placeholder column only: it defaults `False` for
everyone and nothing sets it yet. It exists now so Security Center can
honestly report "0 administrators have MFA enabled" today without a future
migration being needed once real enroll/verify behavior ships in V3.3c-2.
The secret, enrollment timestamp, and recovery-code storage are deliberately
deferred to that same migration rather than added inert here.

## Platform Admin V3.3c-2: MFA + Privilege Protections

Builds real TOTP MFA, mandatory enrollment for privileged platform roles,
step-up re-authentication for sensitive mutations, and session management on
top of the V3.3c-1 telemetry foundation. `User.mfa_enabled` stops being a
placeholder: `MfaService` now actually sets it.

### Secrets abstraction (introduced ahead of need, not full KMS)

`app/security/secrets_provider.py` defines a `SecretsProvider` Protocol
(`encrypt(subject_id, field_name, plaintext) -> str`,
`decrypt(subject_id, field_name, ciphertext) -> str`) with one concrete
implementation, `EnvKeySecretsProvider` (AES-256-GCM, wire format
`{key_version}:{nonce_b64}:{ciphertext_b64}`, AAD binds `subject_id` and
`field_name` so a ciphertext cannot be replayed against a different user or
field). This is a deliberate seam, not full KMS integration: today
`get_mfa_secrets_provider()` returns the env-key implementation, but every
call site depends only on the Protocol, so a future KMS-backed provider
(AWS KMS, GCP KMS, HashiCorp Vault) can be swapped in behind the same
factory function without touching `MfaService`, the model, or any test that
doesn't specifically test `EnvKeySecretsProvider` itself. This mirrors, but
is intentionally separate from, `credential_encryption.py` (shaped around
org/connection-scoped provider credentials, keyed by connection id) since
MFA secrets are per-user, not per-tenant-connection.

`MFA_CREDENTIAL_ENCRYPTION_KEY` is its own key, unconditionally required at
production startup and asserted to differ from
`WHATSAPP_CREDENTIAL_ENCRYPTION_KEY` and `AI_CREDENTIAL_ENCRYPTION_KEY` (the
same fail-fast pattern already established for the AI gateway key) --
distinct product surfaces get distinct key material.

### TOTP primitives (no third-party dependency)

`app/security/totp.py` implements RFC 6238 TOTP directly on top of Python's
standard-library `hmac`/`hashlib` (RFC 4226 HOTP dynamic truncation, 6
digits, 30-second period, base32 secret), verified against the official RFC
6238 Appendix B test vectors rather than trusting an unverified third-party
package. `verify_totp()` allows a `+/-1` step tolerance (90 seconds total)
for clock skew and compares in constant time. `provisioning_uri()` builds
the standard `otpauth://totp/...?secret=...&issuer=Thread%20CRM` URI that
every authenticator app (Google Authenticator, Authy, 1Password) already
knows how to parse from a QR code or manual entry.

### Login-time state machine (no elevated state in the JWT)

`AuthService.login()` now returns a `LoginResult` with three possible
outcomes instead of always minting tokens:

```text
1. user.mfa_enabled=True           -> issue MfaChallengeSession(purpose="challenge")
2. user_requires_mandatory_mfa()   -> issue MfaChallengeSession(purpose="enroll")
3. neither                         -> issue real tokens immediately (unchanged)
```

`MfaChallengeSession` and `StepUpSession` (`app/models/mfa.py`) are DB-backed,
short-lived, opaque-token-hash-only sessions -- the same pattern `RefreshToken`
already established (only `sha256(raw_token)` is persisted; the raw token is
handed to the caller once and never stored). This is a deliberate,
non-negotiable design choice: nothing about "this login is mid-MFA-challenge"
or "this caller recently re-verified their password" is ever encoded in the
15-minute access JWT. A compromised or leaked JWT before either gate resolves
grants nothing; revoking access requires only deleting the DB row, not
waiting for a token to expire.

`resolve_mfa_challenge_session()` / `complete_mfa_login()` on `AuthService`
let `POST /auth/mfa/login-enroll` and `POST /auth/mfa/login-verify`
(`app/api/v1/endpoints/mfa.py`) authenticate purely from the opaque
`mfa_session_token` in the request body -- there is no Bearer token to check
because none has been issued yet. `MFA_MANDATORY_ROLE_TYPES` (in
`app/services/mfa_service.py`) currently covers `platform_owner`,
`platform_admin`, `security_admin`; `user_requires_mandatory_mfa()` resolves
a user's *platform* role (not their tenant role) since mandatory MFA in this
slice is a platform-console protection, not a general tenant policy.
Mandatory enrollment blocks reaching a session at all rather than silently
downgrading permissions -- there is no partial-access state.

### Step-up re-authentication

`StepUpService.create()` re-verifies the caller's own password and a live
TOTP/recovery code, then issues a `StepUpSession` (default 10-minute TTL,
`STEP_UP_SESSION_TTL_MINUTES`). Deliberately **not** single-use: within its
TTL window a caller can complete more than one sensitive action without
re-entering their password each time, treating the elevation as a bounded
time budget for a short admin workflow rather than a one-shot token. Gated
mutations (`PATCH .../platform-role`, `POST .../mfa/reset`,
`POST .../administrators/revoke-all-sessions`) call
`StepUpService.require_valid()`, which checks the token hash, expiry, and
that it belongs to the calling `actor_user_id` -- a step-up token minted for
one administrator cannot be handed to or reused by another.

### Privilege-change hardening

`PlatformAdminService.assign_platform_role()` enforces this exact order:
permission check (`platform.roles.manage`) -> last-owner-still-remains check
-> step-up validity -> mandatory reason -> mutate -> audit with explicit
before/after `platform_role_type` -> emit an `AuthSecurityEvent` where
applicable. A **downgrade** (rank decrease) additionally revokes every
existing session for the target user immediately after the role change
commits; a promotion does not force a re-login, since forcing out a user
mid-promotion has no security benefit and only interrupts their work.

`_role_privilege_rank()` (`app/services/platform_admin_service.py`) is an
explicit ordinal (`platform_owner`=4 down to `read_only_auditor`=1, no
platform role=0), deliberately **not** derived from comparing raw
permission-key sets -- two disjoint specialist roles (e.g. `billing_admin`
vs `support_admin`) aren't orderable by set comparison, but the product
still needs a yes/no answer to "did this change increase or decrease
privilege" for the session-revocation decision. `role_type=None` is rank 0
only when the target's `is_superuser` is `False`; a legacy full-access
admin (`is_superuser=True`, no assigned platform role) ranks as 4
(`platform_owner`-equivalent), computed from the user's pre-mutation
`is_superuser` value -- migrating a legacy admin to any assigned role is
correctly treated as a downgrade, not a promotion from an artificially low
baseline.

### MFA administration (security_admin surface)

`PlatformAdminService.admin_reset_mfa()` (gated by `platform.security.manage`
via `RequirePlatformSecurityManage`, plus step-up) destroys the target's
encrypted secret and every recovery code, forcing fresh enrollment on their
next mandatory-MFA login. The caller -- even a `security_admin` -- never
sees the target's secret, QR code, or recovery codes at any point; the
response confirms only that a reset occurred. There is no "view MFA status
in detail" beyond the boolean/timestamp fields already on
`PlatformAdministratorRead`.

### Session management

`POST /platform-admin/users/{id}/revoke-sessions` (existing, reason-only, no
step-up) and the new step-up-gated
`POST [redacted]` are deliberately
separate actions at different blast radii: the former targets one user, the
latter -- explicitly documented as including the caller's own session --
revokes every active session for every platform administrator system-wide,
reusing `RefreshTokenRepository.revoke_all_for_user()` in a loop rather than
inventing a bulk-SQL variant, since the administrator population is small
and this is an infrequent panic-button action, not a hot path.
`GET /users/{id}/sessions` / `DELETE /users/{id}/sessions/{session_id}`
(gated by `platform.security.read` / `platform.sessions.revoke`
respectively) expose per-session `ip_address`/`user_agent`/`last_seen_at` --
the same columns V3.3c-1 added to `RefreshToken` -- with no `is_current`
concept, since viewing another administrator's sessions has no "current
session" frame of reference.

### Security Center extensions

`PlatformSecurityOverviewRead` gained `mfa_protected_administrators`,
`administrators_without_mandatory_mfa`, `privilege_changes_24h`.
`PlatformAdministratorRead` gained `mfa_enabled`, `mfa_enrolled_at`,
`mfa_last_verified_at`, `active_session_count`. New attention-panel items
follow the same deterministic-threshold-only convention V3.3c-1 established
(never framed as anomaly detection): CRITICAL when a `platform_owner`
lacks MFA, WARNING at 5+ failed attempts against an administrator within 15
minutes (reusing V3.3c-1's existing rule, now administrator-scoped) and for
each remaining legacy unrestricted administrator, INFO when an
administrator holds an unusually high number of concurrent active sessions.

### Desktop wiring

`authApiService.loginWithEmail()` returns a `LoginOutcome` union
(`{kind:"authenticated", session}` or
`{kind:"mfa_required", mfaSessionToken, enrollmentRequired}`) instead of
always resolving a session; `useAppStore`'s `loginWithEmail` action stores a
`mfaChallenge` slice on the latter outcome rather than treating it as a
login error, and `LoginModal` renders `MfaEnrollmentScreen` (QR via the new
`qrcode` dependency, MIT-licensed, plus a manual setup key -- both are
always shown together since not every authenticator flow benefits equally
from a scanned QR) or `MfaChallengeScreen` in place of the credentials form
while a challenge is pending. Settings gained a self-service
`MfaSelfServicePanel` (enable/disable/regenerate recovery codes, API mode
only). Platform Admin's role-assignment and MFA-reset/session-revocation
actions each move through a local select-reason -> `StepUpDialog` (password
+ code) -> mutate sequence before calling their respective endpoints, and
`SecuritySection` gained an admin-scoped `SessionsDialog` (view + revoke one
or all sessions for a single administrator) alongside the new MFA-status
table column and KPI tiles.

### Deployment backlog (carried forward from V3.3c-1, still open)

The auth-telemetry retention worker
(`[redacted].py`, introduced in V3.3c-1) is
still not wired to any systemd unit, cron schedule, or other runtime
supervisor. Nothing currently invokes it outside of tests. Without an
operator wiring it up (mirroring the existing
`vendale-broadcast-worker.service` / `vendale-media-worker.service`
systemd unit pattern under `infrastructure/systemd/`), `auth_security_events`
and now also `mfa_challenge_sessions` / `step_up_sessions` (both already
expiry-bounded and cheap to prune, but currently never pruned) will grow
indefinitely. This is tracked explicitly rather than silently left implicit.

**V6.2 update:** the unit now exists
(`[redacted].service`)
and the worker's per-pass body also prunes expired
`mfa_challenge_sessions`/`step_up_sessions` via the new
`ShortLivedSessionRetentionService`. The unit file is committed; activating
it on the deployment host (`cp` + `systemctl enable --now`) remains an
operator action by design -- the code side of the backlog item is closed.

## Platform Admin V3.4: Feature Flags

Closes V3 (per the user's own stated completion criteria). A real,
server-authoritative feature flag system, deliberately narrow: no A/B
experiments, no conversion tracking or statistical significance, no
arbitrary JSON remote configuration, no plan/pricing entitlements. Two
tables: `FeatureFlag` (key, name, description, enabled, rollout_percentage,
classification, status) and `FeatureFlagTarget` (feature_flag_id,
target_type, target_id, enabled), with target types `workspace`, `user`,
`platform_role`. Plan targeting was explicitly deferred until subscription
pricing has a clean canonical plan identity.

### Resolution order (permanent, must never be silently reordered)

`FeatureFlagService._resolve()` evaluates in this fixed order and stops at
the first matching rule:

```text
1. Explicit user override           (FeatureFlagTarget target_type="user")
2. Explicit workspace override      (FeatureFlagTarget target_type="workspace")
3. Platform-role override           (FeatureFlagTarget target_type="platform_role")
4. Percentage rollout                (deterministic hash bucket < rollout_percentage)
5. Global enabled/default state      (FeatureFlag.enabled)
```

A flag key that has never been created evaluates to `enabled=True`
(`reason="not_configured"`) -- the same fail-open convention billing
enforcement established, so an unconfigured flag never silently disables a
capability nobody has opted to gate yet. An `archived` flag always
evaluates to `enabled=False` (`reason="archived"`) regardless of its other
fields: archiving is a deliberate retirement, not a no-op fallthrough.

### Deterministic rollout, not randomness

```python
def _rollout_bucket(flag_key: str, subject_id: str) -> int:
    digest = hashlib.sha256(f"{flag_key}:{subject_id}".encode()).hexdigest()
    return int(digest[:8], 16) % 100
```

Python's built-in `hash()` is deliberately not used: it is randomized per
process via `PYTHONHASHSEED` for security reasons, so the same
workspace/user could land in a different bucket on every request or after
every restart, defeating the entire point of a gradual rollout. `sha256` is
stable across processes and time. The subject is the organization id
(workspace) when one is present, falling back to the user id only when no
organization context exists -- Vendale is B2B multi-tenant, so gradual
rollout is primarily a per-workspace decision, not per-user.

### Caching and invalidation

A small module-level (process-wide, not per-request-instance) cache holds
every flag + its targets for 45 seconds (`_CACHE_TTL_SECONDS`), refreshed
via `.execution_options(populate_existing=True)` so a freshly-queried
`FeatureFlag`'s `targets` relationship cannot serve a stale collection from
SQLAlchemy's identity map after a target was just added or removed in the
same process. `invalidate_feature_flag_cache()` is called after every
mutation (create/update/target add/target remove), so a Platform Admin
change is visible to the next evaluation almost immediately rather than
waiting out the full TTL. If Vendale later runs many API replicas behind
a load balancer, this in-process cache will need to move to a shared
invalidation channel (Redis pub/sub or similar) so one replica's mutation
invalidates every other replica's cache too -- not required at today's
single-process scale, and deliberately not built ahead of that need.

### Kill-switch classification

`FeatureFlag.classification` is `"normal"` (ordinary product rollout) or
`"operational"` (emergency kill-switch, e.g. `outbound_broadcasts`,
`ai_tool_execution`, `new_automation_runs`, `desktop_auto_update`). This is
a label for Platform Admin's UI, not a different code path: both
classifications resolve through the exact same 5-step order above. The one
real flag wired end-to-end in this slice, `new_automation_runs`, is
classified `operational` -- `AutomationRuntime.run_once()` evaluates it
before `_claim()` on every poll and returns early (after committing, since
`run_once()` always commits its own lease state) when the flag resolves
`false`, so a platform administrator can halt every tenant's automation
execution without touching a single tenant's own `AutomationRule.status`.

### Frontend never decides entitlement

`GET /auth/me` resolves the full `resolve_all_for_subject()` map once at
session bootstrap and returns it as `MeResponse.feature_flags`, carried
into `AuthSession.featureFlags` on the desktop. Reading this map is UX
only -- it renders or hides interface, and only ever an *absent* key means
"no restriction configured," never "disabled." It is never itself an
authorization boundary: `AutomationsPage.tsx` shows an informational banner
when `featureFlags.new_automation_runs === false`, but the actual gate lives
entirely server-side in `AutomationRuntime.run_once()`. Because the map is
captured once at login, a flag flipped in Platform Admin does not live-push
to an already-open desktop session; the next full session bootstrap
(re-login, or a future startup-refresh cycle) picks it up. This is a
known, deliberate property of the current bootstrap-only design, not a bug.

### Permissions and audit

`platform.feature_flags.manage` (already reserved) is joined by a new
`platform.feature_flags.read` (migration `d3f7a2c5e819`), granted to
`platform_owner`, `platform_admin`, `support_admin`, `read_only_auditor` --
mirroring every other read/manage permission pair in the platform RBAC
catalogue. Tenant admins have no path to touch global feature flags at all;
this is a platform-only surface. Every mutation is written to the existing
platform audit ledger with a fixed vocabulary: `feature_flag.created`,
`feature_flag.updated`, `feature_flag.target.added`,
`feature_flag.target.removed`, `feature_flag.rollout.changed`,
`feature_flag.archived` -- `update_flag()` picks the most specific
applicable action name (archival beats a rollout change, which beats a
plain update) rather than always writing the generic `updated` event, so
the audit trail reads as intent rather than a raw field diff. Every
mutation requires a caller-supplied reason (10-500 chars), matching every
other platform-admin action in this codebase; per the user's explicit
instruction, none of this requires step-up re-authentication in this
initial cut -- step-up remains reserved for flags that later get classified
as security-sensitive or infrastructure-critical, not applied blanket to
every harmless UI-rollout toggle.

### Platform Admin UI

New `[redacted].tsx`: a table
(Name / Classification / State / Rollout / Overrides / Actions) plus a
detail dialog (key, global enable toggle, rollout percentage, a targets
list with per-target enable/disable and an add-target form, created-by /
last-changed-by / last-changed metadata). Toggling global state through a
dangerous transition (0%→100%, OFF→ON globally, 100%→OFF) reuses the
existing `ReasonActionDialog` with a dynamically built `impact` bullet list
("Currently: ..." / "After: ...") rather than a new confirmation component --
consistent with how every other Platform Admin destructive action in this
codebase already confirms impact before mutating.

### Validation

14 new unit tests (`test_feature_flag_resolution.py`, no DB: rollout-bucket
determinism/range/variance, the full 5-step resolution order via direct
calls into `FeatureFlagService._resolve()`); 23 new integration tests
(`test_feature_flags.py`: CRUD, targets, permissions, the evaluate-preview
endpoint, direct service-level resolution against a real database,
`/auth/me` exposure, and `AutomationRuntime` kill-switch behavior). A real
SQLAlchemy identity-map staleness bug was found and fixed while writing
these tests: `get_flag()`/`list_flags()`/the cache loader all needed
`populate_existing=True`, or a target added/removed in the same session
would not appear in a subsequently-queried flag's `targets` collection.
Desktop: `AutomationsPage.tsx` reads the flag and renders a warning banner
only when explicitly `false`; live-verified end-to-end against a disposable
local database and a running desktop dev server -- created an
`operational`-classified `new_automation_runs` flag with global state OFF,
confirmed the Automations page banner appeared, re-enabled the flag through
Platform Admin, re-logged-in, and confirmed the banner disappeared.

### Incidental fix found and corrected during this slice's visual verification

While verifying the Feature Flags UI, every Radix-portaled surface in the
whole application (`Dialog`, `DropdownMenu`, `Popover`, `Tooltip`, `Command`)
turned out to render with a fully transparent background -- confirmed by
the user independently (the profile-menu dropdown and "many other dialog
boxes"). Root cause: `ThemeProvider` applies its resolved CSS custom
properties (`--color-app-canvas`, `--color-border`, etc., from
`themeTokens.ts`/`theme-tokens-dark.ts`) only via an inline `style` prop on
an in-tree `<div className="crm-theme-root">`. Radix's `Portal` primitive
renders `Content` directly into `document.body` by default, which sits
outside that div in the real DOM -- CSS custom properties inherit through
the DOM tree, not the React tree, so every portaled node saw those
variables as undefined and fell back to `background-color`'s transparent
initial value. Fixed by also mirroring the same variables onto
`document.documentElement` in a `useEffect` keyed on the resolved
light/dark scheme, since `document.body` is always a descendant of
`document.documentElement`; the existing in-tree div is left unchanged so
nothing about non-portaled rendering changes. Verified visually (profile
dropdown, the Feature Flag create dialog, and a DataTable row-actions
dropdown menu all now render fully opaque) and confirmed via the full
desktop test suite passing unchanged.

## Platform Admin V4.1: Channel Control Center

First V4 increment (the WhatsApp/Channel track, chosen explicitly by the
operator over the AI Control Plane track). Read-only, cross-tenant
drill-down over every `WhatsAppConnection`/`InstagramConnection`/
`FacebookPageConnection` row, closing the gap between `/runtime`'s
system-wide aggregate counts and `/workspaces/{id}/diagnostics`'s
per-workspace detail: neither previously let an operator browse every
connection across every workspace at once.

`PlatformAdminService.list_channels()` deliberately does **not** use a
dict-of-`type[...]` dispatch table to pick which connection model to query
per channel (an earlier draft did, and mypy correctly flagged it: the
dict's value type widens to `type[Base]`, losing the concrete
`WhatsAppConnection`/`InstagramConnection`/`FacebookPageConnection` type
needed for `select(model, ...)` and `self.db.get(model, id)`). Rewritten as
three explicit `_fetch_whatsapp_rows()`/`_fetch_instagram_rows()`/
`_fetch_facebook_rows()` methods instead, matching this file's own existing
precedent for the identical problem (`list_jobs()`'s three
`_list_automation_jobs()`/`_list_broadcast_jobs()`/`_list_media_jobs()`
methods) rather than introducing a new pattern.

Message traffic per connection reuses the parallel FK columns Phase I
established (`Message.provider_connection_id` / `.instagram_connection_id`
/ `.facebook_page_connection_id`); failed-attempt counts do the same via
`MessageProviderAttempt.connection_id` / `.instagram_connection_id` --
except Facebook, where `MessageProviderAttempt` has no per-connection FK at
all (a pre-existing Phase 4E gap, not introduced here), so Facebook falls
back to an organization+provider-scoped count. Documented as a known
precision limit in `docs/API_CONTRACT.md` rather than silently presented as
exact.

`platform.channels.read` (migration `e6a3f8c2d914`) gates all three new
routes -- a deliberate departure from the V3.3a-era precedent of leaving
broad read/list endpoints on the coarse `is_superuser` gate alone. Since
this is new work being added after the platform RBAC system already
exists, it gets a real permission from the start rather than joining that
documented, intentionally-scoped gap.

Desktop: `ChannelsSection.tsx` and `ChannelDetailDialog.tsx` follow the
established `JobsSection.tsx`/`SessionsDialog.tsx` shapes exactly (KPI
tiles, a channel/status/search filter toolbar, a `DataTable`, and a
read-only detail dialog fetched on row-action click) -- no new UI pattern
was introduced.

### Verification note: uncommitted V3.3a-V3.4 work landed first

This session found the fully-implemented V3.3a-V3.4 work (Platform RBAC,
Security Center, auth telemetry, MFA/step-up, Feature Flags) sitting
uncommitted in the working tree, matching what `docs/ROADMAP.md` already
described as complete but never verified or committed. Running the full
validation matrix against it (rather than trusting the documentation)
surfaced one real defect: `alembic check` reported drift on four columns
(`feature_flags.key`, `platform_permissions.key`,
`mfa_challenge_sessions.token_hash`, `step_up_sessions.token_hash`) where
the migration created a redundant `UniqueConstraint` plus a separate
non-unique index instead of the single unique index the ORM models
(`mapped_column(..., unique=True, index=True)`) actually produce. Since
none of the four migrations had ever been applied to any shared database,
they were corrected in place rather than patched with a follow-up
migration, then re-verified (upgrade to head, `alembic check` clean,
downgrade through the corrected chain, re-upgrade to head) before being
committed as their own checkpoint ahead of any V4 work.

## Platform Admin V4.2: WhatsApp Control Center

Second V4 increment, continuing the WhatsApp/Channel track. V4.1's
generic 3-channel connection view already covers WABA/phone metadata,
status, message traffic, and provider errors identically for WhatsApp/
Instagram/Facebook; the one genuinely WhatsApp-specific surface the V4.2
plan calls for that V4.1 doesn't touch is template visibility -- Vendale
already syncs and persists every Meta-approved WhatsApp template per
connection (`WhatsAppProviderTemplate`, built in Phase 4B), but nothing
before this exposed that data cross-tenant.

`PlatformAdminService.whatsapp_templates_summary()` and
`list_whatsapp_templates()` read `status` and `quality_rating` exactly as
`WhatsAppTemplateService` stored them from Meta's own API response --
never normalized into a locally-invented vocabulary, matching the same
"pass through the real value" principle `list_jobs()`'s per-queue status
options already established. `by_quality_rating`'s `None` bucket is
labelled `"unknown"` in the response dict (a template awaiting Meta's
quality assessment), not silently dropped or miscounted.

Both new routes (`/channels/whatsapp/templates/summary`,
`/channels/whatsapp/templates`) had to be inserted **before**
`/channels/{channel}/{connection_id}` in `platform_admin.py`'s route
declaration order -- not after, and not "it doesn't matter since they're
different literal paths." Starlette matches routes in the order they were
registered, not by specificity: a request for `/channels/whatsapp/templates`
would otherwise match the parameterized `{channel}/{connection_id}` route
first (`channel="whatsapp"`, `connection_id="templates"`), which fails
Pydantic's UUID validation and returns 422 rather than ever reaching the
templates handler. This is the identical bug class the Phase F5
`/flows/keypair` fix caught (see that entry above) -- confirmed here by a
dedicated regression test rather than by re-deriving the reasoning from
scratch each time a new nested route is added under an existing
parameterized prefix.

Desktop: `ChannelsSection.tsx` gained a `view` toggle ("Connections" |
"WhatsApp Templates") rather than a new top-level Platform Admin nav item
-- template visibility is a drill-down of the same Channels surface, not a
separate console section. The existing connections-loading `useEffect` is
guarded with `if (view !== "connections") return`, mirroring
`PlatformAdminPage.tsx`'s existing pattern for self-fetching sections, so
switching to the Templates view doesn't also re-fetch connection data in
the background. `WhatsAppTemplatesPanel.tsx` is a new, separate,
self-fetching component (KPI tiles, status/category filters populated from
the real values the summary endpoint returned rather than a hardcoded
list, and a `DataTable`), following `JobsSection.tsx`'s established shape.

### Live verification note

Live-verifying this slice caught a seeding mistake, not a product bug: the
first attempt to log in as the disposable `support_admin` test user showed
no "Platform Admin" nav item at all. The nav item's visibility predicate
checks `session.user.isSuperuser`, which is set by the router-level
`is_superuser` boolean -- a separate DB column from `platform_role_id`, and
the seed script had only set the latter. `is_superuser=True` is the coarse
"has platform access at all" gate every platform route still requires
first (V3.3a); a `platform_role_id` alone, without it, grants nothing. Fixed
by setting both columns, matching how `_assign_platform_role()`'s existing
backend test helper already does it correctly.

## Platform Admin V4.3: Webhook/Event Inspector

Third V4 increment, completing the read-visibility half of the WhatsApp/
Channel track (V4.4 Broadcast Safety and V4.5 Operational Freeze remain,
both of which mutate state rather than just displaying it). Built around
`MetaWebhookEvent`, not `MetaWebhookDelivery`, for a concrete reason:
neither model stores a `channel` column, and `MetaWebhookDelivery.connection_id`
only FKs `whatsapp_connections` -- Instagram and Facebook Page deliveries
always leave it null and reuse `phone_number_id` as a generic "receiving
account id" slot (confirmed by reading `webhook_processing_service.py`'s
three delivery-creation call sites directly, not assumed). The one
reliable per-row channel signal is each event's own `event_key` prefix
(`"message:"`/`"status:"` for WhatsApp, `"ig_message:"` for Instagram,
`"fb_page_message:"` for Facebook), so `_webhook_event_channel()` derives
channel from that string prefix rather than trying to add a stored column
purely for this admin view.

Reuses V4.1's `platform.channels.read` permission again (no new
migration), continuing the established precedent that closely-related
read-only visibility within the same Channel Control Center umbrella
shares one permission rather than fragmenting into a new key per endpoint
group.

Both new routes (`/channels/webhooks/summary`, `/channels/webhooks`) had
to be declared before `/channels/{channel}/{connection_id}` in source,
for the exact same reason as V4.2's templates routes: Starlette matches
by declaration order, and `/channels/webhooks/summary` is the same
2-segment shape as the parameterized detail route.

### Test-isolation lesson: global cross-tenant counts need delta assertions

The first version of this slice's summary test asserted absolute counts
(`by_channel == {"whatsapp": 2, "instagram": 1, "facebook": 1}`) and
passed in isolation but failed when run as part of the full suite --
`by_channel["facebook"]` came back `12`, not `1`. Root cause: unlike
V4.1/V4.2's summary tests (which count `WhatsAppConnection`/
`InstagramConnection`/`FacebookPageConnection`/`WhatsAppProviderTemplate`
rows -- tables no other test file in this suite touches), many unrelated
integration test files legitimately create `MetaWebhookDelivery`/
`MetaWebhookEvent` rows of their own (webhook processing tests, Instagram
tests, Facebook Page tests), and this test suite does not give every test
full transactional rollback isolation from every other test's committed
rows. `webhook_events_summary()` is correctly built as a genuine
cross-tenant global count (that is the whole point of the endpoint), which
means it is fundamentally the wrong kind of assertion to pin to an
absolute value in a shared, non-fully-isolated test database. Fixed by
capturing a baseline summary before seeding and asserting the before/after
delta instead -- the correct general pattern for testing any genuinely
global aggregate endpoint added to this codebase in the future, not a
one-off workaround. List-endpoint assertions were separately hardened by
scoping every query to the test's own freshly-created `workspace_id`
rather than relying on an assumed-clean global list.

## Platform Admin V4.4: Broadcast Safety

Fourth V4 increment, and the first in the WhatsApp/Channel track to
mutate tenant data rather than only display it -- V4.1-V4.3 were
entirely read-only. Every mutation route (`pause`/`resume`/`cancel` on a
single broadcast, and `pause`/`resume` on every currently-sending
broadcast in a workspace) delegates to the existing `BroadcastService`
(Phase 4B) -- the exact code path a tenant admin's own broadcast controls
already call -- rather than manipulating `BroadcastDispatchJob` rows
directly or re-implementing `BroadcastService`'s state-machine validation.
This mirrors the standing principle the program brief itself states in
section 5 ("Platform Owner is powerful, not invisible") and matches the
precedent V3.2's job actions already established: `retry_job()`/`cancel_job()`
explicitly refuse `queue="broadcast"` with a 422 pointing callers at the
parent `Broadcast`'s own pause/resume/cancel, specifically to avoid a raw
per-job mutation bypassing the recipient-status/count-aggregate sync
`BroadcastService` already provides.

`pause_workspace_broadcasts()`/`resume_workspace_broadcasts()` are a
line-for-line structural mirror of V3.2's
`pause_workspace_automations()`/`resume_workspace_automations()`: pause
records the exact set of broadcast ids it paused in the audit event's
`after_state`, and resume looks up that same audit event to restore only
broadcasts still in that set and still `paused` -- a broadcast a tenant
admin has independently cancelled or otherwise changed since the platform
pause is never silently overridden. A dedicated test proves this
concretely (pausing two broadcasts, a tenant "cancelling" one directly,
then confirming resume only touches the other).

`_LARGE_BROADCAST_RECIPIENT_THRESHOLD` (1000) and
`_HIGH_FAILURE_RATE_THRESHOLD` (0.10) are fixed `ClassVar` constants on
`PlatformAdminService`, echoed back in every summary response rather than
hardcoded a second time in the desktop UI -- the same anti-duplication
discipline the program brief's threshold-transparency requirement calls
for. An `outbound_broadcasts` feature-flag kill switch (V3.4 pattern,
fail-open) gates `BroadcastDispatchService.run_once()` exactly the way
`new_automation_runs` already gates `AutomationRuntime.run_once()`: a
platform administrator can halt every workspace's broadcast dispatch in
an emergency without touching individual broadcasts, and it is inert
until deliberately configured.

### Test-isolation lesson #2: a service's own internal `commit()` can leak test state that `db.flush()` never would

Found and fixed during this slice's validation, distinct from V4.3's
"global aggregate needs delta assertions" lesson (same root class --
committed-past-rollback test state -- different trigger). Several new
tests set `admin_user.is_superuser = True` directly (not via
`_assign_platform_role()`) and then called a real pause/resume/cancel
endpoint. `BroadcastService.pause()`/`.resume()`/`.cancel()` each call
`self.db.commit()` internally (required so their own state-machine
changes are durable even if a later step in the same request fails) --
and since this test suite's `db` fixture and the app's request-scoped
session are the same object, that internal commit also permanently
persists the test's own `is_superuser = True` mutation, defeating the
per-test rollback boundary the `db` fixture normally provides.

This is the exact same leak class `test_feature_flags.py`'s
`_revoke_leaked_platform_role()` already documents for
`AutomationRuntime.run_once()`'s internal commit (see the V3.4 entry
above) -- confirmed by re-reading that existing helper's docstring rather
than re-diagnosing from scratch, then adding the mirror-image
`_revoke_leaked_superuser()` helper to `test_platform_admin.py` and
calling it at the end of every test that grants superuser/platform-role
access and then triggers a self-committing mutation. The standing rule
going forward: **any test that grants `is_superuser`/`platform_role_id`
and then calls an endpoint whose service method commits internally must
explicitly revoke that grant before the test ends** -- `db.flush()`
alone is not sufficient protection once a commit happens anywhere
downstream in the same request.

## Platform Admin V4.5: Operational Freeze

Fifth and final V4 increment, closing the WhatsApp/Channel Control Plane
track. Unlike V4.1-V4.4, this slice began with an explicit design pass
(required by V4.4's own ledger note) before any code was written, because
"freeze a workspace's outbound activity" turned out to already have most
of its infrastructure built -- the risk was building a second, redundant
mechanism instead of recognizing that.

### What "frozen" does and does not mean

Reading the actual worker code first (not assumed) settled three
questions the design pass required:

- `Organization.is_active = False` (V3.2's workspace suspend) blocks
  interactive login and all authenticated API access, but never touches
  any background worker loop -- confirmed by grep, not inference. This
  remains true after V4.5; suspend and operational freeze are two
  independent controls, not layers of the same one.
- `MediaIngestionService` needed zero V4.5 changes. Its
  `enqueue_for_asset()` rejects any `direction != "inbound"` at the point
  a job would be created, which means "media ingestion is 100%
  inbound-only" is a structural property of the codebase already, not
  something V4.5 had to add a check to preserve. The one webhook-ingestion
  requirement the design pass carried forward from V4.4's own note
  ("must deliberately preserve inbound webhook ingestion") is satisfied
  by this pre-existing guarantee, not by new code.
- `WebhookProcessingService.handle_delivery()` is a synchronous,
  request-scoped call inside the webhook POST handler, not a worker loop
  polling a queue. There is nothing for a "freeze" to un-claim; it simply
  isn't in scope for this feature by construction.

### Reused infrastructure, not a new mechanism

`FeatureFlagService.evaluate(key, *, organization_id=...)` (V3.4) already
resolves a flag per-subject through its full 5-step precedence order,
including the workspace-override step. V4.4 had already proven the
pattern of a *global* operational kill switch checked once per
`run_once()` poll (`new_automation_runs`, `outbound_broadcasts`). V4.5's
entire implementation is: call that same `evaluate()` a second time,
*inside* the claim loop, resolved with each individual item's own
`organization_id` instead of no organization at all. No new flag
resolution logic, no new caching layer, no new migration -- two existing
flag keys (`new_automation_runs`, `outbound_broadcasts`) are reused
exactly as-is, and one new key (`workspace_outbound_messages`) follows
the identical pattern for the third category (outbound sends), which had
no prior global kill switch to extend.

```text
AutomationRuntime.run_once()
  global gate:  evaluate("new_automation_runs")                     [V3.4, unchanged]
  per-event:    evaluate("new_automation_runs", organization_id=event.organization_id)  [V4.5, new]

BroadcastDispatchService.run_once()
  global gate:  evaluate("outbound_broadcasts")                     [V4.4, unchanged]
  per-job:      evaluate("outbound_broadcasts", organization_id=job.organization_id)    [V4.5, new]

MessageService._is_outbound_frozen(organization_id)                 [V4.5, new -- no prior global gate existed]
  evaluate("workspace_outbound_messages", organization_id=organization_id)
```

A frozen automation event or broadcast job is fully un-claimed, not
failed: status reverts to `queued`, any lease/heartbeat/owner fields are
cleared, and `available_at` is pushed `_WORKSPACE_FREEZE_REQUEUE_SECONDS`
(30s, a module constant in both `automation_runtime.py` and
`broadcast_dispatch_service.py`) into the future -- short enough that
unfreezing resumes delivery quickly, long enough to avoid a tight reclaim
loop racing the feature-flag service's own 45-second in-process cache.
The automation path additionally rolls back the `attempts` counter
`_claim()` had just incremented (`max(event.attempts - 1, 0)`), since a
freeze is not a failed attempt; the broadcast path needs no equivalent
rollback because `_claim_due_jobs()` never increments `attempts` in the
first place -- only `_mark_job_failed()` does, and a frozen job never
reaches that code path.

A frozen outbound send is different in kind from a frozen queue item: it
is not queued for later reconsideration, because `send_message()` has
already committed to a synchronous request/response cycle by the time
`_attempt_provider_send()` runs. Instead, `_is_outbound_frozen()` is
checked before any connection lookup or credential decryption, and on a
freeze the method records a real `MessageProviderAttempt` row
(`succeeded=false`, `error_code="workspace_outbound_frozen"`,
`error_summary="Outbound sends are temporarily frozen for this workspace
by a platform administrator."`) and marks the `Message` `failed` with the
same stable code -- visible to the sending agent as an ordinary failed
send, auditable via the attempt row, never silently swallowed. All three
outbound channels (WhatsApp, Instagram, Facebook Page) get the identical
check at the top of their respective `_attempt_*_provider_send()` method;
WhatsApp's existing manual-retry path re-enters `_attempt_provider_send()`
unchanged, so retry is automatically freeze-aware with zero additional
code.

### No new mutation endpoint

Freezing or unfreezing any of the three categories for a workspace is
exactly the existing V3.4 target mechanism:
`POST /platform-admin/feature-flags/{id}/targets` with
`{"target_type": "workspace", "target_id": <org id>, "enabled": false}`
to freeze, `DELETE .../targets/{target_id}` to unfreeze. The desktop's
new `OperationalFreezeSection.tsx` (rendered on the existing
`WorkspaceDiagnosticsPage.tsx`, directly beneath "Workspace actions")
calls only pre-existing `platformAdminApiService` methods
(`listFeatureFlags`, `getFeatureFlag`, `createFeatureFlag`,
`addFeatureFlagTarget`, `removeFeatureFlagTarget`) -- zero new desktop
service methods were needed. `createFeatureFlag()` is called only the
first time a given key is frozen for any workspace (a platform admin may
never have visited the Feature Flags screen for these three operational
keys before); every subsequent freeze/unfreeze for any workspace reuses
the same flag row. The flag's own global `enabled` field is never written
by this panel -- only its per-workspace target.

### Verification note

Live end-to-end verification surfaced two real, previously-undiscovered
environment gaps, both fixed as part of this slice rather than worked
around:

1. The shared local development database (`vendale`, distinct from the
   disposable `thread_crm_test` the pytest suite uses) was nine
   migrations behind head, missing `users.platform_role_id` entirely
   (registration and login both 500'd). This is exactly the gap flagged
   as a pending step in both the V3.3a and V3-Part-1 ledger entries.
   Resolved with `alembic upgrade head`; `alembic check` confirmed no
   drift afterward.
2. `MFA_CREDENTIAL_ENCRYPTION_KEY` (required since V3.3c-2) was present
   in neither `backend/.env` nor `backend/.env.example`. Because the
   startup fail-fast check for this key only runs when
   `ENVIRONMENT=production`, the gap was invisible until the first actual
   MFA enrollment attempt raised a `ValueError` deep in
   `MfaService.start_enrollment()`. Added a real local-dev value to
   `.env` and a `FAKE...`-prefixed placeholder to `.env.example`,
   mirroring the existing `AI_CREDENTIAL_ENCRYPTION_KEY` documentation
   pattern exactly.

With both fixed, a full real login -> mandatory TOTP MFA enrollment (live
6-digit codes computed with the backend's own `app.security.totp` module
against the QR-code setup key actually rendered in the UI) -> MFA
challenge on a subsequent login -> Platform Admin -> workspace
diagnostics -> a real freeze -> a real unfreeze round trip was completed
through the real desktop UI, the real backend, and the real
Neon-hosted development database, with every state change independently
confirmed via direct database queries (the `FeatureFlag` row, its
`FeatureFlagTarget`, and two `platform_audit_events` rows).

## Platform Admin V4.6: AI Provider/Model Registry

First increment of the AI Control Plane track (V4.6-V4.10). Deliberately
the lightest-weight of the two possible starting points -- `AiProviderConfig`
is a single table with a `UniqueConstraint` on `organization_id` (one row
per workspace), unlike the three-table WhatsApp/Instagram/Facebook shape
V4.1 had to dispatch across, so no `_fetch_whatsapp_rows()`-style per-type
methods were needed here.

### What a row means

`AiService._get_or_create_config()` (built in the SaaS Commercialization
S3 phase) creates a row lazily the first time a workspace calls Copilot
readiness or config -- not at organization creation, and not only when a
workspace actually connects a provider. This means a `status="not_configured"`
row is evidence of "this workspace has opened the Copilot page," not "this
workspace has configured AI." The registry surfaces this honestly (see the
desktop component's own docstring) rather than implying every workspace is
tracked equally: most workspaces will have zero rows, and that absence is
the correct default state, not a loading gap.

`provider_type` is `"axon_gateway"` for every row that has ever been
created, since `AiService.provision()` is the only code path that ever
sets a non-default value and it always sets this one string (see
`CLAUDE.md`'s SaaS Commercialization section: "AI through the existing
Axon gateway rather than an LLM SDK"). The registry returns this
field as real, currently-single-valued data rather than hardcoding a
`Literal["axon_gateway"]` type -- if a second provider is ever integrated,
the column and the API response already accommodate it with zero schema
or contract change.

### Deliberately excluded from this slice

Per-organization gateway usage/quota is real and already fetchable:
`AiService.account_usage(organization_id)` calls `AxonGatewayClient.get_account(api_key=...)`
using that org's own decrypted key, hitting the gateway's real `/v1/account`
endpoint. This was deliberately NOT wired into V4.6's summary or list
endpoint -- calling a live per-tenant gateway endpoint for every row in a
cross-tenant list would be exactly the anti-pattern V4.1's own list/detail
split already established (bulk queries stay local-DB-only; a live
per-connection call is reserved for a single-record detail view). V4.7 (AI
Usage Accounting) is the natural home for exposing `account_usage()`
cross-tenant, most likely per-workspace on demand rather than eagerly
across every configured org.

### Verification note

Live-verified with two disposable organizations sharing one `support_admin`
platform administrator (avoiding the V3.3c-2 mandatory-MFA gate that
correctly applies to `platform_owner`/`platform_admin`/`security_admin`):
one `AiProviderConfig` row seeded `status="configured", enabled=true`, the
other `status="error", enabled=false` with a stored `last_error_code`. Both
KPI tiles and both table rows matched the seeded data exactly through the
real desktop UI; the status filter correctly isolated the error row; the
detail endpoint was confirmed directly via the real API for both a real
workspace id (full record returned) and an unknown UUID (404). Disposable
workspaces and the user were deleted afterward; exactly one organization
(`arche-axon`) remained.

## Platform Admin V4.7: AI Usage Accounting

Second AI Control Plane increment. Two genuinely different signals, kept
structurally separate rather than merged into one "usage" number, because
they answer different questions and have different honesty guarantees.

### A lower bound, stated as one

`ai_usage_summary()` aggregates the persisted `message_drafts` table
(`total_drafts`, `by_status`, `by_source`, `total_input_tokens`,
`total_output_tokens`, `workspaces_with_drafts`). This is real, accurate
data about everything it counts -- but it is not total AI usage, because
`AiService.chat()` (the interactive Copilot chat endpoint) never persists
anything: no `MessageDraft`, no separate chat-log table, nothing. A tenant
that only uses ad-hoc chat and never generates a reply draft shows zero
here despite genuinely using AI. The schema docstring, the API contract
entry, and the desktop panel's own visible description text all state
this lower-bound property explicitly and identically -- this is
deliberate redundancy, not accidental repetition, so no future reader of
any one of the three sees an unqualified "usage" number.

### The real answer, one workspace at a time

`get_workspace_ai_usage()` delegates directly to the existing tenant-facing
`AiService.account_usage(organization_id)`, which already calls the Axon
gateway's real `GET /v1/account` using that workspace's own decrypted key
-- built in the SaaS Commercialization S3 phase for the tenant's own
Copilot status panel, reused here unmodified rather than reimplemented.
This is genuinely complete (whatever the gateway reports is the real
answer), but deliberately never called across a workspace list: doing so
would mean one live external HTTP call per row in a cross-tenant summary,
the exact anti-pattern V4.1's own connections list-vs-detail split already
rejected for local data, now doubly true for a call that leaves this
process entirely.

`PlatformAdminService.get_workspace_ai_usage()` takes an optional
`ai_service: AiService | None` parameter -- not because production needs
it (the real endpoint always omits it, letting `AiService` build its own
gateway lazily), but because tests do, mirroring the exact pattern
`AiService.__init__` itself already established for its own `gateway`
parameter. Without this, a test exercising the successful-response path
would have no way to avoid `AiService(self.db)` building a real
`AxonGatewayClient()` pointed at whatever `AXON_GATEWAY_BASE_URL` happens
to be configured to.

The gateway's account response is returned as an untyped `dict[str, Any]`
pass-through (`PlatformAiWorkspaceUsageRead.account`) rather than a typed
schema, and the desktop dialog renders it generically as key/value pairs
-- this codebase does not control or fully know the Axon gateway's
response contract, so typing it would mean guessing at a shape that could
silently drift. This mirrors `FlowResponseSummary`'s existing precedent
for the same problem (an org-authored WhatsApp Flow's `response_json`
fields, whose shape this codebase also cannot know ahead of time).

### A real bug found while wiring up live verification

Attempting to seed a realistic, encrypted `AiProviderConfig.encrypted_api_key`
for live verification failed with `WHATSAPP_CREDENTIAL_ENCRYPTION_KEY must
be a 64-character hex string` -- from the AI key, not the WhatsApp key,
because `credential_encryption.py`'s `_derive_key()` error message is
hardcoded to name WhatsApp regardless of which caller invoked it (a
separate, minor, pre-existing message-clarity gap, not fixed here since it
is unrelated to this slice's scope). Investigating further: the local
`backend/.env`'s `AI_CREDENTIAL_ENCRYPTION_KEY` was 65 characters, not 64.
Checking `backend/.env.example` found the identical defect in all three
credential-key placeholders (`WHATSAPP_CREDENTIAL_ENCRYPTION_KEY`,
`AI_CREDENTIAL_ENCRYPTION_KEY`, and this session's own V4.5-added
`MFA_CREDENTIAL_ENCRYPTION_KEY`) -- each is the literal text `FAKE`
followed by 61 repeated digits, and the letter `K` in `FAKE` is not a
valid hexadecimal character at all, so `binascii.unhexlify()` would
reject every one of them on both counts (wrong length and invalid
characters) if anyone ever pasted the example value verbatim, despite the
comment directly above each one stating "Must be a 64-character hex
string." This had gone unnoticed because the production-only FATAL
startup check (`ENVIRONMENT == "production"`) never runs in local dev,
and no prior session's live verification had exercised a code path that
actually called `_derive_key()` on any of these three specific keys with
their example value still in place. Fixed all three placeholders in
`.env.example` to genuinely valid, obviously-fake 64-character hex values,
and fixed the real local `.env` `AI_CREDENTIAL_ENCRYPTION_KEY` with an
actual random 64-hex-char value so this session's own live verification
could proceed.

### Verification note

Live-verified with a deliberately fake (but correctly encrypted and
decryptable) stored gateway key: the "View live usage" dialog correctly
reached the real Axon gateway over the network, and the gateway correctly
rejected the fake key, which the dialog displayed safely
(`API error 502: Invalid API key`) rather than crashing, hanging, or
leaking any credential material. This is a genuine, deliberately-sought
proof of the error-handling path, not a fallback taken because the happy
path could not be reached -- the same "no fabricated success" standard
this program has applied throughout the WhatsApp/Channel and AI Control
Plane tracks.

## Platform Admin V4.8: AI Cost Control

Third AI Control Plane increment, and the first in this track to mutate
tenant data rather than only display it.

### Re-scoped from visibility to action, based on real evidence

V4.7's own research note flagged that `account_usage()`'s gateway
response is an untyped pass-through with no fields this codebase could
assume exist, and that a cross-tenant cost rollup would require one live
HTTP call per workspace. Rather than guess at what a "cost control"
feature should look like, the real Axon gateway source was read directly
(`C:\Projects\Axon\gateway\axon\routers\account.py`, an external
dependency, not this codebase, but read the same way any internal
service would be before depending on its behavior) to see what fields
`GET /v1/account` genuinely returns: `wallet_balance_cents`,
`wallet_spend_cap_cents`, `wallet_spent_this_month_cents`,
`wallet_remaining_cap_cents`, `wallet_headroom_cents`,
`token_quota_monthly`, `tokens_used_this_month`, `tokens_remaining`,
`max_members`. This confirmed, rather than merely assumed, that a
cross-tenant cost dashboard is not honestly buildable in this slice --
every one of those fields lives entirely gateway-side, reading it for
one workspace already costs a live external call, and this program's
standing engineering-integrity rule (no fabricated metrics) rules out
synthesizing a rollup from data this codebase does not hold. V4.8 was
re-scoped from "show AI cost across every workspace" to "let a platform
administrator pull an emergency lever for one workspace" -- a smaller,
honest, immediately buildable capability using only data already local
to this codebase (`AiProviderConfig.enabled`).

### Delegation, not direct mutation

`PlatformAdminService._apply_ai_enabled_action()` is a private helper
shared by `disable_workspace_ai()`/`enable_workspace_ai()`: look up the
workspace (404 if unknown), read the current `AiProviderConfig.enabled`
value if a row exists (for the audit `before_state`), call the existing
tenant-facing `AiService.update_config(workspace_id,
CopilotConfigUpdate(enabled=...), actor_user_id=...)` -- the exact method
a tenant's own Settings AI toggle already calls -- then write a platform
audit event and return the refreshed config detail. Neither public
method touches `AiProviderConfig` directly; this is the same
delegation-to-tenant-service discipline `BroadcastService` already
established for V4.4's pause/resume/cancel actions, applied here to a
different tenant service.

Because `AiService.update_config()` calls `_get_or_create_config()`
internally (existing behavior, unmodified), disabling AI for a workspace
that has never opened Copilot at all -- no `AiProviderConfig` row yet --
still succeeds: the row is created in a disabled state rather than the
call 404ing. This was confirmed by reading `update_config()`'s own code
first, then proven concretely with a dedicated test
(`[redacted]`)
rather than left as an inferred assumption.

### Permission scope

`platform.ai.manage` (migration `4ae40777760f`, `is_high_risk=true`) is
a new, dedicated permission -- not a reuse of the existing read-only
`platform.ai.read` -- granted to `platform_owner`, `platform_admin`, and
`support_admin` only. This mirrors the exact three-role grant set
`platform.broadcasts.manage` (V4.4) and `platform.workspaces.manage`
already use for comparable mutation power, and deliberately excludes
`billing_admin`, `security_admin`, and `read_only_auditor` -- none of
whom hold comparable mutation permissions elsewhere in the catalogue
either.

### Verification note

Live-verified a full disable-then-enable round trip through the real
desktop UI against a disposable workspace seeded with an enabled
`AiProviderConfig` row: disabled with a reason, confirmed the row's
status changed to "Disabled" and the row-action button states correctly
reversed (Enable available, Disable/View-usage correctly disabled),
re-enabled with a second reason, confirmed the row returned to
"Enabled". A direct database query (not just the UI's own success toast)
confirmed both `ai.disabled` and `ai.enabled` platform audit events were
written with the correct actor, target workspace, and before/after
`{"enabled": ...}` state -- the same "verify past the UI's own claim of
success" discipline this program has applied to every prior mutation
slice's live verification.

## Platform Admin V4.9: AI Execution Log

Fourth AI Control Plane increment. Read the real code first, again: this
time `AiService.chat()`, `generate_draft()`, and the `generate_ai_draft`
automation action, to determine what execution-level data already exists
before designing a new observability surface over it.

### What the code already does, confirmed by reading it

`AiService.chat()` (the interactive Copilot chat endpoint) persists
nothing at all -- no row, no log, no token count anywhere. It only
updates `AiProviderConfig.last_error_code`/`last_checked_at` on failure
and clears them on success. This is the exact gap V4.7's own "lower
bound" framing already documented for its usage summary; V4.9 does not
change it.

`AiService.generate_draft()` (called both directly by the Copilot
"Suggest reply" button and, with `source="automation"`, by the
`generate_ai_draft` automation action) creates a `MessageDraft` row --
but only after a *successful* gateway response. A failed generation
attempt (an `AxonError`) raises an `HTTPException` before the
`MessageDraft` is ever constructed; only `AiProviderConfig`'s single
most-recent-error field reflects that failure occurred, not a per-attempt
row. This means `message_drafts` is honestly a log of successful
executions only, never a full execution history -- documented, not
silently treated as complete.

### An aggregate-then-drill-down progression, not a new mechanism

This is the same shape the WhatsApp/Channel Control Plane track already
established: V4.1 gave a channel-summary aggregate, V4.3 gave the
per-event drill-down over the same underlying data. V4.7 already built
the aggregate (`ai_usage_summary()`); V4.9 is that track's drill-down --
`list_ai_executions()`, one row per `MessageDraft`, filterable by
workspace/status/source, following `list_ai_configs()`'s own join-and-filter
shape (`MessageDraft` joined to `Organization`, optional `where` filters,
a `COUNT` subquery for `total`, then a paginated `SELECT`).

### The one thing this view must never show

`PlatformAiExecutionRead` deliberately has no `body` field.
`MessageDraft.body` is the AI's drafted reply text, generated by
`_conversation_context()` from real customer message history and the
customer's actual name -- this is genuine customer data, not
business-authored content like V4.2's WhatsApp template bodies (that
entry's own docstring draws exactly this distinction: template text is
safe to expose cross-tenant because it's operator-authored and
Meta-reviewed, not customer-specific). A cross-tenant Platform Admin
surface exposing one workspace's customer conversation content to a
platform administrator working on an unrelated workspace would violate
this program's tenant-isolation principle, so the field is omitted at
the schema level, not merely hidden in the UI -- confirmed by a dedicated
test asserting `"body"` never appears in the response JSON at all, not
just that the desktop table doesn't render it.

### Verification note

Live-verified against a disposable workspace seeded with one enabled
`AiProviderConfig` and two `MessageDraft` rows (one `pending`/`copilot`,
one `accepted`/`automation`). The new "Configurations" / "Execution Log"
view toggle in `AiControlPlaneSection.tsx` renders both tabs; the
Execution Log correctly lists both rows with workspace, status, source,
model, token counts, and timestamps, and the status filter correctly
narrows to one row. Confirmed directly via `curl` that the endpoint's raw
JSON response contains no `body` key at all -- not just that the desktop
UI happens not to render one. Found, along the way, that the backend
process already running from earlier in the session predated this
slice's new route and returned a genuine 404 until restarted; confirmed
this was an environment artifact, not a routing bug, by the identical
request succeeding immediately after a clean restart with no code
change.

## Platform Admin V4.10: AI Governance

Fifth and final AI Control Plane increment, closing V4 in its entirety.

### What "governance" honestly means for this codebase

Before designing anything, the real code was read to see what already
exists rather than assuming a governance feature needed to be invented
from scratch:

- `Settings.reject_autonomous_ai_send()` already refuses to boot the
  process at all if `AI_AUTOMATIC_SEND_ENABLED` is set, in every
  environment, not just production. This is already the strongest
  possible enforcement -- there is no runtime state for a Platform Admin
  view to observe, because the condition it would report on can never be
  true in a running process.
- `AiProviderConfig.pii_redaction_enabled` and `.retention_days` are
  stored, settable via `CopilotConfigUpdate`, and returned by every V4.6
  config endpoint -- but neither is read by any enforcement code path
  anywhere in this codebase. No PII redaction logic exists; no retention
  worker prunes `message_drafts`. This is a real, pre-existing gap,
  documented here rather than silently treated as if these fields did
  something.
- What genuinely did not exist: a platform-wide kill switch for AI
  generation itself, distinct from V4.8's per-workspace
  `AiProviderConfig.enabled` lever. `new_automation_runs` (V3.4) already
  gives this exact capability to automations, and `outbound_broadcasts`
  (V4.4) already gives it to broadcasts; AI Copilot had no equivalent.
  This is the one governance-shaped gap this codebase could honestly
  close without either fabricating a capability or duplicating V4.8.

### Reused mechanism, not a new one

`PlatformAdminService.disable_ai_generation_globally()`/
`enable_ai_generation_globally()` do not talk to `FeatureFlagService`
directly with new bespoke create/update logic. They call the *existing*
`create_feature_flag()`/`update_feature_flag()` methods this same class
already exposes for V3.4's Feature Flags page -- the identical code path,
producing the identical `feature_flag.created`/`feature_flag.updated`
audit actions every other flag mutation in this codebase already
produces. The only new code is the thin `_set_ai_generation_enabled()`
wrapper that resolves whether the `ai_generation_enabled` key already
exists (create vs. update) and returns the resolved
`PlatformAiGovernanceRead` status afterward.

### A permission gap found while designing, not left implicit

The obvious first design was to gate this action with
`platform.feature_flags.manage`, since that is literally what the
underlying mutation is. Checking the actual role grants
(`[redacted].py`) before committing to
that showed a real problem: `support_admin` -- the role meant to handle
exactly this kind of day-to-day operational incident -- holds
`platform.ai.manage` but only `platform.feature_flags.read`, not
`.manage`. Gating the new endpoint behind `platform.feature_flags.manage`
would have meant the one role built for incident response could not use
it. Gating it behind `platform.ai.manage` instead (an AI-domain
permission `support_admin` already holds) fixes this without granting
`support_admin` any broader feature-flag power it does not already need.

### Enforcement, checked before any per-workspace state

`AiService._require_generation_enabled()` is called as the very first
line of both `chat()` and `generate_draft()`, before
`_require_ready_config()` (V4.8's per-workspace check),
`_conversation_context()`, or any gateway key decryption. This ordering
is deliberate and directly proven by the enforcement test: disabling the
global flag and then calling `generate_draft()` with a conversation id
that does not exist in the database still produces the governance
service's own 503, never a 404 or any other error from further down the
call -- confirming the gate genuinely short-circuits before any
per-workspace work begins, not merely that it happens to run somewhere
in the method.

### Verification note

Live-verified a full disable-then-enable round trip through the real
desktop UI: the "Platform-wide AI generation" card rendered the correct
default ("Never configured -- enabled everywhere by default", green
"Enabled" badge), disabling with a reason changed the badge to "Disabled
Platform-Wide" and reversed the button to "Enable", and re-enabling with
a second reason reversed it back to "Disable". A direct database query
independently confirmed both `feature_flag.created` and
`feature_flag.updated` platform audit events were recorded with the
correct before/after state and reason -- the same "verify past the UI's
own claim of success" discipline this program has applied to every prior
mutation slice.

## Platform Admin V5.1: Platform Analytics

First Platform Intelligence unit. Confirmed by reading `overview()`
first that Platform Admin had zero visibility into real product usage
(customers, conversations, messages, orders, tickets) anywhere -- only
workspace/user/subscription counts.

### Never one fabricated total across heterogeneous currencies

`Order.currency` is per-order, not fixed platform-wide (an org's
`organization_settings.currency` sets a default, but nothing prevents an
individual order from using a different one). Summing `grand_total`
across every order in the platform would silently add KES and USD
figures together as if they were the same unit -- a fabricated number
with no honest meaning. `analytics_overview()` instead groups by
currency (`GROUP BY Order.currency`) and returns
`orders_revenue_by_currency: dict[str, str]`, the same "no fabricated
single number from heterogeneous data" discipline the V2 billing
overview already established for MRR/ARR (no stored subscription price
exists platform-wide either).

### Two real PostgreSQL bugs found and fixed while writing the trend query

Both were caught by `[redacted]`
before either ever reached a live environment, not discovered afterward.

**Bug 1: a computed expression bound twice is not the same expression to
asyncpg.** The first draft called `func.date_trunc("day", Customer.created_at)`
separately in both the `SELECT` and `GROUP BY` clauses. SQLAlchemy
compiles each call into its own bound parameter (`$1`, `$3`, ...), and
PostgreSQL's `asyncpg` driver refuses the query outright
(`GroupingError: column "customers.created_at" must appear in the GROUP
BY clause`) because it cannot prove two independently-bound parameters
are equal at parse time, even though their literal values are identical.
Fixed by binding the expression to one Python variable
(`day_bucket = func.date_trunc(...)`) and passing that same object to
both `select()` and `.group_by()` -- SQLAlchemy then compiles it once
and references it in both clauses correctly.

**Bug 2: `date_trunc()` on a `timestamptz` column truncates in the
session's timezone, not UTC.** Even after fixing Bug 1, the test still
failed: a customer created "now" (`datetime.now(UTC)`) was bucketed into
*yesterday's* date. This database's sessions default to
`Africa/Nairobi` (UTC+3, matching `Organization.timezone`'s own
default), and PostgreSQL's 2-argument `date_trunc(field, timestamptz)`
truncates using that session `TimeZone` setting -- silently converting
to local time before truncating, then returning the result still tagged
`timestamptz`. A customer created at, say, 01:00 UTC on the 21st is
04:00 local time on the 21st, which truncates correctly to the 21st --
but a customer created at 22:00 UTC on the 20th is 01:00 local time on
the *21st*, truncating to the 21st when it should bucket as the 20th in
UTC terms, or vice versa depending on which side of the offset the
creation time falls. Fixed with PostgreSQL's 3-argument form,
`date_trunc('day', Customer.created_at, 'UTC')`, which pins the
truncation to UTC regardless of the session's `TimeZone` setting --
matching every other timestamp already stored and returned as UTC
throughout this codebase. Both cutoff computation (`oldest_included_day`)
and the bucketing query now agree on the same UTC calendar-day boundary.

### Verification note

Live-verified against a disposable workspace seeded with one order in
KES and one in USD, plus a customer created moments before the request.
Direct `curl` calls to both endpoints returned exactly the seeded
values -- `orders_revenue_by_currency` showed `{"KES": "2500.0000",
"USD": "75.0000"}`, never a combined total, and the trend correctly
bucketed the new customer to today's UTC date. The desktop UI rendered
identical values, confirmed both via screenshot (KPI tiles) and
extracted page text (the currency and trend panels, which sat below the
visible viewport in this session's fixed-height automation browser --
the same known sandbox-scrolling limitation V4.1's own verification note
already documented, not a product defect).

## Platform Admin V5.3: Workspace Health

Second Platform Intelligence unit.

### A deliberate naming decision, not an accident

The roadmap's own name for this unit is "Customer Health." Implementing
it under that literal name would have created a genuine, damaging
ambiguity: the `Customer` model already means "a workspace's own
end-customer" everywhere else in this codebase (Inbox, Orders, Customer
Detail, the Dashboard, V5.1's own `total_customers` field). This unit is
about something entirely different -- the health of a workspace *as one
of Arche Axon's own paying customers*. Naming the permission
`platform.customer_health.read` and the desktop page "Customer Health"
would have made every future reader guess which meaning applied at each
call site. Renamed to "workspace health" at the permission key, schema,
endpoint path, and desktop copy level, all at once, before any of it
shipped -- not a case of "customer" being cleaned up later after
confusion actually occurred.

### Thresholds, not a score

`_derive_health_status()` evaluates a fixed, documented order and stops
at the first match:

```text
1. subscription_status in {cancelled, read_only}        -> at_risk
2. no message activity ever, or 30+ days inactive        -> at_risk
3. subscription_status in {past_due, grace, pending_setup} -> needs_attention
4. 14-29 days inactive                                    -> needs_attention
5. otherwise                                              -> healthy
```

This is deliberately not a weighted formula, a 0-100 score, or anything
resembling anomaly detection -- there is no principled basis in this
codebase for weighting "12 days inactive" against "past_due for 3 days"
against each other, and inventing weights to produce a single number
would be exactly the kind of fabrication this program's engineering-
integrity rule forbids. Two named `ClassVar` threshold sets
(`[redacted] = 30`,
`[redacted] = 14`) and two named
`ClassVar` status sets are the entire rule -- matching the exact
threshold-not-score convention Security Center's attention items and
V4.4's broadcast-safety thresholds already established, now applied to a
third domain.

Billing status is checked before activity, not the other way around and
not merged into one combined signal: a workspace that is both
billing-lapsed and dormant is reported `at_risk` for the billing reason
specifically (it would have been `at_risk` from the activity check too,
but the ordering means the billing signal is never silently overwritten
by whichever check happens to run second).

### Admin-console-scale, not built to scale past today's volume

`list_workspace_health()` fetches every workspace's health row into
memory (`_workspace_health_rows()`), then filters, searches, sorts, and
paginates in Python -- because `health_status` is a Python-computed
field, not a database column, filtering it with SQL `WHERE` would
require either a computed column or duplicating the threshold logic in
raw SQL. This is the same "admin-console-scale operation, not built to
scale past the volume of connections a real deployment has today"
precedent V4.1's own cross-channel merge already established for an
analogous problem (there, merging three connection tables in Python
rather than a SQL `UNION`).

### Verification note

Live-verified against a disposable local database: one workspace with
an active subscription and a message 2 hours old (correctly `healthy`),
one workspace with a cancelled subscription and zero activity
(correctly `at_risk`). The real production `arche-axon` workspace was
also visible in the same query and correctly, honestly reported
`at_risk` -- its subscription status is `pending_setup` with no recorded
message activity yet, a genuine fact about real data surfaced by this
feature working correctly, not a test artifact or a bug to explain away.

## Platform Admin V5.5: Support Cases

Third Platform Intelligence unit. A cross-tenant drill-down over
individual support tickets -- the natural next step after V5.1's own
basic open/resolved counts and V5.3's per-workspace `open_ticket_count`,
neither of which lets an operator browse or filter individual tickets
across every workspace.

### Metadata yes, content no

`PlatformSupportCaseRead` deliberately never includes `Ticket.title` or
`Ticket.description`. Both may carry customer-specific narrative content
authored by a workspace's own support agents -- this is a cross-tenant
admin surface, and exposing one workspace's support-ticket content to a
platform administrator working on an unrelated workspace would violate
this program's tenant-isolation principle. This is the exact same
boundary `PlatformAiExecutionRead` (V4.9) already drew around
`MessageDraft.body`, applied here to a second content-bearing model.
Every other metadata field (status, priority, category, source, age,
overdue flag, timestamps, and the owning workspace's name/slug) is
included. Confirmed with a dedicated test asserting `"title"` and
`"description"` never appear in the response JSON at all, not just that
the desktop table doesn't render them -- the same discipline V4.9's own
`[redacted]` already
established for this class of test.

### Real columns get real SQL, not V5.3's in-memory pattern

V5.3's `list_workspace_health()` filters/sorts/paginates in Python
because `health_status` is a Python-computed field with no backing
column. Support Cases is different: `Ticket.status`, `.priority`,
`.category`, and `.organization_id` are real, already-existing database
columns, so `list_support_cases()` uses genuine SQL `WHERE` filtering
and `LIMIT`/`OFFSET` pagination, with a separate `COUNT` subquery for
`total` -- the same shape `list_ai_executions()` (V4.9) and
`list_channels()` (V4.1)'s per-channel branches already use for
column-backed filters. The choice was made by checking which fields are
real columns before writing the query, not by defaulting to whichever
pattern the most recently written sibling unit happened to use.

### Overdue is a threshold, not a score

`is_overdue` is `Ticket.due_at < now()` combined with a fixed, named
`_SUPPORT_CASE_OPEN_STATUSES` `ClassVar` set
(`open`/`in_progress`/`waiting_on_customer`/`reopened`) -- a resolved or
closed ticket past its original due date is not "overdue" in any
actionable sense, so the status check is a deliberate second condition,
not an oversight. This continues the same fixed-threshold-never-a-score
convention `_derive_health_status()` (V5.3) and V4.4's broadcast-safety
thresholds already established, now applied to a third domain.

### Verification note

Live-verified against a disposable local database and a running desktop
dev server via `agent-browser`: seeded a `support_admin`-promoted
disposable workspace with six tickets, one per real `Ticket` status
value, each carrying deliberately sensitive `title`/`description` text
(`"SENSITIVE-DO-NOT-LEAK: ..."`). A direct API call confirmed the raw
JSON response contains no `title`, `description`, or `SENSITIVE` text
anywhere. The desktop UI's KPI tiles and all six table rows matched the
seeded data exactly (6 total tickets, 1 open, 2 urgent, 4 overdue --
correctly excluding the resolved and closed tickets from the overdue
count even though both also had a past `due_at`), with the same
zero-leak property holding through the full rendered page text, not
just the raw API response. Switching to and from the pre-existing
Overview and Workspace Health views showed no regression, including the
real `arche-axon` workspace's already-correct `at_risk` Workspace Health
status from the V5.3 verification continuing to render unchanged.
Disposable workspace, tickets, and user were deleted afterward;
confirmed exactly one organization (`arche-axon`) remains.

## Platform Admin V5.4: Workspace Timeline

Fourth Platform Intelligence unit. A single-workspace detail endpoint --
a chronological narrative of one workspace's history -- sitting
alongside the existing V3 Part 1 diagnostics endpoint, which shows
current state rather than history.

### Renamed from "Company Timeline," matching V5.3's own precedent

The roadmap's own name for this unit is "Company Timeline." Nothing in
this codebase is ever called a "Company" -- every prior unit uses
"workspace"/"organization" consistently, including V5.3's own renamed
"Workspace Health." Shipping "Company Timeline" would have introduced a
one-off inconsistency in vocabulary that every other Platform Admin
surface avoids. Renamed to "Workspace Timeline" at the permission key,
schema names, endpoint path, and desktop copy level, all before any of
it shipped -- applying the exact naming-collision discipline V5.3
already established, this time for a stylistic inconsistency rather
than a genuine model-name collision.

### Three sources, one deliberately excluded

`workspace_timeline()` merges exactly three sources for one workspace:

```text
1. Organization.created_at    -> a synthetic "Workspace created" entry
2. BillingEvent (org-scoped)  -> Paystack webhook/billing state history,
                                  never exposed via Platform Admin before
3. PlatformAuditEvent
   (target_organization_id)   -> Vendale's own administrator actions
                                  against this workspace, already a safe
                                  cross-tenant surface since V3 Part 2
```

The tenant `AuditEvent` table is deliberately never read. Unlike
`BillingEvent` (payment state changes only -- no customer content) and
`PlatformAuditEvent` (Vendale's own administrators, not the tenant's
own staff), `AuditEvent.summary`/`.entity_label` are free text written
by a workspace's own agents describing their own customers, orders, and
tickets by name. Exposing that cross-tenant would violate the exact same
"metadata yes, content no" boundary `MessageDraft.body` (V4.9) and
`Ticket.title`/`.description` (V5.5) were already excluded for -- here
enforced even more simply, by never querying the table at all rather
than filtering fields out of a response schema.

### Admin-console-scale merge, applied to a single workspace

All `BillingEvent` and `PlatformAuditEvent` rows for the one requested
workspace are fetched in full (not paginated at the SQL level), merged
with the synthetic milestone entry into one Python list, sorted
descending by `occurred_at`, and paginated in memory. This is safe
because the candidate set is bounded by definition to a single
organization's own history, not a cross-tenant volume concern --
matching V4.1's channel merge and V5.3's health computation, but applied
here to a per-workspace detail view rather than a cross-tenant list.

### A stable-sort pitfall found while writing the test, not shipped

Postgres freezes `now()` at transaction start, not per-statement. Since
this test suite runs each test inside one wrapped transaction, a
`BillingEvent` and a `PlatformAuditEvent` both relying on their model's
`server_default=func.now()` would receive the *exact same* timestamp,
and Python's stable sort preserves original list order for ties --
meaning `entries.sort(key=..., reverse=True)` would NOT reliably put the
more-recently-added entry first when two rows tie, breaking any
ordering assertion. Fixed by passing explicit, Python-controlled
`created_at` values on both seeded rows in the test (never relying on
the server-side default for anything whose relative order the test
actually asserts) -- the general fix for any future test asserting
`occurred_at`-based ordering across two or more rows created in the same
transaction.

### Verification note

Live-verified against a disposable local database and a running desktop
dev server via `agent-browser`: seeded a `support_admin`-promoted
disposable workspace with two `BillingEvent` rows and one
`PlatformAuditEvent` row, each given explicit, distinct `created_at`
timestamps. A direct API call confirmed all four entries (the milestone
plus the three seeded rows) returned in the correct descending order,
with billing detail strings correctly formatted (e.g.
`"pending_setup -> active | 5000.00 KES"` from a stored
`amount_kobo=500000`) and the platform-admin entry's `actor_email`
correctly populated. The desktop's new "Workspace timeline" section,
added to the bottom of the existing Workspace Diagnostics page, rendered
all four entries identically. This session's client-side request
timeout was temporarily raised to 60 seconds to work around the
diagnostics page's already-documented Neon-latency artifact, then
confirmed reverted to 12000ms via a clean `git diff` before finishing.
Disposable workspace and user deleted afterward; confirmed exactly one
organization (`arche-axon`) remains.

## Request Correlation ID Middleware (2026-08-21)

Closes the request-correlation-ID gap that `PlatformAuditEvent.request_id`,
`/platform-admin/search`, and every platform-admin mutation's own before/
after-state documentation had flagged as unpopulated since V3 Part 1/V3
Part 2. This is the first cross-cutting Platform Completion Program item
(program brief section 1, "V4 cross-cutting") -- it touches the whole
backend request pipeline, not one Platform Admin sub-domain, so it was
built as its own module rather than folded into a specific V-numbered unit.

`RequestIDMiddleware` (`app/middleware/request_id.py`) is a
`BaseHTTPMiddleware` subclass, registered in `app/main.py` after
`CORSMiddleware` (Starlette applies added middleware in reverse
registration order, so this places it closer to the actual route handler
-- every response, including one CORS would reject, still gets a
resolvable `X-Request-ID`, matching the same "every response carries a
correlation ID" guarantee industry API gateways provide). A caller-supplied
`X-Request-ID` header is honoured only when it matches a conservative
allowlist (`^[A-Za-z0-9_.-]{1,100}$`); anything else -- oversized, containing
spaces/punctuation, or absent entirely -- gets a freshly generated UUID4
instead. This value flows directly into a persisted audit column and
potentially into log lines, so it is validated the same way any other
externally-supplied string reaching storage would be, not trusted verbatim.

Threading the resolved ID into `PlatformAuditEvent.request_id` required
touching every one of `PlatformAdminService`'s ~34 mutation method
signatures and 19 `AuditService.append_platform()` call sites (plus a
handful of private helper methods like `_record_workspace_lifecycle_audit()`
and `_record_job_action_audit()` that wrap `append_platform()` on behalf of
several public methods), alongside the 29 corresponding endpoint call sites
in `app/api/v1/endpoints/platform_admin.py`. Given the exact, uniform
`ip_address: str | None = None` / `user_agent: str | None = None` parameter
pattern (and the matching `ip_address=ip_address` / `user_agent=user_agent`
call-site pattern) already established by the V3.3c-1 auth-telemetry work,
this was done as a one-shot scripted regex substitution rather than ~50
individual manual edits -- verified safe only because every occurrence was
enumerated and diffed before running it, and CRLF line endings were
explicitly preserved on write (`newline="\r\n"`) so the diff stayed
addition-only with zero line-ending churn.

The blind substitution over-matched three call sites that share the exact
`ip_address=ip_address,` / `user_agent=user_agent,` two-line shape but call
a method that does not accept `request_id` at all: `StepUpService.create()`,
`MfaService.admin_reset()`, and one of five `AuthSecurityEventService.record()`
call sites. `mypy app scripts` caught all three immediately as
`call-arg` errors (`Unexpected keyword argument "request_id"`) -- exactly
the reason the per-module protocol requires a full mypy run before trusting
any mechanical refactor of this shape, not just the two files the change
was scoped to. Fixed by removing the erroneous `request_id=` line from each
of the three call sites; `create_step_up()` itself lost its `request_id`
parameter entirely (unlike the other two, it writes no `PlatformAuditEvent`
at all, so there was no legitimate destination for it in that method).
`AuthSecurityEvent` has no `request_id` column at all (it is a deliberately
separate ledger from `PlatformAuditEvent`, see the V3.3c-1 entry above) --
extending it would need its own migration and is out of scope for this
slice, which is deliberately narrow: `PlatformAuditEvent.request_id` only,
not a platform-wide request-ID propagation into every audit-adjacent table.

Deliberately excluded from this slice, tracked as natural follow-up rather
than silently bundled in or silently left undone: request-scoped structured
logging correlation (no logging filter/contextvar stamps `request_id` onto
log lines yet -- `app/core/logging.py`'s `RedactingFormatter` was not
touched), and a `/platform-admin/search` "request ID" category (the
endpoint's own documented gap is now half-closed: the column is populated,
but search does not index by it).

5 new unit tests (`tests/unit/test_request_id_middleware.py`, using
`httpx.AsyncClient` + `ASGITransport` rather than
`starlette.testclient.TestClient` -- this environment's installed
starlette version has deprecated `TestClient` in favor of an `httpx2`
package that is not installed here, discovered when the first version of
this test file failed to collect at all) and 2 new integration tests
(generated-ID and malformed-header-replacement cases); one existing
integration test (`[redacted]`)
updated from asserting `request_id is None` (the honest gap this slice
closes) to asserting the real echoed/persisted value. 1005 backend tests
passing, confirmed stable across two full-suite runs; `ruff check` clean;
`mypy app scripts` clean (same 4 pre-existing unrelated `facebook_page.py`
errors, untouched). No new migration -- `PlatformAuditEvent.request_id`
already existed as a column since V3 Part 2's `a7f2c9e1b4d6`.

## Platform Admin V5.7: One-click Diagnostics (2026-08-21)

Extends V3 Part 1's `/workspaces/{workspace_id}/diagnostics` with a
synthesized pass/warn/fail checklist and one overall verdict, closing the
gap that six separate panels (connections, webhooks, queues, automation,
provider errors, subscription) required an operator to visually correlate
before answering "is this workspace actually healthy."

`PlatformAdminService._build_diagnostic_checks()` is a pure function of
data `workspace_diagnostics()` already gathers for its existing response
fields -- it issues no new query and makes no new external call. Seven
checks: workspace access, subscription standing, channel connections,
webhook delivery, job queues, automation health, recent activity. Each
check's threshold reuses an existing constant rather than inventing a new
one: subscription/activity reuse V5.3's `_WORKSPACE_HEALTH_AT_RISK_*`/
`_WORKSPACE_HEALTH_NEEDS_ATTENTION_*` sets, webhook failure rate reuses
V4.4's `_HIGH_FAILURE_RATE_THRESHOLD` (0.10).

`_overall_diagnostic_status()` takes the worst of two independently
computed verdicts: V5.3's own `_derive_health_status()` (subscription and
activity only, unchanged) and the worst individual checklist item (which
additionally covers signals V5.3 never considers -- connection errors,
webhook failures, queue failures, automation failures). This is never a
fabricated blend; it is always the more severe of the two independently
valid answers, and it guarantees this endpoint's `overall_status` can
never disagree with the same workspace's Workspace Health row about the
one thing they both compute (subscription + activity).

A deliberate product decision made explicit before writing any check
logic: a workspace with no subscription record, or with zero channel
connections configured, is `pass`, not `warn`. Both are normal early
onboarding states. Treating them as warnings would make every brand-new,
not-yet-onboarded workspace read as `needs_attention` by default, which
would drown out genuinely concerning signals the checklist exists to
surface -- the same "don't flag what isn't actually wrong" discipline
this program has applied to every threshold-based feature since V4.4.

### A real, pre-existing desktop bug found and fixed along the way

While building the desktop UI for this checklist, tracing how V5.3's
`WorkspaceHealthPanel.tsx` renders its own `needs_attention` badge
(`statusTone(status)` returning the string `"warning"`, passed directly
into `<StatusBadge status={...} />`) surfaced a real defect:
`StatusBadge`'s `StatusVariant` union type had no `"warning"` key at all
-- only `"pending"`/`"waiting"`/`"paused"` mapped to the amber color
class. `StatusBadge`'s own fallback logic silently substitutes `"neutral"`
(gray) for any status string not present in `variantColorClass`, so every
`needs_attention` workspace-health row has been rendering as neutral gray
instead of amber warning color since V5.3 shipped, with no runtime error
to surface it. Fixed with a two-line addition (`"warning"` added to the
`StatusVariant` union and to `variantColorClass`, mapping to the existing
`crm-status-badge--warning` CSS class already used by the other
warning-toned statuses) -- this single fix corrects both V5.3's existing,
previously-unnoticed bug and enables this unit's own `checkTone()`/
`overallStatusTone()` helpers to render correctly, confirmed visually in
the live verification below (amber "Warn" badges rendered correctly for
subscription standing, job queues, and automation health).

### Verification note

Live-verified against a disposable local database and a running desktop
dev server via `agent-browser`: seeded a `support_admin`-promoted
workspace and a second target workspace with a WhatsApp connection in an
`error` state and a failed automation trigger event (no message activity,
no subscription record). The rendered checklist showed exactly the
expected per-check colors and details (workspace access: green pass;
subscription standing: amber warn, "pending_setup"; channel connections:
red fail, "1 of 1 connection(s) in an error state"; webhook delivery:
green pass, none recorded; job queues: amber warn, 1 failed; automation
health: amber warn, 1 trigger-queue failure; recent activity: red fail,
no activity recorded) with an overall red "At risk" badge -- confirming
both the synthesis logic and the `StatusBadge` fix rendered correctly
through the real UI, not just in isolated backend tests. The client-side
request timeout was temporarily raised to 60 seconds to work around the
already-documented Neon-latency artifact on this endpoint (V3 Part 1's
own ~18-sequential-query cost), confirmed reverted to 12000ms via a clean
`git diff` before finishing. Disposable workspaces and users deleted
afterward; confirmed exactly one organization (`arche-axon`) remains.

## Platform Admin V5: Incident Management (2026-09-02)

A derived, read-only incident view over failure data the platform already
records. The module's central design decision, made before writing any
code per the ledger's re-scoping rule: an "incident" is not an entity.
There is no incidents table, no state machine, no acknowledge/resolve
workflow. An incident is the grouping, computed at read time, of concrete
failure rows (signals) sharing the same kind, failure signature, and
workspace within a fixed 24-hour lookback window. Every number the
endpoints return is a property of existing rows, never a stored or
fabricated value.

### Signal sources and their exact predicates

Each of the three signal families reuses the failure predicate that
surface's own platform view already established -- incidents never define
a new definition of "failed":

- Failed queue jobs: `status == "failed"` on `automation_trigger_events`,
  `broadcast_dispatch_jobs`, and `media_ingestion_jobs` (V3.2/V4.5's
  queues), windowed by `created_at`. Each queue gets its own signal_type
  (`automation_job`/`broadcast_job`/`media_job`) so a signal names the
  surface that owns its remediation.
- Failed webhook events: `meta_webhook_events.processing_error IS NOT
  NULL` (V4.3's exact predicate), windowed by `created_at`. Rows with a
  NULL `organization_id` -- deliveries that never resolved to a tenant --
  are excluded: they stay visible in V4.3's cross-tenant inspector rather
  than being attributed to a fabricated per-workspace incident.
- Failed provider send attempts: `message_provider_attempts.succeeded IS
  FALSE` (V4.1's recent-errors predicate), windowed by `attempted_at`.

Clustering key: `kind:workspace_id:error_code`. Error summaries are
free-form text and deliberately never part of the key; the error code is
the reliable correlation field the write paths already record. Severity
is a fixed band (>= 10 signals critical, >= 3 warning, else elevated) --
class-level constants, mirroring V5.3/V5.7's derive-from-named-thresholds
precedent, never invented per request.

### Why there is no closed_count (and never will be)

An incident's signals are all open by construction, so the API reports
only `signal_count`. A "closed" tally would be fabricated in both
directions: a retried or cancelled job changes its status in place (no
job-history table exists, so there is nothing to count as closed), and
log-type signals (webhook events, provider attempts) have no mutable
status at all -- they simply age out of the window. Closing is therefore
observable only as absence: a retried job leaves the failed set, and the
next read no longer contains its incident. The lifecycle integration test
proves exactly this by retrying a job through the existing
`/jobs/{id}/retry` endpoint (platform.jobs.retry) and asserting the
signal disappears from the next read -- incidents gain no mutation
endpoints of their own, keeping every remediation action in its existing
audited surface (Jobs, Broadcasts, Channels) per V4.4's delegation rule.