Architecture

Phase 5 Desktop Persistence and Reliability

The desktop uses SQLite only for non-secret read-cache and offline mutation metadata. Tokens are never persisted in Zustand, Web Storage, or SQLite. Session restore remains cookie-only: the backend's httpOnly rt cookie is rotated by /auth/refresh, while the short-lived access token stays in module memory. A native Windows credential-manager bridge is available for future desktop secrets and has no plaintext fallback.

• tauri-plugin-sql (registered in desktop/src-tauri/src/lib.rs) is the only SQL access path; cache_entries stores last-known-good list responses and offline_mutation_queue stores request metadata/payloads for safe non-message writes. No token-bearing data is accepted by the queue.
• keyring with the windows-native backend exposes secure_credential_set/get/delete Tauri commands backed by Windows Credential Manager. secureCredentialStore.ts calls those commands directly and deliberately fails closed when the native bridge is unavailable; it never falls back to localStorage, sessionStorage, a file, or SQLite.
• Cache-then-network, always-refresh, no TTL: loadWithCache() always attempts the network fetch. A cached fallback is used only when the failure is classified offline (ApiError status 0, i.e. backend_unavailable or request_timeout); any real HTTP error, or an offline failure with no prior cached row, rethrows unmasked so existing error UI is unaffected.
• Only the most recent, unfiltered/default list view per entity is cached (one row per entity type in a single generic cache_entries table) — not per-filter-combination result sets.
• API-mode only. loadWithCache checks isApiMode and bypasses the database entirely in mock mode, so synthetic mock data is never persisted to disk.
• This module never imports getAccessToken/setAccessToken/authApiService — the access token remains module-memory-only exactly as before (see "Native API Authentication & Readiness Closure" below); nothing token-related is ever serialized into cache.db.
• A concurrency-safe global isShowingCachedData flag (derived from an internal staleEntities: Set<CacheEntityType> via markEntityCacheState()) drives a global OfflineBanner rendered in AuthenticatedLayout.tsx, so multiple simultaneous loadX() calls (e.g. Inbox loading both conversations and customers) cannot clobber each other's cache-fallback state.
• Offline replay is opt-in and currently covers customer profile PATCH updates. Each row has a unique idempotency key and expected version; transient network/5xx/429 failures use bounded exponential retry, while HTTP 409 version conflicts are terminal and retained for resolution. Outbound message paths are rejected, so this slice never starts customer-facing sends.
• Remaining out of scope: draft persistence and conversation message caching (only the conversation list itself is cached).

Phase 4C Media And Interactive Acceptance Notes

• Inbound WhatsApp button and list replies persist reply metadata and correlate back to the source outbound message, provider attempt, and BroadcastRecipient when the context provider message ID resolves inside the same organization and WhatsApp connection.
• Production media storage uses a private S3-compatible object-storage adapter with tenant-scoped opaque keys, safe API download responses, and no public object URLs. Local MinIO smoke evidence is development evidence only, not live production storage activation.
• Phase 4C native fake-provider acceptance is complete (2026-06-20). The full native acceptance script proved: outbound image send, inbound image/document/ audio/video ingestion, quarantine/recovery, duplicate webhook deduplication, quick-reply source-message correlation, BroadcastRecipient correlation, broadcast pause/resume/cancel lifecycle, media-header broadcast dispatch, mark-as-read provider call, restart/logout persistence, and browser-storage security checks through the Tauri UI.
• Full desktop monitoring states for media-header broadcast campaigns remain partially wired; the monitoring page shows status but not full media-asset preview. This is a UI polish gap, not an acceptance blocker.
• Live Meta connection, live media activation, and Phase 4D start remain blocked until explicit operator authorization.

Phase 4E Facebook Messenger Architecture

Phase 4E adds Facebook Page Messenger (inbound + outbound) alongside the existing WhatsApp Cloud and Instagram scaffolding, reusing the same webhook, identity, conversation, message, and provider-attempt infrastructure.

Backend components

Meta webhook POST /webhooks/meta/facebook_page
        |
        v
WebhookProcessingService._handle_facebook_page_delivery()
        |
        +-- FacebookPageConnection lookup by page id
        +-- CustomerIdentity: identity_type = "facebook_page", normalized_value = sender PSID
        +-- Conversation: channel = "facebook_page", facebook_page_connection_id populated
        +-- Message: direction = "inbound", provider = "facebook_page"

CRM reply POST /api/v1/conversations/{id}/messages
        |
        v
MessageService.send_message()
        |
        +-- channel == "facebook_page" && facebook_page_connection_id != null
        +-- settings.facebook_page_send_enabled must be true
        +-- Message provider = "facebook_page", simulated = false
        +-- _attempt_facebook_page_provider_send()
            +-- FacebookPageConnectionService.get_by_org_or_raise()
            +-- decrypt encrypted_page_access_token
            +-- resolve PSID from CustomerIdentity (identity_type = "facebook_page")
            +-- FacebookPageMessagingProvider.send_text()
            +-- FacebookGraphApiClient.send_text_message()
            +-- POST https://graph.facebook.com/v22.0/{page_id}/messages
            +-- record MessageProviderAttempt (success/failure)

Key patterns

• Separate live-send gate: META_FACEBOOK_PAGE_LIVE_MODE + META_FACEBOOK_PAGE_ALLOW_LIVE_SEND, independent from WhatsApp and Instagram gates.
• Encrypted token: FacebookPageConnection.encrypted_page_access_token uses the same AES-256-GCM credential encryption as WhatsAppConnection.
• No PSID exposure to frontend: the frontend sends {conversation_id, text}; the backend resolves the PSID server-side from CustomerIdentity.
• Idempotent webhook: MetaWebhookDelivery.payload_hash deduplicates identical payloads; MetaWebhookEvent.event_key deduplicates by event id.
• MessageProviderAttempt records every outbound send attempt with HTTP status, provider message id, error summary, and timing.

Frontend components

• desktop/src/types/domain.ts: Conversation.facebookPageConnectionId?: string
• [redacted].ts: maps facebook_page_connection_id from API response into frontend model.
• desktop/src/pages/InboxPage.tsx: ChannelBadge() renders Facebook Page badge for channel === "facebook_page"; composer uses channel-aware isChannelSendable() instead of a global LIVE_SEND_ENABLED=false constant.

Validation

• Live inbound: external Messenger message → webhook → backend processed with HTTP 200.
• Live outbound: CRM reply "Facebook CRM round-trip test" → Meta Graph API HTTP 200 OK → external recipient confirmed receipt.

Target Architecture

Windows desktop app
Tauri 2 + React + TypeScript
        |
        | HTTPS and WebSocket
        v
FastAPI backend
        |
        +-- PostgreSQL
        +-- Redis when needed
        +-- Object storage when needed
        +-- Meta WhatsApp Cloud API

Desktop

The desktop app is the first implementation target. It uses:

• Tauri 2
• React
• TypeScript
• Vite
• React Router
• Reusable accessible components
• Strict TypeScript

The desktop app must eventually produce a Windows installer such as .exe or .msi.

Backend

The backend is planned for a later phase and should use:

• Python
• FastAPI
• Pydantic
• SQLAlchemy
• Alembic
• PostgreSQL
• WebSockets
• Pytest

Every tenant-owned table must include organization_id.

WhatsApp Integration

WhatsApp integration will use the official Meta WhatsApp Cloud API in a later phase. Privileged Meta communication must pass through the backend.

Never embed these in the desktop executable:

• Meta access token
• Meta app secret
• Database password
• Webhook secret
• Private signing keys

Phase 4A Provider Pipeline

Phase 4A adds the backend provider baseline while preserving the desktop/backend security boundary:

• WhatsAppConnection stores organization-scoped WABA/phone identifiers and an encrypted Meta access token. Plaintext tokens are decrypted just in time for provider calls and are never returned to the desktop.
• MetaGraphApiClient is versioned by META_GRAPH_API_VERSION and is wrapped by MetaWhatsAppCloudProvider; mock and Meta providers share the MessagingProvider abstraction.
• MessageService.send_message() routes whatsapp_cloud conversations through the provider only when settings.provider_send_enabled is true.
• WHATSAPP_PROVIDER_TRANSPORT=fake selects a deterministic in-process fake HTTP transport in development only. Production startup rejects this setting.
• Dev-only acceptance endpoints can reset fake provider state, select fake scenarios, create a cloud conversation fixture, and create a low-permission viewer user. These endpoints require development mode and backend permissions.
• Webhook ingestion verifies X-Hub-Signature-256 over raw request bytes before processing inbound messages or status events.

Native Phase 4A acceptance drives the real Tauri UI against the real FastAPI endpoints while only the backend provider client receives deterministic fake Meta responses.

Phase 1 Architecture

Phase 1 is frontend-only plus the Tauri shell:

• Mock phone validation
• Mock OTP send success state
• Mock QR login state
• No network calls
• No database
• No local persistence beyond component state

Phase 2 Stage 2A Architecture

Stage 2A remains frontend-only and native-desktop-first:

• React Router guards protect /app/* routes.
• The existing login modal creates an in-memory mock session after OTP verification or simulated QR scan.
• Zustand holds the mock session, sidebar state, notification state, and global search state.
• Typed mock services live under desktop/src/services/ and return promises with short simulated delays.
• Mock data lives under desktop/src/data/; UI components call services/state rather than importing raw arrays directly.
• The authenticated shell lives beside the Phase 1 shell and includes full CRM navigation, top header, notification center, global search, sidebar collapse, and logout.
• No backend, database, SQLite cache, Redis, or Meta integration has been added.

Current frontend service boundaries:

authService
notificationService
searchService

Future stages should add services for broadcasts, tickets, team, settings, and analytics before implementing full module workflows.

Phase 2 Stage 2B Architecture

Stage 2B is still frontend-only and mock-service-driven. It adds the first coherent CRM vertical slice without changing the Tauri/backend boundary:

• desktop/src/types/domain.ts now models customers, tags, team members, conversations, assignments, messages, dashboard metrics, activity, filters, and toast state.
• desktop/src/data/mockData.ts now follows the minimal-fixture policy: only enough customers, conversations, messages, team members, tags, products, orders, and catalogues to cover required fields and UI states, plus deterministic fixture factories for pagination or load-testing needs.
• Zustand owns the shared in-memory data graph and exposes mutations for message sending, retries, conversation status, assignment, customer create/edit, notes, tags, archive/restore, and filters.
• Dashboard, Inbox, Customers, and Customer Detail pages consume store actions rather than importing mock arrays directly.
• Global search is generated from the current customer and conversation state so newly created or renamed customers are searchable immediately.
• UI feedback uses an in-memory toast host only; there is no persistence or outbound integration.

Current Stage 2B frontend service boundaries:

dashboardService
conversationService
messageService
customerService

No backend, database, SQLite cache, Redis, real OTP, real JWT, or Meta WhatsApp integration has been added.

Phase 2 Stage 2C Architecture

Stage 2C remains frontend-only and mock-service-driven. It adds the commerce slice without changing the Tauri/backend boundary:

• desktop/src/types/domain.ts models orders, order line items, order timelines, payment status, fulfillment status, products, product variants, product availability, catalogues, and commerce filters.
• orderService, productService, and catalogueService expose promise-based mock operations with short simulated delays.
• Zustand owns orders, products, catalogues, selected records, filters, loading states, errors, and success toasts alongside the existing dashboard, inbox, and customer state.
• Dashboard order metrics and recent orders derive from current in-memory order state.
• Customer detail reads real Stage 2C mock orders instead of placeholder order summaries.
• Global search records are generated from current customers, conversations, orders, products, and catalogues.
• Product/catalogue mutations keep membership state consistent in both modules; product rename/price changes update order item display.
• No backend, database, SQLite cache, Redis, real payment flow, Meta WhatsApp integration, or cloud service has been added.

Current Stage 2C frontend service boundaries:

orderService
productService
catalogueService

Phase 2 Stage 2D Architecture

Stage 2D remains frontend-only and mock-service-driven. It adds the communication and analytics slice:

• broadcastService, templateService, segmentService, ticketService, analyticsService expose promise-based mock operations.
• Zustand state expands to cover broadcasts, templates, segments, tickets, and analytics with 60+ actions.
• Dashboard, Inbox, CustomerDetail, and OrderDetail are cross-integrated with Stage 2D modules.
• Global search includes broadcasts, tickets, templates, and segments.

Phase 2 Stage 2E Architecture

Stage 2E remains frontend-only and mock-service-driven. It adds team management, roles, settings, and audit:

• teamService, roleService, invitationService, organizationSettingsService, userPreferencesService, auditService expose promise-based mock operations.
• desktop/src/utils/permissions.ts exports createPermissionChecker(role), PERMISSION_KEYS, PERMISSION_GROUPS, HIGH_RISK_PERMISSIONS, formatPermissionLabel.
• desktop/src/hooks/usePermissions.ts exports usePermissions() — a memoized hook that returns can(key: PermissionKeyLiteral) => boolean from the current user's role.
• [redacted].tsx renders an access-denied screen with optional requiredPermission label and a disclaimer.
• Route-level permission guards are applied to BroadcastsPage, TicketsPage, AnalyticsPage, TeamPage, RolesPage, SettingsPage, and AuditPage using const can = usePermissions(); if (!can("x.view")) return <AccessDenied ... />.
• appendAuditEvent is called after significant business mutations (role create/edit, member invite, settings save, order status, product stock, ticket assign/resolve, broadcast send/cancel).

> IMPORTANT: Client-side permissions are for interface modelling only and are not a security boundary. The future backend must enforce all authorization. can() on the desktop controls which UI elements render — it cannot prevent a determined user from calling store actions directly.

Current Stage 2E frontend service boundaries:

teamService
roleService
invitationService
organizationSettingsService
userPreferencesService
auditService

Phase 2 Stage 2F Architecture

Stage 2F remains frontend-only and mock-service-driven. It hardens the permission layer and settings without changing the Tauri/backend boundary:

• PERMISSION_KEYS expanded from 54 to 92 granular keys covering all module-level actions.
• All 5 mock role permission arrays (owner/admin/manager/agent/viewer) updated to include new keys.
• usePermissions() now returns { can, cannot, canAny, canAll, role, permissions } instead of a single function.
• PermissionGate component added — renders children when can(permission) is true, otherwise renders fallback (default: null).
• DisabledAction component added — renders a visually disabled button with title tooltip and aria-disabled when the user lacks a permission.
• AuthNavItem.permission field added to appConfig.ts; AuthenticatedSidebar filters nav items using can(item.permission).
• SettingsPage guards each section with can("settings.manage_X") and shows AccessDenied for inaccessible sections; nav links replaced with <button> elements that check isDirtyRef before navigating.
• UnsavedChangesDialog added inside SettingsPage with Stay / Discard / Save and continue actions.
• appendAuditEvent calls added to all previously missing store mutations.

> IMPORTANT: Client-side permissions are for interface modelling only and are not a security boundary. The future backend must enforce all authorization. can() on the desktop controls which UI elements render — it cannot prevent a determined user from calling store actions directly.

Current Stage 2F frontend service boundaries (no new services; hardening of existing permission and settings infrastructure):

permissions.ts (utils) — createPermissionChecker, PERMISSION_GROUPS, HIGH_RISK_PERMISSIONS
usePermissions.ts (hook) — { can, cannot, canAny, canAll, role, permissions }
PermissionGate / DisabledAction (components)
UnsavedChangesDialog (inline component in SettingsPage)

Future Backend Model Notes

The following domain models will require backend implementation when Phase 3 begins:

organizations          — multi-tenant root; every table scoped by organization_id
users                  — unified user record across org memberships
organization_members   — links users to organizations with a role
roles                  — per-organization role definitions
permissions            — role-permission join table
sessions               — JWT sessions with refresh tokens
audit_logs             — immutable append-only event ledger (server-enforced)
invitations            — pending email invitations with accept tokens (server-side expiry)

Audit integrity requires server-side append-only enforcement. Client-side appendAuditEvent is a design placeholder only.

Phase 3A Backend Architecture

Phase 3A implements the FastAPI backend under backend/. The desktop remains the primary UI; the backend is the real security boundary.

Stack

Python 3.12 + FastAPI
SQLAlchemy 2 async + asyncpg
Alembic — database migrations
PyJWT — HS256 access tokens
pwdlib[argon2] — Argon2 password hashing
pydantic-settings — env-based configuration
Ruff — linter + formatter
mypy strict — type checking
pytest-asyncio (auto mode) — async test runner

Token Architecture

Access token:
  - HS256 JWT, 15 min expiry
  - Returned in response body
  - Caller keeps in application memory ONLY (never localStorage/sessionStorage)
  - Cleared on application restart, but automatically re-minted from the
    refresh cookie on startup restore — no re-login required (see below)

Refresh token:
  - Opaque random bytes (secrets.token_urlsafe(48))
  - Only sha256 hex digest persisted in refresh_tokens table, alongside the
    organization_id it was issued for (added 2026-07-21 — see fix note below)
  - Raw token sent as httpOnly `rt` cookie
  - Cookie persistence is caller-controlled at login time via `remember_me`
    (added 2026-07-21): `true` (default) issues a 30-day persistent cookie;
    `false` issues a session cookie (no Max-Age/Expires) that the OS clears
    when the browser/WebView process fully exits. Rotation via `/refresh`
    preserves whichever policy was chosen at the original login — it does not
    silently upgrade a session-only login into a persistent one.
  - Token rotation: old revoked, new issued with same family_id
  - Reuse of revoked token → entire family revoked immediately (theft detection)
  - `POST /api/v1/auth/logout-all` revokes every active refresh token for the
    user (all devices/sessions), desktop-wired via Settings → Security →
    "Sign out all other sessions" (2026-07-21)

Fix note (2026-07-21): POST /api/v1/auth/refresh previously required a valid Authorization: Bearer access token in addition to the rt cookie (via a CurrentPrincipal dependency), which meant it could only rotate a token while already logged in — it could never actually restore a session once the access token was gone (app restart, or natural 15-minute expiry), contradicting the "Startup restore" contract documented below. Fixed: the endpoint now authenticates purely from the rt cookie; RefreshToken gained an organization_id column so a cookie-only restore can resolve the exact organization the token was issued for without needing a principal (a user can belong to more than one organization).

Principal

Every protected endpoint receives a Principal dataclass:

@dataclass
class Principal:
    user_id: uuid.UUID
    organization_id: uuid.UUID
    role_type: str
    permissions: list[str]
    jti: str

    def require(self, permission: str) -> None: ...     # raises HTTP 403
    def require_any(self, *permissions: str) -> None: ...

Endpoints call principal.require("permission.key") before any DB work.

Desktop Integration

VITE_AUTH_MODE=mock  — in-memory mock session (default)
VITE_AUTH_MODE=api   — calls FastAPI /api/v1/auth/*; token in memory

desktop/src/services/api/client.ts         — fetch client, token in module var
desktop/src/services/api/authApiService.ts — API-backed auth operations
desktop/src/services/authService.ts        — dispatches mock vs API

Audit Integrity

AuditEvent has no updated_at column and no update or delete API routes. Server-side enforcement (Phase 3A) replaces the client-side appendAuditEvent placeholder from Stage 2E/2F.

Phase 3C Backend Architecture

Phase 3C adds persistent commerce data — Orders, Products, Catalogues, and Inventory — without changing the Tauri/desktop application boundary.

Commerce Models

orders                 — order_number (ORD-YYYY-NNNNNN), status/payment/fulfillment with ClassVar
                         transition maps, Decimal pricing, optimistic version lock
order_items            — snapshot of product_name/sku/unit_price at time of order creation
order_number_counters  — per-org counter; SELECT ... FOR UPDATE prevents duplicate numbers
products               — sku (uppercase normalized), track_inventory, stock_quantity, availability
product_variants       — per-product variants with independent stock and pricing
inventory_movements    — append-only audit log for every stock change (delta, reason, before/after)
catalogues             — named product lists with status (active/archived), position-ordered membership
catalogue_products     — membership join: position, is_featured; unique (catalogue_id, product_id)

Key Patterns

Order number:   SELECT ... FOR UPDATE on order_number_counters; format ORD-{year}-{seq:06}
Snapshot:       OrderItem captures product_name, sku, unit_price at order time (not a FK-live price)
Decimal:        Numeric(19, 4) for all monetary columns; avoid float rounding in financial data
Transitions:    ClassVar dicts (STATUS_TRANSITIONS, PAYMENT_TRANSITIONS, FULFILLMENT_TRANSITIONS)
                tested directly on the class — SQLAlchemy metaclass prevents uninitialised ORM use
Identity map:   CatalogueService add_product uses SELECT products.id (not get_by_org_or_raise)
                to avoid loading Product into identity map before membership insert; expire_all()
                after commit before _reload() clears stale cached relationships

Desktop Commerce Integration

VITE_DATA_MODE=mock  — in-memory Zustand mock state (default)
VITE_DATA_MODE=api   — calls FastAPI commerce endpoints

[redacted].ts      — list, getById, create, update, setStatus,
                                                   setPaymentStatus, setFulfillmentStatus
[redacted].ts    — list, getById, create, update, adjustStock,
                                                   setAvailability, archive, restore, delete
[redacted].ts  — list, getById, create, update, archive,
                                                   restore, addProduct, removeProduct, reorderProducts

useAppStore.ts isApiMode branches cover all 20 commerce actions.

Phase 3D Backend Architecture

Phase 3D adds persistent Tickets, Broadcast Templates, Customer Segments, Broadcasts, and Analytics without changing the Tauri/desktop application boundary. Broadcast delivery remains simulated and deterministic — no Meta WhatsApp Cloud API calls are made (that is Phase 4 scope).

Communication & Analytics Models

tickets                 — TKT-YYYY-NNNNNN generation via SELECT FOR UPDATE on ticket_number_counters;
                         status transitions ([redacted]);
                         version lock; linked to customer/conversation/order
ticket_notes            — internal agent notes (body, author_user_id)
ticket_timeline         — append-only event log per ticket (event, detail, actor)
broadcast_templates     — reusable message templates with category ([redacted]),
                         language, variables (JSONB), archive/restore, duplicate
customer_segments       — JSONB rules with SEGMENT_ALLOWED_FIELDS and SEGMENT_ALLOWED_OPERATORS
                         allowlists (server-enforced validation); estimated_count via rule evaluation
broadcasts              — campaign records with audience_type ([redacted]),
                         simulated deterministic delivery (94% delivered, 61% read of recipient_count)
broadcast_recipients    — per-recipient delivery status (sent/delivered/read/failed timestamps)
broadcast_timeline      — append-only campaign event log

Key Patterns

Ticket number:    SELECT ... FOR UPDATE on ticket_number_counters; format TKT-{year}-{seq:06}
Segment rules:    JSONB with server-side allowlist validation (SEGMENT_ALLOWED_FIELDS,
                  SEGMENT_ALLOWED_OPERATORS) — client cannot inject arbitrary fields/operators
Broadcast send:  Deterministic simulation only — recipient_count, sent_count, delivered_count,
                  read_count computed server-side; no Meta API calls (Phase 4 scope)
_reload() helper: Re-fetch via repo.get_by_org_or_raise() after commit with selectinload to
                  pre-populate timeline/notes relationships — avoids MissingGreenlet in SQLAlchemy 2 async
expire_on_commit=False on both real and test sessions means explicit expire_all() is NOT used
                  (it was harmful: expiring freshly-loaded attributes triggered sync lazy-loads)

Desktop Stage 2D Integration

VITE_DATA_MODE=mock  — in-memory Zustand mock state (default)
VITE_DATA_MODE=api   — calls FastAPI Stage 2D endpoints

[redacted].ts     — list, getById, create, update, assign,
                                                    setStatus, addNote
[redacted].ts   — list, getById, create, update, archive,
                                                    restore, duplicate
[redacted].ts   — list, getById, create, update, delete
[redacted].ts  — list, getById, create, update, startSend,
                                                    completeSend, schedule, pause, resume, cancel, delete
[redacted].ts — getOverview (maps OverviewMetrics → analytics state)

useAppStore.ts isApiMode branches cover all Stage 2D actions (tickets, templates, segments,
broadcasts, analytics) following the same early-return pattern established in Phase 3B/3C.

Native API Authentication & Readiness Closure

Closes the Phase 3A desktop login-UI gap so the native desktop can establish a real backend session through its own UI, plus the /ready endpoint and the seed-script type-check gate. No Phase 4 functionality is added.

Mode boundary

A single runtime selector is the only place the UI reads the auth mode. Components never read import.meta.env directly.

desktop/src/services/api/authMode.ts
  isApiAuthMode / isMockAuthMode

VITE_AUTH_MODE=mock (default) → phone OTP / QR mock login (no network)
VITE_AUTH_MODE=api            → email + password login against FastAPI

API-mode login + session lifecycle

LoginModal (email form)
  → store.loginWithEmail
  → authService.loginWithEmail (dispatches to authApiService)
  → POST /api/v1/auth/login   (access token in body, rt cookie set)
  → setAccessToken (module memory only)
  → GET /api/v1/auth/me       (user/org/role/permissions)
  → AuthSession{ method:"email", permissions:[...] }
  → route guard opens /app/*

Startup restore (API mode only, runs once):

store.restoreSession (concurrent-guarded, no refresh loop)
  → authService.tryRestoreSession
  → POST /api/v1/auth/refresh (uses httpOnly rt cookie)
  → success → /auth/me → load app
  → failure → null/false → Login screen (never surfaced as authError)

Logout:

store.logout
  → authService.logout → POST /api/v1/auth/logout (revoke session, clear cookie)
  → clearAccessToken (memory) — unconditional, even if backend call fails
  → clearSensitiveData() — purge all API-backed records from active UI state
  → session undefined → Login screen

Key files:

desktop/src/services/api/authMode.ts       single mode boundary
desktop/src/services/api/authErrors.ts     classifyAuthError → 7 stable reason codes
desktop/src/services/api/authApiService.ts loginWithEmail, refresh, logout, getMe
desktop/src/services/api/client.ts         fetch client; access token in module memory; bounded timeout;
                                           BASE_URL normalized (strips trailing / and /api/v1) so the
                                           /api/v1 prefix — already on every service path — appears once
desktop/src/services/authService.ts        mock vs API dispatch; tryRestoreSession
desktop/src/components/LoginModal/...      email form (API) + phone/QR (mock)
desktop/src/state/useAppStore.ts           loginWithEmail, restoreSession, clearSensitiveData, logout
desktop/src/App.tsx                        one-shot startup restore + UI hold

Token storage policy (enforced)

Access token: module memory only — never localStorage, sessionStorage, IndexedDB, URL parameters, or logs. Refresh token: httpOnly rt cookie only; only the sha256 digest is persisted server-side. See docs/SECURITY.md.

Error mapping

classifyAuthError maps ApiError (and network TypeError) onto stable codes (invalid_credentials, authentication_required, session_expired, account_inactive, organization_access_denied, backend_unavailable, request_timeout, unknown_auth_error). authErrorMessage resolves each to a concrete sentence with no trace, status code, or credential detail. The store's runLogin catch uses this mapping for both the form and the restore path.

Liveness vs readiness (backend)

GET /health → process liveness (always 200 when responding)
GET /ready  → readiness; bounded SELECT 1 (2s); 200 {ready,ok} / 503 {not_ready,unavailable}

database_reachable is a module-scope dependency override target so the 503 path is tested without a live database. No credentials or traces are exposed.

Seed-script type-check gate

mypy app scripts now type-checks scripts/seed.py alongside the application. The seed uses concrete uuid.UUID types, TypedDict collections for order specs, and narrowly documented Decimal casts (SQLAlchemy Numeric is statically object). The prior runtime seed failure (model constructor field mismatch) is now caught at type-check time without weakening mypy globally.

Phase F — WhatsApp Flows (F1-F2)

Flows ride the same /{phone_number_id}/messages endpoint and the same interactive message type WhatsApp already uses for button/list replies, so this feature is additive to the existing provider architecture rather than a parallel system (unlike the forthcoming Instagram DM channel, which is a real architectural extension — see the SaaS Commercialization research log entry for that comparison).

Outbound send (F1)

FlowSendComponents (flow id, flow token, header/body/footer text, CTA text, flow_action of "navigate" or "data_exchange", initial screen, action payload) is a new dataclass in app/providers/base.py, alongside a new send_flow(...) method on the MessagingProvider Protocol. All three concrete providers implement it:

MetaGraphApiClient.send_flow_message()     builds interactive.flow payload on
                                            the existing _messages_url()
MetaWhatsAppCloudProvider.send_flow()      same 200/error-extraction pattern
                                            as send_template/send_media
MockWhatsAppProvider.send_flow()           wamid.mock.flow.{uuid} deterministic

runtime_fake.py gained a flow_send_success scenario for native desktop acceptance testing; the fake /messages handler already accepted any payload shape, so this only needed a distinct wamid.fake.vendale.flow.* id prefix for evidence readability, not new branching logic.

Authoring/publishing (F2)

WhatsAppFlowDefinition (app/models/whatsapp_flow.py) mirrors WhatsAppProviderTemplate's shape: an org-scoped local record wrapping a synced/published external Meta object, with a status lifecycle (draft → published → deprecated) enforced via a STATUS_TRANSITIONS ClassVar dict, matching the pattern already used by Broadcast/Ticket.

WhatsAppFlowService (app/services/whatsapp_flow_service.py):

create_draft()   local-only; no Meta call
update_draft()   draft-only (422 once published); bumps version
publish()        create_flow (if no provider_flow_id yet) → update_flow_json
                 → publish_flow; requires connection.status in
                 {"verified", "active"}, matching the existing template-sync gate
deprecate()      calls Meta's deprecate endpoint only when a provider_flow_id
                 exists (a never-published draft has nothing to deprecate remotely)

New Meta Flow management client methods on MetaGraphApiClient / MetaWhatsAppCloudProvider, using the same DI-transport pattern as every other provider client method in this codebase:

create_flow(waba_id, name, categories)      POST {waba_id}/flows
update_flow_json(flow_id, flow_json)        POST {flow_id}/assets (multipart FLOW_JSON)
get_flow / get_flow_status(flow_id)         GET  {flow_id}
publish_flow(flow_id)                       POST {flow_id}/publish
deprecate_flow(flow_id)                     POST {flow_id}/deprecate

See docs/RESEARCH_LOG.md (2026-07-31) for the official endpoint research. No API routes exist yet for this service — it is service-layer only in this slice; routes are deferred to F5 alongside the desktop UI that will call them. No live Meta Flow credentials or Flow-enabled WABA have been used; everything is verified against FakeHttpTransport (unit/integration tests) and RuntimeFakeHttpTransport (native desktop acceptance) only.

Encrypted data-exchange endpoint (F3)

app/security/flow_encryption.py is a direct protocol port (not a code merge) of Meta's official WhatsApp/WhatsApp-Flows-Tools Node.js reference ([redacted].js, MIT licensed):

decrypt_flow_request()   RSA-OAEP(SHA-256) unwraps the client's AES-128 key,
                          then AES-128-GCM decrypts the request body. Any
                          failure (wrong key, tampered ciphertext, wrong
                          keypair) raises FlowEndpointError(421, ...) — Meta's
                          documented signal telling the client to refresh its
                          cached public key.
encrypt_flow_response()  Re-encrypts the response with the SAME AES key but a
                          bitwise-flipped IV (Meta's documented scheme, not
                          an accidental quirk) and returns raw base64 TEXT —
                          not a JSON envelope; this is the entire HTTP body.
generate_flow_keypair()  RSA-2048, SPKI public key / PKCS8-encrypted private
                          key (PKCS8 substituted for the reference's PKCS1
                          since it's what `cryptography`'s
                          BestAvailableEncryption targets — only the public
                          key format and RSA-OAEP/AES-128-GCM wire scheme are
                          Meta-client-facing and must match exactly).

WhatsAppFlowKeyPair (app/models/whatsapp_flow.py) stores one RSA keypair per connection. The private key PEM and its passphrase are AES-256-GCM encrypted via the existing credential_encryption.py helper, reusing WHATSAPP_CREDENTIAL_ENCRYPTION_KEY (same secret class as the connection's access token) with distinct field_name bindings ("flow_private_key" / "flow_private_key_passphrase") — not a new master key, since only genuinely distinct product surfaces (WhatsApp vs. the AI gateway) get their own encryption key in this codebase. WhatsAppFlowKeyPairService.generate_for_connection() rotates in place (same row, new keys) rather than creating a new row per rotation.

app/api/whatsapp_flow_endpoint.py is a new, separate router (not folded into webhooks.py) because its response contract is fundamentally different: success returns raw base64 text, not JSON, and its error status codes (421/427/432) are Meta's own Flow-specific codes rather than this codebase's usual 400/403/404 conventions. Route: POST /webhooks/meta/whatsapp/flows/{connection_id}/data-exchange — public, outside the JWT chain, identified by connection UUID in the path because Meta's Flow client sends no bearer token and no organization context, only the encrypted payload (which cannot be parsed until the connection's private key is already known). Signature verification reuses verify_webhook_signature unmodified — the same X-Hub-Signature-256 HMAC-SHA256 scheme Meta uses for both the WhatsApp webhook and this endpoint.

[redacted].py::handle_flow_data_exchange ports flow.js's getNextScreen() action dispatch (ping health check, client error acknowledgement, INIT, data_exchange) but deliberately does not implement per-Flow custom screen logic — WhatsAppFlowDefinition.flow_json can define arbitrary org-authored screens, and mapping "handle this specific screen" to real business logic is deferred to F4/F5 once inbound nfm_reply correlation and the desktop authoring UI exist to define what that even means for a given org's Flow. An unhandled action raises UnhandledFlowActionError (mapped to a plain HTTP 500) — distinct from FlowEndpointError, since an unimplemented screen is this codebase's own gap, not a signal about the client's encryption key or flow_token.

No live Meta Flow-enabled WABA or real Flow client was used to verify this; the full round trip was verified against a simulated client built from the same cryptographic primitives (tests/unit/test_flow_encryption.py, [redacted].py), including negative cases.

Inbound Flow response + desktop send/render (F4-F5)

webhook_processing_service.py's inbound-message parser handles interactive.type == "nfm_reply" (Meta's WhatsApp Flow completion webhook shape) as its own branch, separate from button_reply/list_reply, because its payload is structurally different: nfm_reply.response_json is a JSON string (Flow-defined, arbitrary shape) rather than a flat title/id object. The string is parsed defensively — a malformed response_json still persists the message with an empty payload rather than dropping the webhook. No new correlation logic was needed: the existing generic _resolve_interactive_correlation() already keys off message_type == "interactive" and context_provider_message_id regardless of interactive subtype.

MessageService.send_flow_message() mirrors send_media_message()'s exact shape (require inbox.reply, require an active WhatsApp Cloud connection in provider mode, create the outbound Message row, then attempt the real provider send via _attempt_provider_flow_send()) and additionally requires the target WhatsAppFlowDefinition to already be status="published" with a provider_flow_id — a Flow cannot be sent until it exists on Meta's side. Exposed as POST /conversations/{id}/messages/flow.

ConversationRead.whatsapp_connection_id was added (purely additive) so the desktop can resolve which connection's Flows are sendable for the currently open conversation, without which the composer's Flow picker would have no way to filter to the right connection's published Flows.

Desktop: InboxPage.tsx's composer gained a "Send a Flow…" <select> alongside Attach/Suggest reply, populated from whatsappApiService.listFlows(connectionId) filtered to status === "published", and gated by the exact same LIVE_SEND_ENABLED compile-time constant that already gates every other outbound send — launching a Flow is still an outbound WhatsApp send, not a separate bypass path. FlowResponseSummary renders a completed Flow's response_json fields generically (key/value pairs, flow_token excluded), since Vendale cannot know an org's authored screen field names ahead of time; it reuses the existing interactive-message bubble rather than introducing a new message-type renderer. Settings' WhatsAppConnectionCard gained a "Flows" section (generate/rotate keypair, create draft, publish, deprecate) visible only for status === "active" connections, following the same inline-styled component shape as the adjacent template-sync section.

Phase I — Instagram DM Channel (I1-I2)

Instagram is a real architectural extension, not a small addition: every hardcoded "whatsapp_cloud" string, every phone_number_id/to_phone/ waba_id parameter baked into the MessagingProvider Protocol, and the hard FK from Conversation.whatsapp_connection_id/Message.provider_connection_id to whatsapp_connections.id block a second channel without changes. Per the locked architecture decision, Instagram gets a parallel connection model and separate Protocol — not a generalized multi-channel table/Protocol — so the live WhatsApp integration paying customers depend on is touched by zero lines of this work.

Connection model and provider

InstagramConnection (app/models/instagram.py) is a sibling to WhatsAppConnection, with its own INSTAGRAM_CONNECTION_STATUS_TRANSITIONS dict (not imported from whatsapp.py, so the two lifecycles can diverge independently later) and Instagram-shaped fields: ig_user_id (analogous to phone_number_id — identifies "us") and ig_username in place of waba_id/phone_number_id/phone_number.

InstagramGraphApiClient        reuses the existing HttpTransport Protocol
(instagram_client.py)          and FakeHttpTransport test double from
                                meta_client.py unmodified — only URL builders
                                and payload shapes differ:
                                POST https://graph.instagram.com/{v}/{IG_ID}/messages
                                {"recipient": {"id": IGSID}, "message": {"text": ...}}

InstagramMessagingProviderProtocol   a genuinely separate Protocol (not
(instagram_provider.py)              MessagingProvider) shaped around
                                      Instagram's actual parameters — IGSID
                                      recipient, no phone number, no WABA, no
                                      template system (IG DM has no
                                      marketing-template exception to the
                                      24-hour customer-service window)

MockInstagramProvider          deterministic ig_mid.mock.* fake, mirroring
(instagram_mock.py)            MockWhatsAppProvider's contract exactly

get_instagram_provider()       a NEW separate factory function
(instagram_factory.py)         (not an overload of the existing
                                get_provider(), which returns a different
                                Protocol type)

instagram_runtime_fake.py      development-only deterministic transport for
                                native desktop acceptance testing, mirroring
                                runtime_fake.py's pattern (send_success/
                                [redacted]
                                scenarios)

InstagramConnectionService ([redacted].py) mirrors WhatsAppConnectionService's full shape (CRUD, set_access_token, get_plaintext_token, set_status, verify) with one deliberate deviation: activate() replaces WhatsApp's subscribe() step, because Instagram Login app webhook subscription is configured once at the app level in the Meta developer console — there is no per-connection POST .../subscribed_apps call to make for Instagram, unlike WhatsApp's per-WABA subscription.

Token encryption reuses WHATSAPP_CREDENTIAL_ENCRYPTION_KEY — the same secret class as the WhatsApp connection token — with a distinct field_name="ig_access_token" AAD binding via the existing credential_encryption.py helper, rather than a new master key. This is consistent with the project's established rule that only genuinely distinct product surfaces (e.g. the Axon AI gateway, which has its own AI_CREDENTIAL_ENCRYPTION_KEY) get their own encryption key; Instagram and WhatsApp connection tokens are the same kind of secret (a per-connection Meta bearer token), just for different Meta products.

Config gates are deliberately separate from WhatsApp's: META_INSTAGRAM_LIVE_MODE / META_INSTAGRAM_ALLOW_LIVE_SEND / INSTAGRAM_PROVIDER_TRANSPORT / INSTAGRAM_GRAPH_API_VERSION, with the same production fail-closed rejection of INSTAGRAM_PROVIDER_TRANSPORT=fake that WhatsApp's transport setting already has. Instagram requires its own Meta App Review approval and go-live timeline, independent of WhatsApp's already-approved status — sharing a gate would incorrectly couple the two channels' live-activation readiness.

instagram_connection.view/.manage permission keys were added to the catalogue and flow through owner/admin/manager exactly like whatsapp_connection.* (no special-casing needed in MANAGER_KEYS's exclusion-prefix logic, since it excludes by prefix and Instagram's keys don't match any excluded prefix).

REST routes (GET/POST /instagram/connections, GET/PATCH/DELETE /instagram/connections/{id}, .../token, .../status, .../verify, .../activate) mirror WhatsApp's connection routes 1:1 except for the /subscribe → /activate naming change described above.

No live Instagram Business account or live credentials have been used for any of this; everything is verified against FakeHttpTransport (unit/ integration tests) or the MockInstagramProvider/instagram_mock channel only. See docs/RESEARCH_LOG.md (2026-07-31) for the official Instagram Messaging API research this was built against.

Parallel columns, webhook dispatch, and send path (I3-I5)

Three parallel nullable FK columns were added, each alongside an existing WhatsApp-shaped column that is left completely untouched: Conversation.instagram_connection_id (next to whatsapp_connection_id), Message.instagram_connection_id (next to provider_connection_id), and MessageProviderAttempt.instagram_connection_id (next to connection_id, which is FK'd to whatsapp_connections.id only — a third parallel column the plan's I3 bullet list didn't explicitly itemize but which the same "parallel FK, not shared FK" principle required once I5's send-attempt audit log needed somewhere to point). CONVERSATION_CHANNEL gained "instagram_dm". None of these migrations added an index — whatsapp_connection_id and provider_connection_id aren't indexed either (a precedent from the c2e7f3b5a1d9 migration), and the first draft of these migrations was caught by alembic check for adding indexes SQLAlchemy's own model definitions didn't produce.

WebhookProcessingService.handle_delivery() dispatches on payload.get("object") == "instagram" at the very top, before any of the existing WhatsApp-specific envelope parsing runs. Instagram's envelope is structurally different — entry[].messaging[] arrays of {sender, recipient, timestamp, message}, not WhatsApp's entry[].changes[].value.{messages[],metadata}, and there is no phone_number_id analogue inside the envelope (the receiving IG account's ID is entry[].id itself) — so sibling methods (_process_instagram_entries(), _handle_instagram_inbound_message(), _resolve_instagram_connection(), _resolve_or_create_instagram_customer(), [redacted]()) exist rather than reshaping Instagram's payload to fit the WhatsApp-shaped parser. One genuine wire-format difference required its own helper: Instagram webhook timestamps are Unix epoch milliseconds, not WhatsApp's epoch seconds, so _parse_unix_ts_ms() exists rather than reusing _parse_unix_ts().

Everything else about inbound processing is reused with zero changes: _resolve_or_create_customer()'s CustomerIdentity pattern works unmodified with identity_type="instagram_dm" (the unique constraint on (organization_id, identity_type, normalized_value) already supports a second identity type), and the same emit_safely(..., trigger_type="conversation_message_received") call fires from the Instagram path exactly as it already does from the WhatsApp path — no automation-engine changes were needed for a second channel.

New GET/POST /webhooks/meta/instagram routes (app/api/webhooks.py::meta_instagram_verify/meta_instagram_event) reuse verify_webhook_signature and the hub.challenge handshake with zero modification — Meta's HMAC scheme and one-verify-token-per-app model are identical for both products, so META_WEBHOOK_VERIFY_TOKEN and META_APP_SECRET are shared, not duplicated.

MessageService.send_message() gained is_instagram_dm, a sibling condition to the existing is_whatsapp_cloud (not a rename or in-place modification), gated by the new settings.instagram_provider_send_enabled. A new sibling method _attempt_instagram_provider_send() mirrors _attempt_provider_send()'s exact shape but resolves the recipient's Instagram-scoped ID (IGSID) via the customer's instagram_dm CustomerIdentity row (Instagram has no phone number to look up) rather than Customer.normalized_phone.

A dedicated test ([redacted] in [redacted].py) sends one WhatsApp webhook and one Instagram webhook through the same WebhookProcessingService instance in the same test and asserts both produce fully correct, mutually exclusive Message/Conversation rows — the concrete verification of the plan's stated goal that the parallel design produces zero WhatsApp regressions, not just an assumption.

Phase I closure — AI draft verification and desktop UI (I6-I7)

AiService._conversation_context()/generate_draft() and the generate_ai_draft automation action were verified, not rebuilt: both read Message/Conversation rows keyed purely by conversation_id/ organization_id/direction/body (or, for the automation action, event.entity_type/event.entity_id), with zero reference to channel, provider, or any WhatsApp-specific field. Three new tests in [redacted].py prove this concretely against a real instagram_dm conversation — direct generate_draft() call, the automation action triggered end-to-end through a real AutomationRule + emit() + AutomationRuntime.run_once(), and the internal-note-exclusion safety property — rather than resting on the code read alone. Zero changes were made to ai_service.py or automation_runtime.py.

Desktop: instagramApiService.ts mirrors whatsappApiService.ts's shape, with verify+activate replacing WhatsApp's verify+subscribe (no per-connection subscribe call exists for Instagram — see the I1-I2 connection-service note above). Settings gained an InstagramSettings component alongside the existing WhatsAppSettings, deliberately smaller since Instagram has no templates/Flows sub-features. Conversation.channel was newly exposed to the desktop domain type (it had not been surfaced at all before this) so Inbox rows could render a ChannelBadge — a pure rendering addition; the inbox itself needed no structural change since Conversation.channel already carried this information end-to-end. No Instagram icon exists in the installed lucide-react version (removed for trademark reasons); Camera is used instead for both the Settings section and the Inbox badge.

This closes Phase I (Instagram DM Channel) entirely: 725 backend tests, 549 desktop tests, ruff/mypy/tsc all clean, production build succeeds. No live Instagram Business account or live credentials were used anywhere in this phase.

Platform Admin V3.3c-1: Auth Security Telemetry

Adds a second, deliberately separate append-only ledger, AuthSecurityEvent, alongside the existing PlatformAuditEvent (platform administrative actions) and AuditEvent (tenant business events). The three ledgers answer different questions and are queried independently: AuditEvent for "what changed in this workspace," PlatformAuditEvent for "what did a platform administrator do," and AuthSecurityEvent for "who authenticated, and when." Security Center (Platform Admin) combines the latter two for its overview; nothing merges the three tables.

Event vocabulary is fixed and enforced at the service layer (AUTH_SECURITY_EVENT_TYPES in app/models/auth_security_event.py): auth.login.succeeded, auth.login.failed, auth.session.created, auth.session.revoked. auth.mfa.* and auth.account.locked are reserved for V3.3c-2, when real MFA and account-lockout behavior exist to emit them.

Recording is fail-open by design: AuthSecurityEventService.record() wraps its insert in self.db.begin_nested() (a SAVEPOINT), the exact pattern already established in automation_runtime.py for per-rule action isolation, and swallows any exception. A telemetry write failure must never break login, logout, or token refresh.

RefreshToken gained ip_address, user_agent, last_seen_at, and revoked_at. A token rotation (every /auth/refresh call) carries ip_address/user_agent forward from the row it replaces rather than re-capturing them, because rotation continues the same logical session; it does not emit a new auth.session.created event for the same reason. Only real login, registration, and (once built) MFA re-authentication start a new session.

AuthSecurityEvent.target_is_admin is stamped once at write time from the resolved or attempted account's is_superuser flag. This lets Security Center's admin-scoped rollups avoid a join back to users (whose is_superuser value may have changed since), and keeps the scope decision explicit: the ledger itself records every user's login/session activity system-wide, but Security Center's overview deliberately surfaces only the administrator-scoped subset, matching that console's existing scope (see docs/ROADMAP.md's V3.3c-1 entry for the full reasoning).

The "repeated failed authentication" rule is a fixed, named threshold over real counted rows (AUTH_REPEATED_FAILURE_THRESHOLD / AUTH_REPEATED_FAILURE_WINDOW_MINUTES in app/core/config.py), grouped by attempted_email within a rolling window. It is presented in both the API response and the desktop UI as a threshold being exceeded, never as anomaly detection or a risk score.

Retention: retention_expires_at is computed at insert time (AUTH_TELEMETRY_RETENTION_DAYS, default 90). AuthTelemetryRetentionService and [redacted].py mirror media_retention_service.py's lease/dry-run shape exactly, simplified because deleting an expired auth-telemetry row needs no storage-backend call and no audit trail of its own. The worker is not yet wired to a systemd unit or cron schedule; that activation step is a deliberate, separate follow-up.

User.mfa_enabled is a placeholder column only: it defaults False for everyone and nothing sets it yet. It exists now so Security Center can honestly report "0 administrators have MFA enabled" today without a future migration being needed once real enroll/verify behavior ships in V3.3c-2. The secret, enrollment timestamp, and recovery-code storage are deliberately deferred to that same migration rather than added inert here.

Platform Admin V3.3c-2: MFA + Privilege Protections

Builds real TOTP MFA, mandatory enrollment for privileged platform roles, step-up re-authentication for sensitive mutations, and session management on top of the V3.3c-1 telemetry foundation. User.mfa_enabled stops being a placeholder: MfaService now actually sets it.

Secrets abstraction (introduced ahead of need, not full KMS)

app/security/secrets_provider.py defines a SecretsProvider Protocol (encrypt(subject_id, field_name, plaintext) -> str, decrypt(subject_id, field_name, ciphertext) -> str) with one concrete implementation, EnvKeySecretsProvider (AES-256-GCM, wire format {key_version}:{nonce_b64}:{ciphertext_b64}, AAD binds subject_id and field_name so a ciphertext cannot be replayed against a different user or field). This is a deliberate seam, not full KMS integration: today get_mfa_secrets_provider() returns the env-key implementation, but every call site depends only on the Protocol, so a future KMS-backed provider (AWS KMS, GCP KMS, HashiCorp Vault) can be swapped in behind the same factory function without touching MfaService, the model, or any test that doesn't specifically test EnvKeySecretsProvider itself. This mirrors, but is intentionally separate from, credential_encryption.py (shaped around org/connection-scoped provider credentials, keyed by connection id) since MFA secrets are per-user, not per-tenant-connection.

MFA_CREDENTIAL_ENCRYPTION_KEY is its own key, unconditionally required at production startup and asserted to differ from WHATSAPP_CREDENTIAL_ENCRYPTION_KEY and AI_CREDENTIAL_ENCRYPTION_KEY (the same fail-fast pattern already established for the AI gateway key) -- distinct product surfaces get distinct key material.

TOTP primitives (no third-party dependency)

app/security/totp.py implements RFC 6238 TOTP directly on top of Python's standard-library hmac/hashlib (RFC 4226 HOTP dynamic truncation, 6 digits, 30-second period, base32 secret), verified against the official RFC 6238 Appendix B test vectors rather than trusting an unverified third-party package. verify_totp() allows a +/-1 step tolerance (90 seconds total) for clock skew and compares in constant time. provisioning_uri() builds the standard otpauth://totp/...?secret=...&issuer=Thread%20CRM URI that every authenticator app (Google Authenticator, Authy, 1Password) already knows how to parse from a QR code or manual entry.

Login-time state machine (no elevated state in the JWT)

AuthService.login() now returns a LoginResult with three possible outcomes instead of always minting tokens:

1. user.mfa_enabled=True           -> issue MfaChallengeSession(purpose="challenge")
2. user_requires_mandatory_mfa()   -> issue MfaChallengeSession(purpose="enroll")
3. neither                         -> issue real tokens immediately (unchanged)

MfaChallengeSession and StepUpSession (app/models/mfa.py) are DB-backed, short-lived, opaque-token-hash-only sessions -- the same pattern RefreshToken already established (only sha256(raw_token) is persisted; the raw token is handed to the caller once and never stored). This is a deliberate, non-negotiable design choice: nothing about "this login is mid-MFA-challenge" or "this caller recently re-verified their password" is ever encoded in the 15-minute access JWT. A compromised or leaked JWT before either gate resolves grants nothing; revoking access requires only deleting the DB row, not waiting for a token to expire.

resolve_mfa_challenge_session() / complete_mfa_login() on AuthService let POST /auth/mfa/login-enroll and POST /auth/mfa/login-verify (app/api/v1/endpoints/mfa.py) authenticate purely from the opaque mfa_session_token in the request body -- there is no Bearer token to check because none has been issued yet. MFA_MANDATORY_ROLE_TYPES (in app/services/mfa_service.py) currently covers platform_owner, platform_admin, security_admin; user_requires_mandatory_mfa() resolves a user's platform role (not their tenant role) since mandatory MFA in this slice is a platform-console protection, not a general tenant policy. Mandatory enrollment blocks reaching a session at all rather than silently downgrading permissions -- there is no partial-access state.

Step-up re-authentication

StepUpService.create() re-verifies the caller's own password and a live TOTP/recovery code, then issues a StepUpSession (default 10-minute TTL, STEP_UP_SESSION_TTL_MINUTES). Deliberately not single-use: within its TTL window a caller can complete more than one sensitive action without re-entering their password each time, treating the elevation as a bounded time budget for a short admin workflow rather than a one-shot token. Gated mutations (PATCH .../platform-role, POST .../mfa/reset, POST .../administrators/revoke-all-sessions) call StepUpService.require_valid(), which checks the token hash, expiry, and that it belongs to the calling actor_user_id -- a step-up token minted for one administrator cannot be handed to or reused by another.

Privilege-change hardening

PlatformAdminService.assign_platform_role() enforces this exact order: permission check (platform.roles.manage) -> last-owner-still-remains check -> step-up validity -> mandatory reason -> mutate -> audit with explicit before/after platform_role_type -> emit an AuthSecurityEvent where applicable. A downgrade (rank decrease) additionally revokes every existing session for the target user immediately after the role change commits; a promotion does not force a re-login, since forcing out a user mid-promotion has no security benefit and only interrupts their work.

_role_privilege_rank() (app/services/platform_admin_service.py) is an explicit ordinal (platform_owner=4 down to read_only_auditor=1, no platform role=0), deliberately not derived from comparing raw permission-key sets -- two disjoint specialist roles (e.g. billing_admin vs support_admin) aren't orderable by set comparison, but the product still needs a yes/no answer to "did this change increase or decrease privilege" for the session-revocation decision. role_type=None is rank 0 only when the target's is_superuser is False; a legacy full-access admin (is_superuser=True, no assigned platform role) ranks as 4 (platform_owner-equivalent), computed from the user's pre-mutation is_superuser value -- migrating a legacy admin to any assigned role is correctly treated as a downgrade, not a promotion from an artificially low baseline.

MFA administration (security_admin surface)

PlatformAdminService.admin_reset_mfa() (gated by platform.security.manage via RequirePlatformSecurityManage, plus step-up) destroys the target's encrypted secret and every recovery code, forcing fresh enrollment on their next mandatory-MFA login. The caller -- even a security_admin -- never sees the target's secret, QR code, or recovery codes at any point; the response confirms only that a reset occurred. There is no "view MFA status in detail" beyond the boolean/timestamp fields already on PlatformAdministratorRead.

Session management

POST /platform-admin/users/{id}/revoke-sessions (existing, reason-only, no step-up) and the new step-up-gated POST [redacted] are deliberately separate actions at different blast radii: the former targets one user, the latter -- explicitly documented as including the caller's own session -- revokes every active session for every platform administrator system-wide, reusing RefreshTokenRepository.revoke_all_for_user() in a loop rather than inventing a bulk-SQL variant, since the administrator population is small and this is an infrequent panic-button action, not a hot path. GET /users/{id}/sessions / DELETE /users/{id}/sessions/{session_id} (gated by platform.security.read / platform.sessions.revoke respectively) expose per-session ip_address/user_agent/last_seen_at -- the same columns V3.3c-1 added to RefreshToken -- with no is_current concept, since viewing another administrator's sessions has no "current session" frame of reference.

Security Center extensions

PlatformSecurityOverviewRead gained mfa_protected_administrators, administrators_without_mandatory_mfa, privilege_changes_24h. PlatformAdministratorRead gained mfa_enabled, mfa_enrolled_at, mfa_last_verified_at, active_session_count. New attention-panel items follow the same deterministic-threshold-only convention V3.3c-1 established (never framed as anomaly detection): CRITICAL when a platform_owner lacks MFA, WARNING at 5+ failed attempts against an administrator within 15 minutes (reusing V3.3c-1's existing rule, now administrator-scoped) and for each remaining legacy unrestricted administrator, INFO when an administrator holds an unusually high number of concurrent active sessions.

Desktop wiring

authApiService.loginWithEmail() returns a LoginOutcome union ({kind:"authenticated", session} or {kind:"mfa_required", mfaSessionToken, enrollmentRequired}) instead of always resolving a session; useAppStore's loginWithEmail action stores a mfaChallenge slice on the latter outcome rather than treating it as a login error, and LoginModal renders MfaEnrollmentScreen (QR via the new qrcode dependency, MIT-licensed, plus a manual setup key -- both are always shown together since not every authenticator flow benefits equally from a scanned QR) or MfaChallengeScreen in place of the credentials form while a challenge is pending. Settings gained a self-service MfaSelfServicePanel (enable/disable/regenerate recovery codes, API mode only). Platform Admin's role-assignment and MFA-reset/session-revocation actions each move through a local select-reason -> StepUpDialog (password

• code) -> mutate sequence before calling their respective endpoints, and

SecuritySection gained an admin-scoped SessionsDialog (view + revoke one or all sessions for a single administrator) alongside the new MFA-status table column and KPI tiles.

Deployment backlog (carried forward from V3.3c-1, still open)

The auth-telemetry retention worker ([redacted].py, introduced in V3.3c-1) is still not wired to any systemd unit, cron schedule, or other runtime supervisor. Nothing currently invokes it outside of tests. Without an operator wiring it up (mirroring the existing vendale-broadcast-worker.service / vendale-media-worker.service systemd unit pattern under infrastructure/systemd/), auth_security_events and now also mfa_challenge_sessions / step_up_sessions (both already expiry-bounded and cheap to prune, but currently never pruned) will grow indefinitely. This is tracked explicitly rather than silently left implicit.

V6.2 update: the unit now exists ([redacted].service) and the worker's per-pass body also prunes expired mfa_challenge_sessions/step_up_sessions via the new ShortLivedSessionRetentionService. The unit file is committed; activating it on the deployment host (cp + systemctl enable --now) remains an operator action by design -- the code side of the backlog item is closed.

Platform Admin V3.4: Feature Flags

Closes V3 (per the user's own stated completion criteria). A real, server-authoritative feature flag system, deliberately narrow: no A/B experiments, no conversion tracking or statistical significance, no arbitrary JSON remote configuration, no plan/pricing entitlements. Two tables: FeatureFlag (key, name, description, enabled, rollout_percentage, classification, status) and FeatureFlagTarget (feature_flag_id, target_type, target_id, enabled), with target types workspace, user, platform_role. Plan targeting was explicitly deferred until subscription pricing has a clean canonical plan identity.

Resolution order (permanent, must never be silently reordered)

FeatureFlagService._resolve() evaluates in this fixed order and stops at the first matching rule:

1. Explicit user override           (FeatureFlagTarget target_type="user")
2. Explicit workspace override      (FeatureFlagTarget target_type="workspace")
3. Platform-role override           (FeatureFlagTarget target_type="platform_role")
4. Percentage rollout                (deterministic hash bucket < rollout_percentage)
5. Global enabled/default state      (FeatureFlag.enabled)

A flag key that has never been created evaluates to enabled=True (reason="not_configured") -- the same fail-open convention billing enforcement established, so an unconfigured flag never silently disables a capability nobody has opted to gate yet. An archived flag always evaluates to enabled=False (reason="archived") regardless of its other fields: archiving is a deliberate retirement, not a no-op fallthrough.

Deterministic rollout, not randomness

def _rollout_bucket(flag_key: str, subject_id: str) -> int:
    digest = hashlib.sha256(f"{flag_key}:{subject_id}".encode()).hexdigest()
    return int(digest[:8], 16) % 100

Python's built-in hash() is deliberately not used: it is randomized per process via PYTHONHASHSEED for security reasons, so the same workspace/user could land in a different bucket on every request or after every restart, defeating the entire point of a gradual rollout. sha256 is stable across processes and time. The subject is the organization id (workspace) when one is present, falling back to the user id only when no organization context exists -- Vendale is B2B multi-tenant, so gradual rollout is primarily a per-workspace decision, not per-user.

Caching and invalidation

A small module-level (process-wide, not per-request-instance) cache holds every flag + its targets for 45 seconds (_CACHE_TTL_SECONDS), refreshed via .execution_options(populate_existing=True) so a freshly-queried FeatureFlag's targets relationship cannot serve a stale collection from SQLAlchemy's identity map after a target was just added or removed in the same process. invalidate_feature_flag_cache() is called after every mutation (create/update/target add/target remove), so a Platform Admin change is visible to the next evaluation almost immediately rather than waiting out the full TTL. If Vendale later runs many API replicas behind a load balancer, this in-process cache will need to move to a shared invalidation channel (Redis pub/sub or similar) so one replica's mutation invalidates every other replica's cache too -- not required at today's single-process scale, and deliberately not built ahead of that need.

Kill-switch classification

FeatureFlag.classification is "normal" (ordinary product rollout) or "operational" (emergency kill-switch, e.g. outbound_broadcasts, ai_tool_execution, new_automation_runs, desktop_auto_update). This is a label for Platform Admin's UI, not a different code path: both classifications resolve through the exact same 5-step order above. The one real flag wired end-to-end in this slice, new_automation_runs, is classified operational -- AutomationRuntime.run_once() evaluates it before _claim() on every poll and returns early (after committing, since run_once() always commits its own lease state) when the flag resolves false, so a platform administrator can halt every tenant's automation execution without touching a single tenant's own AutomationRule.status.

Frontend never decides entitlement

GET /auth/me resolves the full resolve_all_for_subject() map once at session bootstrap and returns it as MeResponse.feature_flags, carried into AuthSession.featureFlags on the desktop. Reading this map is UX only -- it renders or hides interface, and only ever an absent key means "no restriction configured," never "disabled." It is never itself an authorization boundary: AutomationsPage.tsx shows an informational banner when featureFlags.new_automation_runs === false, but the actual gate lives entirely server-side in AutomationRuntime.run_once(). Because the map is captured once at login, a flag flipped in Platform Admin does not live-push to an already-open desktop session; the next full session bootstrap (re-login, or a future startup-refresh cycle) picks it up. This is a known, deliberate property of the current bootstrap-only design, not a bug.

Permissions and audit

platform.feature_flags.manage (already reserved) is joined by a new platform.feature_flags.read (migration d3f7a2c5e819), granted to platform_owner, platform_admin, support_admin, read_only_auditor -- mirroring every other read/manage permission pair in the platform RBAC catalogue. Tenant admins have no path to touch global feature flags at all; this is a platform-only surface. Every mutation is written to the existing platform audit ledger with a fixed vocabulary: feature_flag.created, feature_flag.updated, feature_flag.target.added, feature_flag.target.removed, feature_flag.rollout.changed, feature_flag.archived -- update_flag() picks the most specific applicable action name (archival beats a rollout change, which beats a plain update) rather than always writing the generic updated event, so the audit trail reads as intent rather than a raw field diff. Every mutation requires a caller-supplied reason (10-500 chars), matching every other platform-admin action in this codebase; per the user's explicit instruction, none of this requires step-up re-authentication in this initial cut -- step-up remains reserved for flags that later get classified as security-sensitive or infrastructure-critical, not applied blanket to every harmless UI-rollout toggle.

Platform Admin UI

New [redacted].tsx: a table (Name / Classification / State / Rollout / Overrides / Actions) plus a detail dialog (key, global enable toggle, rollout percentage, a targets list with per-target enable/disable and an add-target form, created-by / last-changed-by / last-changed metadata). Toggling global state through a dangerous transition (0%→100%, OFF→ON globally, 100%→OFF) reuses the existing ReasonActionDialog with a dynamically built impact bullet list ("Currently: ..." / "After: ...") rather than a new confirmation component -- consistent with how every other Platform Admin destructive action in this codebase already confirms impact before mutating.

Validation

14 new unit tests (test_feature_flag_resolution.py, no DB: rollout-bucket determinism/range/variance, the full 5-step resolution order via direct calls into FeatureFlagService._resolve()); 23 new integration tests (test_feature_flags.py: CRUD, targets, permissions, the evaluate-preview endpoint, direct service-level resolution against a real database, /auth/me exposure, and AutomationRuntime kill-switch behavior). A real SQLAlchemy identity-map staleness bug was found and fixed while writing these tests: get_flag()/list_flags()/the cache loader all needed populate_existing=True, or a target added/removed in the same session would not appear in a subsequently-queried flag's targets collection. Desktop: AutomationsPage.tsx reads the flag and renders a warning banner only when explicitly false; live-verified end-to-end against a disposable local database and a running desktop dev server -- created an operational-classified new_automation_runs flag with global state OFF, confirmed the Automations page banner appeared, re-enabled the flag through Platform Admin, re-logged-in, and confirmed the banner disappeared.

Incidental fix found and corrected during this slice's visual verification

While verifying the Feature Flags UI, every Radix-portaled surface in the whole application (Dialog, DropdownMenu, Popover, Tooltip, Command) turned out to render with a fully transparent background -- confirmed by the user independently (the profile-menu dropdown and "many other dialog boxes"). Root cause: ThemeProvider applies its resolved CSS custom properties (--color-app-canvas, --color-border, etc., from themeTokens.ts/theme-tokens-dark.ts) only via an inline style prop on an in-tree <div className="crm-theme-root">. Radix's Portal primitive renders Content directly into document.body by default, which sits outside that div in the real DOM -- CSS custom properties inherit through the DOM tree, not the React tree, so every portaled node saw those variables as undefined and fell back to background-color's transparent initial value. Fixed by also mirroring the same variables onto document.documentElement in a useEffect keyed on the resolved light/dark scheme, since document.body is always a descendant of document.documentElement; the existing in-tree div is left unchanged so nothing about non-portaled rendering changes. Verified visually (profile dropdown, the Feature Flag create dialog, and a DataTable row-actions dropdown menu all now render fully opaque) and confirmed via the full desktop test suite passing unchanged.

Platform Admin V4.1: Channel Control Center

First V4 increment (the WhatsApp/Channel track, chosen explicitly by the operator over the AI Control Plane track). Read-only, cross-tenant drill-down over every WhatsAppConnection/InstagramConnection/ FacebookPageConnection row, closing the gap between /runtime's system-wide aggregate counts and /workspaces/{id}/diagnostics's per-workspace detail: neither previously let an operator browse every connection across every workspace at once.

PlatformAdminService.list_channels() deliberately does not use a dict-of-type[...] dispatch table to pick which connection model to query per channel (an earlier draft did, and mypy correctly flagged it: the dict's value type widens to type[Base], losing the concrete WhatsAppConnection/InstagramConnection/FacebookPageConnection type needed for select(model, ...) and self.db.get(model, id)). Rewritten as three explicit _fetch_whatsapp_rows()/_fetch_instagram_rows()/ _fetch_facebook_rows() methods instead, matching this file's own existing precedent for the identical problem (list_jobs()'s three _list_automation_jobs()/_list_broadcast_jobs()/_list_media_jobs() methods) rather than introducing a new pattern.

Message traffic per connection reuses the parallel FK columns Phase I established (Message.provider_connection_id / .instagram_connection_id / .facebook_page_connection_id); failed-attempt counts do the same via MessageProviderAttempt.connection_id / .instagram_connection_id -- except Facebook, where MessageProviderAttempt has no per-connection FK at all (a pre-existing Phase 4E gap, not introduced here), so Facebook falls back to an organization+provider-scoped count. Documented as a known precision limit in docs/API_CONTRACT.md rather than silently presented as exact.

platform.channels.read (migration e6a3f8c2d914) gates all three new routes -- a deliberate departure from the V3.3a-era precedent of leaving broad read/list endpoints on the coarse is_superuser gate alone. Since this is new work being added after the platform RBAC system already exists, it gets a real permission from the start rather than joining that documented, intentionally-scoped gap.

Desktop: ChannelsSection.tsx and ChannelDetailDialog.tsx follow the established JobsSection.tsx/SessionsDialog.tsx shapes exactly (KPI tiles, a channel/status/search filter toolbar, a DataTable, and a read-only detail dialog fetched on row-action click) -- no new UI pattern was introduced.

Verification note: uncommitted V3.3a-V3.4 work landed first

This session found the fully-implemented V3.3a-V3.4 work (Platform RBAC, Security Center, auth telemetry, MFA/step-up, Feature Flags) sitting uncommitted in the working tree, matching what docs/ROADMAP.md already described as complete but never verified or committed. Running the full validation matrix against it (rather than trusting the documentation) surfaced one real defect: alembic check reported drift on four columns (feature_flags.key, platform_permissions.key, mfa_challenge_sessions.token_hash, step_up_sessions.token_hash) where the migration created a redundant UniqueConstraint plus a separate non-unique index instead of the single unique index the ORM models (mapped_column(..., unique=True, index=True)) actually produce. Since none of the four migrations had ever been applied to any shared database, they were corrected in place rather than patched with a follow-up migration, then re-verified (upgrade to head, alembic check clean, downgrade through the corrected chain, re-upgrade to head) before being committed as their own checkpoint ahead of any V4 work.

Platform Admin V4.2: WhatsApp Control Center

Second V4 increment, continuing the WhatsApp/Channel track. V4.1's generic 3-channel connection view already covers WABA/phone metadata, status, message traffic, and provider errors identically for WhatsApp/ Instagram/Facebook; the one genuinely WhatsApp-specific surface the V4.2 plan calls for that V4.1 doesn't touch is template visibility -- Vendale already syncs and persists every Meta-approved WhatsApp template per connection (WhatsAppProviderTemplate, built in Phase 4B), but nothing before this exposed that data cross-tenant.

PlatformAdminService.whatsapp_templates_summary() and list_whatsapp_templates() read status and quality_rating exactly as WhatsAppTemplateService stored them from Meta's own API response -- never normalized into a locally-invented vocabulary, matching the same "pass through the real value" principle list_jobs()'s per-queue status options already established. by_quality_rating's None bucket is labelled "unknown" in the response dict (a template awaiting Meta's quality assessment), not silently dropped or miscounted.

Both new routes (/channels/whatsapp/templates/summary, /channels/whatsapp/templates) had to be inserted before /channels/{channel}/{connection_id} in platform_admin.py's route declaration order -- not after, and not "it doesn't matter since they're different literal paths." Starlette matches routes in the order they were registered, not by specificity: a request for /channels/whatsapp/templates would otherwise match the parameterized {channel}/{connection_id} route first (channel="whatsapp", connection_id="templates"), which fails Pydantic's UUID validation and returns 422 rather than ever reaching the templates handler. This is the identical bug class the Phase F5 /flows/keypair fix caught (see that entry above) -- confirmed here by a dedicated regression test rather than by re-deriving the reasoning from scratch each time a new nested route is added under an existing parameterized prefix.

Desktop: ChannelsSection.tsx gained a view toggle ("Connections" | "WhatsApp Templates") rather than a new top-level Platform Admin nav item -- template visibility is a drill-down of the same Channels surface, not a separate console section. The existing connections-loading useEffect is guarded with if (view !== "connections") return, mirroring PlatformAdminPage.tsx's existing pattern for self-fetching sections, so switching to the Templates view doesn't also re-fetch connection data in the background. WhatsAppTemplatesPanel.tsx is a new, separate, self-fetching component (KPI tiles, status/category filters populated from the real values the summary endpoint returned rather than a hardcoded list, and a DataTable), following JobsSection.tsx's established shape.

Live verification note

Live-verifying this slice caught a seeding mistake, not a product bug: the first attempt to log in as the disposable support_admin test user showed no "Platform Admin" nav item at all. The nav item's visibility predicate checks session.user.isSuperuser, which is set by the router-level is_superuser boolean -- a separate DB column from platform_role_id, and the seed script had only set the latter. is_superuser=True is the coarse "has platform access at all" gate every platform route still requires first (V3.3a); a platform_role_id alone, without it, grants nothing. Fixed by setting both columns, matching how _assign_platform_role()'s existing backend test helper already does it correctly.

Platform Admin V4.3: Webhook/Event Inspector

Third V4 increment, completing the read-visibility half of the WhatsApp/ Channel track (V4.4 Broadcast Safety and V4.5 Operational Freeze remain, both of which mutate state rather than just displaying it). Built around MetaWebhookEvent, not MetaWebhookDelivery, for a concrete reason: neither model stores a channel column, and MetaWebhookDelivery.connection_id only FKs whatsapp_connections -- Instagram and Facebook Page deliveries always leave it null and reuse phone_number_id as a generic "receiving account id" slot (confirmed by reading webhook_processing_service.py's three delivery-creation call sites directly, not assumed). The one reliable per-row channel signal is each event's own event_key prefix ("message:"/"status:" for WhatsApp, "ig_message:" for Instagram, "fb_page_message:" for Facebook), so _webhook_event_channel() derives channel from that string prefix rather than trying to add a stored column purely for this admin view.

Reuses V4.1's platform.channels.read permission again (no new migration), continuing the established precedent that closely-related read-only visibility within the same Channel Control Center umbrella shares one permission rather than fragmenting into a new key per endpoint group.

Both new routes (/channels/webhooks/summary, /channels/webhooks) had to be declared before /channels/{channel}/{connection_id} in source, for the exact same reason as V4.2's templates routes: Starlette matches by declaration order, and /channels/webhooks/summary is the same 2-segment shape as the parameterized detail route.

Test-isolation lesson: global cross-tenant counts need delta assertions

The first version of this slice's summary test asserted absolute counts (by_channel == {"whatsapp": 2, "instagram": 1, "facebook": 1}) and passed in isolation but failed when run as part of the full suite -- by_channel["facebook"] came back 12, not 1. Root cause: unlike V4.1/V4.2's summary tests (which count WhatsAppConnection/ InstagramConnection/FacebookPageConnection/WhatsAppProviderTemplate rows -- tables no other test file in this suite touches), many unrelated integration test files legitimately create MetaWebhookDelivery/ MetaWebhookEvent rows of their own (webhook processing tests, Instagram tests, Facebook Page tests), and this test suite does not give every test full transactional rollback isolation from every other test's committed rows. webhook_events_summary() is correctly built as a genuine cross-tenant global count (that is the whole point of the endpoint), which means it is fundamentally the wrong kind of assertion to pin to an absolute value in a shared, non-fully-isolated test database. Fixed by capturing a baseline summary before seeding and asserting the before/after delta instead -- the correct general pattern for testing any genuinely global aggregate endpoint added to this codebase in the future, not a one-off workaround. List-endpoint assertions were separately hardened by scoping every query to the test's own freshly-created workspace_id rather than relying on an assumed-clean global list.

Platform Admin V4.4: Broadcast Safety

Fourth V4 increment, and the first in the WhatsApp/Channel track to mutate tenant data rather than only display it -- V4.1-V4.3 were entirely read-only. Every mutation route (pause/resume/cancel on a single broadcast, and pause/resume on every currently-sending broadcast in a workspace) delegates to the existing BroadcastService (Phase 4B) -- the exact code path a tenant admin's own broadcast controls already call -- rather than manipulating BroadcastDispatchJob rows directly or re-implementing BroadcastService's state-machine validation. This mirrors the standing principle the program brief itself states in section 5 ("Platform Owner is powerful, not invisible") and matches the precedent V3.2's job actions already established: retry_job()/cancel_job() explicitly refuse queue="broadcast" with a 422 pointing callers at the parent Broadcast's own pause/resume/cancel, specifically to avoid a raw per-job mutation bypassing the recipient-status/count-aggregate sync BroadcastService already provides.

pause_workspace_broadcasts()/resume_workspace_broadcasts() are a line-for-line structural mirror of V3.2's pause_workspace_automations()/resume_workspace_automations(): pause records the exact set of broadcast ids it paused in the audit event's after_state, and resume looks up that same audit event to restore only broadcasts still in that set and still paused -- a broadcast a tenant admin has independently cancelled or otherwise changed since the platform pause is never silently overridden. A dedicated test proves this concretely (pausing two broadcasts, a tenant "cancelling" one directly, then confirming resume only touches the other).

_LARGE_BROADCAST_RECIPIENT_THRESHOLD (1000) and _HIGH_FAILURE_RATE_THRESHOLD (0.10) are fixed ClassVar constants on PlatformAdminService, echoed back in every summary response rather than hardcoded a second time in the desktop UI -- the same anti-duplication discipline the program brief's threshold-transparency requirement calls for. An outbound_broadcasts feature-flag kill switch (V3.4 pattern, fail-open) gates BroadcastDispatchService.run_once() exactly the way new_automation_runs already gates AutomationRuntime.run_once(): a platform administrator can halt every workspace's broadcast dispatch in an emergency without touching individual broadcasts, and it is inert until deliberately configured.

Test-isolation lesson #2: a service's own internal commit() can leak test state that db.flush() never would

Found and fixed during this slice's validation, distinct from V4.3's "global aggregate needs delta assertions" lesson (same root class -- committed-past-rollback test state -- different trigger). Several new tests set admin_user.is_superuser = True directly (not via _assign_platform_role()) and then called a real pause/resume/cancel endpoint. BroadcastService.pause()/.resume()/.cancel() each call self.db.commit() internally (required so their own state-machine changes are durable even if a later step in the same request fails) -- and since this test suite's db fixture and the app's request-scoped session are the same object, that internal commit also permanently persists the test's own is_superuser = True mutation, defeating the per-test rollback boundary the db fixture normally provides.

This is the exact same leak class test_feature_flags.py's _revoke_leaked_platform_role() already documents for AutomationRuntime.run_once()'s internal commit (see the V3.4 entry above) -- confirmed by re-reading that existing helper's docstring rather than re-diagnosing from scratch, then adding the mirror-image _revoke_leaked_superuser() helper to test_platform_admin.py and calling it at the end of every test that grants superuser/platform-role access and then triggers a self-committing mutation. The standing rule going forward: any test that grants is_superuser/platform_role_id and then calls an endpoint whose service method commits internally must explicitly revoke that grant before the test ends -- db.flush() alone is not sufficient protection once a commit happens anywhere downstream in the same request.

Platform Admin V4.5: Operational Freeze

Fifth and final V4 increment, closing the WhatsApp/Channel Control Plane track. Unlike V4.1-V4.4, this slice began with an explicit design pass (required by V4.4's own ledger note) before any code was written, because "freeze a workspace's outbound activity" turned out to already have most of its infrastructure built -- the risk was building a second, redundant mechanism instead of recognizing that.

What "frozen" does and does not mean

Reading the actual worker code first (not assumed) settled three questions the design pass required:

• Organization.is_active = False (V3.2's workspace suspend) blocks interactive login and all authenticated API access, but never touches any background worker loop -- confirmed by grep, not inference. This remains true after V4.5; suspend and operational freeze are two independent controls, not layers of the same one.
• MediaIngestionService needed zero V4.5 changes. Its enqueue_for_asset() rejects any direction != "inbound" at the point a job would be created, which means "media ingestion is 100% inbound-only" is a structural property of the codebase already, not something V4.5 had to add a check to preserve. The one webhook-ingestion requirement the design pass carried forward from V4.4's own note ("must deliberately preserve inbound webhook ingestion") is satisfied by this pre-existing guarantee, not by new code.
• WebhookProcessingService.handle_delivery() is a synchronous, request-scoped call inside the webhook POST handler, not a worker loop polling a queue. There is nothing for a "freeze" to un-claim; it simply isn't in scope for this feature by construction.

Reused infrastructure, not a new mechanism

FeatureFlagService.evaluate(key, *, organization_id=...) (V3.4) already resolves a flag per-subject through its full 5-step precedence order, including the workspace-override step. V4.4 had already proven the pattern of a global operational kill switch checked once per run_once() poll (new_automation_runs, outbound_broadcasts). V4.5's entire implementation is: call that same evaluate() a second time, inside the claim loop, resolved with each individual item's own organization_id instead of no organization at all. No new flag resolution logic, no new caching layer, no new migration -- two existing flag keys (new_automation_runs, outbound_broadcasts) are reused exactly as-is, and one new key (workspace_outbound_messages) follows the identical pattern for the third category (outbound sends), which had no prior global kill switch to extend.

AutomationRuntime.run_once()
  global gate:  evaluate("new_automation_runs")                     [V3.4, unchanged]
  per-event:    evaluate("new_automation_runs", organization_id=event.organization_id)  [V4.5, new]

BroadcastDispatchService.run_once()
  global gate:  evaluate("outbound_broadcasts")                     [V4.4, unchanged]
  per-job:      evaluate("outbound_broadcasts", organization_id=job.organization_id)    [V4.5, new]

MessageService._is_outbound_frozen(organization_id)                 [V4.5, new -- no prior global gate existed]
  evaluate("workspace_outbound_messages", organization_id=organization_id)

A frozen automation event or broadcast job is fully un-claimed, not failed: status reverts to queued, any lease/heartbeat/owner fields are cleared, and available_at is pushed _WORKSPACE_FREEZE_REQUEUE_SECONDS (30s, a module constant in both automation_runtime.py and broadcast_dispatch_service.py) into the future -- short enough that unfreezing resumes delivery quickly, long enough to avoid a tight reclaim loop racing the feature-flag service's own 45-second in-process cache. The automation path additionally rolls back the attempts counter _claim() had just incremented (max(event.attempts - 1, 0)), since a freeze is not a failed attempt; the broadcast path needs no equivalent rollback because _claim_due_jobs() never increments attempts in the first place -- only _mark_job_failed() does, and a frozen job never reaches that code path.

A frozen outbound send is different in kind from a frozen queue item: it is not queued for later reconsideration, because send_message() has already committed to a synchronous request/response cycle by the time _attempt_provider_send() runs. Instead, _is_outbound_frozen() is checked before any connection lookup or credential decryption, and on a freeze the method records a real MessageProviderAttempt row (succeeded=false, error_code="workspace_outbound_frozen", error_summary="Outbound sends are temporarily frozen for this workspace by a platform administrator.") and marks the Message failed with the same stable code -- visible to the sending agent as an ordinary failed send, auditable via the attempt row, never silently swallowed. All three outbound channels (WhatsApp, Instagram, Facebook Page) get the identical check at the top of their respective _attempt_*_provider_send() method; WhatsApp's existing manual-retry path re-enters _attempt_provider_send() unchanged, so retry is automatically freeze-aware with zero additional code.

No new mutation endpoint

Freezing or unfreezing any of the three categories for a workspace is exactly the existing V3.4 target mechanism: POST /platform-admin/feature-flags/{id}/targets with {"target_type": "workspace", "target_id": <org id>, "enabled": false} to freeze, DELETE .../targets/{target_id} to unfreeze. The desktop's new OperationalFreezeSection.tsx (rendered on the existing WorkspaceDiagnosticsPage.tsx, directly beneath "Workspace actions") calls only pre-existing platformAdminApiService methods (listFeatureFlags, getFeatureFlag, createFeatureFlag, addFeatureFlagTarget, removeFeatureFlagTarget) -- zero new desktop service methods were needed. createFeatureFlag() is called only the first time a given key is frozen for any workspace (a platform admin may never have visited the Feature Flags screen for these three operational keys before); every subsequent freeze/unfreeze for any workspace reuses the same flag row. The flag's own global enabled field is never written by this panel -- only its per-workspace target.

Verification note

Live end-to-end verification surfaced two real, previously-undiscovered environment gaps, both fixed as part of this slice rather than worked around:

1. The shared local development database (vendale, distinct from the disposable thread_crm_test the pytest suite uses) was nine migrations behind head, missing users.platform_role_id entirely (registration and login both 500'd). This is exactly the gap flagged as a pending step in both the V3.3a and V3-Part-1 ledger entries. Resolved with alembic upgrade head; alembic check confirmed no drift afterward.
2. MFA_CREDENTIAL_ENCRYPTION_KEY (required since V3.3c-2) was present in neither backend/.env nor backend/.env.example. Because the startup fail-fast check for this key only runs when ENVIRONMENT=production, the gap was invisible until the first actual MFA enrollment attempt raised a ValueError deep in MfaService.start_enrollment(). Added a real local-dev value to .env and a FAKE...-prefixed placeholder to .env.example, mirroring the existing AI_CREDENTIAL_ENCRYPTION_KEY documentation pattern exactly.

With both fixed, a full real login -> mandatory TOTP MFA enrollment (live 6-digit codes computed with the backend's own app.security.totp module against the QR-code setup key actually rendered in the UI) -> MFA challenge on a subsequent login -> Platform Admin -> workspace diagnostics -> a real freeze -> a real unfreeze round trip was completed through the real desktop UI, the real backend, and the real Neon-hosted development database, with every state change independently confirmed via direct database queries (the FeatureFlag row, its FeatureFlagTarget, and two platform_audit_events rows).

Platform Admin V4.6: AI Provider/Model Registry

First increment of the AI Control Plane track (V4.6-V4.10). Deliberately the lightest-weight of the two possible starting points -- AiProviderConfig is a single table with a UniqueConstraint on organization_id (one row per workspace), unlike the three-table WhatsApp/Instagram/Facebook shape V4.1 had to dispatch across, so no _fetch_whatsapp_rows()-style per-type methods were needed here.

What a row means

AiService._get_or_create_config() (built in the SaaS Commercialization S3 phase) creates a row lazily the first time a workspace calls Copilot readiness or config -- not at organization creation, and not only when a workspace actually connects a provider. This means a status="not_configured" row is evidence of "this workspace has opened the Copilot page," not "this workspace has configured AI." The registry surfaces this honestly (see the desktop component's own docstring) rather than implying every workspace is tracked equally: most workspaces will have zero rows, and that absence is the correct default state, not a loading gap.

provider_type is "axon_gateway" for every row that has ever been created, since AiService.provision() is the only code path that ever sets a non-default value and it always sets this one string (see CLAUDE.md's SaaS Commercialization section: "AI through the existing Axon gateway rather than an LLM SDK"). The registry returns this field as real, currently-single-valued data rather than hardcoding a Literal["axon_gateway"] type -- if a second provider is ever integrated, the column and the API response already accommodate it with zero schema or contract change.

Deliberately excluded from this slice

Per-organization gateway usage/quota is real and already fetchable: AiService.account_usage(organization_id) calls AxonGatewayClient.get_account(api_key=...) using that org's own decrypted key, hitting the gateway's real /v1/account endpoint. This was deliberately NOT wired into V4.6's summary or list endpoint -- calling a live per-tenant gateway endpoint for every row in a cross-tenant list would be exactly the anti-pattern V4.1's own list/detail split already established (bulk queries stay local-DB-only; a live per-connection call is reserved for a single-record detail view). V4.7 (AI Usage Accounting) is the natural home for exposing account_usage() cross-tenant, most likely per-workspace on demand rather than eagerly across every configured org.

Verification note

Live-verified with two disposable organizations sharing one support_admin platform administrator (avoiding the V3.3c-2 mandatory-MFA gate that correctly applies to platform_owner/platform_admin/security_admin): one AiProviderConfig row seeded status="configured", enabled=true, the other status="error", enabled=false with a stored last_error_code. Both KPI tiles and both table rows matched the seeded data exactly through the real desktop UI; the status filter correctly isolated the error row; the detail endpoint was confirmed directly via the real API for both a real workspace id (full record returned) and an unknown UUID (404). Disposable workspaces and the user were deleted afterward; exactly one organization (arche-axon) remained.

Platform Admin V4.7: AI Usage Accounting

Second AI Control Plane increment. Two genuinely different signals, kept structurally separate rather than merged into one "usage" number, because they answer different questions and have different honesty guarantees.

A lower bound, stated as one

ai_usage_summary() aggregates the persisted message_drafts table (total_drafts, by_status, by_source, total_input_tokens, total_output_tokens, workspaces_with_drafts). This is real, accurate data about everything it counts -- but it is not total AI usage, because AiService.chat() (the interactive Copilot chat endpoint) never persists anything: no MessageDraft, no separate chat-log table, nothing. A tenant that only uses ad-hoc chat and never generates a reply draft shows zero here despite genuinely using AI. The schema docstring, the API contract entry, and the desktop panel's own visible description text all state this lower-bound property explicitly and identically -- this is deliberate redundancy, not accidental repetition, so no future reader of any one of the three sees an unqualified "usage" number.

The real answer, one workspace at a time

get_workspace_ai_usage() delegates directly to the existing tenant-facing AiService.account_usage(organization_id), which already calls the Axon gateway's real GET /v1/account using that workspace's own decrypted key -- built in the SaaS Commercialization S3 phase for the tenant's own Copilot status panel, reused here unmodified rather than reimplemented. This is genuinely complete (whatever the gateway reports is the real answer), but deliberately never called across a workspace list: doing so would mean one live external HTTP call per row in a cross-tenant summary, the exact anti-pattern V4.1's own connections list-vs-detail split already rejected for local data, now doubly true for a call that leaves this process entirely.

PlatformAdminService.get_workspace_ai_usage() takes an optional ai_service: AiService | None parameter -- not because production needs it (the real endpoint always omits it, letting AiService build its own gateway lazily), but because tests do, mirroring the exact pattern AiService.__init__ itself already established for its own gateway parameter. Without this, a test exercising the successful-response path would have no way to avoid AiService(self.db) building a real AxonGatewayClient() pointed at whatever AXON_GATEWAY_BASE_URL happens to be configured to.

The gateway's account response is returned as an untyped dict[str, Any] pass-through (PlatformAiWorkspaceUsageRead.account) rather than a typed schema, and the desktop dialog renders it generically as key/value pairs -- this codebase does not control or fully know the Axon gateway's response contract, so typing it would mean guessing at a shape that could silently drift. This mirrors FlowResponseSummary's existing precedent for the same problem (an org-authored WhatsApp Flow's response_json fields, whose shape this codebase also cannot know ahead of time).

A real bug found while wiring up live verification

Attempting to seed a realistic, encrypted AiProviderConfig.encrypted_api_key for live verification failed with WHATSAPP_CREDENTIAL_ENCRYPTION_KEY must be a 64-character hex string -- from the AI key, not the WhatsApp key, because credential_encryption.py's _derive_key() error message is hardcoded to name WhatsApp regardless of which caller invoked it (a separate, minor, pre-existing message-clarity gap, not fixed here since it is unrelated to this slice's scope). Investigating further: the local backend/.env's AI_CREDENTIAL_ENCRYPTION_KEY was 65 characters, not 64. Checking backend/.env.example found the identical defect in all three credential-key placeholders (WHATSAPP_CREDENTIAL_ENCRYPTION_KEY, AI_CREDENTIAL_ENCRYPTION_KEY, and this session's own V4.5-added MFA_CREDENTIAL_ENCRYPTION_KEY) -- each is the literal text FAKE followed by 61 repeated digits, and the letter K in FAKE is not a valid hexadecimal character at all, so binascii.unhexlify() would reject every one of them on both counts (wrong length and invalid characters) if anyone ever pasted the example value verbatim, despite the comment directly above each one stating "Must be a 64-character hex string." This had gone unnoticed because the production-only FATAL startup check (ENVIRONMENT == "production") never runs in local dev, and no prior session's live verification had exercised a code path that actually called _derive_key() on any of these three specific keys with their example value still in place. Fixed all three placeholders in .env.example to genuinely valid, obviously-fake 64-character hex values, and fixed the real local .env AI_CREDENTIAL_ENCRYPTION_KEY with an actual random 64-hex-char value so this session's own live verification could proceed.

Verification note

Live-verified with a deliberately fake (but correctly encrypted and decryptable) stored gateway key: the "View live usage" dialog correctly reached the real Axon gateway over the network, and the gateway correctly rejected the fake key, which the dialog displayed safely (API error 502: Invalid API key) rather than crashing, hanging, or leaking any credential material. This is a genuine, deliberately-sought proof of the error-handling path, not a fallback taken because the happy path could not be reached -- the same "no fabricated success" standard this program has applied throughout the WhatsApp/Channel and AI Control Plane tracks.

Platform Admin V4.8: AI Cost Control

Third AI Control Plane increment, and the first in this track to mutate tenant data rather than only display it.

Re-scoped from visibility to action, based on real evidence

V4.7's own research note flagged that account_usage()'s gateway response is an untyped pass-through with no fields this codebase could assume exist, and that a cross-tenant cost rollup would require one live HTTP call per workspace. Rather than guess at what a "cost control" feature should look like, the real Axon gateway source was read directly (C:\Projects\Axon\gateway\axon\routers\account.py, an external dependency, not this codebase, but read the same way any internal service would be before depending on its behavior) to see what fields GET /v1/account genuinely returns: wallet_balance_cents, wallet_spend_cap_cents, wallet_spent_this_month_cents, wallet_remaining_cap_cents, wallet_headroom_cents, token_quota_monthly, tokens_used_this_month, tokens_remaining, max_members. This confirmed, rather than merely assumed, that a cross-tenant cost dashboard is not honestly buildable in this slice -- every one of those fields lives entirely gateway-side, reading it for one workspace already costs a live external call, and this program's standing engineering-integrity rule (no fabricated metrics) rules out synthesizing a rollup from data this codebase does not hold. V4.8 was re-scoped from "show AI cost across every workspace" to "let a platform administrator pull an emergency lever for one workspace" -- a smaller, honest, immediately buildable capability using only data already local to this codebase (AiProviderConfig.enabled).

Delegation, not direct mutation

PlatformAdminService._apply_ai_enabled_action() is a private helper shared by disable_workspace_ai()/enable_workspace_ai(): look up the workspace (404 if unknown), read the current AiProviderConfig.enabled value if a row exists (for the audit before_state), call the existing tenant-facing AiService.update_config(workspace_id, CopilotConfigUpdate(enabled=...), actor_user_id=...) -- the exact method a tenant's own Settings AI toggle already calls -- then write a platform audit event and return the refreshed config detail. Neither public method touches AiProviderConfig directly; this is the same delegation-to-tenant-service discipline BroadcastService already established for V4.4's pause/resume/cancel actions, applied here to a different tenant service.

Because AiService.update_config() calls _get_or_create_config() internally (existing behavior, unmodified), disabling AI for a workspace that has never opened Copilot at all -- no AiProviderConfig row yet -- still succeeds: the row is created in a disabled state rather than the call 404ing. This was confirmed by reading update_config()'s own code first, then proven concretely with a dedicated test ([redacted]) rather than left as an inferred assumption.

Permission scope

platform.ai.manage (migration 4ae40777760f, is_high_risk=true) is a new, dedicated permission -- not a reuse of the existing read-only platform.ai.read -- granted to platform_owner, platform_admin, and support_admin only. This mirrors the exact three-role grant set platform.broadcasts.manage (V4.4) and platform.workspaces.manage already use for comparable mutation power, and deliberately excludes billing_admin, security_admin, and read_only_auditor -- none of whom hold comparable mutation permissions elsewhere in the catalogue either.

Verification note

Live-verified a full disable-then-enable round trip through the real desktop UI against a disposable workspace seeded with an enabled AiProviderConfig row: disabled with a reason, confirmed the row's status changed to "Disabled" and the row-action button states correctly reversed (Enable available, Disable/View-usage correctly disabled), re-enabled with a second reason, confirmed the row returned to "Enabled". A direct database query (not just the UI's own success toast) confirmed both ai.disabled and ai.enabled platform audit events were written with the correct actor, target workspace, and before/after {"enabled": ...} state -- the same "verify past the UI's own claim of success" discipline this program has applied to every prior mutation slice's live verification.

Platform Admin V4.9: AI Execution Log

Fourth AI Control Plane increment. Read the real code first, again: this time AiService.chat(), generate_draft(), and the generate_ai_draft automation action, to determine what execution-level data already exists before designing a new observability surface over it.

What the code already does, confirmed by reading it

AiService.chat() (the interactive Copilot chat endpoint) persists nothing at all -- no row, no log, no token count anywhere. It only updates AiProviderConfig.last_error_code/last_checked_at on failure and clears them on success. This is the exact gap V4.7's own "lower bound" framing already documented for its usage summary; V4.9 does not change it.

AiService.generate_draft() (called both directly by the Copilot "Suggest reply" button and, with source="automation", by the generate_ai_draft automation action) creates a MessageDraft row -- but only after a successful gateway response. A failed generation attempt (an AxonError) raises an HTTPException before the MessageDraft is ever constructed; only AiProviderConfig's single most-recent-error field reflects that failure occurred, not a per-attempt row. This means message_drafts is honestly a log of successful executions only, never a full execution history -- documented, not silently treated as complete.

An aggregate-then-drill-down progression, not a new mechanism

This is the same shape the WhatsApp/Channel Control Plane track already established: V4.1 gave a channel-summary aggregate, V4.3 gave the per-event drill-down over the same underlying data. V4.7 already built the aggregate (ai_usage_summary()); V4.9 is that track's drill-down -- list_ai_executions(), one row per MessageDraft, filterable by workspace/status/source, following list_ai_configs()'s own join-and-filter shape (MessageDraft joined to Organization, optional where filters, a COUNT subquery for total, then a paginated SELECT).

The one thing this view must never show

PlatformAiExecutionRead deliberately has no body field. MessageDraft.body is the AI's drafted reply text, generated by _conversation_context() from real customer message history and the customer's actual name -- this is genuine customer data, not business-authored content like V4.2's WhatsApp template bodies (that entry's own docstring draws exactly this distinction: template text is safe to expose cross-tenant because it's operator-authored and Meta-reviewed, not customer-specific). A cross-tenant Platform Admin surface exposing one workspace's customer conversation content to a platform administrator working on an unrelated workspace would violate this program's tenant-isolation principle, so the field is omitted at the schema level, not merely hidden in the UI -- confirmed by a dedicated test asserting "body" never appears in the response JSON at all, not just that the desktop table doesn't render it.

Verification note

Live-verified against a disposable workspace seeded with one enabled AiProviderConfig and two MessageDraft rows (one pending/copilot, one accepted/automation). The new "Configurations" / "Execution Log" view toggle in AiControlPlaneSection.tsx renders both tabs; the Execution Log correctly lists both rows with workspace, status, source, model, token counts, and timestamps, and the status filter correctly narrows to one row. Confirmed directly via curl that the endpoint's raw JSON response contains no body key at all -- not just that the desktop UI happens not to render one. Found, along the way, that the backend process already running from earlier in the session predated this slice's new route and returned a genuine 404 until restarted; confirmed this was an environment artifact, not a routing bug, by the identical request succeeding immediately after a clean restart with no code change.

Platform Admin V4.10: AI Governance

Fifth and final AI Control Plane increment, closing V4 in its entirety.

What "governance" honestly means for this codebase

Before designing anything, the real code was read to see what already exists rather than assuming a governance feature needed to be invented from scratch:

• Settings.reject_autonomous_ai_send() already refuses to boot the process at all if AI_AUTOMATIC_SEND_ENABLED is set, in every environment, not just production. This is already the strongest possible enforcement -- there is no runtime state for a Platform Admin view to observe, because the condition it would report on can never be true in a running process.
• AiProviderConfig.pii_redaction_enabled and .retention_days are stored, settable via CopilotConfigUpdate, and returned by every V4.6 config endpoint -- but neither is read by any enforcement code path anywhere in this codebase. No PII redaction logic exists; no retention worker prunes message_drafts. This is a real, pre-existing gap, documented here rather than silently treated as if these fields did something.
• What genuinely did not exist: a platform-wide kill switch for AI generation itself, distinct from V4.8's per-workspace AiProviderConfig.enabled lever. new_automation_runs (V3.4) already gives this exact capability to automations, and outbound_broadcasts (V4.4) already gives it to broadcasts; AI Copilot had no equivalent. This is the one governance-shaped gap this codebase could honestly close without either fabricating a capability or duplicating V4.8.

Reused mechanism, not a new one

PlatformAdminService.disable_ai_generation_globally()/ enable_ai_generation_globally() do not talk to FeatureFlagService directly with new bespoke create/update logic. They call the existing create_feature_flag()/update_feature_flag() methods this same class already exposes for V3.4's Feature Flags page -- the identical code path, producing the identical feature_flag.created/feature_flag.updated audit actions every other flag mutation in this codebase already produces. The only new code is the thin _set_ai_generation_enabled() wrapper that resolves whether the ai_generation_enabled key already exists (create vs. update) and returns the resolved PlatformAiGovernanceRead status afterward.

A permission gap found while designing, not left implicit

The obvious first design was to gate this action with platform.feature_flags.manage, since that is literally what the underlying mutation is. Checking the actual role grants ([redacted].py) before committing to that showed a real problem: support_admin -- the role meant to handle exactly this kind of day-to-day operational incident -- holds platform.ai.manage but only platform.feature_flags.read, not .manage. Gating the new endpoint behind platform.feature_flags.manage would have meant the one role built for incident response could not use it. Gating it behind platform.ai.manage instead (an AI-domain permission support_admin already holds) fixes this without granting support_admin any broader feature-flag power it does not already need.

Enforcement, checked before any per-workspace state

AiService._require_generation_enabled() is called as the very first line of both chat() and generate_draft(), before _require_ready_config() (V4.8's per-workspace check), _conversation_context(), or any gateway key decryption. This ordering is deliberate and directly proven by the enforcement test: disabling the global flag and then calling generate_draft() with a conversation id that does not exist in the database still produces the governance service's own 503, never a 404 or any other error from further down the call -- confirming the gate genuinely short-circuits before any per-workspace work begins, not merely that it happens to run somewhere in the method.

Verification note

Live-verified a full disable-then-enable round trip through the real desktop UI: the "Platform-wide AI generation" card rendered the correct default ("Never configured -- enabled everywhere by default", green "Enabled" badge), disabling with a reason changed the badge to "Disabled Platform-Wide" and reversed the button to "Enable", and re-enabling with a second reason reversed it back to "Disable". A direct database query independently confirmed both feature_flag.created and feature_flag.updated platform audit events were recorded with the correct before/after state and reason -- the same "verify past the UI's own claim of success" discipline this program has applied to every prior mutation slice.

Platform Admin V5.1: Platform Analytics

First Platform Intelligence unit. Confirmed by reading overview() first that Platform Admin had zero visibility into real product usage (customers, conversations, messages, orders, tickets) anywhere -- only workspace/user/subscription counts.

Never one fabricated total across heterogeneous currencies

Order.currency is per-order, not fixed platform-wide (an org's organization_settings.currency sets a default, but nothing prevents an individual order from using a different one). Summing grand_total across every order in the platform would silently add KES and USD figures together as if they were the same unit -- a fabricated number with no honest meaning. analytics_overview() instead groups by currency (GROUP BY Order.currency) and returns orders_revenue_by_currency: dict[str, str], the same "no fabricated single number from heterogeneous data" discipline the V2 billing overview already established for MRR/ARR (no stored subscription price exists platform-wide either).

Two real PostgreSQL bugs found and fixed while writing the trend query

Both were caught by [redacted] before either ever reached a live environment, not discovered afterward.

Bug 1: a computed expression bound twice is not the same expression to asyncpg. The first draft called func.date_trunc("day", Customer.created_at) separately in both the SELECT and GROUP BY clauses. SQLAlchemy compiles each call into its own bound parameter ($1, $3, ...), and PostgreSQL's asyncpg driver refuses the query outright (GroupingError: column "customers.created_at" must appear in the GROUP BY clause) because it cannot prove two independently-bound parameters are equal at parse time, even though their literal values are identical. Fixed by binding the expression to one Python variable (day_bucket = func.date_trunc(...)) and passing that same object to both select() and .group_by() -- SQLAlchemy then compiles it once and references it in both clauses correctly.

Bug 2: date_trunc() on a timestamptz column truncates in the session's timezone, not UTC. Even after fixing Bug 1, the test still failed: a customer created "now" (datetime.now(UTC)) was bucketed into yesterday's date. This database's sessions default to Africa/Nairobi (UTC+3, matching Organization.timezone's own default), and PostgreSQL's 2-argument date_trunc(field, timestamptz) truncates using that session TimeZone setting -- silently converting to local time before truncating, then returning the result still tagged timestamptz. A customer created at, say, 01:00 UTC on the 21st is 04:00 local time on the 21st, which truncates correctly to the 21st -- but a customer created at 22:00 UTC on the 20th is 01:00 local time on the 21st, truncating to the 21st when it should bucket as the 20th in UTC terms, or vice versa depending on which side of the offset the creation time falls. Fixed with PostgreSQL's 3-argument form, date_trunc('day', Customer.created_at, 'UTC'), which pins the truncation to UTC regardless of the session's TimeZone setting -- matching every other timestamp already stored and returned as UTC throughout this codebase. Both cutoff computation (oldest_included_day) and the bucketing query now agree on the same UTC calendar-day boundary.

Verification note

Live-verified against a disposable workspace seeded with one order in KES and one in USD, plus a customer created moments before the request. Direct curl calls to both endpoints returned exactly the seeded values -- orders_revenue_by_currency showed {"KES": "2500.0000", "USD": "75.0000"}, never a combined total, and the trend correctly bucketed the new customer to today's UTC date. The desktop UI rendered identical values, confirmed both via screenshot (KPI tiles) and extracted page text (the currency and trend panels, which sat below the visible viewport in this session's fixed-height automation browser -- the same known sandbox-scrolling limitation V4.1's own verification note already documented, not a product defect).

Platform Admin V5.3: Workspace Health

Second Platform Intelligence unit.

A deliberate naming decision, not an accident

The roadmap's own name for this unit is "Customer Health." Implementing it under that literal name would have created a genuine, damaging ambiguity: the Customer model already means "a workspace's own end-customer" everywhere else in this codebase (Inbox, Orders, Customer Detail, the Dashboard, V5.1's own total_customers field). This unit is about something entirely different -- the health of a workspace as one of Arche Axon's own paying customers. Naming the permission platform.customer_health.read and the desktop page "Customer Health" would have made every future reader guess which meaning applied at each call site. Renamed to "workspace health" at the permission key, schema, endpoint path, and desktop copy level, all at once, before any of it shipped -- not a case of "customer" being cleaned up later after confusion actually occurred.

Thresholds, not a score

_derive_health_status() evaluates a fixed, documented order and stops at the first match:

1. subscription_status in {cancelled, read_only}        -> at_risk
2. no message activity ever, or 30+ days inactive        -> at_risk
3. subscription_status in {past_due, grace, pending_setup} -> needs_attention
4. 14-29 days inactive                                    -> needs_attention
5. otherwise                                              -> healthy

This is deliberately not a weighted formula, a 0-100 score, or anything resembling anomaly detection -- there is no principled basis in this codebase for weighting "12 days inactive" against "past_due for 3 days" against each other, and inventing weights to produce a single number would be exactly the kind of fabrication this program's engineering- integrity rule forbids. Two named ClassVar threshold sets ([redacted] = 30, [redacted] = 14) and two named ClassVar status sets are the entire rule -- matching the exact threshold-not-score convention Security Center's attention items and V4.4's broadcast-safety thresholds already established, now applied to a third domain.

Billing status is checked before activity, not the other way around and not merged into one combined signal: a workspace that is both billing-lapsed and dormant is reported at_risk for the billing reason specifically (it would have been at_risk from the activity check too, but the ordering means the billing signal is never silently overwritten by whichever check happens to run second).

Admin-console-scale, not built to scale past today's volume

list_workspace_health() fetches every workspace's health row into memory (_workspace_health_rows()), then filters, searches, sorts, and paginates in Python -- because health_status is a Python-computed field, not a database column, filtering it with SQL WHERE would require either a computed column or duplicating the threshold logic in raw SQL. This is the same "admin-console-scale operation, not built to scale past the volume of connections a real deployment has today" precedent V4.1's own cross-channel merge already established for an analogous problem (there, merging three connection tables in Python rather than a SQL UNION).

Verification note

Live-verified against a disposable local database: one workspace with an active subscription and a message 2 hours old (correctly healthy), one workspace with a cancelled subscription and zero activity (correctly at_risk). The real production arche-axon workspace was also visible in the same query and correctly, honestly reported at_risk -- its subscription status is pending_setup with no recorded message activity yet, a genuine fact about real data surfaced by this feature working correctly, not a test artifact or a bug to explain away.

Platform Admin V5.5: Support Cases

Third Platform Intelligence unit. A cross-tenant drill-down over individual support tickets -- the natural next step after V5.1's own basic open/resolved counts and V5.3's per-workspace open_ticket_count, neither of which lets an operator browse or filter individual tickets across every workspace.

Metadata yes, content no

PlatformSupportCaseRead deliberately never includes Ticket.title or Ticket.description. Both may carry customer-specific narrative content authored by a workspace's own support agents -- this is a cross-tenant admin surface, and exposing one workspace's support-ticket content to a platform administrator working on an unrelated workspace would violate this program's tenant-isolation principle. This is the exact same boundary PlatformAiExecutionRead (V4.9) already drew around MessageDraft.body, applied here to a second content-bearing model. Every other metadata field (status, priority, category, source, age, overdue flag, timestamps, and the owning workspace's name/slug) is included. Confirmed with a dedicated test asserting "title" and "description" never appear in the response JSON at all, not just that the desktop table doesn't render them -- the same discipline V4.9's own [redacted] already established for this class of test.

Real columns get real SQL, not V5.3's in-memory pattern

V5.3's list_workspace_health() filters/sorts/paginates in Python because health_status is a Python-computed field with no backing column. Support Cases is different: Ticket.status, .priority, .category, and .organization_id are real, already-existing database columns, so list_support_cases() uses genuine SQL WHERE filtering and LIMIT/OFFSET pagination, with a separate COUNT subquery for total -- the same shape list_ai_executions() (V4.9) and list_channels() (V4.1)'s per-channel branches already use for column-backed filters. The choice was made by checking which fields are real columns before writing the query, not by defaulting to whichever pattern the most recently written sibling unit happened to use.

Overdue is a threshold, not a score

is_overdue is Ticket.due_at < now() combined with a fixed, named _SUPPORT_CASE_OPEN_STATUSES ClassVar set (open/in_progress/waiting_on_customer/reopened) -- a resolved or closed ticket past its original due date is not "overdue" in any actionable sense, so the status check is a deliberate second condition, not an oversight. This continues the same fixed-threshold-never-a-score convention _derive_health_status() (V5.3) and V4.4's broadcast-safety thresholds already established, now applied to a third domain.

Verification note

Live-verified against a disposable local database and a running desktop dev server via agent-browser: seeded a support_admin-promoted disposable workspace with six tickets, one per real Ticket status value, each carrying deliberately sensitive title/description text ("SENSITIVE-DO-NOT-LEAK: ..."). A direct API call confirmed the raw JSON response contains no title, description, or SENSITIVE text anywhere. The desktop UI's KPI tiles and all six table rows matched the seeded data exactly (6 total tickets, 1 open, 2 urgent, 4 overdue -- correctly excluding the resolved and closed tickets from the overdue count even though both also had a past due_at), with the same zero-leak property holding through the full rendered page text, not just the raw API response. Switching to and from the pre-existing Overview and Workspace Health views showed no regression, including the real arche-axon workspace's already-correct at_risk Workspace Health status from the V5.3 verification continuing to render unchanged. Disposable workspace, tickets, and user were deleted afterward; confirmed exactly one organization (arche-axon) remains.

Platform Admin V5.4: Workspace Timeline

Fourth Platform Intelligence unit. A single-workspace detail endpoint -- a chronological narrative of one workspace's history -- sitting alongside the existing V3 Part 1 diagnostics endpoint, which shows current state rather than history.

Renamed from "Company Timeline," matching V5.3's own precedent

The roadmap's own name for this unit is "Company Timeline." Nothing in this codebase is ever called a "Company" -- every prior unit uses "workspace"/"organization" consistently, including V5.3's own renamed "Workspace Health." Shipping "Company Timeline" would have introduced a one-off inconsistency in vocabulary that every other Platform Admin surface avoids. Renamed to "Workspace Timeline" at the permission key, schema names, endpoint path, and desktop copy level, all before any of it shipped -- applying the exact naming-collision discipline V5.3 already established, this time for a stylistic inconsistency rather than a genuine model-name collision.

Three sources, one deliberately excluded

workspace_timeline() merges exactly three sources for one workspace:

1. Organization.created_at    -> a synthetic "Workspace created" entry
2. BillingEvent (org-scoped)  -> Paystack webhook/billing state history,
                                  never exposed via Platform Admin before
3. PlatformAuditEvent
   (target_organization_id)   -> Vendale's own administrator actions
                                  against this workspace, already a safe
                                  cross-tenant surface since V3 Part 2

The tenant AuditEvent table is deliberately never read. Unlike BillingEvent (payment state changes only -- no customer content) and PlatformAuditEvent (Vendale's own administrators, not the tenant's own staff), AuditEvent.summary/.entity_label are free text written by a workspace's own agents describing their own customers, orders, and tickets by name. Exposing that cross-tenant would violate the exact same "metadata yes, content no" boundary MessageDraft.body (V4.9) and Ticket.title/.description (V5.5) were already excluded for -- here enforced even more simply, by never querying the table at all rather than filtering fields out of a response schema.

Admin-console-scale merge, applied to a single workspace

All BillingEvent and PlatformAuditEvent rows for the one requested workspace are fetched in full (not paginated at the SQL level), merged with the synthetic milestone entry into one Python list, sorted descending by occurred_at, and paginated in memory. This is safe because the candidate set is bounded by definition to a single organization's own history, not a cross-tenant volume concern -- matching V4.1's channel merge and V5.3's health computation, but applied here to a per-workspace detail view rather than a cross-tenant list.

A stable-sort pitfall found while writing the test, not shipped

Postgres freezes now() at transaction start, not per-statement. Since this test suite runs each test inside one wrapped transaction, a BillingEvent and a PlatformAuditEvent both relying on their model's server_default=func.now() would receive the exact same timestamp, and Python's stable sort preserves original list order for ties -- meaning entries.sort(key=..., reverse=True) would NOT reliably put the more-recently-added entry first when two rows tie, breaking any ordering assertion. Fixed by passing explicit, Python-controlled created_at values on both seeded rows in the test (never relying on the server-side default for anything whose relative order the test actually asserts) -- the general fix for any future test asserting occurred_at-based ordering across two or more rows created in the same transaction.

Verification note

Live-verified against a disposable local database and a running desktop dev server via agent-browser: seeded a support_admin-promoted disposable workspace with two BillingEvent rows and one PlatformAuditEvent row, each given explicit, distinct created_at timestamps. A direct API call confirmed all four entries (the milestone plus the three seeded rows) returned in the correct descending order, with billing detail strings correctly formatted (e.g. "pending_setup -> active | 5000.00 KES" from a stored amount_kobo=500000) and the platform-admin entry's actor_email correctly populated. The desktop's new "Workspace timeline" section, added to the bottom of the existing Workspace Diagnostics page, rendered all four entries identically. This session's client-side request timeout was temporarily raised to 60 seconds to work around the diagnostics page's already-documented Neon-latency artifact, then confirmed reverted to 12000ms via a clean git diff before finishing. Disposable workspace and user deleted afterward; confirmed exactly one organization (arche-axon) remains.

Request Correlation ID Middleware (2026-08-21)

Closes the request-correlation-ID gap that PlatformAuditEvent.request_id, /platform-admin/search, and every platform-admin mutation's own before/ after-state documentation had flagged as unpopulated since V3 Part 1/V3 Part 2. This is the first cross-cutting Platform Completion Program item (program brief section 1, "V4 cross-cutting") -- it touches the whole backend request pipeline, not one Platform Admin sub-domain, so it was built as its own module rather than folded into a specific V-numbered unit.

RequestIDMiddleware (app/middleware/request_id.py) is a BaseHTTPMiddleware subclass, registered in app/main.py after CORSMiddleware (Starlette applies added middleware in reverse registration order, so this places it closer to the actual route handler -- every response, including one CORS would reject, still gets a resolvable X-Request-ID, matching the same "every response carries a correlation ID" guarantee industry API gateways provide). A caller-supplied X-Request-ID header is honoured only when it matches a conservative allowlist (^[A-Za-z0-9_.-]{1,100}$); anything else -- oversized, containing spaces/punctuation, or absent entirely -- gets a freshly generated UUID4 instead. This value flows directly into a persisted audit column and potentially into log lines, so it is validated the same way any other externally-supplied string reaching storage would be, not trusted verbatim.

Threading the resolved ID into PlatformAuditEvent.request_id required touching every one of PlatformAdminService's ~34 mutation method signatures and 19 AuditService.append_platform() call sites (plus a handful of private helper methods like _record_workspace_lifecycle_audit() and _record_job_action_audit() that wrap append_platform() on behalf of several public methods), alongside the 29 corresponding endpoint call sites in app/api/v1/endpoints/platform_admin.py. Given the exact, uniform ip_address: str | None = None / user_agent: str | None = None parameter pattern (and the matching ip_address=ip_address / user_agent=user_agent call-site pattern) already established by the V3.3c-1 auth-telemetry work, this was done as a one-shot scripted regex substitution rather than ~50 individual manual edits -- verified safe only because every occurrence was enumerated and diffed before running it, and CRLF line endings were explicitly preserved on write (newline="\r\n") so the diff stayed addition-only with zero line-ending churn.

The blind substitution over-matched three call sites that share the exact ip_address=ip_address, / user_agent=user_agent, two-line shape but call a method that does not accept request_id at all: StepUpService.create(), MfaService.admin_reset(), and one of five AuthSecurityEventService.record() call sites. mypy app scripts caught all three immediately as call-arg errors (Unexpected keyword argument "request_id") -- exactly the reason the per-module protocol requires a full mypy run before trusting any mechanical refactor of this shape, not just the two files the change was scoped to. Fixed by removing the erroneous request_id= line from each of the three call sites; create_step_up() itself lost its request_id parameter entirely (unlike the other two, it writes no PlatformAuditEvent at all, so there was no legitimate destination for it in that method). AuthSecurityEvent has no request_id column at all (it is a deliberately separate ledger from PlatformAuditEvent, see the V3.3c-1 entry above) -- extending it would need its own migration and is out of scope for this slice, which is deliberately narrow: PlatformAuditEvent.request_id only, not a platform-wide request-ID propagation into every audit-adjacent table.

Deliberately excluded from this slice, tracked as natural follow-up rather than silently bundled in or silently left undone: request-scoped structured logging correlation (no logging filter/contextvar stamps request_id onto log lines yet -- app/core/logging.py's RedactingFormatter was not touched), and a /platform-admin/search "request ID" category (the endpoint's own documented gap is now half-closed: the column is populated, but search does not index by it).

5 new unit tests (tests/unit/test_request_id_middleware.py, using httpx.AsyncClient + ASGITransport rather than starlette.testclient.TestClient -- this environment's installed starlette version has deprecated TestClient in favor of an httpx2 package that is not installed here, discovered when the first version of this test file failed to collect at all) and 2 new integration tests (generated-ID and malformed-header-replacement cases); one existing integration test ([redacted]) updated from asserting request_id is None (the honest gap this slice closes) to asserting the real echoed/persisted value. 1005 backend tests passing, confirmed stable across two full-suite runs; ruff check clean; mypy app scripts clean (same 4 pre-existing unrelated facebook_page.py errors, untouched). No new migration -- PlatformAuditEvent.request_id already existed as a column since V3 Part 2's a7f2c9e1b4d6.

Platform Admin V5.7: One-click Diagnostics (2026-08-21)

Extends V3 Part 1's /workspaces/{workspace_id}/diagnostics with a synthesized pass/warn/fail checklist and one overall verdict, closing the gap that six separate panels (connections, webhooks, queues, automation, provider errors, subscription) required an operator to visually correlate before answering "is this workspace actually healthy."

PlatformAdminService._build_diagnostic_checks() is a pure function of data workspace_diagnostics() already gathers for its existing response fields -- it issues no new query and makes no new external call. Seven checks: workspace access, subscription standing, channel connections, webhook delivery, job queues, automation health, recent activity. Each check's threshold reuses an existing constant rather than inventing a new one: subscription/activity reuse V5.3's _WORKSPACE_HEALTH_AT_RISK_*/ _WORKSPACE_HEALTH_NEEDS_ATTENTION_* sets, webhook failure rate reuses V4.4's _HIGH_FAILURE_RATE_THRESHOLD (0.10).

_overall_diagnostic_status() takes the worst of two independently computed verdicts: V5.3's own _derive_health_status() (subscription and activity only, unchanged) and the worst individual checklist item (which additionally covers signals V5.3 never considers -- connection errors, webhook failures, queue failures, automation failures). This is never a fabricated blend; it is always the more severe of the two independently valid answers, and it guarantees this endpoint's overall_status can never disagree with the same workspace's Workspace Health row about the one thing they both compute (subscription + activity).

A deliberate product decision made explicit before writing any check logic: a workspace with no subscription record, or with zero channel connections configured, is pass, not warn. Both are normal early onboarding states. Treating them as warnings would make every brand-new, not-yet-onboarded workspace read as needs_attention by default, which would drown out genuinely concerning signals the checklist exists to surface -- the same "don't flag what isn't actually wrong" discipline this program has applied to every threshold-based feature since V4.4.

A real, pre-existing desktop bug found and fixed along the way

While building the desktop UI for this checklist, tracing how V5.3's WorkspaceHealthPanel.tsx renders its own needs_attention badge (statusTone(status) returning the string "warning", passed directly into <StatusBadge status={...} />) surfaced a real defect: StatusBadge's StatusVariant union type had no "warning" key at all -- only "pending"/"waiting"/"paused" mapped to the amber color class. StatusBadge's own fallback logic silently substitutes "neutral" (gray) for any status string not present in variantColorClass, so every needs_attention workspace-health row has been rendering as neutral gray instead of amber warning color since V5.3 shipped, with no runtime error to surface it. Fixed with a two-line addition ("warning" added to the StatusVariant union and to variantColorClass, mapping to the existing crm-status-badge--warning CSS class already used by the other warning-toned statuses) -- this single fix corrects both V5.3's existing, previously-unnoticed bug and enables this unit's own checkTone()/ overallStatusTone() helpers to render correctly, confirmed visually in the live verification below (amber "Warn" badges rendered correctly for subscription standing, job queues, and automation health).

Verification note

Live-verified against a disposable local database and a running desktop dev server via agent-browser: seeded a support_admin-promoted workspace and a second target workspace with a WhatsApp connection in an error state and a failed automation trigger event (no message activity, no subscription record). The rendered checklist showed exactly the expected per-check colors and details (workspace access: green pass; subscription standing: amber warn, "pending_setup"; channel connections: red fail, "1 of 1 connection(s) in an error state"; webhook delivery: green pass, none recorded; job queues: amber warn, 1 failed; automation health: amber warn, 1 trigger-queue failure; recent activity: red fail, no activity recorded) with an overall red "At risk" badge -- confirming both the synthesis logic and the StatusBadge fix rendered correctly through the real UI, not just in isolated backend tests. The client-side request timeout was temporarily raised to 60 seconds to work around the already-documented Neon-latency artifact on this endpoint (V3 Part 1's own ~18-sequential-query cost), confirmed reverted to 12000ms via a clean git diff before finishing. Disposable workspaces and users deleted afterward; confirmed exactly one organization (arche-axon) remains.

Platform Admin V5: Incident Management (2026-09-02)

A derived, read-only incident view over failure data the platform already records. The module's central design decision, made before writing any code per the ledger's re-scoping rule: an "incident" is not an entity. There is no incidents table, no state machine, no acknowledge/resolve workflow. An incident is the grouping, computed at read time, of concrete failure rows (signals) sharing the same kind, failure signature, and workspace within a fixed 24-hour lookback window. Every number the endpoints return is a property of existing rows, never a stored or fabricated value.

Signal sources and their exact predicates

Each of the three signal families reuses the failure predicate that surface's own platform view already established -- incidents never define a new definition of "failed":

• Failed queue jobs: status == "failed" on automation_trigger_events, broadcast_dispatch_jobs, and media_ingestion_jobs (V3.2/V4.5's queues), windowed by created_at. Each queue gets its own signal_type (automation_job/broadcast_job/media_job) so a signal names the surface that owns its remediation.
• Failed webhook events: meta_webhook_events.processing_error IS NOT NULL (V4.3's exact predicate), windowed by created_at. Rows with a NULL organization_id -- deliveries that never resolved to a tenant -- are excluded: they stay visible in V4.3's cross-tenant inspector rather than being attributed to a fabricated per-workspace incident.
• Failed provider send attempts: message_provider_attempts.succeeded IS FALSE (V4.1's recent-errors predicate), windowed by attempted_at.

Clustering key: kind:workspace_id:error_code. Error summaries are free-form text and deliberately never part of the key; the error code is the reliable correlation field the write paths already record. Severity is a fixed band (>= 10 signals critical, >= 3 warning, else elevated) -- class-level constants, mirroring V5.3/V5.7's derive-from-named-thresholds precedent, never invented per request.

Why there is no closed_count (and never will be)

An incident's signals are all open by construction, so the API reports only signal_count. A "closed" tally would be fabricated in both directions: a retried or cancelled job changes its status in place (no job-history table exists, so there is nothing to count as closed), and log-type signals (webhook events, provider attempts) have no mutable status at all -- they simply age out of the window. Closing is therefore observable only as absence: a retried job leaves the failed set, and the next read no longer contains its incident. The lifecycle integration test proves exactly this by retrying a job through the existing /jobs/{id}/retry endpoint (platform.jobs.retry) and asserting the signal disappears from the next read -- incidents gain no mutation endpoints of their own, keeping every remediation action in its existing audited surface (Jobs, Broadcasts, Channels) per V4.4's delegation rule.