cloud.abridge.services

Architecture

Updated 2026-08-29

APEX is event-driven and stateless. A Slack mention flows through a gate, spawns a Cursor agent that follows DIRECTIVE.md, calls MCP tools against live systems, and posts a threaded reply with an evidence tier.

Request flow

flowchart TD
  A["@APEX mention in Slack"] --> B["Socket Mode pods + watcher backup"]
  B --> C["auto_responder.py"]
  C --> G{"Gate: org member? HIPAA fail-closed"}
  G -->|"blocked"| X["Ignore"]
  G -->|"pass"| H["Add hourglass, spawn Cursor agent"]
  H --> DR["DIRECTIVE.md: 19 investigation steps"]
  DR --> T["MCP tools: Slack, BigQuery, Datadog, Temporal, Firestore, Logs, Notion"]
  T --> RES["Draft answer with evidence tier A/B/C/D"]
  RES --> M{"Confidence >= 0.72?"}
  M -->|"yes"| Y["Post threaded reply, checkmark"]
  M -->|"no"| Z["Route to human"]

Key components

  • auto_responder.py — core event handler on the Slack Bolt framework with Socket Mode (real-time WebSocket). Handles user gating, an emoji-based state machine for idempotency (hourglass → checkmark), a concurrent worker pool (up to 10 agents per pod), and graceful SIGTERM drain.
  • DIRECTIVE.md — the canonical, single source of truth for agent behavior: 19 investigation steps, HIPAA/PHI rules, evidence-tier classification, and PR/doc/ticket workflows.
  • watcher.py — backup scanner that polls Slack search.messages every 30s to catch mentions missed by Socket Mode, claiming each atomically via reactions.add (a distributed lock, so no double-answers). Scales 1→5 pods on an HPA.
  • config.py — model selection, thresholds, and limits (CURSOR_MODEL, CONFIDENCE_TRIAGE_THRESHOLD = 0.72, AGENT_TIMEOUT_SECONDS = 3600, MAX_CONCURRENT_WORKERS = 10).

MCP tool ecosystem

APEX combines nine custom MCP servers (built with esbuild on the MCP SDK) plus external HTTP MCPs:

Server Purpose
slack Search, post messages, Socket Mode
dbt-cloud Job status, runs, model failures
cloud-logging GCP logs via Workload Identity
identity-platform Firebase Auth lookups
temporal Read-only workflow debugging
firestore Real-time document lookups
notion Pages, databases, search
research arXiv + PubMed paper search
google-workspace Drive/Docs/Sheets/Slides creation

Plus external HTTP MCPs: Linear, Datadog, and incident.io (a Go binary).

Production scale

Dimension Value
Namespace apex (GKE)
Replicas 10 Socket Mode pods + 1–5 watcher pods (HPA)
Capacity ~100 concurrent agents, ~200 questions/min at full load
Model claude-4.6-opus-high via Cursor CLI
Timeout 60 min per agent, 62-min pod termination grace
Rolling updates maxUnavailable: 0, maxSurge: 2 (zero downtime)
Pod restart Every ~3 hours — all in-memory state wiped

Deployment is Docker → GKE via Helm + ArgoCD, with secrets from GCP Secret Manager through the External Secrets Operator, and read-only Workload Identity access to the observability, data, and Kubernetes surfaces needed for investigation.

Safety & compliance

  • HIPAA fail-closed — external users blocked; aggregates-only queries; never raw clinical text.
  • Confidence triage — answers below 0.72 are flagged for human review.
  • Evidence tiers — every answer is classified A/B/C/D by verification level.
  • Destructive actions — blocked for non-admins; explicit confirmation required for admins.
  • Idempotency — the emoji state machine plus marker strings prevent duplicate answers.