Every one of these taps is the transaction event this system scores in real time.
Real-Time Transaction Fraud Detection
Multi-Agent Architecture — LangGraph Orchestration with Layered Guardrails
Three specialist agents feed a decision orchestrator over a parallel + conditional graph. Every request passes through input, runtime, and output guardrails; every node is traced in LangSmith.
Status
built & verified
designed, not yet wired
Guardrail stage
input
runtime / HITL
output
observability tap
● incoming transaction event {customer_id, amount, category, merchant, V1..V28}
▼
01
Input guardrails
runs before the graph is invoked
Prompt injection / jailbreak
Keyword pre-filter + Llama Guard 4 12B (via HF Inference Providers) on any free-text note. Verified empirically: Llama Guard alone misses plain "ignore previous instructions" with no unsafe payload — hence the paired keyword filter.
PII / data-sensitivity scan
Regex + Luhn-validated detector (SSN, card PAN, email, phone) on free-text fields; blocks before the payload enters any LLM prompt. Verified Llama Guard does not catch this — it's a content-safety classifier, not a PII detector.
Action authorization
Validates the caller's JWT role via auth/main.py — e.g. only analyst+ may trigger a re-score, only admin may override a verdict.
▼
02
Orchestration & agents
LangGraph StateGraph
Orchestrator
Entry node. Validates state, then fans out to Agent A and Agent B in parallel branches of the graph.
Agent A · ML inference
score_propensity(V1..V28) → fraud_probability
Calls the model-inference tool (LogisticRegression, ml/artifacts/model.joblib). ROC-AUC 0.971 on holdout.
Note: trained on V1–V28 only — this dataset has no Amount column, so the "V1–V28 + Amount" spec was adjusted to match available data.
Agent B · behavior
score_behavior(amount, category_mean, category_std, merchant_count) → anomaly_score
Redesigned from the original per-customer-history plan: credit_card_transaction_flow has exactly one row per customer (no repeat history to deviate from). Rebuilt as a peer-cohort z-score — this amount vs. the category's population mean/std — which turned out to have clean, well-separated stats per category (e.g. Travel ~$1540±$830 vs Restaurant ~$55±$26).
Tool layer (pure math) vs. agent layer (DB fetch) are split: agents/tools/behavior_tool.py vs agents/agent_b_behavior.py.
final_score = 0.7 × propensity + 0.3 × behavior
Agent C · policy consultation (conditional)
consult_policy(query, k) → cited passages
Semantic search over document_chunks (pgvector) — Citi T&Cs, benefits guide, OCC Comptroller's Handbook. Invoked whenever final_score ≥ 0.5 (review or block tier) — corrected from the original 0.35–0.85 spec to match what route_after_combine actually checks.
Decision agent
Threshold-only for auto-approve (no LLM call, no Agent C consultation — saves the cost/latency entirely). For review/block tiers, one LLM call reasons over all three signals and cites Agent C's passages.
The original "ReAct loop, re-invoke Agent C" design was never built — it's a single reasoning pass, not iterative.
▼
03
Runtime guardrails
enforced during graph execution
Tool-usage policy
Each agent node checks a whitelist (guardrails/runtime_guardrails.py) before calling its tool, plus a per-agent rate limit (30 calls/min).
Autonomy control — HITL
Real interrupt() / Command(resume=...) pause-and-resume, backed by an InMemorySaver checkpointer — not a placeholder. Requires a single shared compiled graph instance across the pause/resume boundary (a per-call rebuild would silently lose the checkpoint).
Human-in-the-loop thresholds (live in guardrails/runtime_guardrails.py)
| final_score | action | autonomy |
|---|---|---|
| < 0.50 | auto-approve | ● autonomous |
| 0.50 – 0.85 | hold for analyst review | ● HITL pause |
| ≥ 0.85 | auto-block, notify admin | ● HITL pause |
▼
04
Output guardrails
runs on the decision agent's response
Harmful content
Llama Guard classifies the generated rationale (role="assistant"). Hit a real API constraint building this: an assistant-only message is rejected — Llama Guard needs a preceding user turn for context.
PII leak scan
Same regex/Luhn detector as the input stage, run on the rationale text.
Citation groundedness & presence
Groundedness (severity: warning) catches a hallucinated .pdf citation not actually retrieved. Presence (severity: warning, keyed on Agent C's policy_context being non-empty — not on decision tier, which only coincidentally lines up today) flags Agent C context that was retrieved but never cited.
Decision schema
Verdict must be a known value with a non-empty rationale — a regression guard, not something that fires on real traffic today (the auto-approve path always sets a rationale).
▼
● verdict returned {decision, final_score, rationale, citations} — gated by caller's JWT role
LangSmith observability
taps every stage & node above
Node/tool tracing, cost, latencyfree
Turned out to need zero extra code: LangGraph auto-traces every node and @tool call with full I/O; ChatOpenAI calls report exact token counts and cost (confirmed down to the dollar, e.g. $0.00014775 on one run).
Citation / doc usagefree
Also automatic: Agent C's node return value (source, page, content, distance) is already the recorded output of its trace — no separate logging needed.
Quality & hallucination feedbackmanual
common/observability.py's log_quality_feedback() attaches each output-guardrail result as a scored LangSmith feedback entry. Caught a real gap live: one run scored citation_presence: 0.0 — the LLM used retrieved context without naming the source.
Compliance & incidentsmanual
log_incident() — standalone, tag="incident" runs for events with no in-flight LangGraph trace at all: input/output guardrail blocks, MCP tool rejections, auth 401/403s. Verified all four land in LangSmith via direct API query, not just log output.
05
Added after this diagram was first drawn
not originally in this design
Memory (memory/)
Working memory: rolling per-session buffer, auto-compacts past 8 turns (summary-of-summary). Episodic memory: durable, pgvector-embedded checkpoints, semantically searchable across sessions — plus a semantic cache (exact + cosine-similarity match) in front of recall.
Event bus (event_bus.py)
Async pub-sub chain: UserQueryEvent → InputGuardrailService → OrchestrationService → OutputGuardrailService → ResultPublisher. Generic mechanics live in common/pubsub.py (named to avoid a self-collision with this file's own name).
MCP server + client
mcp_server.py exposes Agent A/B/C as MCP tools, each wrapped in a @guarded(...) decorator running the same input guardrails — without it, an MCP caller could bypass every check this system enforces. mcp_client.py proves the real stdio transport works, separate from the direct-call path.