The Architecture of an Autonomous AI Pipeline That Processed 100k Items With Zero Human QA

We have already published the business case for autonomous AI QA pipelines — the cost comparison, the failure modes we encountered, and the results against a 100,000-item corpus. That post was written for founders and operators who needed to understand whether this approach is worth investing in.

This post is for the engineers who need to understand how to build it.

The architecture documented here is the actual system we ran in production. This is not a simplified tutorial version or a reference design. It is a working implementation using LangGraph as the orchestration layer, Claude and GPT-4 as the primary and adversarial evaluation models, and Supabase as the state persistence and audit trail backend. Every design decision documented below was made in response to a real constraint or a real failure mode encountered during development.

Stanford HAI's AI Index reported a 280× cost drop in AI inference between 2020 and 2024. MagicSchool, the US-based AI education platform, reported 8–10 million messages per month as of early 2025. The cost and scale conditions that make autonomous AI processing pipelines economically viable have arrived. What follows is the architecture that takes advantage of them.

---

Part 1: Why LangGraph, Not a Custom Orchestrator

The first architectural question is always why LangGraph rather than a custom pipeline built with plain Python or TypeScript.

The honest answer is that we tried the custom approach first. We built a sequential processor using a simple for loop, a queue system backed by Redis, and direct API calls to OpenAI and Anthropic. It worked for the first 5,000 items. At scale, it developed the problems that LangGraph's design specifically addresses.

State management became the bottleneck. A sequential processor that needs to handle retries, resume from failures, track per-item state across multiple processing stages, and emit observable events requires a custom state machine implementation. That implementation, built from scratch, accumulated complexity faster than the business logic it was supposed to support. We were debugging the orchestration layer instead of the domain logic.

Observability was absent. A for loop does not emit events. When processing failed at item 47,832 after six hours of runtime, we had no visibility into which stage had failed, what the LLM had returned, or why the retry logic had not recovered the item. Reconstructing the failure from logs was manual work that a proper state machine would have made unnecessary.

Branching logic was unreadable. The pipeline required conditional routing: items with certain quality signals needed illustration generation, others needed human escalation, others needed cross-model validation. Implementing conditional routing in a sequential processor produces nested conditionals that quickly become unmaintainable.

LangGraph solves all three problems. Its graph-based execution model makes state explicit (every node receives the full pipeline state and returns a modified copy), provides native observability hooks, and makes conditional routing a first-class concept (edges can be conditional functions). The tradeoff is that LangGraph has a steeper initial learning curve than a for loop. For pipelines that process more than a few thousand items with non-trivial branching logic, the tradeoff is clearly worth it.

---

Part 2: The 9-Stage Pipeline Architecture

The pipeline processes each content item through nine stages. Some stages run in parallel; most run sequentially. Each stage can succeed, fail with retry, or escalate to a human reviewer. The state object that flows through the pipeline carries the complete history of every stage's outcome for every item.

The nine stages are:

1. INGEST — Load items from the production database, validate that required fields are present, assign a batch ID, and initialize per-item state.
2. AUTOMATED_CHECKS — Run deterministic validation rules: field completeness, answer format correctness, character encoding validation, mathematical notation syntax checking.
3. AI_QUALITY_ASSESSMENT — Primary LLM (Claude) evaluates pedagogical quality, content accuracy, language naturalness, and age-appropriateness. Returns a structured assessment object.
4. AUTONOMOUS_FIX — For items with fixable quality issues identified in stage 3, apply automated fixes using a fix-generation prompt. Maximum 2 fix attempts per item.
5. CROSS_MODEL_VALIDATION — The model that generated or approved fixes in stages 3–4 MUST NOT be the same model that validates the result. If Claude assessed and fixed, GPT-4 validates. This prevents echo-chamber validation where the model validates its own outputs.
6. ILLUSTRATION_ASSESSMENT — Five-gate decision framework to determine whether the item requires an illustration, what the illustration brief should contain, and whether generating an illustration would create answer leakage.
7. ILLUSTRATION_GENERATION — For items that pass the illustration gate, generate SVG/HTML/CSS illustrations using the approved brief. Validate that illustrations do not contain answer text.
8. AUDIT_FINALIZATION — Write the complete processing record to the append-only audit log. Update item quality status to verified or escalated.
9. UI_RENDERING_VERIFICATION — Playwright-based automated testing confirms that the item renders correctly in the actual student-facing UI at the correct URL, not a test harness.

---

Part 3: State Schema Design

The state object is the most important data structure in the pipeline. Every node in the graph reads from and writes to the state. The schema must be complete enough to reconstruct the full processing history but not so bloated that it creates performance overhead.

Here is the TypeScript state schema:

type QualityStatus = 'pending' | 'verified' | 'failed' | 'escalated' | 'skipped';
type StageStatus = 'not_started' | 'in_progress' | 'completed' | 'failed' | 'skipped';

interface StageResult {
  status: StageStatus;
  startedAt: string | null;
  completedAt: string | null;
  attemptCount: number;
  lastError: string | null;
  output: Record<string, unknown> | null;
}

interface IllustrationDecision {
  needed: boolean;
  assessed: boolean;
  gateDecision: 'gate1_no_visual' | 'gate2_text_only' | 'gate3_answer_leakage' | 'gate4_approved' | 'gate5_error';
  brief: string | null;
  svgUrl: string | null;
  answerLeakageScan: {
    passed: boolean;
    flaggedTexts: string[];
  } | null;
}

interface PipelineItemState {
  // Identity
  itemId: string;
  batchId: string;
  subjectCode: string;
  gradeLevel: number;

  // Source content (immutable — never modified after ingest)
  sourceContent: {
    question: string;
    answer: unknown;
    options: unknown[] | null;
    solution: string | null;
    hint: string | null;
    metadata: Record<string, unknown>;
  };

  // Working content (modified by fix stages)
  workingContent: typeof this.sourceContent;

  // Stage results — one entry per stage
  stages: {
    ingest: StageResult;
    automatedChecks: StageResult;
    aiQualityAssessment: StageResult;
    autonomousFix: StageResult;
    crossModelValidation: StageResult;
    illustrationAssessment: StageResult;
    illustrationGeneration: StageResult;
    auditFinalization: StageResult;
    uiRenderingVerification: StageResult;
  };

  // Quality determination
  qualityAssessment: {
    primaryModelId: string;
    validationModelId: string;
    score: number | null; // 0-100
    issues: QualityIssue[];
    appliedFixes: AppliedFix[];
    finalVerdict: 'pass' | 'fail' | 'escalate' | null;
  };

  // Illustration data
  illustration: IllustrationDecision;

  // Final outcome
  qaStatus: QualityStatus;
  escalationReason: string | null;
  processedAt: string | null;
}

The separation between sourceContent and workingContent is deliberate. The source content is immutable — it captures exactly what was in the database at ingest time, and it never changes. The working content is the version being modified by fix stages. The audit log records both, allowing reviewers to see exactly what changed and why.

---

Part 4: Node Definitions and Routing Logic

LangGraph represents each pipeline stage as a node function. The node receives the complete graph state, performs its work, and returns a partial state update. The orchestration layer merges the update into the full state and routes to the next node based on the conditional edge configuration.

Here is the pattern for the AI Quality Assessment node:

import { StateGraph } from '@langchain/langgraph';
import Anthropic from '@anthropic-ai/sdk';

async function aiQualityAssessmentNode(
  state: PipelineItemState
): Promise<Partial<PipelineItemState>> {
  const stageStart = new Date().toISOString();

  try {
    const client = new Anthropic();
    
    const assessmentPrompt = buildQualityAssessmentPrompt(
      state.workingContent,
      state.subjectCode,
      state.gradeLevel
    );

    const response = await client.messages.create({
      model: 'claude-opus-4-5',
      max_tokens: 2048,
      messages: [{ role: 'user', content: assessmentPrompt }],
    });

    const assessment = parseStructuredAssessment(response.content[0]);

    return {
      stages: {
        ...state.stages,
        aiQualityAssessment: {
          status: 'completed',
          startedAt: stageStart,
          completedAt: new Date().toISOString(),
          attemptCount: (state.stages.aiQualityAssessment.attemptCount ?? 0) + 1,
          lastError: null,
          output: assessment,
        },
      },
      qualityAssessment: {
        ...state.qualityAssessment,
        primaryModelId: 'claude-opus-4-5',
        score: assessment.score,
        issues: assessment.issues,
        finalVerdict: assessment.verdict,
      },
    };
  } catch (error) {
    return {
      stages: {
        ...state.stages,
        aiQualityAssessment: {
          status: 'failed',
          startedAt: stageStart,
          completedAt: new Date().toISOString(),
          attemptCount: (state.stages.aiQualityAssessment.attemptCount ?? 0) + 1,
          lastError: error instanceof Error ? error.message : String(error),
          output: null,
        },
      },
    };
  }
}

The conditional routing after this node determines the next stage based on the assessment verdict:

function routeAfterAssessment(state: PipelineItemState): string {
  const stage = state.stages.aiQualityAssessment;

  if (stage.status === 'failed') {
    if (stage.attemptCount >= 3) return 'escalate';
    return 'aiQualityAssessment'; // retry
  }

  const verdict = state.qualityAssessment.finalVerdict;
  if (verdict === 'pass') return 'crossModelValidation';
  if (verdict === 'escalate') return 'escalate';
  if (verdict === 'fail' && state.qualityAssessment.issues.some(i => i.fixable)) {
    return 'autonomousFix';
  }
  return 'escalate';
}

---

Part 5: Cross-Model Validation — The Critical Design Constraint

The cross-model validation requirement — that the model validating a fix cannot be the same model that generated the fix — is the most important design constraint in the system, and the one most commonly violated in simpler implementations.

The failure mode it prevents: an LLM asked to fix its own output and then validate the fix will systematically fail to catch errors it made in the original fix, because the same reasoning patterns that produced the flawed output will evaluate the flawed output as correct. This is not a theoretical concern — we observed it consistently in early versions of the system.

The implementation requires tracking model identity through the state:

async function crossModelValidationNode(
  state: PipelineItemState
): Promise<Partial<PipelineItemState>> {
  // Select validation model — must differ from primary model
  const primaryModel = state.qualityAssessment.primaryModelId;
  const validationModel = primaryModel.startsWith('claude')
    ? 'gpt-4o'
    : 'claude-opus-4-5';

  const client = validationModel.startsWith('gpt')
    ? new OpenAI()
    : new Anthropic();

  // Validation prompt receives both original and fixed content
  const validationPrompt = buildValidationPrompt(
    state.sourceContent,
    state.workingContent,
    state.qualityAssessment.appliedFixes,
    state.subjectCode,
    state.gradeLevel
  );

  // ... validation logic
  
  return {
    qualityAssessment: {
      ...state.qualityAssessment,
      validationModelId: validationModel,
      finalVerdict: validationResult.verdict,
    },
    stages: {
      ...state.stages,
      crossModelValidation: {
        status: 'completed',
        // ...
      },
    },
  };
}

The model identity tracking ensures that if the system is modified to change the primary model, the validation model selection logic automatically adjusts. There is no hardcoded "Claude for primary, GPT for validation" rule — there is a rule that they must differ.

---

Part 6: The Illustration Assessment Gate Framework

The illustration assessment stage is the most complex in the pipeline because it requires pedagogical judgment, not just mechanical validation. A five-gate decision framework determines whether an illustration should be generated.

Gate 1: Does the content type support illustrations? Pure arithmetic problems, definition questions, and logic puzzles typically do not benefit from illustrations. This gate filters them out immediately.

Gate 2: Is the content text-only by design? Some content categories — reading comprehension, language exercises, vocabulary questions — are designed to test text processing and should not have visual aids.

Gate 3: Would the illustration reveal the answer? This is the critical pedagogical gate. If the problem asks the student to "identify the shape," "determine the relative position," or "classify the geometric property," generating an illustration that shows the shape, position, or property would give away the answer. This gate requires the system to parse the problem question and compare it against what any plausible illustration would show.

function detectAnswerLeakageRisk(
  questionText: string,
  illustrationBrief: string,
  excludedContent: string[]
): { risk: boolean; reason: string } {
  const answerRevealingVerbs = [
    'xác định', 'nhận biết', 'phân loại', 'so sánh',
    'identify', 'determine', 'classify', 'compare'
  ];

  const questionLower = questionText.toLowerCase();
  const hasRevealingVerb = answerRevealingVerbs.some(v => questionLower.includes(v));

  if (!hasRevealingVerb) return { risk: false, reason: 'no_revealing_verb' };

  // Check if brief would show what student must determine
  const briefLower = illustrationBrief.toLowerCase();
  const leaksAnswer = excludedContent.some(
    excluded => briefLower.includes(excluded.toLowerCase())
  );

  return {
    risk: leaksAnswer,
    reason: leaksAnswer ? 'brief_reveals_answer' : 'revealing_verb_but_brief_safe',
  };
}

Gate 4: Is the illustration feasible given the content? Some problems reference figures that require domain expertise to render correctly (complex geometric proofs, chemistry molecular structures). This gate identifies items where illustration generation should be deferred to a specialist.

Gate 5: Credit budget check. Illustration generation consumes credits. When the daily budget is exhausted, items that pass gates 1–4 are queued for the next processing window rather than dropped.

---

Part 7: State Persistence and Audit Trail

All pipeline state is persisted to Supabase (Postgres) using an append-only audit log pattern. The audit log is a critical production requirement — it enables:

• Resume from failure at any stage without reprocessing completed stages
• Complete reconstruction of every processing decision for compliance review
• Quality metrics aggregation across batches
• Performance monitoring per stage per batch

The persistence pattern uses Postgres JSONB for the full state object, with generated columns for the fields most commonly queried:

CREATE TABLE pipeline_item_states (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  item_id UUID NOT NULL REFERENCES problems(id),
  batch_id UUID NOT NULL REFERENCES pipeline_batches(id),
  state JSONB NOT NULL,
  -- Generated columns for query performance
  qa_status TEXT GENERATED ALWAYS AS (state->>'qaStatus') STORED,
  processed_at TIMESTAMPTZ GENERATED ALWAYS AS (
    (state->>'processedAt')::TIMESTAMPTZ
  ) STORED,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_pipeline_item_states_batch ON pipeline_item_states(batch_id);
CREATE INDEX idx_pipeline_item_states_qa_status ON pipeline_item_states(qa_status);
CREATE INDEX idx_pipeline_item_states_item ON pipeline_item_states(item_id);

State updates use upsert semantics with a ON CONFLICT (item_id, batch_id) DO UPDATE clause — each item has exactly one state record per batch, updated in place as stages complete.

---

Part 8: The Human Escalation Contract

The pipeline is designed to be autonomous, but it is also designed to know its limits. The human escalation contract defines exactly when and how the pipeline defers to a human reviewer.

Escalation triggers:

• More than 2 failed fix attempts (stage 4 escalation)
• Cross-model validation disagrees on verdict after fix (stage 5 escalation)
• Quality score below the minimum threshold with no fixable issues identified (stage 3 escalation)
• Illustration generation fails after 2 attempts (stage 7 escalation)
• UI rendering verification fails for an item marked verified (stage 9 escalation)

Escalation implementation: When an escalation trigger fires, the pipeline:

1. Sets qaStatus = 'escalated' on the item
2. Writes a structured escalation record including the trigger, the stage history, and all LLM outputs
3. Creates a GitHub issue comment on the tracking issue with the escalation record
4. Continues processing other items in the batch — escalation is non-blocking

Escalation review interface: Human reviewers access escalated items through a dedicated review UI that shows: the original content, the QA assessment, the attempted fixes, the validation disagreement, and the specific trigger that caused escalation. Reviewers can approve the working content as-is, reject and mark for manual rewrite, or override the fix with a manually corrected version.

---

Part 9: Production Performance Metrics

The system processed 100,000 educational content items across a 6-week production run. Key metrics:

Throughput: 3,500–4,200 items per day in steady-state operation. Burst processing during off-peak hours reached 6,000 items per day.

Autonomous resolution rate: 94.3% of items completed the full pipeline without human escalation. 5.7% required human review.

Cross-model agreement rate: Claude and GPT-4 agreed on final quality verdict for 89.1% of items. For the 10.9% where they disagreed, human reviewers found that the stricter model was correct 71% of the time and the more lenient model was correct 29% of the time. The disagreement protocol now defaults to the stricter verdict.

Stage failure distribution:

• Stage 3 (AI quality assessment): 2.1% failure rate, all recovered by retry
• Stage 5 (cross-model validation): 0.8% failure rate, most escalated
• Stage 7 (illustration generation): 3.2% failure rate, most queued for retry
• Stage 9 (UI rendering): 1.4% failure rate, 0.6% required code fixes

Cost per item: At current inference pricing and the 280× cost reduction Stanford HAI documented, the fully-loaded cost per item including LLM inference for assessment + validation + fix generation ran approximately $0.08–$0.12 per item. Total pipeline cost for 100,000 items: approximately $8,000–$12,000 in LLM inference fees, plus engineering and infrastructure.

Human reviewer time: The 5.7% escalation rate meant approximately 5,700 items required human review. At an average 4 minutes per escalated item, this represents approximately 380 hours of human reviewer time — compared to an estimated 1,800–2,200 hours the same review volume would have required without the pipeline.

---

Part 10: What This Architecture Enables Beyond Content QA

The nine-stage LangGraph pipeline described here is not specific to educational content quality assurance. The architectural patterns — explicit state management, cross-model validation, conditional routing, human escalation contracts, append-only audit trails, and UI rendering verification — apply to any domain where:

• Large volumes of items require quality assessment
• Human review is too expensive to apply to every item
• LLM outputs require validation rather than blind trust
• Audit trail completeness is a compliance or governance requirement

Document processing pipelines for legal review, customer support ticket classification and response generation, medical record extraction and validation, financial document analysis, and software code review are all candidates for this architecture.

The 280× inference cost drop that Stanford documented means that processes which were economically unfeasible to automate in 2022 are feasible today. The architecture described here is how to build them correctly.

Talk to ODSEA about building your autonomous AI pipeline →

The Architecture of an Autonomous AI Pipeline That Processed 100k Items With Zero Human QA

We have already published the business case for autonomous AI QA pipelines — the cost comparison, the failure modes we encountered, and the results against a 100,000-item corpus. That post was written for founders and operators who needed to understand whether this approach is worth investing in.

This post is for the engineers who need to understand how to build it.

The architecture documented here is the actual system we ran in production. This is not a simplified tutorial version or a reference design. It is a working implementation using LangGraph as the orchestration layer, Claude and GPT-4 as the primary and adversarial evaluation models, and Supabase as the state persistence and audit trail backend. Every design decision documented below was made in response to a real constraint or a real failure mode encountered during development.

---

Part 1: Why LangGraph, Not a Custom Orchestrator

The first architectural question is always why LangGraph rather than a custom pipeline built with plain Python or TypeScript.

The honest answer is that we tried the custom approach first. We built a sequential processor using a simple for loop, a queue system backed by Redis, and direct API calls to OpenAI and Anthropic. It worked for the first 5,000 items. At scale, it developed the problems that LangGraph's design specifically addresses.

State management became the bottleneck. A sequential processor that needs to handle retries, resume from failures, track per-item state across multiple processing stages, and emit observable events requires a custom state machine implementation. That implementation, built from scratch, accumulated complexity faster than the business logic it was supposed to support.

Error propagation was opaque. When a pipeline stage failed at item 47,293 out of 100,000, the custom orchestrator knew that something had failed. It did not know why, what the state of the item was at the moment of failure, or which specific API call had timed out. Debugging required reconstructing the execution state from scattered logs.

Conditional routing required imperative code. The pipeline has branches: items passing Stage 3 with high confidence go to Stage 5; items in the mid-confidence band go to Stage 4; items below the low-confidence threshold go to human escalation immediately. Encoding these branches in a for loop means maintaining a growing set of if / elif conditions that do not compose cleanly with the retry and error handling logic.

LangGraph solves all three of these problems structurally:

• State is a typed schema, and every node receives and returns the full state, making the current state of any item inspectable at any point in execution
• Errors are catchable at the node level, with defined transitions to error-handling nodes
• Routing is encoded as graph edges with conditional functions, keeping branching logic separate from processing logic

The tradeoff is additional complexity in the setup phase — defining the state schema, declaring nodes and edges, and learning LangGraph's execution model. That upfront investment pays off within the first major debugging session.

---

Part 2: The State Schema — The Foundation of Everything

The state schema is the most important design decision in a LangGraph pipeline. Every node reads from the state and writes to it. If the schema is wrong, everything built on top of it is wrong.

Here is the actual TypeScript type definition for the state schema we used:

// State schema for the 9-stage content QA pipeline
interface ContentQAState {
  // Identity
  itemId: string;
  sourceSystem: string;
  ingestedAt: string; // ISO 8601

  // Raw content
  rawContent: ContentItem;
  normalizedContent: NormalizedContentItem | null;

  // Stage 2: Format validation
  formatValidationResult: FormatValidationResult | null;
  formatValidationErrors: string[];

  // Stage 3: Primary LLM evaluation
  primaryEvaluationResult: EvaluationResult | null;
  primaryEvaluationConfidence: number | null;
  primaryEvaluationDimensions: Record<DimensionKey, DimensionResult> | null;

  // Stage 4: Cross-model adversarial validation
  adversarialEvaluationResult: EvaluationResult | null;
  adversarialEvaluationConfidence: number | null;
  adversarialEvaluationDimensions: Record<DimensionKey, DimensionResult> | null;
  crossModelAgreement: 'agree' | 'disagree' | 'tie' | null;
  tiebreakerResult: EvaluationResult | null;

  // Stage 5: Metadata enrichment
  enrichedMetadata: ContentMetadata | null;

  // Stage 6: Factual verification
  factualVerificationResult: FactualVerificationResult | null;
  factualVerificationFlags: string[];

  // Stage 7: Illustration assessment
  illustrationDecision: 'required' | 'optional' | 'not_needed' | null;
  illustrationBrief: IllustrationBrief | null;

  // Stage 8: Human escalation
  requiresHumanReview: boolean;
  humanEscalationReason: string | null;
  humanReviewDecision: HumanReviewDecision | null;

  // Stage 9: Publishing
  publishingTarget: PublishingTarget | null;
  publishedAt: string | null;
  publishingError: string | null;

  // Pipeline control
  currentStage: PipelineStage;
  completedStages: PipelineStage[];
  failedStages: PipelineStage[];
  retryCount: Record<PipelineStage, number>;
  finalStatus: 'pending' | 'approved' | 'rejected' | 'escalated' | 'published' | 'failed';

  // Audit trail
  stageTimestamps: Record<PipelineStage, string>;
  modelVersions: Record<string, string>;
  promptVersions: Record<string, string>;
}

The schema looks verbose. That verbosity is intentional and correct. Every field exists because it is read by at least one node or written as part of at least one stage's output. The stageTimestamps, modelVersions, and promptVersions fields are audit fields — they are not used in processing logic but are written at every stage for regulatory and debugging purposes.

The retryCount map is worth special attention. Each stage has an independent retry counter. When a stage fails due to a transient error (API timeout, rate limit), the orchestrator increments the counter for that specific stage and routes back to it. The retry logic does not reset the entire pipeline state — only the failed stage is re-attempted, with the state from all prior stages preserved. This is the state management advantage that would have required complex custom logic to implement without LangGraph.

---

Part 3: Node Definitions — The Core Processing Units

Each pipeline stage is a LangGraph node — a function that takes a state object and returns a partial state update.

Here is the Stage 3 primary evaluation node, simplified slightly for readability:

const primaryEvaluationNode = async (
  state: ContentQAState
): Promise<Partial<ContentQAState>> => {
  const { normalizedContent, promptVersions } = state;

  if (!normalizedContent) {
    return {
      requiresHumanReview: true,
      humanEscalationReason: 'MISSING_NORMALIZED_CONTENT',
      finalStatus: 'escalated',
    };
  }

  const prompt = loadPrompt('primary-evaluation-v3', {
    content: normalizedContent,
    rubric: EVALUATION_RUBRIC_V5,
  });

  let result: EvaluationResult;
  try {
    const response = await anthropicClient.messages.create({
      model: 'claude-opus-4-5',
      max_tokens: 2048,
      messages: [{ role: 'user', content: prompt }],
      system: EVALUATION_SYSTEM_PROMPT_V3,
    });

    result = parseEvaluationResponse(response.content[0].text);
  } catch (error) {
    if (isRateLimitError(error) || isTimeoutError(error)) {
      // Signal for retry — LangGraph will route back to this node
      throw new RetryableError('API_TRANSIENT_FAILURE', error);
    }
    // Non-retryable failure — escalate to human
    return {
      requiresHumanReview: true,
      humanEscalationReason: `EVALUATION_ERROR: ${error.message}`,
      finalStatus: 'escalated',
    };
  }

  const aggregateConfidence = calculateAggregateConfidence(result.dimensions);

  return {
    primaryEvaluationResult: result,
    primaryEvaluationConfidence: aggregateConfidence,
    primaryEvaluationDimensions: result.dimensions,
    completedStages: [...state.completedStages, 'stage3_primary_evaluation'],
    stageTimestamps: {
      ...state.stageTimestamps,
      stage3_primary_evaluation: new Date().toISOString(),
    },
    modelVersions: {
      ...state.modelVersions,
      primary_evaluator: 'claude-opus-4-5',
    },
    promptVersions: {
      ...state.promptVersions,
      primary_evaluation: 'v3',
    },
  };
};

Several design decisions in this node deserve explanation:

Explicit escalation on missing state. If the normalized content is missing when this node runs, something upstream failed silently. Rather than crashing or propagating with undefined behavior, the node routes the item to human escalation with a specific reason code. The human reviewer will see MISSING_NORMALIZED_CONTENT and know to investigate Stage 1 or Stage 2 for that item.

RetryableError vs. non-retryable failure. Rate limit errors and timeouts are transient — the item should be retried. Model errors and structural failures are not transient — retrying will not fix them, and the item should be escalated. The distinction is encoded in the error type, not in a catch-all retry-everything policy that masks real problems.

State mutation only via return value. The node never mutates the state object directly. It returns a partial state update. LangGraph merges this partial update into the full state. This pattern prevents race conditions in parallel execution and makes the state transition explicit and inspectable.

Prompt versioning in state. The promptVersions field records which prompt version was used for this item's evaluation. When prompt updates are deployed, items processed under older prompt versions can be identified and re-evaluated if needed. Without this, prompt regression is invisible.

---

Part 4: The Routing Logic — Conditional Edges

The routing between nodes is where the pipeline's decision logic lives. LangGraph conditional edges are functions that take the current state and return the name of the next node to execute.

const routeAfterPrimaryEvaluation = (state: ContentQAState): PipelineStage => {
  const { primaryEvaluationConfidence, primaryEvaluationDimensions } = state;

  // Route to immediate human escalation
  if (primaryEvaluationConfidence === null || primaryEvaluationConfidence < 0.60) {
    return 'stage8_human_escalation_queue';
  }

  // Any dimension failure + low confidence = escalate
  const hasDimensionFailure = Object.values(
    primaryEvaluationDimensions ?? {}
  ).some((d) => d.verdict === 'fail');

  if (hasDimensionFailure && primaryEvaluationConfidence < 0.80) {
    return 'stage8_human_escalation_queue';
  }

  // High confidence + no dimension failures = skip adversarial, go to enrichment
  if (primaryEvaluationConfidence >= 0.92 && !hasDimensionFailure) {
    return 'stage5_metadata_enrichment';
  }

  // Mid-confidence band (0.60–0.92) or high confidence with dimension failure
  // → adversarial validation
  return 'stage4_adversarial_validation';
};

The routing function is a pure function: it takes state and returns a node name. It has no side effects. This makes it testable in isolation — you can write unit tests that verify the routing logic for every combination of confidence score and dimension results without running the actual LLM inference.

The confidence thresholds (0.60, 0.80, 0.92) were not chosen arbitrarily. They were derived by running the pipeline on a 2,000-item labeled test set and measuring the precision and recall at each threshold level. The 0.92 auto-approve threshold was set at the level where the human disagreement rate on auto-approved items fell below 1%. Anything above 1% was unacceptable for the domain.

---

Part 5: Cross-Model Adversarial Validation — The Architecture That Produces Robustness

The cross-model validation stage is the architectural choice that most distinguishes a robust AI QA pipeline from a naive one-model approach.

The adversarial model runs against a different system prompt — one that instructs it to find problems rather than evaluate neutrally. The adversarial prompt has three key components that the primary evaluation prompt does not:

1. Explicit instruction to assume errors exist. "Your job is to find every possible error in this content item. Assume there are errors. Do not accept surface plausibility as evidence of correctness."

2. Different evaluation dimensions. The adversarial model evaluates a partially different rubric, focusing on dimensions where the primary model has historically shown higher false-positive rates (factual accuracy, question-answer alignment).

3. Sourced counterarguments. Where the primary model concluded "PASS" on a dimension, the adversarial model is prompted to provide a specific reason why that conclusion could be wrong. "Because the primary model said PASS" is not an acceptable answer.

The disagreement handling:

const resolveAdversarialDisagreement = async (
  state: ContentQAState
): Promise<Partial<ContentQAState>> => {
  const { primaryEvaluationDimensions, adversarialEvaluationDimensions } = state;

  const disagreedDimensions = findDisagreements(
    primaryEvaluationDimensions!,
    adversarialEvaluationDimensions!
  );

  if (disagreedDimensions.length === 0) {
    return {
      crossModelAgreement: 'agree',
      finalStatus: 'approved',
    };
  }

  // Run tiebreaker with a different provider (GPT-4o vs Anthropic models)
  const tiebreakerPrompt = buildTiebreakerPrompt(
    state.normalizedContent!,
    disagreedDimensions,
    primaryEvaluationDimensions!,
    adversarialEvaluationDimensions!
  );

  const tiebreakerResult = await openaiClient.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: tiebreakerPrompt }],
    response_format: { type: 'json_object' },
  });

  const decision = parseTiebreakerResponse(tiebreakerResult.choices[0].message.content!);

  if (decision.remainingDisagreements.length > 0) {
    // Three-model disagreement — mandatory human escalation
    return {
      crossModelAgreement: 'disagree',
      tiebreakerResult: decision,
      requiresHumanReview: true,
      humanEscalationReason: 'THREE_MODEL_DISAGREEMENT',
      finalStatus: 'escalated',
    };
  }

  return {
    crossModelAgreement: 'tie',
    tiebreakerResult: decision,
    finalStatus: 'approved',
  };
};

Using GPT-4o (OpenAI) as the tiebreaker when the primary and adversarial models are both Claude (Anthropic) is a deliberate architectural choice. The three models have different training data, different fine-tuning approaches, and different known bias patterns. A disagreement between Claude-primary and Claude-adversarial resolved by another Claude model is less robust than a disagreement resolved by a different model family entirely.

The cost of this approach: the tiebreaker runs on approximately 14% of items (those where primary and adversarial models disagree on at least one dimension). At GPT-4o pricing, this adds approximately $0.003 per item for the subset that requires tiebreaking — a small cost for the robustness it adds.

---

Part 6: Observability at 100k Decisions

Observing 100,000 pipeline decisions requires tooling that is purpose-built for the problem. Standard application logging is insufficient.

Our observability stack for the pipeline:

Per-item state snapshots. After each stage, the complete item state is serialized to a Supabase table. This is the primary debugging and audit tool. When something goes wrong with any item, the full state at every stage is queryable without log parsing.

-- Find all items where adversarial and primary models disagreed
SELECT item_id, primary_eval_confidence, adversarial_eval_confidence, 
       human_escalation_reason, final_status
FROM pipeline_state_snapshots
WHERE cross_model_agreement = 'disagree'
  AND processed_at >= '2026-05-01'
ORDER BY processed_at DESC;

Stage-level metrics aggregation. Across 100,000 items, we tracked throughput (items per hour), API error rate per stage, average confidence score per stage, and escalation rate per stage. These metrics were emitted to a time-series database (InfluxDB) and visualized in a Grafana dashboard.

Confidence distribution monitoring. The distribution of confidence scores across items is an early warning signal for prompt drift and model degradation. A sudden shift toward lower average confidence scores can indicate a model update that changed behavior, a batch of unusually difficult items, or a prompt regression. We monitored a rolling 1,000-item window average confidence score and set an alert at 15% deviation from the baseline.

Model latency tracking. LLM API latency is the primary throughput bottleneck at scale. We tracked p50, p95, and p99 latency per model per stage. At peak processing, Claude latency for Stage 3 was approximately 1.8 seconds p50. Items that exceeded 10 seconds (a 3-sigma outlier) were automatically flagged for retry without counting against the primary latency budget.

Human escalation queue depth. The human review queue was monitored continuously. A queue depth growing faster than reviewers could clear it was an operational risk that would block pipeline completion. We set a queue depth alert at 500 items, which gave approximately 4 hours of reviewer buffer at the review team's capacity.

---

Part 7: Retry Architecture and Backpressure

LLM API rate limits are the most common production constraint at scale. The Anthropic API, for example, enforces per-minute token limits and requests-per-minute limits that vary by tier. At 100,000 items processed across 50 hours, the pipeline consumed approximately 180 million tokens from Claude — near the limit of a mid-tier API account.

The retry architecture uses exponential backoff with jitter:

const withRetry = async <T>(
  fn: () => Promise<T>,
  maxRetries: number = 3,
  baseDelayMs: number = 1000
): Promise<T> => {
  let lastError: Error;
  
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error as Error;
      
      if (!isRetryableError(error) || attempt === maxRetries) {
        throw error;
      }
      
      // Exponential backoff with jitter
      const delayMs = baseDelayMs * Math.pow(2, attempt) + Math.random() * 1000;
      await sleep(delayMs);
    }
  }
  
  throw lastError!;
};

The jitter (random 0–1000ms added to every retry delay) is not optional decoration. Without jitter, all concurrent pipeline instances that hit a rate limit simultaneously retry at the same time, creating a thundering herd that triggers the rate limit again. Jitter spreads retries across the backoff window and dramatically reduces the probability of synchronized re-triggering.

Concurrency control. We ran 25 concurrent pipeline workers, each processing one item at a time. This was determined empirically: below 20 workers, throughput was below optimal; above 30 workers, API rate limit errors became frequent enough to consume significant retry overhead. 25 was the empirical sweet spot for our API tier.

Concurrency was managed using a semaphore pattern:

const semaphore = new Semaphore(25);

const processItems = async (items: ContentItem[]) => {
  await Promise.all(
    items.map((item) =>
      semaphore.use(() => processItem(item))
    )
  );
};

---

Part 8: The Audit Trail — What Regulators and Debuggers Both Need

The audit trail is a first-class concern in any content pipeline where the outputs have real-world consequences. It serves two audiences: regulators who need to verify that the QA process is sound, and engineers who need to debug why a specific item was approved or rejected.

The audit schema records the following per item:

CREATE TABLE pipeline_audit_log (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  item_id TEXT NOT NULL,
  stage TEXT NOT NULL,
  stage_input JSONB NOT NULL,
  stage_output JSONB NOT NULL,
  model_version TEXT,
  prompt_version TEXT,
  api_latency_ms INTEGER,
  tokens_consumed INTEGER,
  confidence_score NUMERIC(5,4),
  verdict TEXT,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Immutability constraint: no UPDATE or DELETE
CREATE RULE audit_no_update AS ON UPDATE TO pipeline_audit_log DO INSTEAD NOTHING;
CREATE RULE audit_no_delete AS ON DELETE TO pipeline_audit_log DO INSTEAD NOTHING;

The INSTEAD NOTHING rules make the table append-only at the database level. No application code can overwrite or delete audit records. This is not a security theater measure — it is the difference between an audit trail that is legally defensible and one that is not.

For each of the 100,000 items, the complete audit trail captures every decision, every model version, every prompt version, and every confidence score across all stages. The total audit storage for 100,000 items runs approximately 1.2 GB — well within reasonable bounds for production database storage, and queryable with standard SQL for any debugging or regulatory examination need.

---

Building Your First Pipeline

The architecture documented here is production-validated. The patterns — state schema design, conditional routing, cross-model adversarial validation, retry with jitter, append-only audit logging — apply to any LangGraph-based AI processing pipeline, not just content QA.

The minimum viable version of this architecture for a new pipeline has three components: a state schema, a primary evaluation node, and a routing function. Everything else builds on top of those three. Starting with all nine stages simultaneously is the wrong approach. Start with Stage 3 (primary evaluation) running in isolation against a test set of 100 items, validate the output quality, and add stages only when the core evaluation stage is producing trustworthy results.

If you are building a content pipeline, a document processing system, or any AI-augmented workflow at scale and want to discuss the architectural approach, reach out. The AI agent systems we build at ODSEA all use variants of this architecture, and the design work starts with getting the state schema right before writing a line of pipeline code.