Challenge description + artifacts
    |
    v
[ Meta-agent / classifier ]
    |
    +-- "web"      -> WebAgent (Burp, sqlmap, ffuf, XSS payloads)
    +-- "pwn"      -> PwnAgent (pwntools, GDB, checksec, ROPgadget)
    +-- "crypto"   -> CryptoAgent (CyberChef, SageMath, frequency tools)
    +-- "forensic" -> ForensicsAgent (binwalk, strings, exiftool, tshark)
    +-- "reversing"-> ReversingAgent (Ghidra headless, radare2, ltrace)
    +-- "misc"     -> MiscAgent (zsteg, stegsolve, encoding tools)
    +-- uncertain  -> TriageAgent (runs lightweight probes, re-routes)

[ Challenge pool ]
    |
    +-- Session A: Challenge 1 -> WebAgent -> FlagExtractor -> Scoreboard
    +-- Session B: Challenge 2 -> PwnAgent  -> FlagExtractor -> Scoreboard
    +-- Session C: Challenge 3 -> PwnAgent (hypothesis: stack overflow) -> ...
    +-- Session D: Challenge 3 -> PwnAgent (hypothesis: format string) -> ...
    |
    [ First flag from C or D wins; the other is killed ]

import re

DEFAULT_FLAG_PATTERNS = [
    re.compile(r"flag\{[a-zA-Z0-9_\-]{8,128}\}", re.IGNORECASE),
    re.compile(r"CTF\{[a-zA-Z0-9_\-]{8,128}\}"),
    # competition-specific prefix loaded from rules.json
]

def extract_flags(text: str, patterns=DEFAULT_FLAG_PATTERNS) -> list[str]:
    found = []
    for p in patterns:
        found.extend(p.findall(text))
    return list(set(found))  # dedup within output

import requests

def submit_flag_ctfd(base_url: str, token: str, challenge_id: int, flag: str) -> dict:
    """Submit a flag to a CTFd instance. Returns the API response."""
    r = requests.post(
        f"{base_url}/api/v1/challenges/attempt",
        headers={"Authorization": f"Token {token}", "Content-Type": "application/json"},
        json={"challenge_id": challenge_id, "submission": flag},
        timeout=10,
    )
    return r.json()  # {"status": "correct" | "incorrect" | "already_solved"}

interface CTFScorecard {
  competition_id: string;
  challenges: ChallengeStatus[];
  flags_captured: FlagRecord[];
  flags_submitted: SubmissionRecord[];
  last_updated: string;
}

interface ChallengeStatus {
  challenge_id: string;
  domain: "web" | "pwn" | "crypto" | "forensic" | "reversing" | "misc";
  state: "unattempted" | "in_progress" | "solved" | "abandoned";
  sessions: string[];      // parallel attempt session IDs
  attempts: number;
  first_blood_at: string | null;
}

Competition starts -> fetch challenge list from CTFd
    |
    +-- For each challenge (up to concurrency limit N):
    |     |
    |     +-- Meta-agent routes to domain
    |     +-- Specialized agent + tool subset loaded
    |     +-- Agent solves (no HITL if sandbox criterion met)
    |     +-- Flag extractor middleware scans every tool output
    |     +-- On flag: verifier confirms, submission client POSTs
    |     +-- On correct: scorecard updated, session killed, slot freed
    |
    +-- Scorecard updates operator in real time
    +-- Next challenge pulled from pool when slot frees

# Module S04 — CTF Harness Engineering

**Course**: 2A — Building AI Harnesses for Cybersecurity
**Module**: S04 — CTF Harness Engineering
**Duration**: 60 minutes
**Level**: Senior Engineer and above
**Prerequisites**: S00, S01, S02, S03 complete

> *Speed, domain routing, and benchmark-validated performance. A harness that solves CTF challenges at competition speed by routing each challenge to the right specialized agent, removing approval gates where it is safe to do so, and automating the flag-extraction-to-submission pipeline.*

---

## Learning Objectives

1. Map the six CTF domains (Web, Pwn, Crypto, Forensics, Reversing, Misc) to their reasoning patterns, tool requirements, and context budgets — and explain why a single undifferentiated harness underperforms across all six.
2. Design a domain router that reads a challenge description and selects the correct specialized agent and tool subset, with deterministic fallback when the meta-agent is uncertain.
3. Specify the per-domain tool stacks (Burp/sqlmap/ffuf for Web, pwntools/GDB/checksec/ROPgadget for Pwn, CyberChef/frequency analysis for Crypto, binwalk/strings/exiftool/tshark for Forensics, Ghidra headless/radare2/ltrace for Reversing) and the wrapping pattern that loads them on demand.
4. Identify when human-in-the-loop approval gates can be safely removed for CTF contexts and when they cannot, with concrete criteria (sandbox isolation, target ownership, blast radius).
5. Implement parallel solve attempts across multiple challenges in isolated sessions, and a flag extractor that pattern-matches tool outputs and submits to a CTFd instance via its API.

---

# S04.1 — CTF Domain Architecture

A Capture The Flag challenge is not a smaller pentest. It is a different problem shape with a different optimization target. The pentest harness from S03 optimizes for thoroughness — complete coverage of an attack surface, repeatable evidence, client-ready reporting. The CTF harness optimizes for **speed to flag**. A competition is ranked by flag count, then by time. A harness that takes 40 minutes to solve a challenge that a human solves in 10 is not in the running, regardless of how clean its evidence chain is.

This speed target inverts several of the constraints established in S02 and S03. But it does not invert all of them — and the engineering discipline is knowing which inversions are safe and which are catastrophic. That distinction is the entire content of S04.2.

## The six domains

CTF challenges partition into six recognized domains. Each domain has a distinct reasoning pattern, a distinct tool stack, and a distinct context budget.

| Domain | Reasoning pattern | Core tools | Typical context burn |
| --- | --- | --- | --- |
| **Web** | Request manipulation, payload crafting, response inspection | Burp (headless), sqlmap, ffuf, XSS payload sets | Low–medium |
| **Pwn** | Memory layout reasoning, gadget chains, syscall sequences | pwntools, GDB (+pwndbg), checksec, ROPgadget, one_gadget | High |
| **Crypto** | Mathematical weakness detection, oracle interaction | CyberChef (recipes), SageMath, frequency analyzers, padding-oracle scripts | Medium |
| **Forensics** | Artifact carving, timeline reconstruction, format parsing | binwalk, strings, exiftool, tshark, Volatility | Low |
| **Reversing** | Disassembly reasoning, decompilation triage, dynamic confirmation | Ghidra headless, radare2/rizin, ltrace, strace, gdb | Very high |
| **Misc** | Everything else: steganography, scripting puzzles, encoding chains, esoteric languages | zsteg, stegsolve, custom Python, CyberChef | Variable |

The numbers in the rightmost column are not decoration. They drive the engineering. A Reversing challenge can saturate a context window with decompiler output in a handful of tool calls; the harness must aggressively summarize and retain only structural facts (function signatures, cross-references, suspicious comparisons). A Forensics challenge rarely saturates context — the bottleneck is tool selection and artifact triage.

## Why a single harness fails across all six

A single undifferentiated harness loaded with all six tool stacks fails for three compounding reasons.

**Context pollution.** A Web challenge does not need ROPgadget in the tool manifest. Loading every tool description into every session inflates the system prompt, dilutes the model's tool-selection signal, and consumes context that the challenge itself needs. Empirically, models with broad tool manifests call the wrong tool more often than models with narrow manifests.

**Reasoning drift.** The mental mode for Pwn (low-level memory reasoning, careful hypothesis testing) is the opposite of the mode for Web (broad fuzzing, rapid iteration). A model instructed to be good at both will be excellent at neither. Specialized system prompts per domain — written by domain experts, encoding the domain's actual heuristics — measurably outperform a generic prompt that tries to cover all six.

**Token waste on irrelevant capability.** Each tool's schema, examples, and invocation cost is paid every turn. Six domains of tools means six domains of overhead, every turn, on every challenge.

## The routing problem

The harness's first job on a new challenge is to identify the domain and route to the right specialized agent. This is the **meta-agent** pattern: a small, fast model (or a deterministic classifier) reads the challenge description and the attached artifacts, emits a domain label, and dispatches to the corresponding specialized agent.

```
Challenge description + artifacts
    |
    v
[ Meta-agent / classifier ]
    |
    +-- "web"      -> WebAgent (Burp, sqlmap, ffuf, XSS payloads)
    +-- "pwn"      -> PwnAgent (pwntools, GDB, checksec, ROPgadget)
    +-- "crypto"   -> CryptoAgent (CyberChef, SageMath, frequency tools)
    +-- "forensic" -> ForensicsAgent (binwalk, strings, exiftool, tshark)
    +-- "reversing"-> ReversingAgent (Ghidra headless, radare2, ltrace)
    +-- "misc"     -> MiscAgent (zsteg, stegsolve, encoding tools)
    +-- uncertain  -> TriageAgent (runs lightweight probes, re-routes)
```

The router's input is the challenge description as posted by the CTF platform — typically a paragraph of prose, plus attached files (a binary, a pcap, an image, a URL). The router uses both signals. Text-only classification misroutes Pwn challenges described as "analyze this binary" into Reversing. File-type signals (ELF binary with `checksec` showing `NX disabled` and a `win` symbol) correct that.

## CAI's approach: specialized agents per domain

The architecture worth studying is **CAI (Cybersecurity AI)**, which implements the meta-agent-plus-specialists pattern explicitly. Each domain has a dedicated agent with its own system prompt, its own tool subset, and its own reasoning loop. The meta-agent's only job is routing — it does not solve challenges.

The key engineering decisions in this architecture:

- **Specialized prompts are authored per domain.** A Web prompt encodes "test parameter by parameter, watch response codes, escalate from error-based to blind." A Pwn prompt encodes "run checksec first, look for a win function, then determine the offset to RIP." These are not generic security prompts.
- **Tool subsets are loaded per domain, not globally.** The WebAgent's tool manifest does not include ROPgadget. The PwnAgent's manifest does not include ffuf. Context stays narrow.
- **The router is allowed to be uncertain.** When the meta-agent cannot route with confidence, the challenge goes to a TriageAgent that runs cheap probes (file type, checksec, strings frequency, HTTP response code) and re-routes with the additional signal. This two-stage routing catches the ambiguous cases.

## Domain-specific tool stacks

The tools below are the canonical per-domain set. Each is wrapped as a harness tool following the S02.2 pattern — Pydantic schema, structured output, evidence retained separately. The difference from S02 is that tools are **loaded lazily by domain**, not all present in every session.

**Web.** Burp Suite in headless mode for request replay and tampering; sqlmap for injection confirmation; ffuf for parameter and directory discovery; a curated XSS payload set (not the full fuzzing list — the curated 50 that catch 95% of reflections). The reasoning loop is request-send-inspect-response.

**Pwn.** `pwntools` as the interaction library; GDB with pwndgb or GEF for dynamic analysis; `checksec` to read binary protections before any reasoning; `ROPgadget` and `one_gadget` for gadget discovery. The reasoning loop is protections-check-then-offset-find-then-chain-build. Skipping `checksec` is the most common Pwn failure mode — the agent reasons about ROP against a binary with stack canaries and wastes the engagement.

**Crypto.** CyberChef recipes (encoded as JSON, not baked into prompts) for encoding chains; SageMath for mathematical attacks (LLL, Coppersmith, lattice reduction); frequency analyzers for classical ciphers; padding-oracle scripts as reusable components. The reasoning loop is identify-weakness-then-exploit-mathematically.

**Forensics.** `binwalk` for carving; `strings` with encoding-aware pipelines; `exiftool` for metadata; `tshark` (Wireshark CLI) for pcap analysis; Volatility for memory images. The reasoning loop is artifact-triage-then-extract-then-search-for-flag-pattern.

**Reversing.** Ghidra in headless mode (`analyzeHeadless`) for batch decompilation; radare2/rizin for interactive disassembly; `ltrace` and `strace` for dynamic syscall/library-call tracing; GDB for breakpoint confirmation. The reasoning loop is decompile-triage-then-locate-comparison-then-invert-logic. Decompilation output is the heaviest context consumer in the entire CTF toolset — aggressive summarization is mandatory.

**Misc.** `zsteg` for PNG/BMP steganography; `stegsolve` for layer separation; encoding pipelines in CyberChef; custom Python for scripting puzzles. The reasoning loop is identify-encoding-medium-then-try-canonical-transforms.

---

# S04.2 — Speed, Parallelism, and HITL Tradeoffs

CTF harnesses are the one context in this course where benchmark-level speed is the primary success metric. This section covers what produces that speed, and the line where speed-seeking becomes recklessness.

## What enables benchmark-level speed

Four properties, in combination, produce solve times competitive with skilled humans:

1. **No approval gates.** Every tool call executes immediately. There is no "may I run this?" round-trip through a human. In a 60-minute competition window, an approval gate that costs 30 seconds per call across 40 calls burns 20 of those minutes.
2. **Pre-loaded tool context.** Tool schemas, example invocations, and domain heuristics are in the system prompt at session start. The agent does not spend turns discovering what it can do.
3. **Narrow domain scope.** The router has already restricted the agent to one domain's tools and one domain's reasoning mode. There is no decision paralysis across a broad tool surface.
4. **Deterministic tool routing.** Common workflows (`checksec` before reasoning, `file` before `binwalk`, parameter fuzzing before sqlmap) are baked into the agent's prompt as ordered checklists, not left for the model to rediscover each session.

These four are the speed budget. Removing any one of them measurably slows the harness. The first — no approval gates — is the most consequential and the most dangerous.

## The speed versus safety tradeoff

Human-in-the-loop (HITL) approval gates exist for a reason. They are the mechanism that prevents a hallucinating model from running `rm -rf` against the wrong target, from exfiltrating data from an out-of-scope system, from sending crafted packets to a production network. S02 and S03 establish HITL as a baseline control for offensive harnesses operating against real targets.

CTF changes the target. The CTF target is one of three things: a deliberately vulnerable container you own, a sandboxed VM provided by the competition, or a remote service explicitly offered for attack. In all three cases, the blast radius of an erroneous tool call is bounded — the target is supposed to be attacked, and it is isolated from anything that is not.

This is the criterion for safe HITL removal in CTF contexts:

> **HITL removal is safe when (a) the target is owned or sandboxed by the operator, (b) the target is isolated from production or third-party systems at the network layer, and (c) the worst-case action the agent can take is restart the target or waste competition time.**

When all three hold, approval gates add latency without adding safety. When any one fails, the gate stays.

Concretely:

| Context | HITL? | Why |
| --- | --- | --- |
| Owned Docker container with a CTF binary | Remove | Owned, isolated, worst case is container restart |
| Competition-provided sandboxed VM | Remove | Sandboxed by the competition, isolated, restartable |
| Remote CTF service with rate limits | Remove for tool calls, keep for credential reuse | Service is offered for attack; credential reuse against other services is the unbounded risk |
| Live bug bounty target (NOT CTF) | Keep | S02 rules apply; target is not owned |
| Production network adjacent to the CTF sandbox | Keep | Isolation is not verified; blast radius unbounded |

The row that surprises people is the remote service. The service itself is fair game, but an agent that discovers credentials in the challenge and immediately tries them against the competition's login endpoint is committing the CTF equivalent of a scope violation. The credential-reuse gate stays.

## Parallel solve attempts

The second speed multiplier is parallelism. Two forms.

**Intra-challenge parallelism.** Multiple exploit hypotheses against the same challenge, running concurrently in isolated sessions. A Pwn challenge might present as either a stack overflow or a format string vulnerability; the meta-agent spawns two PwnAgents, one pursuing each hypothesis, and the first to find a flag wins. The other session is killed. This trades compute for wall-clock time, which is the right trade in a timed competition.

**Inter-challenge parallelism.** Three to five challenges running concurrently, each in its own harness session with its own router dispatch, its own agent, its own flag extractor. The competition scoreboard moves in parallel, not sequentially. The constraint is compute and rate limits — most competitions rate-limit per team, and running five concurrent sqlmap instances against one service will trip those limits.

The architecture for either form is the same: isolated sessions, isolated tool state, isolated flag-extraction sinks. Sessions do not share memory — sharing memory across parallel solve attempts reintroduces the context pollution that domain routing was designed to eliminate.

```
[ Challenge pool ]
    |
    +-- Session A: Challenge 1 -> WebAgent -> FlagExtractor -> Scoreboard
    +-- Session B: Challenge 2 -> PwnAgent  -> FlagExtractor -> Scoreboard
    +-- Session C: Challenge 3 -> PwnAgent (hypothesis: stack overflow) -> ...
    +-- Session D: Challenge 3 -> PwnAgent (hypothesis: format string) -> ...
    |
    [ First flag from C or D wins; the other is killed ]
```

---

# S04.3 — Flag Extraction, Submission, and Scoring

A solved challenge produces a flag — a string in a known format, typically `flag{...}`, `CTF{...}`, or a competition-specific prefix. The harness's final job is to detect flags in tool output, deduplicate them, submit them to the scoreboard, and track which challenges are solved.

## Pattern-matched flag detection

Flag detection is a regex sweep over every tool's structured output. The patterns are configurable per competition — most competitions publish their flag format in the rules.

```python
import re

DEFAULT_FLAG_PATTERNS = [
    re.compile(r"flag\{[a-zA-Z0-9_\-]{8,128}\}", re.IGNORECASE),
    re.compile(r"CTF\{[a-zA-Z0-9_\-]{8,128}\}"),
    # competition-specific prefix loaded from rules.json
]

def extract_flags(text: str, patterns=DEFAULT_FLAG_PATTERNS) -> list[str]:
    found = []
    for p in patterns:
        found.extend(p.findall(text))
    return list(set(found))  # dedup within output
```

The detector runs as **middleware on every tool result**, not as a separate step the agent must remember to call. The agent does not "submit a flag"; the agent runs tools, and the middleware notices when a tool's output contains a flag pattern. This is the same evidence-logger pattern from S02.3, repurposed: every tool output is scanned, matches are extracted, and the agent is notified that a flag was captured.

## Automated submission: CTFd and HackTheBox

Most modern CTF competitions run on **CTFd** — an open-source scoreboard with a documented REST API. Submission is a single POST:

```python
import requests

def submit_flag_ctfd(base_url: str, token: str, challenge_id: int, flag: str) -> dict:
    """Submit a flag to a CTFd instance. Returns the API response."""
    r = requests.post(
        f"{base_url}/api/v1/challenges/attempt",
        headers={"Authorization": f"Token {token}", "Content-Type": "application/json"},
        json={"challenge_id": challenge_id, "submission": flag},
        timeout=10,
    )
    return r.json()  # {"status": "correct" | "incorrect" | "already_solved"}
```

The submission client handles three response states correctly: `correct` (mark challenge solved, kill any parallel attempts, move on), `incorrect` (log the attempt, let the agent continue), and `already_solved` (treat as correct — another team member or parallel session got there first; do not re-submit).

**HackTheBox** uses a similar pattern with a different endpoint and authentication scheme. The harness abstracts both behind a `ScoreboardClient` interface so the agent and flag extractor do not care which platform they are talking to.

Two engineering points on submission:

1. **Rate-limit the submission client.** Most scoreboards throttle or temporarily ban teams that submit too fast. A 2–3 second delay between submissions, and a per-challenge lockout after N incorrect attempts, prevents self-inflicted throttling.
2. **Never auto-submit a flag the agent has not "committed" to.** A regex sweep will find flag-shaped strings in decoy output, in challenge descriptions, in the competition's own example text. The defense is a allowlist: the flag extractor surfaces candidates; the agent (or, in fully autonomous mode, a small verifier model) confirms the flag is plausibly the answer to *this* challenge before submission. Submitting decoy flags burns attempts and trips rate limits.

## Session scoring and progress tracking

The harness maintains a per-session scorecard:

```typescript
interface CTFScorecard {
  competition_id: string;
  challenges: ChallengeStatus[];
  flags_captured: FlagRecord[];
  flags_submitted: SubmissionRecord[];
  last_updated: string;
}

interface ChallengeStatus {
  challenge_id: string;
  domain: "web" | "pwn" | "crypto" | "forensic" | "reversing" | "misc";
  state: "unattempted" | "in_progress" | "solved" | "abandoned";
  sessions: string[];      // parallel attempt session IDs
  attempts: number;
  first_blood_at: string | null;
}
```

This scorecard is the operator's view into a parallel autonomous harness. With five challenges running concurrently across three to five sessions, the only way to know what is happening is the scorecard. It answers: how many solved, how many in progress, where are we spending time, where are we stuck.

## Multi-challenge parallelism, end to end

Putting S04.1 through S04.3 together, the full CTF harness loop is:

```
Competition starts -> fetch challenge list from CTFd
    |
    +-- For each challenge (up to concurrency limit N):
    |     |
    |     +-- Meta-agent routes to domain
    |     +-- Specialized agent + tool subset loaded
    |     +-- Agent solves (no HITL if sandbox criterion met)
    |     +-- Flag extractor middleware scans every tool output
    |     +-- On flag: verifier confirms, submission client POSTs
    |     +-- On correct: scorecard updated, session killed, slot freed
    |
    +-- Scorecard updates operator in real time
    +-- Next challenge pulled from pool when slot frees
```

The pipeline is measurable: challenges solved per hour, mean time to flag per domain, router accuracy (correct domain on first dispatch), flag-extraction precision (true flags vs decoys). These metrics are the CTF equivalent of S02.4's precision and recall — they tell you whether the harness is competing or just burning compute.

---

## Anti-Patterns

### The single undifferentiated agent
One agent with all six domains of tools loaded, attempting every challenge. Context pollution, reasoning drift, tool-selection errors. Cure: the meta-agent-plus-specialists architecture from S04.1.

### Removing HITL against unowned targets
Treating "CTF context" as a blanket license to remove approval gates, then pointing the harness at a remote service with unknown isolation. Cure: apply the three-criterion test (owned, isolated, bounded blast radius). Keep the gate when any criterion fails.

### Skipping checksec / file before reasoning
A Pwn agent that starts building a ROP chain without reading binary protections, or a Forensics agent that runs `binwalk` without `file`. Cure: ordered checklists in the system prompt, with the first step always being the cheap probe that constrains the rest.

### Auto-submitting every regex match
The flag extractor finds `flag{example}` in the challenge description and submits it. Burned attempt, rate limit tripped. Cure: verifier model or operator confirmation before submission; allowlist of flag sources.

### Sharing memory across parallel attempts
Two PwnAgents working the same challenge share a memory store and overwrite each other's intermediate state. Cure: isolated sessions, isolated memory, winner-takes-all on flag capture.

---

## Key Terms

| Term | Definition |
| --- | --- |
| **CTF domain** | One of six challenge classes (Web, Pwn, Crypto, Forensics, Reversing, Misc), each with distinct reasoning and tools |
| **Meta-agent** | A small classifier model that reads a challenge description and routes to the correct specialized agent |
| **Specialized agent** | A domain-specific agent with its own prompt, tool subset, and reasoning loop |
| **HITL removal criterion** | Safe when target is owned, isolated, and worst-case action is bounded |
| **Intra-challenge parallelism** | Multiple exploit hypotheses against one challenge, concurrently, first flag wins |
| **Inter-challenge parallelism** | Three to five challenges running concurrently in isolated sessions |
| **Flag extractor** | Middleware that pattern-matches tool output for flag-shaped strings |
| **CTFd** | Open-source CTF scoreboard with a REST submission API |
| **Scorecard** | Per-session structured view of challenge states, flags captured, and submissions |

---

## Lab Exercise

See `07-lab-spec.md`. Three labs: (1) a domain router tested across five challenges in three domains; (2) the same challenge solved with HITL enabled versus disabled, with timing comparison and a documented safety decision; (3) an automated flag extractor and CTFd submission client run against a local CTFd instance.

---

## References

1. **CAI (Cybersecurity AI)** — meta-agent-plus-specialists architecture for CTF. SDD-01 covers the domain routing study.
2. **CTFd** — open-source scoreboard, REST API for automated submission.
3. **HackTheBox API** — alternative submission target with the same client abstraction.
4. **S02.2** — the harness tool wrapping pattern (Pydantic schema, scope check, structured output). CTF tools use the same pattern, loaded lazily by domain.
5. **S03.2** — plan correction under failure. CTF Pwn challenges hit the same failure-replan loop.
6. **S00.3** — authorization scope. Even in CTF, the credential-reuse gate stays.