Teaching an agent that severity isn’t difficulty

The problem

The Deep Space Network is NASA’s array of large radio antennas in California, Spain, and Australia, and the link to every spacecraft operating beyond the Moon.¹ A missed communication pass can mean lost science data, so the network’s most important property is availability.

That makes routine security work delicate. The standard fix for a vulnerability is to patch and reboot, and rebooting a mission-critical machine during an active tracking pass is not an option. Scanners rank findings by CVSS,³ a severity score that is deliberately context-free: it can’t know what a machine does, or whether it can go down tonight. Analysts fill that gap by hand, cross-checking every alert against documentation, configuration records, and years of institutional knowledge.

Meanwhile the number of disclosed vulnerabilities keeps climbing, and LLMs are beginning to speed up how attackers find and exploit them. Defense that depends entirely on manual correlation can’t keep pace, and correlation is exactly what a well-equipped LLM agent is good at.

What I built

An end-to-end pipeline that turns a raw scanner finding into a reviewed, structured remediation recommendation. The agent decides which tools to call to gather context, reasons about the vulnerability, and returns a validated assessment. It recommends; it never acts. Step through it below, and switch between the two example alerts.

{
  "vuln_id": "SYN-A",
  "severity": "High",
  "difficulty": "easy",
  "reason": "Routine package update on a
    redundant, non-mission host. Rolls back
    cleanly; fits a normal maintenance window.",
  "considerations": ["Update one node at a time"],
  "known_exploited": false
}

{
  "vuln_id": "SYN-B",
  "severity": "Medium",
  "difficulty": "hard",
  "reason": "Needs a reboot of a mission-critical
    host with no hot spare. Schedule outside
    tracking passes.",
  "considerations": ["Coordinate with the tracking
    schedule", "Verify signal processing after reboot"],
  "known_exploited": false
}

The central result is that inversion. A high-severity flaw whose fix is a routine, easily rolled-back update is classified easy. A moderate one whose fix needs a reboot of core services, a firmware change, or a performance-costing patch is classified hard. The agent gets there by consulting the same internal documentation an analyst would, and it has to cite what it used.

Structured output as a contract

Free-form prose can’t feed a pipeline. The agent is built on Pydantic AI,⁴ which constrains the model to fill a strictly typed schema. When the output violates it, the validation error is fed back and the model corrects itself, up to a fixed number of retries. That loop is what turned a promising demo into a component other tools can rely on. Inference runs at temperature zero, and every assessment is stamped with the prompt and schema versions, the model ID, and the backend it ran on, so any result can be traced and reproduced.

Context on demand, through MCP

Putting every potentially relevant document into the prompt would be expensive and would overflow the context window. Instead, each context source sits behind a Model Context Protocol⁵ server, and the agent retrieves only what a given vulnerability needs. The sources are the engineering documentation (the primary, authoritative source), an internal wiki of procedures and known issues, an acronym database, a configuration-item database, and the existing vulnerability data. Each server runs as a local process that talks only to internal resources, so sensitive data stays inside approved boundaries.

The backend is model-agnostic. Switching between three approved backends takes one configuration value, so analysis can be routed by data sensitivity, and the tool improves as the underlying models do.

Chains, not lists

Attackers rarely use one vulnerability. They chain them,⁶ turning a minor foothold into a position from which a serious compromise becomes possible, and a flat list of findings hides that. A second agent reasons about how vulnerabilities across interdependent systems could combine, and the dashboard draws the result as an interactive attack-path diagram.

Built to run for real

Caching

Repeated context queries are cached, cutting cost on repeated queries by an estimated 50–70%.

Checkpoints

Progress is saved after every assessment, so an interrupted run resumes and loses at most one item.

Adaptive pacing

When a backend rate-limits, the delay between requests widens and honors cooldown hints, then relaxes as pressure clears.

Cost tracking

Input, output, and cached tokens are recorded per assessment, with actual and cloud-equivalent cost.

What comes next

The next step is measuring how good the recommendations really are, by comparing the agent’s calls with what experienced DSN analysts would decide. Doing that well takes a purpose-built dataset: real vulnerabilities from the network, each paired with an analyst’s judgment and the reasoning behind it. Assembling that takes real analyst time and care, which is why it comes after the prototype rather than alongside it. Until it exists, I’m not putting an accuracy number on the system.

Two capabilities are already prototyped and ready for the same hardening: vision, so the agent can read the diagrams in DSN documentation directly, and a tightly sandboxed container that tries to reproduce an exploit to see whether it’s actually feasible.

Two research questions came out of the internship: whether structured vulnerability-test data improves the agent’s grasp of real-world impact, and whether it can reason over firewall rule sets to prioritize by actual reachability⁷ instead of assuming worst-case connectivity.