PreviousNext
AI-assisted engineering/Verifying AI output

Fix the citation check for a reverse-engineered design doc

Review the AI's codeEasy10 minAnthropic style

Your task

You asked an AI to read your codebase and write a design doc for a new team, "describing how the system actually works". Its failure-handling section reads cleanly: retries with backoff, a circuit breaker. Some of that may be how such systems usually work, not how yours does. Your team now requires each claim to cite the code that implements it, and checks the citations against an index of the codebase.

An assistant wrote find_unverified_claims(claim_lines, code_index). It is in the editor. It has a bug. Find it and fix it.

Input and output:

  • claim_lines: the doc's claims, one per line, written section|claim|citation. A citation is a file:function symbol, like src/http.rs:retry_with_backoff. A line can have empty fields or fewer than three.
  • code_index: the file:function symbols that really exist, taken from the repo.
  • Return "<section>: <claim>" for every claim that is not verified, in doc order.

Rules the tests check:

  • Split a line on |. The first field is the section, the second the claim and the third the citation. A missing field counts as empty, and fields after the third are ignored.
  • Trim whitespace from both ends of each field and of each index entry.
  • A claim is verified only when its citation is not empty and equals an index entry exactly, case included. A file alone, like src/http.rs, does not match src/http.rs:retry.
  • Every other line is reported, including a line with no citation field at all.
  • Report the trimmed section and claim, even when one of them is empty. A repeated line is reported each time.

Examples

find_unverified_claims(claim_lines: list[str], code_index: list[str]) -> list[str]
Example 1
Inputfind_unverified_claims(["request lifecycle|handler parses then dispatches|src/server.rs:dispatch", "failure-handling|we retry with exponential backoff|src/http.rs:retry_with_backoff", "data model|orders hold line items|src/db/schema.rs:Order"], ["src/server.rs:dispatch", "src/db/schema.rs:Order", "src/http.rs:send_once"])
Output["failure-handling: we retry with exponential backoff"]
Example 2
Inputfind_unverified_claims(["failure-handling|circuit breaker opens after 5 failures|"], ["src/http.rs:send_once"])
Output["failure-handling: circuit breaker opens after 5 failures"]
Example 3
Inputfind_unverified_claims(["data model|orders hold line items|src/db/schema.rs:Order"], ["src/db/schema.rs:Order"])
Output[]