PreviousNext
AI-assisted engineering/Agent evaluation

Keep the generated tests that catch real bugs

Agent buildEasy10 minMeta style

Your task

An agent writes tests for a legacy service that has almost none. Coverage will rise either way, so each generated test is judged by mutation testing instead.

Write triage_generated_tests(assertions, mutants_killed): the rule your agent follows to keep, review or throw away each test.

  • assertions (list of str): what each generated test asserts, in plain words, for example "cart total equals 42 for three items".
  • mutants_killed (list of int): how many mutants each test killed. A mutant is a copy of the code with one small bug planted on purpose, and a test kills it by failing on it. mutants_killed[i] belongs to assertions[i].
  • Return one verdict per pair, in order: "accept", "review" or "reject". If the lists differ in length, stop at the end of the shorter one.

Rules, for each pair:

  1. Lowercase the assertion and trim whitespace from both ends. If it is empty, or contains not null, not none, is truthy or does not throw anywhere: "reject", however many mutants it killed.
  2. 0 or fewer mutants killed: "reject".
  3. Exactly 1: "review".
  4. 2 or more: "accept".

Examples

triage_generated_tests(assertions: list[str], mutants_killed: list[int]) -> list[str]
Example 1
Inputtriage_generated_tests(["result is not null", "cart total equals 42 for three items"], [0, 4])
Output["reject", "accept"]
Example 2
Inputtriage_generated_tests(["response status is 200", "handler does not throw"], [1, 5])
Output["review", "reject"]
Example 3
Inputtriage_generated_tests(["discount is applied once per order", "user id is not none"], [3, 2])
Output["accept", "reject"]