Hold out a test set without splitting paraphrases

Agent buildMedium25 min

Your task

Your agent's eval set has paraphrases: tasks that ask the same thing in different words. You are holding out part of the set as a final test set. If one paraphrase lands in the test set and another in the dev set, an agent tuned on dev has already seen the test answer.

Write test_set_size(n, pairs, target) that returns the test set size, as close to target as the rules allow.

Input and output:

  • n: the number of tasks, from 1 to 1,000. Tasks are numbered 0 to n - 1.
  • pairs: up to 5,000 pairs [a, b] of tasks that are paraphrases of each other. A pair may repeat, or name the same task twice.
  • target: the test set size you want, a whole number from 0 to n.
  • Return a whole number.

Rules:

  • Every task goes into exactly one of the two sets, test or dev. The test set may be empty or hold every task.
  • The two tasks of a pair go into the same set. So do any tasks joined by a chain of pairs.
  • Return the test set size closest to target that a split following these rules can reach. If two sizes are equally close, return the smaller one.

Examples

test_set_size(n: int, pairs: list[list[int]], target: int) -> int
Example 1
Inputtest_set_size(6, [[0, 1], [2, 3]], 3)
Output3
Example 2
Inputtest_set_size(7, [[0, 1], [1, 2], [2, 3], [4, 5], [5, 6]], 3)
Output3
Example 3
Inputtest_set_size(10, [[3, 7], [2, 8], [1, 0], [3, 2], [5, 6], [9, 6], [0, 4]], 6)
Output6