Hold out a test set without splitting paraphrases
Your task
Your agent's eval set has paraphrases: tasks that ask the same thing in different words. You are holding out part of the set as a final test set. If one paraphrase lands in the test set and another in the dev set, an agent tuned on dev has already seen the test answer.
Write test_set_size(n, pairs, target) that returns the test set size, as close to target as the rules allow.
Input and output:
n: the number of tasks, from 1 to 1,000. Tasks are numbered 0 ton - 1.pairs: up to 5,000 pairs[a, b]of tasks that are paraphrases of each other. A pair may repeat, or name the same task twice.target: the test set size you want, a whole number from 0 ton.- Return a whole number.
Rules:
- Every task goes into exactly one of the two sets, test or dev. The test set may be empty or hold every task.
- The two tasks of a pair go into the same set. So do any tasks joined by a chain of pairs.
- Return the test set size closest to
targetthat a split following these rules can reach. If two sizes are equally close, return the smaller one.
Examples
test_set_size(n: int, pairs: list[list[int]], target: int) -> intExample 1
Input
test_set_size(6, [[0, 1], [2, 3]], 3)Output
3Example 2
Input
test_set_size(7, [[0, 1], [1, 2], [2, 3], [4, 5], [5, 6]], 3)Output
3Example 3
Input
test_set_size(10, [[3, 7], [2, 8], [1, 0], [3, 2], [5, 6], [9, 6], [0, 4]], 6)Output
6