Fill a labeling queue without starving the rare class

Agent buildEasy12 minOpenAI and Anthropic style

Your task

Your labeling team is the bottleneck for your classifier, and most of what they label the model already gets right. An agent picks which predictions go to the labelers each day, without starving a rare class.

Write build_label_queue(margins, predicted_classes, budget, rare_class, rare_floor, control_slots): the rule your agent follows to fill the queue.

  • margins (list of float): for each prediction, the gap between the model's top two class scores. A small margin means the model is unsure.
  • predicted_classes (list of str): the predicted class of each prediction.
  • budget (int): how many examples the labelers can take.
  • rare_class (str): the class that must not be starved.
  • rare_floor (int): how many examples of the rare class to include, if there are enough.
  • control_slots (int): how many examples to pick in input order, as an unbiased control.
  • Return the indices of the selected examples, in ascending order.

Rules:

  1. The queue holds at most budget examples, and never more than there are predictions. If that limit is 0 or less: return [].
  2. Rank all indices by margin, smallest first, breaking ties by the smaller index.
  3. Fill the queue in 3 passes. Never pick an index twice, and stop as soon as the queue is full.
    • Pass 1: walk the ranking and pick examples of rare_class until this pass has picked rare_floor.
    • Pass 2: walk the indices in input order (0, 1, 2 and so on) and pick unpicked ones until this pass has picked control_slots.
    • Pass 3: walk the ranking and pick unpicked ones until the queue is full.

Examples

build_label_queue(margins: list[float], predicted_classes: list[str], budget: int, rare_class: str, rare_floor: int, control_slots: int) -> list[int]
Example 1
Inputbuild_label_queue([0.05, 0.9, 0.62, 0.42, 0.07], ["spam", "spam", "fraud", "spam", "spam"], 3, "fraud", 1, 1)
Output[0, 2, 4]
Example 2
Inputbuild_label_queue([0.31, 0.02, 0.77], ["a", "b", "a"], 2, "z", 2, 0)
Output[0, 1]
Example 3
Inputbuild_label_queue([], [], 5, "fraud", 1, 1)
Output[]