PreviousNext
AI-assisted engineering/Agent evaluation

Score whether a capacity forecast's ranges hold up

Agent buildEasy10 minGoogle style

Your task

An agent forecasts infrastructure demand and publishes a range for each period: a low and a high value it expects demand to fall between. Each month you check whether reality landed inside those ranges as often as the agent promised.

Write score_band_calibration(low, high, actual, target_pct): the monthly scorecard.

  • low, high (list of float): the forecast range for each period.
  • actual (list of float): the demand that actually happened.
  • target_pct (int): the share of periods, in percent, the agent promised its ranges would cover.
  • Return exactly 5 strings, in this order: "periods=<n>", "covered=<c>", "coverage_pct=<p>", "under_provisioned=<u>", "verdict=<v>".

Rules:

  1. Index i in each list is one period. Stop at the end of the shortest list. n is the number of periods scored.
  2. A period is covered when low <= actual <= high, both ends included. Otherwise, if actual > high, it is under-provisioned.
  3. With no periods, return ["periods=0", "covered=0", "coverage_pct=0", "under_provisioned=0", "verdict=insufficient_data"].
  4. coverage_pct is c * 100 // n, rounded down.
  5. The verdict is overconfident if coverage_pct is below target_pct - 5, underconfident if it is above target_pct + 5, and calibrated otherwise.

Examples

score_band_calibration(low: list[float], high: list[float], actual: list[float], target_pct: int) -> list[str]
Example 1
Inputscore_band_calibration([80, 90, 100, 110, 120, 130, 140, 150, 160, 170], [120, 130, 140, 150, 160, 170, 180, 190, 200, 210], [100, 95, 135, 148, 155, 131, 175, 189, 199, 240], 90)
Output["periods=10", "covered=9", "coverage_pct=90", "under_provisioned=1", "verdict=calibrated"]
Example 2
Inputscore_band_calibration([95, 95, 95, 95, 95, 95, 95, 95, 95, 95], [105, 105, 105, 105, 105, 105, 105, 105, 105, 105], [100, 130, 99, 140, 101, 160, 96, 180, 104, 200], 90)
Output["periods=10", "covered=5", "coverage_pct=50", "under_provisioned=5", "verdict=overconfident"]
Example 3
Inputscore_band_calibration([], [], [], 90)
Output["periods=0", "covered=0", "coverage_pct=0", "under_provisioned=0", "verdict=insufficient_data"]