Price an agent run when prompt prefixes are cached

Agent buildMedium25 min

Your task

Your eval reports what each agent run cost. Every model call in a run sends a prompt made of blocks: the system prompt, the tool list, and one block per message. Your harness bills a block at a cheaper cached rate when the call's prompt opens with blocks that an earlier call in the same run also opened with.

Write run_cost(calls, tokens, full_rate, cached_rate) that returns the run's total cost.

Input and output:

  • calls: the run's model calls in order, up to 10,000, each a list of block ids. There are up to 500,000 ids across all calls. A call may be empty, and it may hold the same id more than once.
  • tokens: the size of each block in tokens, a whole number from 0 to 100,000, keyed by block id. Every id in calls is a key. Ids are non-empty strings, compared exactly, and one id always means the same content.
  • full_rate, cached_rate: the price of one token, whole numbers from 0 to 1,000. cached_rate is at most full_rate.
  • Return the total as a whole number.

Rules:

  • A call's cached part is the longest run of blocks at its start that matches, block by block, the start of some earlier call in the same run.
  • Each block in the cached part costs its tokens times cached_rate. Every other block costs its tokens times full_rate.
  • The first call has no cached part.

Examples

run_cost(calls: list[list[str]], tokens: dict[str,int], full_rate: int, cached_rate: int) -> int
Example 1
Inputrun_cost([["sys", "tools", "u1"], ["sys", "tools", "u1", "a1", "t1"]], {"sys": 1200, "tools": 800, "u1": 40, "a1": 300, "t1": 900}, 10, 1)
Output34440
Example 2
Inputrun_cost([["sys", "u1", "a1"], ["sys", "u1", "a1", "t1"], ["sys", "u1", "a2"]], {"sys": 1200, "tools": 800, "u1": 40, "a1": 300, "t1": 900, "a2": 250, "t2": 1400, "u2": 60, "a3": 200}, 10, 1)
Output29680
Example 3
Inputrun_cost([["sys", "tools"], ["tools", "sys"]], {"sys": 1200, "tools": 800, "u1": 40, "a1": 300, "t1": 900, "a2": 250, "t2": 1400, "u2": 60, "a3": 200}, 5, 2)
Output20000