Price an agent run when prompt prefixes are cached
Your task
Your eval reports what each agent run cost. Every model call in a run sends a prompt made of blocks: the system prompt, the tool list, and one block per message. Your harness bills a block at a cheaper cached rate when the call's prompt opens with blocks that an earlier call in the same run also opened with.
Write run_cost(calls, tokens, full_rate, cached_rate) that returns the run's total cost.
Input and output:
calls: the run's model calls in order, up to 10,000, each a list of block ids. There are up to 500,000 ids across all calls. A call may be empty, and it may hold the same id more than once.tokens: the size of each block in tokens, a whole number from 0 to 100,000, keyed by block id. Every id incallsis a key. Ids are non-empty strings, compared exactly, and one id always means the same content.full_rate,cached_rate: the price of one token, whole numbers from 0 to 1,000.cached_rateis at mostfull_rate.- Return the total as a whole number.
Rules:
- A call's cached part is the longest run of blocks at its start that matches, block by block, the start of some earlier call in the same run.
- Each block in the cached part costs its tokens times
cached_rate. Every other block costs its tokens timesfull_rate. - The first call has no cached part.
Examples
run_cost(calls: list[list[str]], tokens: dict[str,int], full_rate: int, cached_rate: int) -> intExample 1
Input
run_cost([["sys", "tools", "u1"], ["sys", "tools", "u1", "a1", "t1"]], {"sys": 1200, "tools": 800, "u1": 40, "a1": 300, "t1": 900}, 10, 1)Output
34440Example 2
Input
run_cost([["sys", "u1", "a1"], ["sys", "u1", "a1", "t1"], ["sys", "u1", "a2"]], {"sys": 1200, "tools": 800, "u1": 40, "a1": 300, "t1": 900, "a2": 250, "t2": 1400, "u2": 60, "a3": 200}, 10, 1)Output
29680Example 3
Input
run_cost([["sys", "tools"], ["tools", "sys"]], {"sys": 1200, "tools": 800, "u1": 40, "a1": 300, "t1": 900, "a2": 250, "t2": 1400, "u2": 60, "a3": 200}, 5, 2)Output
20000