Third-party tool integration

You run an agent platform that routes work through a few hundred third-party tools. The schemas are syntactically accurate and semantically thin. Rate limits are undocumented and keyed to the vendor's global state, so they move. Large results are silently truncated. Retries are not always idempotent. A field's meaning changed when the vendor shipped and nothing in the schema moved. Some endpoints return success and do nothing.

The constraints are the point. You may not call billable or irreversible tools speculatively to explore their behaviour. No vendor gives you a sandbox whose semantics you trust. No vendor will run a contract suite for you. And the only behavioural data you have is production traces, which is to say calls your planner chose to make while believing the model of the tool that you are now trying to correct. The tool paths your model considers bad are the ones your traces do not contain.

Design the subsystem that builds and maintains a behavioural model of each tool and serves it to the planner. Cover acquisition from that biased corpus, how you decide a tool's model has gone stale, what the planner does when its model and reality disagree in the middle of a plan, and how you notice semantic failures that arrive wearing a success code. Then do the failure analysis: name the class of tool behaviour your design will systematically never learn, and say why you are accepting that rather than what you would do with a bigger budget.

Ties break toward designs that treat the trace corpus as a biased sample and say concretely how, over designs whose answer to every gap is more logging or a wider retry policy.