Choose the model tier for in-editor autocomplete

Judgment callMedium20 minOpenAI and Microsoft style

Your task

You are shipping autocomplete in your company's code editor: as the user types, your service suggests the rest of the line or block. Product wants "the smartest model, so suggestions are great".

You measured 2 options. The large model takes 1.8 seconds at the median. The small one takes 250 ms, but in an offline test on hard completions, its suggestions were accepted noticeably less often.

Which do you ship, and how do you defend the choice to product?

It covers

  • Model tier
  • Latency budget