Research assistant answers on patchy phone connections
Your task
Your team runs a research assistant app. A person asks a question, and a hosted model writes a long answer with sources. The app shows the answer word by word as it arrives. An answer takes 30 to 90 seconds to write and costs about $0.04 in model fees. The first word can take up to 5 seconds.
The app writes about 8 million answers a day, and at peak 20,000 are being written at once. Most people use the phone app on their commute. About 1 answer in 5 is cut off on the way, when the phone changes network or enters a tunnel. About a tenth of those cuts come before the first word reaches the phone. Some people reopen the app only an hour later.
Today a cut-off answer is gone, and the app sends the question again. Your team wants to pay for each question once, and the person should still get the whole answer.
What happens to an answer when its phone disappears, and what does the phone send when it comes back?
It covers
- Streamed answers
- Mobile connections
- Model costs