A customer's whole workspace suddenly disappears

On-call incidentEasy15 min

Your task

It is 01:35 UTC. Your project tool has about 9,000 customer workspaces. Support staff can close one from the admin console when a customer asks. Closing sets status = 'closed', and the app then answers "Workspace not found". Every night at 02:00 UTC a Kubernetes CronJob, purge-closed, permanently deletes the rows and files of every closed workspace. This page comes in:

UrgentTicket: priority customer ticket open for 10 minutes with no reply (threshold 10 minutes)

What you see:

  • The ticket is from Acme Labs, a customer in Sydney with 300 users. All of them have seen "Workspace not found" since about 00:52 UTC.
  • The admin audit log has one entry for them: workspace.close id=w_8841 name="Acme Labs" by=dana@support ticket=55102 at=00:52.
  • Ticket 55102 reads: "Please close our trial workspace. Thanks, Acme Logistics."
  • 12 workspaces, this one included, are closed and waiting for tonight's purge. The database has nightly backups, and the files have no backup.

What do you do in the next 25 minutes, and what do you tell Acme Labs and Acme Logistics?

It covers

  • Admin tools
  • Scheduled jobs
  • Data recovery

About this question

A customer's whole workspace suddenly disappears

Admin toolsScheduled jobsData recovery

Model answer

Free on every account

First move: suspend the purge. It deletes for good at 02:00, and the files have no backup.

kubectl patch cronjob purge-closed -p '{"spec":{"suspend":true}}'
kubectl get cronjob purge-closed

Add -n with your namespace. SUSPEND should read True. No new run starts while it is suspended. A run already under way would continue, so also check that ACTIVE reads 0 in the same output. At 01:35 it should. kubectl get jobs also lists finished runs, so it need not be empty. This does not bring Acme Labs back, and it leaves 11 other closed workspaces unchecked.

Then reopen w_8841. Use the console if it can, or set the status back to what it was before the close:

UPDATE workspaces SET status = 'active' WHERE id = 'w_8841' AND status = 'closed';

Expect 1 row. Their 300 users have been out for 43 minutes at 01:35, so reopen right after the suspend. Ask someone at Acme Labs to sign in and check their projects and files. Reopening also takes it out of tonight's run, if the purge reads status when it runs and not from a list made earlier. So the order of these 2 steps does not matter, as long as both finish before 02:00. Suspending takes seconds and covers the other 11 too, which is why it goes first. If the close did anything else in your system, such as billing or integrations, check those too.

Check the other 11 before the purge runs. Compare each closed workspace with its ticket: does the requester match the workspace? 12 of about 9,000 workspaces is a short list, so you can read every ticket before 02:00. Keep the purge suspended until they all match. When you resume, take care: a run skipped while suspended counts as missed, and with no starting deadline a resumed CronJob starts a missed run immediately. So resume only when the list is right. Note who resumes it, so deletions do not stay off by accident. If one of the 11 has a promised deletion date, check that with the support lead.

What you tell them. Reply on Acme Labs' ticket now, since the page is about silence: "We are on this and will write again as soon as your workspace is back." Then, once it is checked, Acme Labs: "Our support team closed your workspace by mistake at 00:52 UTC. We reopened it at [time]. Nothing was deleted, but your users could not get in for that time." Acme Logistics: "We have not closed your trial yet. We will confirm the exact workspace with you, then close it." Find their real workspace through the account that sent the ticket. Name neither customer to the other.

Read the signals. The audit line shows Dana closed Acme Labs under ticket 55102, and that ticket came from Acme Logistics. Both names start with Acme. "Workspace not found" is what the app says for any closed workspace, so this is a status, not lost data. Last night's purge ran before 00:52, so nothing is deleted yet. After 02:00 the rows could come back from a backup and the files could not, and that is why the first move is about the clock.

NoYes, under an Acme Logistics ticketAcme Labs sees Workspacenot foundIs it closed in theaudit log?Look at the app and thedatabaseSuspend purge-closedbefore 02:00Reopen w_8841, have AcmeLabs checkCheck the other 11against their ticketsResume the purge oncethe list is rightClose the real trial,add a grace period

Confirm. kubectl get cronjob purge-closed shows it suspended. SELECT status FROM workspaces WHERE id = 'w_8841'; no longer says closed. A user at Acme Labs sees their files.

Fix.

  • Purge only workspaces closed at least 14 days ago, so a mistake can be undone for days and not minutes.
  • In the console, show the ticket's requester beside the workspace name, and make the agent type the name to confirm.
  • Give support a reopen button.
  • Back up the files, or turn on versioning where they are stored.

Follow-ups.

  • Have the account manager call Acme Labs today. It is around midday in Sydney, so their staff are at work.
  • Review this without blame: the tool let one click on a similar name start a delete with a 68-minute fuse.

Concept guide

Related questions