A write rejected when the quota runs out

The idea

Every tenant has a storage quota — a hard cap on how much space their data may occupy. A write that would push usage past that cap is rejected, often with HTTP 507, EDQUOT, or a 403 QuotaExceeded result.

The real danger is timing. If the rejection lands mid-operation — partway through a multi-part upload, or mid-batch — and writes aren't atomic, you can be left with a half-written, inconsistent object that still consumed space. The calm response is to triage (confirm it's quota, find what consumed it, check for partial junk) and then contain (stop the writer, fail cleanly, abort the fragment, free space or raise the cap, then resume).

See it work

How it works

The fix is to decide before you start writing. Atomically reserve the space, check it against what remains, and reject up front if it won't fit. Then a rejection costs nothing and never leaves a fragment behind. If a write still fails mid-stream, abort the partial object so it doesn't linger as orphaned bytes.

def write(tenant, obj, size):
    # Pre-flight: reserve space ATOMICALLY before a single byte lands.
    # used + size is computed under a lock (or a conditional update / CAS)
    # so two concurrent writers can't both squeak past the same headroom.
    with tenant.reservation_lock:
        if tenant.used + size > tenant.limit:
            reject(QUOTA_EXCEEDED)        # 507 / EDQUOT — fail fast, clean
            return
        tenant.used += size               # the reservation is now committed

    try:
        store.put(tenant, obj)            # only now do bytes actually flow
    except WriteFailed:
        store.abort(tenant, obj)          # roll back the partial object...
        tenant.used -= size               # ...and release the reservation
        raise

Signals

Watch out for

• No pre-flight. Starting a write you can't finish means the reject lands mid-stream — leaving a partial, possibly corrupt object instead of a clean refusal.
• Treating a quota reject as transient. A 507 isn't a blip. Blind retry-with-backoff becomes a retry storm that can never succeed until space is freed.
• Counting quota only on success. If the partial write already consumed space but you only debit usage on success, the freed bytes are never reclaimed — a slow leak toward the cap.
• No per-tenant isolation. If quotas share one pool, one tenant's overflow can starve writes for everyone else.
• Alerting at 100%. Paging only when the cap is hit leaves no runway. Alert at 85% so there's time to raise quota or free space first.

Worked example

A tenant sits at 9.8 / 10 GB and starts a 0.5 GB upload. Without a pre-flight check, the store happily writes 0.2 GB, hits the cap, and rejects — leaving a 0.2 GB partial fragment behind.

Triage: the error is QuotaExceeded, not a generic failure; the cause is this tenant at the cap; an orphaned 0.2 GB partial is confirmed.

Contain: abort the partial (usage drops back to 9.8), then free 1 GB of expired temp data (down to 8.8). Now 8.8 + 0.5 = 9.3 < 10 — the upload retries and succeeds.

Check yourself