Social feed (fan-out design)

The idea

A home timeline shows you recent posts from everyone you follow. The hard question is when to do the merging work: once per post (at write time) or once per app open (at read time).

Fan-out on write (push) copies each new post into every follower’s precomputed inbox, so a read is just “return my inbox.” Fan-out on read (pull) stores each post once and gathers posts from everyone you follow at open time. Reads and writes trade places; real systems do both.

How it works

Pick where the cost lands. With fan-out on write, the author’s post() loops over followers and appends the post id to each follower’s precomputed feed, so read() is a single lookup. With fan-out on read, post() writes once to the author’s own outbox, and read() gathers and merges recent posts from every account you follow.

# --- fan-out on write (push): heavy post(), trivial read() ---
def post_write(author, post_id):
    save_post(post_id, author)
    for follower in followers_of(author):     # O(followers) writes
        feed[follower].prepend(post_id)        # fill each inbox now

def read_write(user):
    return feed[user][:PAGE]                    # 1 read: just your inbox

# --- fan-out on read (pull): trivial post(), heavy read() ---
def post_read(author, post_id):
    outbox[author].prepend(post_id)            # 1 write to your own outbox

def read_read(user):
    posts = []
    for author in following_of(user):          # O(following) fetches
        posts += outbox[author][:PAGE]
    return merge_by_time(posts)[:PAGE]          # merge at read time

The cost is conserved, not removed: push pays O(followers) per post and reads for free; pull posts for free and pays O(following) per open. Which is cheaper depends on the read:write ratio and on how skewed the follower counts are.

Cost / trade-offs

The real-world answer is hybrid: push posts from normal authors into inboxes, but leave celebrity authors on pull and merge their recent posts in at read time. You pay a small per-open merge for the handful of huge accounts and avoid the million-write storm.

Watch out for

• The celebrity fan-out storm. One post from a 50M-follower account becomes 50M inbox writes under pure push — a single post can saturate the write path. Exempt high-follower authors from push and pull them in at read time instead.
• Unbounded feed storage. Duplicating every post id into every inbox multiplies storage and grows without limit. Cap each inbox to the most recent N entries and trim on write; let older history come from pull.
• Deletes and edits must propagate. A deleted or edited post lives in thousands of inbox copies. Either fan out a tombstone to every copy, or store only post ids and resolve content (and its “deleted” flag) at read time so a delete is honoured everywhere.
• Ranking and recency, not pure chronological. A precomputed inbox locks in insertion order; a ranked feed needs scoring at read time, which pushes you back toward pull or toward storing candidates and ranking on open.
• Thundering herd on hot accounts. Under pull, a celebrity’s outbox is fetched by every viewer at once and becomes a hot key. Cache the hot author’s recent posts and share that cached slice across readers.

Worked example

A user follows three normal friends and one celebrity. The normal friends’ posts are pushed into the user’s inbox at write time; the celebrity is pulled and merged at read time.

# write time — normal authors push, celebrity does not
friend_a.post(p101)   # pushed -> inbox[user].prepend(p101)
friend_b.post(p102)   # pushed -> inbox[user].prepend(p102)
celeb.post(p900)       # 40M followers -> NO push; outbox[celeb].prepend(p900)

# read time — open the app
inbox = feed[user][:PAGE]               # 1 read: p102, p101, ... (pushed)
celeb_posts = []
for a in celebrities_followed(user):    # tiny set, usually 0-5 accounts
    celeb_posts += cache_or_outbox(a)[:PAGE]   # p900 (pulled, cached)
home = merge_by_time(inbox + celeb_posts)[:PAGE]

The two friend posts cost two inbox writes total; the celebrity post cost zero writes instead of 40 million. At read time the user pays one inbox read plus a tiny merge over the few celebrities they follow — the cost lands where it is cheapest in each case.

Check yourself