Delayed job scheduler

Design a distributed delayed-job / scheduled-task system that can hold hundreds of millions of pending jobs, each scheduled to fire at an arbitrary future time (seconds from now to a year out), e.g., 'send this reminder in 3 days', 'expire this hold in 15 minutes', 'retry this charge at 9am tomorrow'. Jobs must fire close to on time (within a few seconds), exactly-ish once, and the system must absorb spikes where millions of jobs are due in the same minute (e.g., a midnight batch). How do you store, schedule, and dispatch these at scale?