Making RocksDB and SQLite fly without blowing up your server's RAM.
Many modern databases (like CockroachDB, Cassandra, or custom data pipelines) don't write their own storage engines. Instead, they embed a library like RocksDB (a Key-Value store) directly inside their application code. While RocksDB is incredibly fast, it requires massive tuning of its memory buffers (MemTables) and disk compaction algorithms (LSM Trees). If you leave it on default settings, your application will randomly freeze (stall) or crash from Out-of-Memory (OOM) errors under high load.
Writes in RocksDB first go to a fast RAM buffer called a MemTable. When the MemTable fills up, it flushes to a permanent file on disk (SSTable). If you write data faster than the disk can flush it, the MemTable memory grows endlessly. To prevent crashing your server, RocksDB triggers a Write Stall—it intentionally pauses your application code to let the disk catch up. This looks like a massive latency spike to your users.
// Tuning RocksDB in Go (preventing write stalls)
opts := gorocksdb.NewDefaultOptions()
// 1. Give it more RAM to absorb sudden write spikes
opts.SetWriteBufferSize(256 * 1024 * 1024) // 256MB per MemTable
opts.SetMaxWriteBufferNumber(4) // Allow 4 MemTables (1GB total)
// 2. Allow it to use more CPU threads to flush to disk faster
env := gorocksdb.NewDefaultEnv()
env.SetBackgroundThreads(4)
opts.SetMaxBackgroundCompactions(4)
opts.SetEnv(env)Tuning embedded databases is a zero-sum game. If you give RocksDB a 4GB MemTable, writes will be blazing fast, but you steal 4GB of RAM from your main application. If you give it 8 background threads for compaction, you steal CPU cores from serving web requests. You have to balance the engine's greed against the application's needs.