LiteFS
published on 2022/09/25
LiteFS extends the idea of Litestream with fine-grained transactional control. Where Litestream simply copies the raw SQLite WAL file, LiteFS can inspect and ship individual transactions, which span pages, and are the true unit of change in a SQL database.
LiteFS works by interposing a very thin virtual filesystem between your app and your on-disk database file. It's not a file system like ext4, but rather a pass-through. Think of it as a file system proxy. What that proxy does is track SQLite databases to spot transactions and then LiteFS copies out those transactions to be shipped to replicas.
You see where this is going. SQLite's exquisitely documented file format makes it easy for LiteFS to replicate whole databases. Now we've got transaction boundaries. So we roll those transactions up into a simple file format we call LTX. LiteFS replicas can replay those transactions back to recreate the current (or any previous) transaction state of a LiteFS-tracked SQLite database — without touching app code. It seems like magic, but it's a natural consequence of SQLite's strong design.
We really like Fly.IO. It is an interesting company creating open source products and services with a lot of interesting ideas.