Discussion thread for new storage engine ideas

I think some form of compressing storage engine could be very useful. Prefix compression might be especially handy given that many layers want to represent data hierarchically (and tuples/directories encourage this).

The straw-man argument for why this should be low priority is that you could easily implement it in a layer, and it could be done “adaptively” by storing the versioned compression dictionary itself in the database, keeping metadata about what dictionary version(s) different portions of keyspace were written at, and having a background process that periodically re-compresses old data and garbage collects old dictionary versions.

You could probably do better by writing something special-purpose for each layer that takes advantage of what it knows about the layer’s data model. But that seems like an optimization, and it’d be nice to have a default besides “no compression”.