Storing one billion floats with dense keys


I want to store a large rectangular volume of floats, and it’d be nice if every single lattice point had its own entry in the database (so I could pick it out individually). If you include the namespace, that means that the values will be much smaller than their keys. They keys will also be dense.

In many environments, feeding the lattice into a key-value store would be one of the worst possible ideas, but FDB may be clever enough to make it work. Will FDB’s internal optimizations handle this abuse gracefully, or would there be something to gain in chunking my data into larger values?

(Ryan Worl) #2

There are no optimizations today for compressing keys with shared prefixes. You should probably write your data into 4k chunks at the low end if you need fine grained access. If you go smaller than that, you won’t get much benefit in terms of IO because disk pages are 4k and you’ll be spending more bytes on keys.

You should also explore larger than 4k chunks if your workload uses more than 4k worth of data points at a time. You won’t get much additional performance, but you will spend less storing keys. If you didn’t need all that extra data, it would be a waste. So it depends on the workload.

1 Like