Is the rocksdb storage engine ready for production use?

Using FoundationDB
1

Is the rocksdb storage engine ready for production use in the latest release 7.3.x?

2

I can’t answer your question (though I’m pretty sure the answer is No) but is there some reason you’re interested in the rocksdb engine specifically?

Have you tried the Redwood engine? It has been production ready for a while - Snowflake finished migrating our entire fleet to Redwood about a year ago and we’ve been using it in production since FDB 7.1 about two years ago. We’ve encountered zero data corruption / loss / wrong result / stability issues, and it has eliminated very frequent storage engine bottlenecks in our fleet. Compared to the ssd-2 engine it provides lower latency and much higher throughput while using less CPU, IO, and disk space.

1 Like
3

Thanks for your reply.

Our workload is write-intensive and there are very few data updates, in this scenario, I think rocksdb will have better performance compared to B±tree based storage engines.

4

And yes, we have tried the Redwood engine. Compared to the ssd-2 engine, It does perform better.

5

write-intensive and there are very few data updates

This sounds like a heavy insert workload. If the inserts are extremely random with almost no key locality then yes you might see better write performance at least for a time with an LSM, but I would be surprised if you see reliable sustained performance long term. Also, regardless of your workload pattern FDB’s data movement alone does not execute well on the rocksdb engine.

If you haven’t done so already, I suggest you run performance tests which drive a workload continuously for several days. This gets past the “honeymoon period” which LSMs tend to have where writes are overly cheap because compaction debt is being built up.

6

Also, if you’d like to share more information about your workload and the performance you saw with Redwood I might be able to suggest some tweaks to improve it.

1 Like
7

To your question, we have been starting using 7.3.43 with RocksDB in production in a few clusters, and many more clusters for several months in Testing Storage Server (TSS). In a few months, we can probably report back our experience with RocksDB engine. So far we haven’t encountered any issues. For write heavy workload, RocksDB should perform better than ssd-2.

4 Likes
8

Hello @jzhou

We are starting to evaluate rocksdb storage engine for our mixed read/write workloads. I was curious to know your experience running rocksdb in production so far.

Did you run into any significant problems with data movement on rocksdb? I see release 8.0 adding support for sharded RocksDB. Is that feature targeted at streamlining the data movement performance? And is it going to be prod ready in the upcoming March release?

Thank you!

9

We have dozens of production clusters running on RocksDB now (currently on 7.3.57). For our workload, we don’t see much difference in terms of latency. During data movement, there could be slightly increase of latency. Note our production workload is far from saturation.

Sharded RocksDB in 8.0 is still experimental and we hope to qualify the engine in that release. Yes, the main benefit is to improve data movement performance, i.e., copying files instead of writing through the storage engine on the destination storage servers.