# Scaling Record Layer for Bulk Writes

**URL:** <https://forums.foundationdb.org/t/scaling-record-layer-for-bulk-writes/2249>\
**Category:** Using Layers\
**Created:** [July 16, 2020, 4:18pm UTC](https://forums.foundationdb.org/t/scaling-record-layer-for-bulk-writes/2249 "2020-07-16T16:18:26Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![rahul-nitkkr](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/rahul-nitkkr/32/754_2.png) [@rahul-nitkkr](https://forums.foundationdb.org/u/rahul-nitkkr)\
**Post date:** [July 16, 2020, 4:18pm UTC](https://forums.foundationdb.org/t/scaling-record-layer-for-bulk-writes/2249/1 "2020-07-16T16:18:26Z")

</div>

i know this has been asked before , but this post covers a slightly different issue as compared to the other [one](https://forums.foundationdb.org/t/bulk-load-in-record-layer-is-slow/1794).

We are using Record Layer to model our data and dealing with volumes in the range of ~10TB.  
Below captures the essence of our Data model -

```auto
message Parent {
    int primary_id = 1; //Primary Key
    Nested1 nested1 = 2;
    Nested2 nested2 = 3;
}

message Nested1 {
    int foo = 1;
    int bar = 2;
}

message Nested2 {
    string baz = 1;
    string bat = 2;
}

```

We have a legacy Relational Database , from which we read records and attempt to first hydrate or bulk load the data into a record layer fronted FDB Cluster ( 6.2.19 ).  
We have repeatedly run into bottlenecks on the write speed we can see on the cluster.

Cluster shape -

```auto
Number of machines - 16
Config - 
     RAM - 600 GB
     DISK - 5.5 TB ( NVMe SSDs )
     CORES - 40 
Resolvers - 4
Proxies - 4
Storage Processes - 84 ( 12 dedicated machines as storage servers )
Tlogs - 4 ( separate from Storage )

```

We use a **SINGLE** record store , since model-wise that makes sense for us.  
Essentially, our keyspace looks as follows -

```auto
/<some_parent>/<some_static_id>/<enviroment_value>/<record-store-name>/<message_type>/<primary_key>

```

The prefix is constant for all records , varying only at the level of the `primary_key`, which is the identifier.

**Our bulk loading processes run in parallel, to make more FDB network threads available to scale the write throughput, but we notice that invariably, after some time we see one of the storage processes running at 100% CPU, and as soon as that happens the cluster write throughput comes to a crawl.**  
**I’m guessing `RateKeeper` kicks in here and limits the throughput.**

Record Layer Specifics -  
This is how we get the Record store keyspace -

```auto
private static final com.apple.foundationdb.record.provider.foundationdb.keyspace.KeySpace KEY_SPACE =
                new com.apple.foundationdb.record.provider.foundationdb.keyspace.KeySpace(
                        new DirectoryLayerDirectory("some_parent")
                                .addSubdirectory(new DirectoryLayerDirectory("some_static_id")
                                        .addSubdirectory(new DirectoryLayerDirectory("enviroment_value"))));

```

Finally ,

- **Is there a way we can scale the write throughput with the above model in place ?**
- We currently see write rate range around ~800000 hz, but the rate of increase of key-val sizes is ~3.5GB/min. Which is almost the same when the write rate were ~400000Hz. **Is there a way to have a better control on the shard assignment?** since i guess our prefix for keyspace is not allowing better partitioning of the data

---

<div class="post-metadata">

**Author:** ![rahul-nitkkr](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/rahul-nitkkr/32/754_2.png) [@rahul-nitkkr](https://forums.foundationdb.org/u/rahul-nitkkr)\
**Post date:** [July 16, 2020, 4:20pm UTC](https://forums.foundationdb.org/t/scaling-record-layer-for-bulk-writes/2249/2 "2020-07-16T16:20:24Z")

</div>

Hey @alloc , Tagging you in case you miss this. 🙂

---

<div class="post-metadata">

**Author:** ![sears](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/sears/32/887_2.png) [@sears](https://forums.foundationdb.org/u/sears)\
**Post date:** [July 23, 2020, 8:54pm UTC](https://forums.foundationdb.org/t/scaling-record-layer-for-bulk-writes/2249/3 "2020-07-23T20:54:11Z")

</div>

Are the parallel workers writing disjoint key ranges? The best case for FDB is when the writes are spread across the key space, so breaking the primary key space up into contiguous ranges, and giving each load process one range to process should help.

If you are already doing that, it’s likely that the writes will initially hit a single shard. FDB should detect this, and split the shard, allowing the writes to hit multiple machines.

When you see 100% load on one storage server, how busy are the other storage servers?

---

<div class="post-metadata">

**Author:** ![mengxu](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/mengxu/32/893_2.png) [@mengxu](https://forums.foundationdb.org/u/mengxu)\
**Post date:** [July 24, 2020, 5:07am UTC](https://forums.foundationdb.org/t/scaling-record-layer-for-bulk-writes/2249/4 "2020-07-24T05:07:00Z")

</div>

Besides what @sears’s valuable points,

Empirically, the tLog to storage server ratio is 1:14. (It may change based on hardware types.)

For a better write throughput, it is, empirically, better to have more tLogs.

---

<div class="post-metadata">

**Author:** ![rahul-nitkkr](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/rahul-nitkkr/32/754_2.png) [@rahul-nitkkr](https://forums.foundationdb.org/u/rahul-nitkkr)\
**Post date:** [July 24, 2020, 5:31am UTC](https://forums.foundationdb.org/t/scaling-record-layer-for-bulk-writes/2249/5 "2020-07-24T05:31:35Z")

</div>

There are a range of keys which are written by each parallel process.  
eg - Let’s suppose we have to write 100000 Ids ( Keys ) , and there are 10 processes , the distribution would be , P1 - 0-10000, P2 - 10001-20000 and so on.

`When you see 100% load on one storage server, how busy are the other storage servers?`  
That is the interesting part, when one of the storage processes hits 100% , the rest are usually \< 10%.  
One fact which i probably did not mention is the updation of record layer indexes also happens in each transaction. We have around 30 indexes defined.

---

<div class="post-metadata">

**Author:** ![sears](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/sears/32/887_2.png) [@sears](https://forums.foundationdb.org/u/sears)\
**Post date:** [July 24, 2020, 6:40pm UTC](https://forums.foundationdb.org/t/scaling-record-layer-for-bulk-writes/2249/6 "2020-07-24T18:40:45Z")

</div>

The distribution of keys looks OK.

For the sake of debugging, can you get rid of the indexes? If it fixes the load imbalance, then perhaps one of the indexes has a problematic access pattern. Add them back in a few at a time until the problem reproduces.

---

<div class="post-metadata">

**Author:** ![MMcM](https://avatars.discourse-cdn.com/v4/letter/m/d07c76/32.png) [@MMcM](https://forums.foundationdb.org/u/MMcM)\
**Post date:** [July 24, 2020, 8:28pm UTC](https://forums.foundationdb.org/t/scaling-record-layer-for-bulk-writes/2249/7 "2020-07-24T20:28:03Z")

</div>

Your workload seems simple enough that adjusting it to narrow down the issue may be the quickest way.

But an alternative to consider is turning on client request sampling with

```auto
profile client set <RATE> <SIZE>

```

in `fdbcli` and then using `DatabaseClientLogEvents` and `KeySpaceCountTree` in Record Layer to analyze them.

Specifically, this allows you to get a hierarchical breakdown of sampled read/write requests by storage server by key range, decoded according to your key-space paths. It understands subspaces inside the record store keyspace like particular indexes.

We use it as part of some higher level tooling, but the unit tests should be enough to understand the API.