# Slicing a key range to work with analytical engines (e.g. Spark)

**URL:** <https://forums.foundationdb.org/t/slicing-a-key-range-to-work-with-analytical-engines-e-g-spark/2231>\
**Category:** Using FoundationDB\
**Created:** [July 7, 2020, 3:50am UTC](https://forums.foundationdb.org/t/slicing-a-key-range-to-work-with-analytical-engines-e-g-spark/2231 "2020-07-07T03:50:38Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![omidb](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/omidb/32/925_2.png) [@omidb](https://forums.foundationdb.org/u/omidb)\
**Post date:** [July 7, 2020, 3:50am UTC](https://forums.foundationdb.org/t/slicing-a-key-range-to-work-with-analytical-engines-e-g-spark/2231/1 "2020-07-07T03:50:38Z")

</div>

Most analytical engines like Apache Spark slice the data into chunks and perform some sort of map/reduce action on them. In FDB, reading a range is well documented and there is this [hint](https://apple.github.io/foundationdb/known-limitations.html#dont-use-key-selectors-for-paging) to not use offset for pagination through the data.  
I’m using the Java Api, is there any way to slice the data into N ranges or multiple ranges with a fixed size? If not, what would be the best way to implement it?

---

<div class="post-metadata">

**Author:** ![SteavedHams](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/steavedhams/32/18_2.png) [@SteavedHams](https://forums.foundationdb.org/u/SteavedHams)\
**Post date:** [July 7, 2020, 4:24am UTC](https://forums.foundationdb.org/t/slicing-a-key-range-to-work-with-analytical-engines-e-g-spark/2231/2 "2020-07-07T04:24:56Z")

</div>

This feature is being worked on, the issue tracking it is here:

> <https://github.com/apple/foundationdb/issues/3391>
>
> The new API would allow clients to ask FDB to return a list of keys(split points) that would split the given...

---

<div class="post-metadata">

**Author:** ![omidb](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/omidb/32/925_2.png) [@omidb](https://forums.foundationdb.org/u/omidb)\
**Post date:** [July 7, 2020, 3:08pm UTC](https://forums.foundationdb.org/t/slicing-a-key-range-to-work-with-analytical-engines-e-g-spark/2231/3 "2020-07-07T15:08:22Z")

</div>

Thanks @SteavedHams … what would be the best workaround in the meantime? Does the `LocalityUtil` an option?

---

<div class="post-metadata">

**Author:** ![andrew.noyes](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/andrew.noyes/32/443_2.png) [@andrew.noyes](https://forums.foundationdb.org/u/andrew.noyes)\
**Post date:** [July 7, 2020, 5:03pm UTC](https://forums.foundationdb.org/t/slicing-a-key-range-to-work-with-analytical-engines-e-g-spark/2231/4 "2020-07-07T17:03:13Z")

</div>

[https://apple.github.io/foundationdb/javadoc/com/apple/foundationdb/LocalityUtil.html#getBoundaryKeys-com.apple.foundationdb.Database-byte:A-byte:A-](https://apple.github.io/foundationdb/javadoc/com/apple/foundationdb/LocalityUtil.html#getBoundaryKeys-com.apple.foundationdb.Database-byte:A-byte:A-) gets you keys that split your keyrange into similarly sized chunks. This should be appropriate if you’re ok with your chunks being as large as [500mb](https://github.com/apple/foundationdb/blob/739767b8381f22995b996cb0384a310ea9147209/fdbserver/Knobs.cpp#L139)

---

<div class="post-metadata">

**Author:** ![omidb](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/omidb/32/925_2.png) [@omidb](https://forums.foundationdb.org/u/omidb)\
**Post date:** [July 7, 2020, 5:22pm UTC](https://forums.foundationdb.org/t/slicing-a-key-range-to-work-with-analytical-engines-e-g-spark/2231/5 "2020-07-07T17:22:47Z")

</div>

500mb is too big 😟  
These little things make the adoption very hard.