# Abstracting hundred-thousands of collections of items

**URL:** <https://forums.foundationdb.org/t/abstracting-hundred-thousands-of-collections-of-items/1543>\
**Category:** Using FoundationDB\
**Created:** [July 29, 2019, 7:52pm UTC](https://forums.foundationdb.org/t/abstracting-hundred-thousands-of-collections-of-items/1543 "2019-07-29T19:52:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![janderland](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/janderland/32/517_2.png) [@janderland](https://forums.foundationdb.org/u/janderland)\
**Post date:** [July 29, 2019, 7:52pm UTC](https://forums.foundationdb.org/t/abstracting-hundred-thousands-of-collections-of-items/1543/1 "2019-07-29T19:52:10Z")

</div>

I want to create an FDB instance holding holding several hundred-thousand collections of items. Each of these collections have a name, and need to be re-namable. I’d appreciate the community’s opinion on this.

# 1. Separate Name Index

First idea is to use a separate index for the name. The main data entries would look like this:

```auto
(app_dir, collections_dir, 1, item1) = data1
(app_dir, collections_dir, 1, item2) = data2
...

```

Then the index would look like this:

```auto
(app_dir, names_dir, collection_name) = 1

```

# 2. Collection Directories

The second idea is to use directories for each collection which are inherently re-namable:

```auto
(app_dir, collections_dir, collection1_dir, item1) = data1
(app_dir, collections_dir, collection1_dir, item2) = data2
...

```

I’m leaning towards the first plan because I don’t know of a way to stream a list of subdirectory names. One of the things I need to be able to do is list all collection names. Because several hundred thousand names may fail to be read in a single transaction, I wanted the ability to perform a long multi-transaction range-read on the contents of the `collections_dir`. The directory API doesn’t currently provide this, correct?

---

<div class="post-metadata">

**Author:** ![ryanworl](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/ryanworl/32/440_2.png) [@ryanworl](https://forums.foundationdb.org/u/ryanworl)\
**Post date:** [July 29, 2019, 8:17pm UTC](https://forums.foundationdb.org/t/abstracting-hundred-thousands-of-collections-of-items/1543/2 "2019-07-29T20:17:16Z")

</div>

If you choose to go the do-it-yourself route with creating an index of names to internal IDs, you should check out the metadata version feature. This will allow you to keep a consistent cache on the client of names to IDs and know when 1) collections are added or removed, and 2) when collections are re-named.

This will decrease the latency of your transactions if each requires mapping a collection name to ID before doing anything else.

The directory layer currently doesn’t implement this, so that is a benefit to doing it yourself here.

---

<div class="post-metadata">

**Author:** ![alexmiller](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alexmiller/32/326_2.png) [@alexmiller](https://forums.foundationdb.org/u/alexmiller)\
**Post date:** [July 29, 2019, 9:23pm UTC](https://forums.foundationdb.org/t/abstracting-hundred-thousands-of-collections-of-items/1543/3 "2019-07-29T21:23:04Z")

</div>

Without looking at your proposed solutions, my thought would be to use a random ID / [high contention allocator](https://activesphere.com/blog/2018/08/05/high-contention-allocator) to allocate IDs, and then maintain a separate index that maps ID to name. This appears to be your (1) solution.

I vaguely recall some weird caveats about directories being renamed, but it’s left my mental cache… @alloc would probably know.