# Release: NodeJS FoundationDB mixed order data encoder

**URL:** <https://forums.foundationdb.org/t/release-nodejs-foundationdb-mixed-order-data-encoder/2188>\
**Category:** Using FoundationDB\
**Created:** [June 16, 2020, 1:57pm UTC](https://forums.foundationdb.org/t/release-nodejs-foundationdb-mixed-order-data-encoder/2188 "2020-06-16T13:57:53Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![endyjasmi](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/endyjasmi/32/674_2.png) [@endyjasmi](https://forums.foundationdb.org/u/endyjasmi)\
**Post date:** [June 16, 2020, 1:57pm UTC](https://forums.foundationdb.org/t/release-nodejs-foundationdb-mixed-order-data-encoder/2188/1 "2020-06-16T13:57:53Z")

</div>

Hello, I have just released an encoder library build on top of the nodejs foundationdb binding by @josephg.

[https://github.com/couchplus/collator](https://github.com/couchplus/collator)

There are one particular feature that I am proud of, which is the ability to collate mixed order data.

A lot of effort has been given to make this works, I hope you guys will like it.

Do leave me any feedback and thanks in advance for that.

---

<div class="post-metadata">

**Author:** ![amirouche](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/amirouche/32/2096_2.png) [@amirouche](https://forums.foundationdb.org/u/amirouche)\
**Post date:** [June 21, 2020, 9:11am UTC](https://forums.foundationdb.org/t/release-nodejs-foundationdb-mixed-order-data-encoder/2188/2 "2020-06-21T09:11:27Z")

</div>

If I understand correctly that is a replacement `tuple.pack` and `tuple.unpack` functions that offer the ability to change the order between object of different types. Like for example `true` can be bigger than `0.1` given a particular configuration or the inverse that is `0.1` can be bigger than `true`. Is that correct?

Does `msgpack.lite` preserve lexical ordering. Pseudo code:

```auto
if (a > b) then
  assert msgpack.encode(a) > msgpack.encode(b)

```

In Python bindings of msgpack I used, there was a problem with utf8 / string and bytes, I may be wrong but my understanding was that it was not possible to encode the equivalent of a JS Object that contains both bytes and strings of characters in values. Is it the case with the current javascript `msgpack-lite`?

In the readme there is no mention of support for JavaScript [Int8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Int8Array), that would be the equivalent of Scheme’s `bytevector` type or Python 3 `byte`. `Int8Array` are very useful in my experience, because [spatial indices relying on z-curve need to play with the binary form of the coordinates](https://apple.github.io/foundationdb/spatial-indexing.html). Converting the bytes into a number and storing that number is not necessarily always possible, depending on the coordinates precision and whether the host language support big integers.

I considered changing the byte encoding in my Scheme bindings, to better map with Scheme (eg. add support for pair type, the primitive of what lists in lisp are made of), so far I did not do it, because re-using the same byte encoding as Python allows having direct access to the same FoundationDB cluster from several services possibly written in different languages. My understanding is using CouchPlus:Collator, it will require to re-implement the byte encoding algorithm. Is that correct?

> [@endyjasmi](#):
>
> the ability to collate mixed order data

I am not familiar with that feature. Can you give an example, please?

There is two typos in the `README.md`:

> After that, you can start using it as follows;

Maybe:

> After that, you can start using it as follow:

The sed command would look like:

```plaintext
s/s;/:/

```

Thanks for sharing.

---

<div class="post-metadata">

**Author:** ![endyjasmi](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/endyjasmi/32/674_2.png) [@endyjasmi](https://forums.foundationdb.org/u/endyjasmi)\
**Post date:** [June 21, 2020, 12:14pm UTC](https://forums.foundationdb.org/t/release-nodejs-foundationdb-mixed-order-data-encoder/2188/3 "2020-06-21T12:14:02Z")

</div>

Hello @amirouche

Your understanding of this library is a replacement for `tuple.pack` and `tuple.unpack` is correct. The combination of these methods is called encoder. There are key encoder and value encoder for the `foundationdb` library. The msgpack described in the readme serve the purpose of the value encoder in which you can replace with any other value encoder. (Side note: `msgpack-lite` encoding by default is not lexicographically sorted.) My library is best used as key encoder.

By default, the key is ordered in following order;

```auto
asc - null
asc - false
asc - true
asc - NaN
asc - number
asc - string
asc - array
asc - object
asc - undefined
desc - undefined
desc - object
desc - array
desc - string
desc - number
desc - NaN
desc - true
desc - false
desc - null

```

and you can change the collation order by changing the `CustomCode` prefix value as shown below;

```ts
import { 
  BooleanCollator, 
  ComplexCollator, 
  NumberCollator, 
  StringCollator 
} from "@couchplus/collator";

const CustomCode = { / **Custom order** / };

const collator = new ComplexCollator(CustomCode);
collator.register(new BooleanCollator(CustomCode));
collator.register(new NumberCollator(CustomCode));
collator.register(new StringCollator(CustomCode));

```

For `CustomCode` configuration, please refers to [https://github.com/couchplus/collator/blob/master/source/DefaultCode.ts](https://github.com/couchplus/collator/blob/master/source/DefaultCode.ts)

The library does not currently supports `Uint8Array` as key but I do have plan to support spatial data as part of the key type in the future. _Though as a workaround, maybe you can use my library at certain key space and use the default binary encoding at the key space where you requires spatial index_

> [@amirouche](#):
>
> My understanding is using CouchPlus:Collator, it will require to re-implement the byte encoding algorithm. Is that correct?

Yes, currently my library have it’s own byte encoding.

> [@amirouche](#):
>
> I am not familiar with that feature. Can you give an example, please?

As for the explanation for mixed order data, given I want use a list/array as key, the catch is, I want to order the first element of the array in ascending order but I want to order the second element in descending order, I can do that as follow;

```ts
transaction.set([collator.order.asc(first), collator.order.desc(second)], value);

```

The resulting order inside the database will be as follows;

```auto
["abc", 5]
["abc", 4]
["abc", 3]
["abc", 2]
["abc", 1]
["def", 10]
["def", 9]
["def", 8]
["def", 7]
["def", 6]

```

note: the above is lexicographically sorted. means, you will get the benefit of range scan while ordering element in an array in mixed order.

Thank you for your attention at my library, feel free to ask me anything.

---

<div class="post-metadata">

**Author:** ![amirouche](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/amirouche/32/2096_2.png) [@amirouche](https://forums.foundationdb.org/u/amirouche)\
**Post date:** [June 21, 2020, 1:35pm UTC](https://forums.foundationdb.org/t/release-nodejs-foundationdb-mixed-order-data-encoder/2188/4 "2020-06-21T13:35:19Z")

</div>

> [@endyjasmi](#):
>
> note: the above is lexicographically sorted. means, you will get the benefit of range scan while ordering element in an array in mixed order.

That is great!

The idea of getting together a specific type for geospatial index makes sense.

---

<div class="post-metadata">

**Author:** ![josephg](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/josephg/32/84_2.png) [@josephg](https://forums.foundationdb.org/u/josephg)\
**Post date:** [June 23, 2020, 11:48pm UTC](https://forums.foundationdb.org/t/release-nodejs-foundationdb-mixed-order-data-encoder/2188/5 "2020-06-23T23:48:39Z")

</div>

Cool! I’m glad that abstraction was useful to you!

---

<div class="post-metadata">

**Author:** ![dallen](https://avatars.discourse-cdn.com/v4/letter/d/3d9bf3/32.png) [@dallen](https://forums.foundationdb.org/u/dallen)\
**Post date:** [August 31, 2020, 12:17am UTC](https://forums.foundationdb.org/t/release-nodejs-foundationdb-mixed-order-data-encoder/2188/6 "2020-08-31T00:17:30Z")

</div>

> [@endyjasmi](#):
>
> couchplus/collator/blob/master

The link is not working any more. I am search for how to do this, can you please share how this was done.

Thanks

---

<div class="post-metadata">

**Author:** ![endyjasmi](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/endyjasmi/32/674_2.png) [@endyjasmi](https://forums.foundationdb.org/u/endyjasmi)\
**Post date:** [September 6, 2020, 2:56am UTC](https://forums.foundationdb.org/t/release-nodejs-foundationdb-mixed-order-data-encoder/2188/7 "2020-09-06T02:56:42Z")

</div>

@dallen Sorry for the late reply, due to some reason I have to pull down the previous project repository.

But I have rewrite the whole library to be more performance then the previous release and released it to [https://github.com/himpun/collator](https://github.com/himpun/collator)

Do let me know if you have any question.

---

<div class="post-metadata">

**Author:** ![dallen](https://avatars.discourse-cdn.com/v4/letter/d/3d9bf3/32.png) [@dallen](https://forums.foundationdb.org/u/dallen)\
**Post date:** [September 7, 2020, 12:35am UTC](https://forums.foundationdb.org/t/release-nodejs-foundationdb-mixed-order-data-encoder/2188/8 "2020-09-07T00:35:54Z")

</div>

@endyjasmi Many thanks for re-sharing your code.