# Streaming data out of FoundationDB **URL:** \ **Category:** Using FoundationDB\ **Created:** [September 7, 2018, 2:30pm UTC](https://forums.foundationdb.org/t/streaming-data-out-of-foundationdb/683 "2018-09-07T14:30:24Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![tobad357](https://avatars.discourse-cdn.com/v4/letter/t/49beb7/32.png) [@tobad357](https://forums.foundationdb.org/u/tobad357)\ **Post date:** [September 7, 2018, 2:30pm UTC](https://forums.foundationdb.org/t/streaming-data-out-of-foundationdb/683/1 "2018-09-07T14:30:25Z")
Hello I’m wondering if FoundationDB can be used successfully to stream data out. We have a use case where we ingest thousand of transactions per second and then will have hundreds of clients connecting doing range reads to get the latest data. Queries like give me all tx between these two timestamps for key x.y.z. I understand the limitation of the five second transaction timeout which would mean we would have to chunk out data from fdb in chunks of X tx. Beyond that is there anything else that would cause issues for fdb in this case? Also, is old data that hardly every gets queried something that causes issues for fdb. If as an example 100TB was stored but only the last 1 TB was frequently used, would the response times be affected by the other 99TB? ---
**Author:** ![alloc](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/alloc/32/9_2.png) [@alloc](https://forums.foundationdb.org/u/alloc)\ **Post date:** [September 8, 2018, 4:11pm UTC](https://forums.foundationdb.org/t/streaming-data-out-of-foundationdb/683/2 "2018-09-08T16:11:35Z")
Are all of the transactions hitting the same subspace? I’m asking because if there is effectively one giant change queue, you can run into issues where all of the writes are hitting one team of storage servers, then data distribution kicks in and routes all writes to another team, put then that team gets hot, etc. Similarly, if all of the readers are hitting the same keyspace, then you might have exactly the same hot shard problem, but this time at read time. This will limit the theoretical qps, but if you are writing to a bunch of different queues, then that might be fine. I’d also say that the most compelling reason (in my mind) to do this would be because you have some dataset and you want to be able to read the latest data _and_ you want to perform other queries on it (or really need it durable for some reason). This way gets you transactional updates between your change feed and your database, which can be nice, but if you never use the database, then maybe some other queue solution would be better. One issue you can run into using timestamps as the mechanism for grabbing changes is that if you can sometimes miss data. Even ignoring instances of clock drift, you can do something like: 1. Some DB-reader has done a scan and has read everything through timestamp _t0_. 2. At timestamp _t1_, one transaction begins. 3. At timestamp _t2_, another transaction begins. 4. The second transaction commits at timestamp _t3_, but it writes timestamp _t2_ (because it can’t know at what time it will commit). 5. The DB-reader performs another scan, starting at _t0_. It will end its scan at _t2_ because that’s the most recent successful commit. 6. The transaction that began at _t1_ commits, writing its data _before_ the transaction that committed at timestamp _t2_. This means that now that the reader thinks that it has everything through _t2_, it will **never** go back and grab the data from the _t1_ transaction. One way around that is to use versionstamp mutations (i.e., [SET\_VERSIONSTAMPED\_KEY](https://apple.github.io/foundationdb/api-python.html#fdb.Transaction.set_versionstamped_key) and [SET\_VERSIONSTAMPED\_VALUE](https://apple.github.io/foundationdb/api-python.html#fdb.Transaction.set_versionstamped_value)) to ensure that transaction mutations get written in commit order instead of by timestamp. This closes out the gaps insofar as it ensures that later committing transactions _always_ get written after earlier-committing transactions. We don’t have a lot of documentation on how you would do this, but see this discussion about change feeds: [Changefeeds (watching and getting updates on ranges of keys)](https://forums.foundationdb.org/t/changefeeds-watching-and-getting-updates-on-ranges-of-keys/511) See also this discussion about how versionstamps don’t necessarily play nicely with restoring from backup and how you might fix this: [VersionStamp uniqueness and monotonicity](https://forums.foundationdb.org/t/versionstamp-uniqueness-and-monotonicity/602) As to having cold data lie around, I don’t think it will affect your response times too much. The underlying storage mechanism is a B-tree, which means that the number of disk pages your query might touch is equal to the depth of the B-tree (plus any overflow pages), which might be larger with a larger data set. That being said, the storage servers have an in-memory cache of recently read and written pages, so if you are reading and writing 1% of your dataset, presumably most of it is in the cache? Also, for larger datasets, there will be more shards (as a dataset gets larger, we both increase the number of shards and the size of each shard, so it’s not like its linear with the dataset size). Having more shards can lead to a few performance penalties, though I don’t think it’s too bad. Though I’ll also admit that that isn’t my area of expertise, so if someone else knows the trade-offs of a database with a large number of shards, that would be nice to know. Also, I guess all of the problems with having a dataset like the one you’ve described come mainly from its size rather than the coldness of some of its data. I suspect (because of caching), the coldness of most of your data probably won’t actually matter too much. ---
**Author:** ![tobad357](https://avatars.discourse-cdn.com/v4/letter/t/49beb7/32.png) [@tobad357](https://forums.foundationdb.org/u/tobad357)\ **Post date:** [September 11, 2018, 12:25pm UTC](https://forums.foundationdb.org/t/streaming-data-out-of-foundationdb/683/3 "2018-09-11T12:25:21Z")
Thanks for the response. We have done some performance tests and so far it looks like we can use foundationdb for our usecase