FDB has two parameters under moving_data category:
We don’t know how to interpret the parameters under different scenarios, such as removing a node, adding a node and cross-DC data movement, and need help.
When I remove a node, the moving_data_in_flight parameter starts at 80GB, then gradually and consistently decreases to 70, 60, … and 0. I interpret this as follows: FDB knows the total amount of data (80GB) to be rebalanced out of the node being removed. This parameter represents the amount of data left to be rebalanced. As the rebalancing continues, the parameter decreases.
When I add a node, however, moving_data_in_flight seems to have a different meaning. It starts at about 5GB, fluctuates a little bit, then becomes almost constant at 4.7GB. I check the disk usage of /var/lib/foundationdb at the newly added node, it is increasing from 10GB to 20, 30, …, 120, etc.
I have a few questions:
- How should I interpret moving_data_in_flight (4.7GB) when I add a node?
- When cross-region migration/replication is performed, what does moving_data_in_flight represent?
- What’s the meaning of moving_data_in_queue?
Also if you can give some details of how FDB rebalance data when a node is added, I’ll highly appreciate it. For example,
- Does FDB predetermine the total amount of data to be distributed to the new node, or does it rebalance as it sees fit (dynamically computed, without a predetermined number)?
- How much data (in terms of bytes or shards) is pushed to the new node at a time?
Attached is a screenshot of our FDB-Grafana monitoring platform, when I add a node.