Storage server running out of space

We love you for your reports, and apologize for your troubles. :slight_smile:

If you just add new machines without the exclude, data distribution tries to form groups of three machines using the new ones and old ones randomly, and then adding machines to a nearly full cluster can make it go completely full.

The largest hammer here is to exclude all of your current machines, and attach an equal number of fresh processes, and then let data distribution copy from the old machines to the new ones, and then MOVE THE COORDINATORS, and tear down the old machines.

If you don’t have the excess machines to be able to do that, then you could configure from triple replication to double, let it calm down with the added machines, and then configure back to triple. As you’ll need to “remove” data from the cluster somehow to free up space while it rebalances.

Data Distribution Stopped - How to Restart? has related graphs where you can see the on disk size of files jump when machines are added to the cluster, and a bit of discussion about it.