I am trying to load roughly 5 GB worth of data in a three_data_hall cluster with 9 logs, 6 storages, 9 coordinators and 1 instance of other roles.
I see that if the number of clients (processes) I use to ingest the data is more than 2, then the cluster becomes nonresponsive after ingesting a few MB of data.
The status changes from this
Redundancy mode - three_data_hall Storage engine - ssd-2 Coordinators - 9 Desired Proxies - 1 Desired Resolvers - 1 Desired Logs - 9 Usable Regions - 1 Cluster: FoundationDB processes - 42 Zones - 27 Machines - 24 Memory availability - 7.5 GB per process on machine with least available Retransmissions rate - 2 Hz Fault Tolerance - 2 zones Server time - 08/19/21 11:29:14 Data: Replication health - (Re)initializing automatic data distribution Moving data - unknown (initializing) Sum of key-value sizes - unknown Disk space used - 1.715 GB Operating space: Storage server - 65.6 GB free on most full server Log server - 130.2 GB free on most full server Workload: Read rate - 51 Hz Write rate - 63182 Hz Transactions started - 3188 Hz Transactions committed - 3163 Hz Conflict rate - 0 Hz Performance limited by process: Storage server performance (storage queue). Most limiting process: 10.240.0.15:4500
Unable to read database configuration. Configuration: Redundancy mode - unknown Storage engine - unknown Coordinators - unknown Usable Regions - unknown Cluster: FoundationDB processes - 42 Zones - 27 Machines - 24 Memory availability - 7.5 GB per process on machine with least available Retransmissions rate - 3 Hz Server time - 08/19/21 11:29:22 Data: Replication health - unknown Moving data - unknown Sum of key-value sizes - unknown Disk space used - unknown Operating space: Unable to retrieve operating space status Workload: Read rate - unknown Write rate - unknown Transactions started - unknown Transactions committed - unknown Conflict rate - unknown
What could I possibly look for to explain this behavior (and what additional info should I provide here to get better support)?