We see here that _fallback can used as a temporary measure to restore cluster health during a datacenter outage.
My cluster is currently in three_datacenter mode.
fdb> status
Using cluster file `/etc/foundationdb/fdb.cluster'.
Configuration:
Redundancy mode - three_datacenter
Storage engine - ssd-2
Coordinators - 1
Usable Regions - 1
Cluster:
FoundationDB processes - 9
Zones - 9
Machines - 1
Memory availability - 5.4 GB per process on machine with least available
Fault Tolerance - 0 zones
Server time - 05/05/23 16:32:32
Data:
Replication health - Healthy
Moving data - 0.000 GB
Sum of key-value sizes - 10 MB
Disk space used - 1.192 GB
Operating space:
Storage server - 132.1 GB free on most full server
Log server - 132.1 GB free on most full server
Workload:
Read rate - 15 Hz
Write rate - 0 Hz
Transactions started - 4 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz
Backup and DR:
Running backups - 0
Running DRs - 0
Client time: 05/05/23 16:32:32
When I execute configure three_datacenter_fallback, its Replication health remains in (Re)initializing automatic data distribution.
After a few hours, it’s still like this.
fdb> status details
Using cluster file `/etc/foundationdb/fdb.cluster'.
Configuration:
Redundancy mode - three_datacenter_fallback
Storage engine - ssd-2
Coordinators - 1
Usable Regions - 1
Cluster:
FoundationDB processes - 9 (less 0 excluded; 1 with errors)
Zones - 9
Machines - 1
Memory availability - 4.9 GB per process on machine with least available
Retransmissions rate - 1 Hz
Fault Tolerance - 0 zones
Server time - 05/05/23 16:57:41
Data:
Replication health - (Re)initializing automatic data distribution
Moving data - unknown (initializing)
Sum of key-value sizes - unknown
Disk space used - 980 MB
Operating space:
Storage server - 132.3 GB free on most full server
Log server - 132.3 GB free on most full server
Workload:
Read rate - 21 Hz
Write rate - 0 Hz
Transactions started - 7 Hz
Transactions committed - 0 Hz
Conflict rate - 0 Hz
Backup and DR:
Running backups - 0
Running DRs - 0
Process performance details:
xx.xxx.xxx.xxx:4500 ( 2% cpu; 16% machine; 0.000 Gbps; 0% disk IO; 0.3 GB / 4.9 GB RAM )
xx.xxx.xxx.xxx:4501 ( 1% cpu; 16% machine; 0.000 Gbps; 0% disk IO; 0.2 GB / 4.9 GB RAM )
xx.xxx.xxx.xxx:4502 ( 1% cpu; 16% machine; 0.000 Gbps; 0% disk IO; 0.3 GB / 4.9 GB RAM )
xx.xxx.xxx.xxx:4510 ( 4% cpu; 16% machine; 0.000 Gbps; 0% disk IO; 0.1 GB / 4.9 GB RAM )
Last logged error: DataDistribution: internal_error at Fri May 5 16:57:26 2023
xx.xxx.xxx.xxx:4511 ( 1% cpu; 16% machine; 0.000 Gbps; 0% disk IO; 0.2 GB / 4.9 GB RAM )
xx.xxx.xxx.xxx:4512 ( 1% cpu; 16% machine; 0.000 Gbps; 0% disk IO; 0.2 GB / 4.9 GB RAM )
xx.xxx.xxx.xxx:4520 ( 0% cpu; 16% machine; 0.000 Gbps; 0% disk IO; 0.1 GB / 4.9 GB RAM )
xx.xxx.xxx.xxx:4521 ( 1% cpu; 16% machine; 0.000 Gbps; 0% disk IO; 0.3 GB / 4.9 GB RAM )
xx.xxx.xxx.xxx:4522 ( 2% cpu; 16% machine; 0.000 Gbps; 0% disk IO; 0.3 GB / 4.9 GB RAM )
Coordination servers:
xx.xxx.xxx.xxx:4500 (reachable)
Client time: 05/05/23 16:57:41
The class of the 4510 process is stateless.
fdbcli --exec "kill; kill all" restarting all processes do not work.
It seems that there was an error in the recruitment DD process：
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="TeamCollectionInfo" ID="02e621b904023a43" Primary="1" AddedTeams="0" TeamsToBuild="0" CurrentServerTeams="1" DesiredTeams="25" MaxTeams="125" StorageTeamSize="4" CurrentMachineTeams="1" CurrentHealthyMachineTeams="1" DesiredMachineTeams="25" MaxMachineTeams="125" TotalHealthyMachines="5" MinTeamsOnServer="0" MaxTeamsOnServer="1" MinMachineTeamsOnMachine="0" MaxMachineTeamsOnMachine="1" DoBuildTeams="0" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" TrackLatestType="Original" />
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="BuildTeams" ID="02e621b904023a43" ServerCount="5" UniqueMachines="5" Primary="1" StorageTeamSize="4" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="BuildTeamsBegin" ID="02e621b904023a43" TeamsToBuild="24" DesiredTeams="25" MaxTeams="125" BadServerTeams="0" PerpetualWigglingTeams="0" UniqueMachines="5" TeamSize="4" Servers="5" HealthyServers="5" CurrentTrackedServerTeams="1" HealthyTeamCount="1" TotalTeamCount="1" MachineTeamCount="1" MachineCount="5" DesiredTeamsPerServer="5" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="ChosenMachine" ID="0000000000000000" MachineInfo="dc3-zone3" LeaseUsedMachinesSize="1" ForcedAttributesSize="1" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="40" ErrorKind="BugDetected" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="InternalError" ID="0000000000000000" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" FailedAssertion="team.size() == configuration.storageTeamSize" LeftValue="5" RightValue="4" File="/home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/DDTeamCollection.actor.cpp" Line="4175" ThreadID="5379302038607344942" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x3753c3c 0x3752860 0x3752c4e 0x36a3d6f 0x13af6cc 0x138683f 0x138a01a 0x13db8f3 0x13dbf83 0x13dc06b 0x13e2eb5 0x11e8760 0x36ee806 0xa776e9 0x7f9459bd3555" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="40" ErrorKind="Unset" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="SystemError" ID="0000000000000000" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" ThreadID="5379302038607344942" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x3753c3c 0x3752860 0x3752c4e 0x36a347c 0x36a3d81 0x13af6cc 0x138683f 0x138a01a 0x13db8f3 0x13dbf83 0x13dc06b 0x13e2eb5 0x11e8760 0x36ee806 0xa776e9 0x7f9459bd3555" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="BuildMachineTeams" ID="0000000000000000" Primary="1" TotalMachines="5" TotalHealthyMachine="5" HealthyMachineTeamCount="1" DesiredMachineTeams="25" MaxMachineTeams="125" TotalMachineTeams="1" MachineTeamsToBuild="24" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="BgDDMountainChopper" ID="02e621b904023a43" SuppressedEventCount="0" PollingInterval="10" Enabled="1" QueuedRelocations="0" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="40" ErrorKind="Unset" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="DataDistributionQueueError" ID="02e621b904023a43" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" ThreadID="5379302038607344942" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x3753c3c 0x3752860 0x3752c4e 0x131b25e 0x131b35a 0xb03879 0xb03879 0x131d95e 0x131df37 0x131b79d 0x131b840 0x131bab2 0x1379e66 0x1379f4e 0xb03879 0x136a955 0xb03879 0x13adf9f 0xb03879 0x13dbff1 0x13dc06b 0x13e2eb5 0x11e8760 0x36ee806 0xa776e9 0x7f9459bd3555" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="BgDDValleyFiller" ID="02e621b904023a43" SuppressedEventCount="0" PollingInterval="10" Enabled="1" QueuedRelocations="0" Error="operation_cancelled" ErrorDescription="Asynchronous operation cancelled" ErrorCode="1101" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="40" ErrorKind="Unset" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="DDQueue" ID="02e621b904023a43" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" ThreadID="5379302038607344942" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x3753c3c 0x3752860 0x3752c4e 0xff3e5b 0xff3f0e 0xb03879 0x131b279 0x131b35a 0xb03879 0xb03879 0x131d95e 0x131df37 0x131b79d 0x131b840 0x131bab2 0x1379e66 0x1379f4e 0xb03879 0x136a955 0xb03879 0x13adf9f 0xb03879 0x13dbff1 0x13dc06b 0x13e2eb5 0x11e8760 0x36ee806 0xa776e9 0x7f9459bd3555" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="DataDistributorDestroyTeamCollections" ID="0000000000000000" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="DDTeamCollectionDestructed" ID="02e621b904023a43" Primary="1" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="TeamTrackerStopping" ID="02e621b904023a43" ServerPrimary="1" Team="TeamID 505953935a02c308; Size 4; 10.xxx.xxx.xxx:4511 1118f35c50ee9297, 10.xxx.xxx.xxx:4501 8c51d777f53a7f59, 10.xxx.xxx.xxx:4512 921bfa039101bd93, 10.xxx.xxx.xxx:4500 a546bdcc9c6236f1" Priority="140" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="20" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="ZeroTeamsHealthySignalling" ID="02e621b904023a43" ServerPrimary="1" SignallingServerTeam="TeamID 505953935a02c308; Size 4; 10.xxx.xxx.xxx:4511 1118f35c50ee9297, 10.xxx.xxx.xxx:4501 8c51d777f53a7f59, 10.xxx.xxx.xxx:4512 921bfa039101bd93, 10.xxx.xxx.xxx:4500 a546bdcc9c6236f1" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="StorageServerTrackerCancelled" ID="02e621b904023a43" Error="operation_cancelled" ErrorDescription="Asynchronous operation cancelled" ErrorCode="1101" SuppressedEventCount="4" Primary="1" Server="1118f35c50ee9297" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="DataDistributorTeamCollectionsDestroyed" ID="0000000000000000" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="40" ErrorKind="Unset" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="DataDistribution" ID="02e621b904023a43" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" ThreadID="5379302038607344942" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x3753c3c 0x3752860 0x3752c4e 0xff3e5b 0xff3f0e 0xb03879 0x12cf6bd 0x12cfea3 0x12cff89 0xb03879 0xb03879 0xff3e82 0xff3f0e 0xb03879 0x131b279 0x131b35a 0xb03879 0xb03879 0x131d95e 0x131df37 0x131b79d 0x131b840 0x131bab2 0x1379e66 0x1379f4e 0xb03879 0x136a955 0xb03879 0x13adf9f 0xb03879 0x13dbff1 0x13dc06b 0x13e2eb5 0x11e8760 0x36ee806 0xa776e9 0x7f9459bd3555" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="DataDistributorError" ID="02e621b904023a43" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.091284" DateTime="2023-05-05T08:59:56Z" Type="Role" ID="02e621b904023a43" Error="success" ErrorDescription="Success" ErrorCode="0" Transition="End" As="DataDistributor" Reason="Error" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,DD,GP,MS,RK,RV" TrackLatestType="Original" />
<Event Severity="10" Time="1683277196.094667" DateTime="2023-05-05T08:59:56Z" Type="ClusterControllerWorkerAlreadyRegistered" ID="151acc620b0cfc95" SuppressedEventCount="1" WorkerId="09998554446e44cd" ProcessId="2d1f24fe162db4191ec09f383fe640fa" ZoneId="dc2-zone1" DataHall="hall2" PClass="stateless" Workers="9" Degraded="0" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.095160" DateTime="2023-05-05T08:59:56Z" Type="EndpointNotFound" ID="0000000000000000" SuppressedEventCount="0" Address="10.xxx.xxx.xxx:4510" Token="9c2f687727c1d5f3" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.095160" DateTime="2023-05-05T08:59:56Z" Type="CCDataDistributorDied" ID="151acc620b0cfc95" DDID="02e621b904023a43" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.095160" DateTime="2023-05-05T08:59:56Z" Type="DBInfoStartBroadcast" ID="151acc620b0cfc95" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.095654" DateTime="2023-05-05T08:59:56Z" Type="GotServerDBInfoChange" ID="0000000000000000" ChangeID="57bd5665043a6efa" InfoGeneration="28" MasterID="98d45106b817cd79" RatekeeperID="ab8c30f259c53708" DataDistributorID="0000000000000000" BlobManagerID="0000000000000000" EncryptKeyProxyID="0000000000000000" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,GP,MS,RK,RV" />
<Event Severity="10" Time="1683277196.095654" DateTime="2023-05-05T08:59:56Z" Type="DataDistributorDied" ID="57bd5665043a6efa" ThreadID="5379302038607344942" Machine="10.xxx.xxx.xxx:4510" LogGroup="default" Roles="CC,CP,GP,MS,RK,RV" />