Status of the fdbcli shows that cluster is healthy and is moving data (around 5-6GB). We have turned off almost all our application connections to the database, yet we see a read rate of about 5k Hz.
**$ fdbcli -v
FoundationDB CLI 6.2 (v6.2.20)
source version 77b5171e81754f2fda8869703d662e59d85b7f23
protocol fdb00b062010001
Two storage logs in particular has high durable byte input. In one of their tracelog, we see
<Event Severity="10" Time="1597355347.427617" Type="StorageServerNoLongerBehind" ID="26167eb650f76e1a" CursorVersion="43960427902726" TLogVersion="0" Machine="10.5.1.7:4611" LogGroup="default" Roles="SS" />
<Event Severity="20" Time="1597355347.427617" Type="StorageServerUpdateLag" ID="26167eb650f76e1a" Version="43960427902725" DurableVersion="43960422902201" Machine="10.5.1.7:4611" LogGroup="default" Roles="SS" />
<Event Severity="10" Time="1597355347.943870" Type="StorageServerNoLongerBehind" ID="26167eb650f76e1a" CursorVersion="43960427904116" TLogVersion="0" Machine="10.5.1.7:4611" LogGroup="default" Roles="SS" />
<Event Severity="20" Time="1597355347.943870" Type="StorageServerUpdateLag" ID="26167eb650f76e1a" Version="43960427904115" DurableVersion="43960422902725" Machine="10.5.1.7:4611" LogGroup="default" Roles="SS" />
<Event Severity="10" Time="1597355348.481322" Type="StorageServerNoLongerBehind" ID="26167eb650f76e1a" CursorVersion="43960427904545" TLogVersion="0" Machine="10.5.1.7:4611" LogGroup="default" Roles="SS" />
<Event Severity="20" Time="1597355348.481322" Type="StorageServerUpdateLag" ID="26167eb650f76e1a" Version="43960427904544" DurableVersion="43960422904115" Machine="10.5.1.7:4611" LogGroup="default" Roles="SS" />
We have tried excluding one of the storage log process, but it didn’t do anything. Anything else that we can check to make sure that work is being done and the cluster is not stuck?