# Suspected bug in config broadcaster

**URL:** <https://forums.foundationdb.org/t/suspected-bug-in-config-broadcaster/4501>\
**Category:** Development\
**Created:** [June 6, 2024, 10:51am UTC](https://forums.foundationdb.org/t/suspected-bug-in-config-broadcaster/4501 "2024-06-06T10:51:04Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![gm42](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/gm42/32/1784_2.png) [@gm42](https://forums.foundationdb.org/u/gm42)\
**Post date:** [June 6, 2024, 10:51am UTC](https://forums.foundationdb.org/t/suspected-bug-in-config-broadcaster/4501/1 "2024-06-06T10:51:04Z")

</div>

Hi there,

I believe there could be a bug in the config broadcaster; let me give some context: I am using clusters with FoundationDB 7.3.27 and operator v1.33; k8s services are enabled for routing, and I find that the `tailing-sidecar` contains a lot of `N2_ConnectError` log lines (see copy/paste below).

I produced these log lines by grepping for the IP address which a stateless process is trying to reach.

I think that this is an IP of the k8s service of an older coordinator which does not exist anymore. In some cases I can find this IP in the `exclude` list, sometimes not; also, it seems that only stateless processes are generating this traffic towards old coordinators.

Restarting the `fdbserver` process is ineffective, however if I instead delete all the `stateless` pods which generate this traffic, one by one, the issue is gone for some time and I can see no more traffic towards non-existing old service IPs. This indicates that some kind of state in the stateless pods is responsible for reaching out to these ghost coordinators.  
After ~30min I can see again the IPs, which leads me to believe that the processes are bouncing around information about these older coordinator IPs, and it’s never really forgotten.

I would also be tempted to use `--no-config-db` on the `fdbserver` processes’ command line, but I am concerned of possible safety issues and it’s not supported by the operator anyways.

I do not want to open a bug if there is no clear proof of it; I could not find in the git log specific commits about this possible issue, and I am available for further question or testcases as I can reliably see this problem on clusters.

Thanks for any insight you can provide!

```auto
{ "Severity": "10", "Type": "PingLatency", "ID": "0000000000000000", "Elapsed": "75.0007", "PeerAddr": "172.20.31.95:4501", "MinLatency": "1.79769e+308", "MaxLatency": "-1.79769e+308", "MeanLatency": "0", "MedianLatency": "0", "P90Latency": "0", "Count": "0", "BytesReceived": "0", "BytesSent": "0", "TimeoutCount": "0", "ConnectOutgoingCount": "25", "ConnectIncomingCount": "0", "ConnectFailedCount": "25", "ConnectMinLatency": "1.79769e+308", "ConnectMaxLatency": "-1.79769e+308", "ConnectMeanLatency": "0", "ConnectMedianLatency": "0", "ConnectP90Latency": "0", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "PingLatency", "ID": "0000000000000000", "Elapsed": "75.0007", "PeerAddr": "172.20.31.95:4501", "MinLatency": "1.79769e+308", "MaxLatency": "-1.79769e+308", "MeanLatency": "0", "MedianLatency": "0", "P90Latency": "0", "Count": "0", "BytesReceived": "0", "BytesSent": "0", "TimeoutCount": "0", "ConnectOutgoingCount": "25", "ConnectIncomingCount": "0", "ConnectFailedCount": "25", "ConnectMinLatency": "1.79769e+308", "ConnectMaxLatency": "-1.79769e+308", "ConnectMeanLatency": "0", "ConnectMedianLatency": "0", "ConnectP90Latency": "0", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "PingLatency", "ID": "0000000000000000", "Elapsed": "75.0006", "PeerAddr": "172.20.31.95:4501", "MinLatency": "1.79769e+308", "MaxLatency": "-1.79769e+308", "MeanLatency": "0", "MedianLatency": "0", "P90Latency": "0", "Count": "0", "BytesReceived": "0", "BytesSent": "0", "TimeoutCount": "0", "ConnectOutgoingCount": "25", "ConnectIncomingCount": "0", "ConnectFailedCount": "25", "ConnectMinLatency": "1.79769e+308", "ConnectMaxLatency": "-1.79769e+308", "ConnectMeanLatency": "0", "ConnectMedianLatency": "0", "ConnectP90Latency": "0", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "PingLatency", "ID": "0000000000000000", "Elapsed": "75.0007", "PeerAddr": "172.20.31.95:4501", "MinLatency": "1.79769e+308", "MaxLatency": "-1.79769e+308", "MeanLatency": "0", "MedianLatency": "0", "P90Latency": "0", "Count": "0", "BytesReceived": "0", "BytesSent": "0", "TimeoutCount": "0", "ConnectOutgoingCount": "25", "ConnectIncomingCount": "0", "ConnectFailedCount": "25", "ConnectMinLatency": "1.79769e+308", "ConnectMaxLatency": "-1.79769e+308", "ConnectMeanLatency": "0", "ConnectMedianLatency": "0", "ConnectP90Latency": "0", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "PingLatency", "ID": "0000000000000000", "Elapsed": "75.0008", "PeerAddr": "172.20.31.95:4501", "MinLatency": "1.79769e+308", "MaxLatency": "-1.79769e+308", "MeanLatency": "0", "MedianLatency": "0", "P90Latency": "0", "Count": "0", "BytesReceived": "0", "BytesSent": "0", "TimeoutCount": "0", "ConnectOutgoingCount": "25", "ConnectIncomingCount": "0", "ConnectFailedCount": "25", "ConnectMinLatency": "1.79769e+308", "ConnectMaxLatency": "-1.79769e+308", "ConnectMeanLatency": "0", "ConnectMedianLatency": "0", "ConnectP90Latency": "0", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "PingLatency", "ID": "0000000000000000", "Elapsed": "75.0006", "PeerAddr": "172.20.31.95:4501", "MinLatency": "1.79769e+308", "MaxLatency": "-1.79769e+308", "MeanLatency": "0", "MedianLatency": "0", "P90Latency": "0", "Count": "0", "BytesReceived": "0", "BytesSent": "0", "TimeoutCount": "0", "ConnectOutgoingCount": "25", "ConnectIncomingCount": "0", "ConnectFailedCount": "25", "ConnectMinLatency": "1.79769e+308", "ConnectMaxLatency": "-1.79769e+308", "ConnectMeanLatency": "0", "ConnectMedianLatency": "0", "ConnectP90Latency": "0", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "PingLatency", "ID": "0000000000000000", "Elapsed": "81.0007", "PeerAddr": "172.20.31.95:4501", "MinLatency": "1.79769e+308", "MaxLatency": "-1.79769e+308", "MeanLatency": "0", "MedianLatency": "0", "P90Latency": "0", "Count": "0", "BytesReceived": "0", "BytesSent": "0", "TimeoutCount": "0", "ConnectOutgoingCount": "27", "ConnectIncomingCount": "0", "ConnectFailedCount": "27", "ConnectMinLatency": "1.79769e+308", "ConnectMaxLatency": "-1.79769e+308", "ConnectMeanLatency": "0", "ConnectMedianLatency": "0", "ConnectP90Latency": "0", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionClosed", "ID": "0000000000000000", "Error": "connection_failed", "ErrorDescription": "Network connection failed", "ErrorCode": "1026", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectingTo", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "PeerReferences": "5", "FailureStatus": "FAILED", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }
{ "Severity": "10", "Type": "ConnectionTimedOut", "ID": "0000000000000000", "SuppressedEventCount": "1", "PeerAddr": "172.20.31.95:4501", "ThreadID": "5345288753128754575", "Machine": "172.20.200.234:4501", "LogGroup": "my-cluster-c", "Roles": "CS" }

```

---

<div class="post-metadata">

**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\
**Post date:** [June 10, 2024, 11:55am UTC](https://forums.foundationdb.org/t/suspected-bug-in-config-broadcaster/4501/2 "2024-06-10T11:55:25Z")

</div>

> I think that this is an IP of the k8s service of an older coordinator which does not exist anymore. In some cases I can find this IP in the `exclude` list, sometimes not; also, it seems that only stateless processes are generating this traffic towards old coordinators.
> 
> Restarting the `fdbserver` process is ineffective, however if I instead delete all the `stateless`pods which generate this traffic, one by one, the issue is gone for some time and I can see no more traffic towards non-existing old service IPs. This indicates that some kind of state in the stateless pods is responsible for reaching out to these ghost coordinators.  
> After ~30min I can see again the IPs, which leads me to believe that the processes are bouncing around information about these older coordinator IPs, and it’s never really forgotten.

This is a known issue that fdbserver processes try to reach old addresses. I’m not sure if we have a GitHub issue for that (CC @jzhou). Restarting the cluster with `fdbcli` should also reset this state.

> I would also be tempted to use `--no-config-db` on the `fdbserver` processes’ command line, but I am concerned of possible safety issues and it’s not supported by the operator anyways.

That should be support with the [customParameters](https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/cluster_spec.md#processsettings) as this flag must be set on the fdbserver process. Have you tried setting it and it didn’t work?

---

<div class="post-metadata">

**Author:** ![gm42](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/gm42/32/1784_2.png) [@gm42](https://forums.foundationdb.org/u/gm42)\
**Post date:** [June 12, 2024, 9:09am UTC](https://forums.foundationdb.org/t/suspected-bug-in-config-broadcaster/4501/3 "2024-06-12T09:09:44Z")

</div>

Thanks for the reply!

> [@johscheuer](#):
>
> Restarting the cluster with `fdbcli` should also reset this state.

Forgive my ignorance, but how do you restart a cluster via `fdbcli`?

> [@johscheuer](#):
>
> That should be support with the [customParameters](https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/cluster_spec.md#processsettings) as this flag must be set on the fdbserver process. Have you tried setting it and it didn’t work?

Haven’t tried it because `customParameters` would cover [this aspect of the functionality](https://github.com/apple/foundationdb/commit/cd2bbffa4c412c4c1d599d0dc568d3b28fce4d6b) but not [this other aspect](https://github.com/apple/foundationdb/commit/1a33515934a9c9701c120e4dfc1e478538ab8ffe) e.g. operator never specifies that option when running the `coordinators` command via `fdbcli`, so I concluded it’s not supported.

---

<div class="post-metadata">

**Author:** ![johscheuer](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@johscheuer](https://forums.foundationdb.org/u/johscheuer)\
**Post date:** [June 12, 2024, 9:48am UTC](https://forums.foundationdb.org/t/suspected-bug-in-config-broadcaster/4501/4 "2024-06-12T09:48:09Z")

</div>

> Forgive my ignorance, but how do you restart a cluster via `fdbcli` ?

I wouldn’t call that ignorance 🙂 The command would be `kill; kill all` obviously that will cause a minimal downtime. Docs for fdbcli kill: [Command Line Interface — FoundationDB 7.1](https://apple.github.io/foundationdb/command-line-interface.html#kill)

> Haven’t tried it because `customParameters` would cover [this aspect of the functionality](https://github.com/apple/foundationdb/commit/cd2bbffa4c412c4c1d599d0dc568d3b28fce4d6b) but not [this other aspect](https://github.com/apple/foundationdb/commit/1a33515934a9c9701c120e4dfc1e478538ab8ffe) e.g. operator never specifies that option when running the `coordinators` command via `fdbcli` , so I concluded it’s not supported.

Good point, could you open a GitHub issue for that in the operator repo. I’m not sure what will happen when the fdbserver processes are started with that flag but the fdbcli is not setting it 🤔

---

<div class="post-metadata">

**Author:** ![gm42](https://sea1.discourse-cdn.com/foundationdb/user_avatar/forums.foundationdb.org/gm42/32/1784_2.png) [@gm42](https://forums.foundationdb.org/u/gm42)\
**Post date:** [June 12, 2024, 10:21am UTC](https://forums.foundationdb.org/t/suspected-bug-in-config-broadcaster/4501/5 "2024-06-12T10:21:37Z")

</div>

> [@johscheuer](#):
>
> Good point, could you open a GitHub issue for that in the operator repo. I’m not sure what will happen when the fdbserver processes are started with that flag but the fdbcli is not setting it 🤔

According to that commit message it should make the command stuck:

> Failing to specify this option  
> when the configuration database is not active will not affect the  
> correctness of the command, but it will hang instead of returning.

Created a new issue [here](https://github.com/FoundationDB/fdb-kubernetes-operator/issues/2059)

---

<div class="post-metadata">

**Author:** ![ljoswiak](https://avatars.discourse-cdn.com/v4/letter/l/977dab/32.png) [@ljoswiak](https://forums.foundationdb.org/u/ljoswiak)\
**Post date:** [July 8, 2024, 9:36pm UTC](https://forums.foundationdb.org/t/suspected-bug-in-config-broadcaster/4501/6 "2024-07-08T21:36:45Z")

</div>

Requiring extra flags from clients when the configuration database is disabled on the server is not a great experience. I submitted a PR to fix this by always serving the relevant interfaces even if the config DB is disabled [Serve no-op configuration database interface when disabled by sfc-gh-ljoswiak · Pull Request #11491 · apple/foundationdb · GitHub](https://github.com/apple/foundationdb/pull/11491). This allows the coordinator change command to succeed regardless of whether the config DB is running or not, without any extra client side logic.