Backup to s3(ali cloud oss) through backup agent pod error Net2TLSInitError

Using FoundationDB
1

I am a new to FDB, I want to backup to s3 through backup agent pod and run command:

FDB_TLS_CA_FILE=/etc/ssl/certs/ca-bundle.trust.crt fdbbackup start --knob_http_request_aws_v4_header=true -C /mnt/config-volume/cluster-file -d 'blobstore://key:secret@oss-cn-hangzhou-internal.aliyuncs.com:443/hzbyconityfdbtest/backup/test0924?bucket=backuptest&secure_connection=1'  --log

`it returns The backup on tag `default' was successfully submitted.but it acturaly not works,
```
5 
<Event Severity="10" Time="1727261796.363343" DateTime="2024-09-25T10:56:36Z" Type="Net2Starting" ID="0000000000000000" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.364038" DateTime="2024-09-25T10:56:36Z" Type="ProgramStart" ID="0000000000000000" SourceVersion="dfe8dacba348815b4b2d4746ffa056303c72259d" Version="7.1.15" PackageName="7.1" ActualTime="1727261796" CommandLine="fdbbackup start --knob_http_request_aws_v4_header=true -s 60 -C /mnt/config-volume/cluster-file -d blobstore://LTAI5tHFdBp8bxzRWch4ME3W:tJJmrlyPRMJ0DORYlfWdEWWj049diT@func-saas-private.oss-cn-hangzhou-internal.aliyuncs.com:443/hzbyconityfdbtest/backup/test0924?bucket=func-saas-private&amp;secure_connection=1 --log --tls-certificate-file=/etc/ssl/certs/ca-bundle.trust.crt --tls-ca-file=/etc/pki/tls/cert.pem" MemoryLimit="8589934592" ThreadID="16023373979936087421" TrackLatestType="Original" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.363414" DateTime="2024-09-25T10:56:36Z" Type="ClientStart" ID="0000000000000000" SourceVersion="dfe8dacba348815b4b2d4746ffa056303c72259d" Version="7.1.15" PackageName="7.1" ActualTime="1727261796" ApiVersion="-1" ClientLibrary="fdbbackup" ImageOffset="(nil)" Primary="1" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" TrackLatestType="Original" />
<Event Severity="10" Time="1727261796.363414" DateTime="2024-09-25T10:56:36Z" Type="MachineLoadDetail" ID="0000000000000000" User="314244774" Nice="2336" System="121219453" Idle="5888471696" IOWait="15371106" IRQ="0" SoftIRQ="4922008" Steal="0" Guest="0" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.363414" DateTime="2024-09-25T10:56:36Z" Type="Net2TLSConfig" ID="0000000000000000" CAPath="/etc/pki/tls/cert.pem" CertificatePath="/etc/ssl/certs/ca-bundle.trust.crt" KeyPath="" HasPassword="0" VerifyPeers="Check.Valid=1" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.363414" DateTime="2024-09-25T10:56:36Z" Type="NotifyAddressHealthy" ID="0000000000000000" SuppressedEventCount="0" Address="10.16.11.66:4501" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.363414" DateTime="2024-09-25T10:56:36Z" Type="ConnectToDatabase" ID="f90fa90affc980f2" Version="7.1.15" ClusterFile="file:///mnt/config-volume/cluster-file" ConnectionString="lc_hz_test_byconity_fdb:fJhErMsD9ZmhkRSwTOl7BAmWgCNJpax0@10.16.11.66:4501,10.16.35.45:4501,10.16.149.189:4501" ClientLibrary="fdbbackup" Primary="1" Internal="1" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" TrackLatestType="Original" />
<Event Severity="10" Time="1727261796.363414" DateTime="2024-09-25T10:56:36Z" Type="FBA_SubmitBackup" ID="0000000000000000" TagName="default" StopWhenDone="1" UsePartitionedLog="0" OutContainer="blobstore://LTAI5tHFdBp8bxzRWch4ME3W:tJJmrlyPRMJ0DORYlfWdEWWj049diT@func-saas-private.oss-cn-hangzhou-internal.aliyuncs.com:443/hzbyconityfdbtest/backup/test0924?bucket=func-saas-private&amp;secure_connection=1" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.363414" DateTime="2024-09-25T10:56:36Z" Type="CodeCoverage" ID="0000000000000000" File="/home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/ReadYourWrites.actor.cpp" Line="1529" Condition="true" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.363414" DateTime="2024-09-25T10:56:36Z" Type="StartingRunLoopProfilingThread" ID="0000000000000000" Interval="0.125" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.363414" DateTime="2024-09-25T10:56:36Z" Type="Net2Running" ID="0000000000000000" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.363414" DateTime="2024-09-25T10:56:36Z" Type="ProcessTimeOffset" ID="0000000000000000" ProcessTime="1727261796.376338" SystemTime="1727261796.376338" OffsetFromSystemTime="-0.000000" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.376371" DateTime="2024-09-25T10:56:36Z" Type="ConnectingTo" ID="0000000000000000" SuppressedEventCount="0" PeerAddr="10.16.11.66:4501" PeerReferences="3" FailureStatus="OK" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.376371" DateTime="2024-09-25T10:56:36Z" Type="ConnectionExchangingConnectPacket" ID="97de8151843d6fb2" SuppressedEventCount="0" PeerAddr="10.16.11.66:4501" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.377612" DateTime="2024-09-25T10:56:36Z" Type="ConnectionEstablished" ID="97de8151843d6fb2" SuppressedEventCount="0" Peer="10.16.11.66:4501" ConnectionId="1" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.377612" DateTime="2024-09-25T10:56:36Z" Type="ConnectedOutgoing" ID="0000000000000000" SuppressedEventCount="0" PeerAddr="10.16.11.66:4501" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.403044" DateTime="2024-09-25T10:56:36Z" Type="S3BlobStoreEndpointNewConnection" ID="0000000000000000" SuppressedEventCount="0" RemoteEndpoint="100.118.28.49:443:tls(fromHostname)" ExpiresIn="120" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
<Event Severity="10" Time="1727261796.414485" DateTime="2024-09-25T10:56:36Z" Type="S3BlobStoreEndpointReusingConnected" ID="0000000000000000" SuppressedEventCount="0" RemoteEndpoint="100.118.28.49:443:tls(fromHostname)" ExpiresIn="119.989" ThreadID="16023373979936087421" Machine="10.20.3.31:502" LogGroup="default" ClientDescription="primary-7.1.15-16023373979936087421" />
6

fdbbackup status -C /mnt/config-volume/cluster-file
The previous backup on tag `default’ at blobstore://key:secret@func-saas-private.oss-cn-hangzhou-internal.aliyuncs.com:443/backup/test0924?bucket=func-saas-private&secure_connection=1 has been aborted.
BackupUID: 79423dd12a0b29369e388860b1602326
BackupURL: blobstore://key:secret@func-saas-private.oss-cn-hangzhou-internal.aliyuncs.com:443/hzbyconityfdbtest/backup/test0924?bucket=func-saas-private&secure_connection=1
Recent Errors (since initialization)
20.57 minute(s) ago : ‘Network connection failed’ on ‘file_backup_write_logs_5.2’

The logs do not show any errors, only that the --tls-key-file configuration was not set because I did not find this file in the backup agent’s Pod

I would be extremely grateful if anyone familiar with this area could provide an answer to this question.

7

Just to make sure, the environment variable FDB_TLS_CA_FILE is also set on the backup_agent pods? How did you start the backup agent pods?

edit: The logs above are from the pod where you ran the fdbbackup start command? Or from one of the backup_agent pods?

8

yes logs from one of backup_agent pod where i run fdbbackup command and the pod create by deployment file

---- Replied Message ----

From | Johannes Scheuermann via FoundationDBnotifications@foundationdb.discoursemail.com |

  • | - |
    Date | 09/26/2024 00:21 |
    To | 13362133754@163.com |
    Cc | |
    Subject | [FoundationDB] [Using FoundationDB] Backup to s3(ali cloud oss) through backup agent pod error Net2TLSInitError |

| johscheuer Johannes Scheuermann
September 25 |

  • | - |

Just to make sure, the environment variable FDB_TLS_CA_FILE is also set on the backup_agent pods? How did you start the backup agent pods?

edit: The logs above are from the pod where you ran the fdbbackup start command? Or from one of the backup_agent pods?