Hi, I’m configuring backups to s3. My S3 bucket uses SSE with AWS KMS, which requires AWS Signature Version 4, according to the HTTP error I got:
<Error><Code>InvalidArgument</Code><Message>Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.</Message><ArgumentName>Authorization</ArgumentName>[...]
It would be great if fdbbackup used V4 auth in order to support KMS SSE. Also, I am using instance profile temporary S3 credentials, which requires specifying the security token as a HTTP header (x-amz-security-token). I can currently add it using the header arg at the end of the backupURL. If fdbbackup were to support V4, note that the security token must be part of the auth signature
What does this mean? I am not familiar with AWS credentials. All workers for a backup or restore are going to use the same URL, so if the thing passed via the header option is per-agent or expires then you would have to use fdbbackup modify to update the URL for a running backup, and for restore currently there is no way to modify the source URL of an active restore process.
Instance profiles grant an IAM role to an EC2 instance so that you can access resources like ListBucket, Get/PutObject, etc. without directly using Access/Secret key string credentials: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html Under the hood it uses rotating temporary credentials. I fetched the credentials from the s3 client (I’m using golang) and formed the backup URL that way. These temporary credentials require the session token.
V4 is just the auth header in the HTTP request. All backup workers can use the same credentials, so the same backup URL (this is how we currently get fdbbackup to work). The requested change would just be how the auth header is created: https://github.com/apple/foundationdb/blob/master/fdbclient/BlobStore.actor.cpp#L971-L1020 (The V4 page I linked describes how to calculate the new signature)
Upon second read looks like the V4 signature for session token still uses the same header name x-amz-security-token , which is already part of the current calculation, so nothing would really need to change to specifically support the session token in a V4 auth header. However it would be really great if the credentials file included the security token. The command is logged in the fdbbackup logs, and to avoid leaking the s3 credentials I used a blob_credentials file. But I have to add the security token as a url param and there is no way to prevent it from being logged unless I disable logging altogether.
This is what I was wondering about, and it sounds like it simply has to for a correct implementation, due to visibility but also because it sounds like this token is meant to be used from a single EC2 instance, so wouldn’t backup agents on the same cluster but running in difference EC2 instances require different security tokens? With the security token in the blob credentials file, deployment tooling can generate per-backup_agent credentials files and rotate them as needed (which fdbbackup will pick up on fairly quickly).
Ah, that’s right, that’s a good point. You can use the credentials (access, secret, token) from any machine, it’s not locked to the EC2 instance. (That’s how we currently get it to work) And deployment tooling can regenerate a credentials file per-backup agent. But as you’d expect, you can’t use access + secret keys from your own machine with a token from a different machine. So in the current state, if you want to use a credentials file you must standard S3 static credentials that don’t require a token. If you want to use temporary credentials, you must include the credentials and token in the backup URL so that the same one is used on all agents.
But if the credentials file included the token, using the file or stuffing creds in the backup URL would both be valid options
We should probably add the v4 signature and make it the default but keep v2 around for compatibility as there are non-Amazon S3 implementations that might not support V4.
In the credentials file, I’m thinking we just add a “header_*” instead of a specific “aws_security_token” concept. So basically a way to pass sensitive headers vs with the existing URL parameter method.
Just wanted to check in, is there a tracker for these two requests? (AWS V4 signature, adding headers to credentials file)?
Another benefit is that fdbbackup list will list all backups with the credentials and tokens, and the output sanitizes the token. Makes sense why that’s the case, but I got stuck for a bit before I realized I can’t just pass in one of those backup URLs from fdbbackup list. Having creds + token in the file would fix this as well