I’d like to propose an optional prefix URL parameter for blobstore backup containers.
Motivation
Today, BackupContainerBlobStore always places backup objects under two bucket-root trees:
data/<name>/...
backups/<name>
This makes it difficult to use a bucket that is shared with other applications or FDB deployments. In particular, IAM policies, replication rules, lifecycle policies, inventory, cleanup, and cost accounting are often scoped to one object key prefix. With the current layout, a deployment-specific prefix cannot cover both the backup data and its index entry.
Proposed behavior
Add an optional prefix parameter to blobstore backup URLs:
The relative data/ and backups/ layout is unchanged. A base URL carrying the same prefix makes fdbbackup list enumerate only <prefix>/backups/, and the returned container URLs retain the parameter so they can be reopened directly. Backup, restore, describe, expire, delete, and BulkDump/BulkLoad all use the same prefixed layout.
Compatibility and validation
The parameter is optional. Without it, object keys are byte-for-byte identical to the current layout.
Older clients and backup agents reject this unknown backup-specific parameter, so all participating agents and tools must be upgraded before a prefixed URL is used.
Leading and trailing / characters are stripped. After that normalization, / separates path segments; each segment is limited to ASCII [A-Za-z0-9._-]. Internal empty segments and . or .. segments are rejected, and the normalized first segment cannot be data or backups. Percent-encoding is not decoded. If normalization leaves an empty value, the existing bucket-root layout is used.
The prefix only changes object-key construction; FoundationDB does not enforce isolation between other overlapping layouts. Any security boundary must be enforced by the object store/IAM configuration. As backup names may contain /, other prefix/name combinations whose resulting object key trees overlap must be avoided; behavior for overlapping trees is undefined.
Implementation scope
The implementation:
centralizes the prefixed data and index paths in BackupContainerBlobStore;
carries the parameter through container listing and returned URLs;
aligns the direct BulkDump/BulkLoad object path with the backup container layout;
documents the new URL parameter; and
adds unit and end-to-end coverage for both prefixed and unprefixed layouts.
file:// and azure:// containers are unchanged.
Validation
The implementation has been validated with:
focused unit tests for prefix parsing, normalization, URL semantics, path construction, and BulkDump/BulkLoad paths;
prefixed and unprefixed MockS3 backup/restore end-to-end tests, with the prefixed variant additionally verifying the physical object layout and round-tripping a URL returned by fdbbackup list;
prefixed and unprefixed BulkDump/BulkLoad end-to-end tests; and
prefixed rangefile backup/restore runs against Tencent COS and Alibaba Cloud OSS through their S3-compatible APIs, including physical layout, list isolation, and restored-data equality checks.
An implementation is available for review here:
Feedback is welcome, particularly on the parameter name, the validation/overlap semantics, and whether this is the preferred way to scope all objects belonging to a blobstore backup under one key prefix.