This adds a new configuration setting: AzurePublishEndpoints, similar
to the existing S3PublishEndpoints and SwiftPublishEndpoints.
For each endpoint, the following has to be defined:
- accountName
- accountKey
- container
- prefix
Azure tests require the following environment variables to be set:
- AZURE_STORAGE_ACCOUNT
- AZURE_STORAGE_ACCESS_KEY
With either of these not set, Azure-specific tests are skipped.
The S3 backend relies on ETag S3 returns being equal to the MD5 of the
object, but it’s not necessarily true. When the value returned clearly
doesn’t look like a valid MD5 hash (length isn’t exactly 32 characters),
attempt to retrieve the MD5 hash possibly stored in the metadata.
We cannot always do this since user-defined metadata isn’t returned by
the ListObjects call, so verifying it for each object is expensive as it
requires one HEAD request per each object.
This commit fixes#923.
Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>
The S3 backend relies on ETag S3 returns being equal to the MD5 of the
object, but it’s not necessarily true. For that purpose we store the MD5
object in a separate metadata field as well to make sure it isn’t lost.
From https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html:
> The entity tag is a hash of the object. The ETag reflects changes only
> to the contents of an object, not its metadata. The ETag may or may not
> be an MD5 digest of the object data. Whether or not it depends on how
> the object was created and how it is encrypted as described below:
>
> Objects created by the PUT Object, POST Object, or Copy operation,
> or through the AWS Management Console, and are encrypted by SSE-S3 or
> plaintext, have ETags that are an MD5 digest of their object data.
>
> Objects created by the PUT Object, POST Object, or Copy operation,
> or through the AWS Management Console, and are encrypted by SSE-C or
> SSE-KMS, have ETags that are not an MD5 digest of their object data.
>
> If an object is created by either the Multipart Upload or Part Copy
> operation, the ETag is not an MD5 digest, regardless of the method
> of encryption.
Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>
Specifically, I have MacGPG installed instead of upstream GPG, which
results in the version string reading
gpg (GnuPG/MacGPG2) 2.2.17
instead of the expected
gpg (GnuPG) 2.2.17
