The S3 backend relies on ETag S3 returns being equal to the MD5 of the
object, but it’s not necessarily true. For that purpose we store the MD5
object in a separate metadata field as well to make sure it isn’t lost.
From https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html:
> The entity tag is a hash of the object. The ETag reflects changes only
> to the contents of an object, not its metadata. The ETag may or may not
> be an MD5 digest of the object data. Whether or not it depends on how
> the object was created and how it is encrypted as described below:
>
> Objects created by the PUT Object, POST Object, or Copy operation,
> or through the AWS Management Console, and are encrypted by SSE-S3 or
> plaintext, have ETags that are an MD5 digest of their object data.
>
> Objects created by the PUT Object, POST Object, or Copy operation,
> or through the AWS Management Console, and are encrypted by SSE-C or
> SSE-KMS, have ETags that are not an MD5 digest of their object data.
>
> If an object is created by either the Multipart Upload or Part Copy
> operation, the ETag is not an MD5 digest, regardless of the method
> of encryption.
Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>
Specifically, I have MacGPG installed instead of upstream GPG, which
results in the version string reading
gpg (GnuPG/MacGPG2) 2.2.17
instead of the expected
gpg (GnuPG) 2.2.17
This commit closes: #145
The dependency format "pkg:arch" (e.g. "python3:any") was not well
parsed if not any version is given. This commit splits the dependency
name and architecture in all cases.
For any action which is multi-step (requires updating more than 1 DB
key), use transaction to make update atomic.
Also pack big chunks of updates (importing packages for importing and
mirror updates) into single transaction to improve aptly performance and
get some isolation.
Note that still layers up (Collections) provide some level of isolation,
so this is going to shine with the future PRs to remove collection
locks.
Spin-off of #459
This is spin-off of changes from #459.
Transactions are not being used yet, but batches are updated to work
with the new API.
`database/` package was refactored to split abstract interfaces and
implementation via goleveldb. This should make it easier to implement
new database types.
Use CDN-backed Debian mirror to make tests run faster hopefully for
everyone. Redirects might be important to know what exactly is going on
when items are being downloaded.
Apply retries as global, config-level option `downloadRetries` so that
it can be applied to any aptly command which downloads objects.
Unwrap `errors.Wrap` which is used in downloader.
Unwrap `*url.Error` which should be the actual error returned from the
HTTP client, catch more cases, be more specific around failures.
