- log import errors for test modules
- log output only on test failure
- improve docker system test container
- use go 1.19 in docker system tests
- download go dependencies in docker container
- system tests: color failues output
- imrpove test result output
- do not install golangci-lint in system tests
If the S3 bucket used to house a repo has KMS encryption enabled then
the etag of an object may not match the MD5 of the file. This may
cause an incorrect error to be reported stating the file already
exists and is different.
A mechanism exists to work around this issue by using the MD5 stored
in object metadata. This check doesn't always cover the case where KMS
is enabled as the fallback is only used if the etag is not 32
characters long.
This commit changes the fallback mechanism so that it is used in any
case where the object's etag does not match the source MD5. This will
incur a performance penalty of an extra head request for each object
with a mismatch.
None of the commands' output is ever treated as binary, so we can just
always decode it as text.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
This adds support for storing packages directly on Azure, with no truly
"local" (on-disk) repo used. The existing Azure PublishedStorage
implementation was refactored to move the shared code to a separate
context struct, which can then be re-used by the new PackagePool. In
addition, the files package's mockChecksumStorage was made public so
that it could be used in the Azure PackagePool tests as well.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Several sections of the code *required* a LocalPackagePool, but they
could still perform their operations with a standard PackagePool.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
The contents of `os.Stat` are rather fitted towards local package pools,
but the method is in the generic PackagePool interface. This moves it to
LocalPackagePool, and the use case of simply finding a file's size is
delegated to a new, more generic PackagePool.Size() method.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Before, a "partial" URL (either "localhost:port" or an endpoint URL
*without* the account name as the subdomain) would be specified, and the
full one would automatically be inferred. Although this is somewhat
nice, it means that the endpoint string doesn't match the official Azure
syntax:
https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string
This also raises issues for the creation of functional tests for Azure,
as the code to determine the endpoint string needs to be duplicated
there as well.
Instead, it's just easiest to follow Azure's own standard, and then
sidestep the need for any custom logic in the functional tests.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
read_path() can read in binary, which the S3 tests don't support (simply
because they don't need it)...but it needs to be able to take the `mode`
argument anyway.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
This commit blocks concurrent calls to RunTaskInBackground which is
intended to fix the quirky behaviour where concurrent PUT calls to
api/publish/<prefix>/<distribution> would immedietly reuturn an error.
The solution proposed in this commit is not elegant and probaly has
unintended side-effects. The intention of this commit is to highlight
the area that actually needs to be addressed.
Ideally this patch is amended or dropped entierly in favor of a better
fixup.
This commit introduces a test which runs concurrent publishes (which
could be parallell with multiproccessing, python is fun).
The test purposly fails (at the point in history that this patch is
written) in order to make it as easy as possible to verify later patches,
which hopefully addresses concurrency problems.
The same behaviour can easily be tested outside of the system tests with
the following (or similar) shell
$ aptly serve -listen=:8080 -no-lock
$ aptly repo create create -distributions=testing local-repo
$ atply publish repo -architectures=amd64 local-repo
$ apt download aptly
$ aptly repo add local-repo ./aptly*.deb
$ for _ in $(seq 10); do curl -X PUT 127.0.0.1:8080/api/publish//testing
In the local testing perfomed (on a dual core vm) the first 1-4 jobs
would typically succeed and the rest would error out.
André Roth
5hir0kur0
Sylvain Nieuwlandt
Valentin BRICE
hudeng
Kevin Martin
Ryan Gonzalez
Ramón N.Rodriguez