Commit Graph

2240 Commits

Author SHA1 Message Date
André Roth
ff1557afee update go.mod 2024-07-31 22:16:00 +02:00
André Roth
67771795ca etcd: implement transactions
- use temporary db for lookups in transactions
- use batch implementation to commit transaction
2024-07-31 22:16:00 +02:00
André Roth
7a01c9c62d etcd: implement batch operations
- cache the operations internally in a list
- Write() applies the list to etcd
2024-07-31 22:16:00 +02:00
André Roth
9768ecef22 etcd: implement temporary db support
- temporary db support is implemented with a unique key prefix
- prevent closing etcd connection when closing temporary db
2024-07-31 22:16:00 +02:00
André Roth
640c202ee5 etcd: implement separate system tests
- add t13_etcd test directory
- etcd will be started for the unit tests and each system test
- etcd will load fixture DB export if requested by the test
- existing tests are reused for etcd testing
2024-07-31 22:16:00 +02:00
André Roth
f10acb3df8 etcd: fix db config and test
fix unit test for adapted config file
2024-07-31 22:16:00 +02:00
André Roth
5b74f82edb etcd: fix int overflow
goxc fails with:

Error: database/etcddb/database.go:17:25: cannot use 2048 * 1024 * 1024 (untyped int constant 2147483648) as int value in struct literal (overflows)
2024-07-31 22:16:00 +02:00
hudeng
59bf4501e8 feat: Use databaseBackend config repace databaseEtcd
databaseBackend config contains type and url sub config, It can facilitate the expansion of other types of databases in the future.
2024-07-31 22:16:00 +02:00
hudeng
f29449db14 feat: Add system test for etcd 2024-07-31 22:16:00 +02:00
hudeng
78172d11d7 feat: Add etcd database support
improve concurrent access and high availability of aptly with the help of the characteristics of etcd
2024-07-31 22:16:00 +02:00
NeroBurner
f42ff697d4 CONTRIBUTING: Python3 is supportet
Explicitly state that Python3 is supported and required.

Since aptly `v1.5.1` (with commit 035d5314b0)
the tests are ported to Python3.

With a687df2f4f the system
tests are run with `python3` per default.

With f4a152ab22 (after `v1.5.0` aptly tag)
the tests run against Python 3.11.
2024-07-25 10:10:05 +02:00
André Roth
57e2c5c670 api tests: use check_task_fail 2024-07-24 21:19:47 +02:00
André Roth
8b41ec48c8 use consistent golangci-lint version 2024-07-24 21:19:47 +02:00
André Roth
49ff832f94 reenable lost tests 2024-07-24 21:19:47 +02:00
André Roth
09a44ba409 fix empty mirror check 2024-07-24 21:19:47 +02:00
André Roth
deae90485a fix DirIsAccessible
perms 0000 need to be checked explicitly
2024-07-24 21:19:47 +02:00
André Roth
4a0bdcbb64 improve system tests
- log import errors for test modules
- log output only on test failure
- improve docker system test container
- use go 1.19 in docker system tests
- download go dependencies in docker container
- system tests: color failues output
- imrpove test result output
- do not install golangci-lint in system tests
2024-07-24 21:19:47 +02:00
André Roth
9f1860dff7 fix unit test 2024-07-24 21:19:47 +02:00
André Roth
fe25414b45 api: repo copy handle package not found
and add tests for error proper handling.
2024-07-24 21:19:47 +02:00
André Roth
49184c9163 fix apiReposCopyPackage getting corrupt file name
it seems c.Params.ByName("file") should not be used
inside maybeRunTaskInBackground, as the content may be corrupted sometimes.
2024-07-24 21:19:47 +02:00
André Roth
440c3debdc improve api tests and error output
show only relevant aptly logs if a test fails.
for async tasks, show task output, as it contains the error message.
2024-07-24 21:19:47 +02:00
André Roth
8029305d32 dependencies: remove duplicates / missing deps from test 2024-07-11 18:25:49 +02:00
5hir0kur0
02bdb7c76a Deduplicate missing dependency list 2024-07-11 18:25:49 +02:00
5hir0kur0
8d537b4e3e Fix bug in dependency resolution 2024-07-11 18:25:49 +02:00
Sylvain Nieuwlandt
11401ca472 [api/copy] create system tests for new copy api endpoint 2024-07-10 16:43:03 +02:00
Valentin BRICE
66429bff45 [api/repos] Add copy API 2024-07-10 16:43:03 +02:00
Sylvain Nieuwlandt
8114786179 Declare the Copy API 2024-07-10 16:43:03 +02:00
André Roth
d8c1e432c6 add test 2024-07-03 18:08:58 +02:00
André Roth
3a286ae07f fix unit tests 2024-07-03 18:08:58 +02:00
André Roth
a93ccd4100 fix tests 2024-07-03 18:08:58 +02:00
André Roth
c1f7e5fe96 handle GpgDisableVerify and ignore-signatures consistently
and be less verbose
2024-07-03 18:08:58 +02:00
André Roth
d16110068c allow not signed mirrors without InRelease file 2024-07-03 18:08:58 +02:00
André Roth
4661913265 add system test for maximumVersion filter 2024-06-24 17:44:40 +02:00
hudeng
ecc88e7a40 feat: repo and snapshots packages filter api add 'maximumVersion' query parameter support
example: `curl http://localhost:8080/api/repos/test/packages\?maximumVersion\=1`

Change-Id: Ie9ffd36146bf017bbb353737f32360f7b73d6b0a
2024-06-24 17:44:40 +02:00
André Roth
5cf8c54cb2 fix test 2024-06-20 23:40:46 +02:00
André Roth
b758033ccb fix compilation 2024-06-20 23:40:46 +02:00
Kevin Martin
13f4bb441d Check if S3 bucket is encrypted by default.
Adds check to see if the S3 bucket is encrypted by default. If so this
uses the existing workaround for object etags not matching file MD5s.
2024-06-20 23:40:46 +02:00
Kevin Martin
1af09069f7 Check both MD5 locations for S3 KMS support.
If the S3 bucket used to house a repo has KMS encryption enabled then
the etag of an object may not match the MD5 of the file. This may
cause an incorrect error to be reported stating the file already
exists and is different.

A mechanism exists to work around this issue by using the MD5 stored
in object metadata. This check doesn't always cover the case where KMS
is enabled as the fallback is only used if the etag is not 32
characters long.

This commit changes the fallback mechanism so that it is used in any
case where the object's etag does not match the source MD5. This will
incur a performance penalty of an extra head request for each object
with a mismatch.
2024-06-20 23:40:46 +02:00
Ryan Gonzalez
b5bf2cbcda Fix functional tests' '--capture' on Python 3
None of the commands' output is ever treated as binary, so we can just
always decode it as text.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2024-06-17 11:51:18 +02:00
André Roth
fb3436b23d fix golangci-lint errors 2024-06-17 11:51:18 +02:00
André Roth
1a3cfea348 replace io/ioutil
fixes golangci-lint errors
2024-06-17 11:51:18 +02:00
André Roth
c33141d49b fix golangci-lint and compilation errors 2024-06-17 11:51:18 +02:00
Ryan Gonzalez
f9325fbc91 Add support for Azure package pools
This adds support for storing packages directly on Azure, with no truly
"local" (on-disk) repo used. The existing Azure PublishedStorage
implementation was refactored to move the shared code to a separate
context struct, which can then be re-used by the new PackagePool. In
addition, the files package's mockChecksumStorage was made public so
that it could be used in the Azure PackagePool tests as well.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2024-06-17 11:51:18 +02:00
Ryan Gonzalez
810df17009 Clean up temporary files when mirroring
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2024-06-17 11:51:18 +02:00
Ryan Gonzalez
19255debb9 Reduce required usage of LocalPackagePool
Several sections of the code *required* a LocalPackagePool, but they
could still perform their operations with a standard PackagePool.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2024-06-17 11:51:18 +02:00
Ryan Gonzalez
1ebd37f9ad Move Stat() into LocalPackagePool
The contents of `os.Stat` are rather fitted towards local package pools,
but the method is in the generic PackagePool interface. This moves it to
LocalPackagePool, and the use case of simply finding a file's size is
delegated to a new, more generic PackagePool.Size() method.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2024-06-17 11:51:18 +02:00
Ryan Gonzalez
8e37813129 Add support for custom package pool locations
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2024-06-17 11:51:18 +02:00
Ryan Gonzalez
91574b53d9 Add functional tests for Azure publishing
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2024-06-17 11:51:18 +02:00
Ryan Gonzalez
b8e0aba3cf Document Azure configuration
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2024-06-17 11:51:18 +02:00
Ryan Gonzalez
0eccaf2b60 Change Azure endpoint syntax to require a full URL
Before, a "partial" URL (either "localhost:port" or an endpoint URL
*without* the account name as the subdomain) would be specified, and the
full one would automatically be inferred. Although this is somewhat
nice, it means that the endpoint string doesn't match the official Azure
syntax:

https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string

This also raises issues for the creation of functional tests for Azure,
as the code to determine the endpoint string needs to be duplicated
there as well.

Instead, it's just easiest to follow Azure's own standard, and then
sidestep the need for any custom logic in the functional tests.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2024-06-17 11:51:18 +02:00