publishRenameConcurrency (publish_rename_concurrency in YAML) sets how
many index files RenameFiles renames at once. It defaults to 4, the
value that was hard-coded, and 1 renames them one at a time. Values
below 1 are treated as 1, since a semaphore with no capacity would
never admit a rename.
The setting is global rather than per publishing endpoint because the
renames run against every storage backend, not only S3.
Acquire-By-Hash entries are written for MD5Sum, SHA1, SHA256 and SHA512,
hardcoded in two places in deb/index_files.go. Every algorithm is a full
set of index copies on each publish, and on S3-like storage each of those
is a network round trip rather than a link.
Measured on an R2-backed archive of 25 packages across three suites, one
`aptly publish update` per suite:
Acquire-By-Hash off 38 S3 requests
Acquire-By-Hash on 222 S3 requests (88 HeadObject, 83 CopyObject,
35 DeleteObject, 13 PutObject)
Modern apt verifies with SHA256; MD5Sum and SHA1 are deprecated and
Debian has been phasing them out of Release files for years. Publishers
who do not need them are paying three quarters of that for nothing, with
no way to say so.
acquireByHashAlgorithms in aptly.conf now selects the set. An empty or
absent value keeps all four, so existing configurations are unaffected.
The order entries are written in comes from the code rather than the
config, so the directories created in one loop and the entries written in
another cannot disagree. An unrecognised name is an error rather than
something to skip: skipping a typo would produce "no by-hash entries for
that algorithm", which looks exactly like having configured it away.
Set per publish invocation from the config, the way SkipBz2 already is,
rather than persisted on the publish point.
Pre-existing test failures on this tree are unchanged by this commit:
deb goes 173->177 passed with the same 11 failures, utils keeps its one
(TestCompress), s3 stays green.
- Remove unused pathCache field from PublishedStorage struct
- Remove unused md5s accumulation in Filelist()
- Fix unchecked error return on expectedOut.Write in config_test.go
- Suppress unused linter on configFileYAML/configFileYAMLError constants
(retained to avoid merge conflicts with feat/pls/gcs-support)
- Add --timeout=10m to golangci-lint workflow
- add back removed tests
This adds support for storing packages directly on Azure, with no truly
"local" (on-disk) repo used. The existing Azure PublishedStorage
implementation was refactored to move the shared code to a separate
context struct, which can then be re-used by the new PackagePool. In
addition, the files package's mockChecksumStorage was made public so
that it could be used in the Azure PackagePool tests as well.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Martin Simon
James Munson
André Roth
Pierig Le Saux
Ales Bregar
JupiterRider
hudeng
Ryan Gonzalez
Mauro Regli
Markus Muellner
boxjan
Benj Fassbind
Sjoerd Simons
Chuan Liu
Lorenzo Bolla