Acquire-By-Hash entries are written for MD5Sum, SHA1, SHA256 and SHA512,
hardcoded in two places in deb/index_files.go. Every algorithm is a full
set of index copies on each publish, and on S3-like storage each of those
is a network round trip rather than a link.
Measured on an R2-backed archive of 25 packages across three suites, one
`aptly publish update` per suite:
Acquire-By-Hash off 38 S3 requests
Acquire-By-Hash on 222 S3 requests (88 HeadObject, 83 CopyObject,
35 DeleteObject, 13 PutObject)
Modern apt verifies with SHA256; MD5Sum and SHA1 are deprecated and
Debian has been phasing them out of Release files for years. Publishers
who do not need them are paying three quarters of that for nothing, with
no way to say so.
acquireByHashAlgorithms in aptly.conf now selects the set. An empty or
absent value keeps all four, so existing configurations are unaffected.
The order entries are written in comes from the code rather than the
config, so the directories created in one loop and the entries written in
another cannot disagree. An unrecognised name is an error rather than
something to skip: skipping a typo would produce "no by-hash entries for
that algorithm", which looks exactly like having configured it away.
Set per publish invocation from the config, the way SkipBz2 already is,
rather than persisted on the publish point.
Pre-existing test failures on this tree are unchanged by this commit:
deb goes 173->177 passed with the same 11 failures, utils keeps its one
(TestCompress), s3 stays green.
in the API, snapshotPull never finished reading the target repository
within the async block. So the target was treated as an empty
snapshot, and a pull would only carry over the content from the sources
Make sure pathCache is properly locked for concurrent access.
Add RWMutex to the PublishedStorage struct:
- Cache initialization
Read-lock to test for nil, then write-lock with a second nil check before populating
- Cache reads
RLock/RUnlock, allowing concurrent readers
- Cache writes / deletes
Lock/Unlock
* load data inside background tasks
Perform collection.LoadComplete inside maybeRunTaskInBackground
Have tasks use a fresh copy of taskCollectionFactory, taskCollection
* perform collection.LoadComplete inside maybeRunTaskInBackground
* have tasks use a fresh copy of taskCollectionFactory, taskCollection
* fix locking for snapshots of snapshots by locking SourceSnapshots
* use uuids, since names can be renamed
* load data inside background tasks
Perform collection.LoadComplete inside maybeRunTaskInBackground
Have tasks use a fresh copy of taskCollectionFactory, taskCollection
* use uuids, since names can be renamed
* remove useless resource lock
Resource locks need to be before the background task. creating same publish endpoint at the same time is unlikely...
* load data inside background tasks
This fixes a flaw in async apis, which loaded the published repo from the DB and mutated it outside the task closure, before the task lock was acquired.
Perform collection.LoadComplete inside maybeRunTaskInBackground and have tasks use a fresh copy of taskCollectionFactory, taskCollection
* lock source repos/snapshots for publish operations
Concurrent tasks were not properly locking their resources, leading to inconsistent published indexes:
SourceLocalRepo: iterate published.Sources (component -> source UUID), look up each local repo via localRepoCollection.ByUUID and append string(repo.Key()) to resources
SourceSnapshot: iterate b.Snapshots,look up each snapshot via snapshotCollection.ByName and append string(snapshot.ResourceKey()) to resources.
* lock pool on non MultiDist publish
* revert mutex on LinkFromPool
* use uuids, since names can be renamed
* add test for MultiDist change
add API response wrappers with NumPackages derived from RefList length; keep show endpoint payloads unchanged for backward compatibility; add API tests for list endpoint NumPackages; update swagger response schemas for list endpoints
Hansjoerg Maurer
Rascalov
Martin Simon
Luan Dang
Luan DangandCopilot Autofix powered by AI
James Munson
André Roth
adam magued
Eleni Michalaki
Thomas Berger
Pierig Le Saux
Benoît Allard
Philip Cramer
Russell Greene
tom