Concurrent tasks were not properly locking their resources, leading to
inconsistent published indexes:
Task A: apiPublishUpdateSwitch loads published, reads source repo/snapshot
Request B: modifies same source repo or snapshot (add/remove packages, etc)
Task A: Update() + Publish() reads stale/modified source -> inconsistent
published index, or partial write if source deleted mid-task.
This changes introduces resource locking for publishing:
* SourceLocalRepo: iterate published.Sources (component -> source UUID), look up each local repo via localRepoCollection.ByUUID and append
string(repo.Key()) to resources
* SourceSnapshot: iterate b.Snapshots,look up each snapshot via snapshotCollection.ByName and append
string(snapshot.ResourceKey()) to resources.
This fixes a flaw in async apis, which loaded the published repo from the
DB and mutated it outside the task closure, before the task lock was acquired.
* perform collection.LoadComplete inside maybeRunTaskInBackground
* have tasks use a fresh copy of taskCollectionFactory, taskCollection
https://wiki.debian.org/DebianRepository/Format#Signed-By says:
> **Signed-By**
> An optional field containing a comma separated list of
> OpenPGP key fingerprints to be used for validating
> the next Release file. The fingerprints must consist
> only of hex digits and may not contain spaces.
> The fingerprint specifies either the key the Release file
> must be signed with or the key the signature key must be
> a subkey of. The later match can be disabled by appending
> an exclamation mark to the fingerprint.
>
> If the field is present, a client should only accept future updates
> to the repository that are signed with keys listed in the field.
> The field should be ignored if the Valid-Until field is not present
> or if it is expired.
For both the CLI tools and JSON, the field is taken as a string verbatim.
When specified, we must also provide `Valid-Until` field,
and i'm not sure there is an 'infinity' value for it,
so 100 years will have to do?
Fixes https://github.com/aptly-dev/aptly/issues/1497
This commit allows to add, remove and update components of published repositories without the need to recreate them.
Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
Do all relevant database reading/modifying inside `maybeRunTaskInBackground`.
Notably, `LoadComplete` will load the reflist of a repo. if this is done outside of a background operation,
the data might be outdated when the background tasks runs.
This commit modifies the behavior of the publish switch method in the way, that also new components can be added to an already published repository. It is no longer necessary to drop and recreate the whole publish.
Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
This change makes it possible to publish multiple distributions
with packages named the same but with different content by changing
structure of the generated pool hierarchy. The option not enabled
by default as this changes the structure of the output which could
break the expectations of other tools.
The output doesn't actually depend on the reflists, and loading them for
every published repo starts to take substantial time and memory.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
André Roth
Zhang Xiao
Ales Bregar
Roman Lebedev
chesseed
Mauro Regli
Christoph Fiehe
Noa Resare
Ryan Gonzalez
Markus Muellner