Update Task-Race-Conditions.md with complete final assessment:
Results:
- 3 real data races found and fixed (Issues 1, 2, 4-NEW)
- 4 false alarms identified (Issues 3, 4, 5, 6)
- 1 low-severity logic race — won't-fix (Issue 7)
False alarm analysis:
- Issue 3: ResourcesSet map is protected indirectly by list.Lock()
(all callers hold list.Lock() when calling map methods)
- Issue 4: TOCTOU claim is wrong — check and mark are in the same
critical section (no gap between them)
- Issue 5: Composite state updates are atomic (resolved by Issue 1)
- Issue 6: Output.Write() is a no-op stub (doesn't access shared state)
Won't-fix rationale (Issue 7):
- WaitForTaskByID post-deletion requires user to simultaneously wait
for AND delete the same task (conflicting API calls)
- No data corruption or panic — just a confusing error message
- User-error scenario, not a code defect
RunTaskInBackground() previously returned *task AFTER releasing list.Lock()
and sending the task to the consumer queue. This created a data race:
1. list.queue <- task (consumer receives)
2. Consumer: list.Lock() → task.State = RUNNING → list.Unlock()
3. RunTaskInBackground: return *task (struct copy WITHOUT lock)
Steps 2 and 3 can execute concurrently — consumer writes task.State
while RunTaskInBackground reads the entire struct via copy.
Fix: Copy the task struct BEFORE unlocking, while list.Lock() is still
held. At this point the task was just created and no other goroutine can
access it, so the copy is guaranteed consistent (always State=IDLE).
The returned copy is a snapshot of the initial task state, which is what
callers expect — the task ID and name for tracking purposes.
Safety invariant maintained:
- I4: All struct copies happen while list.Lock() is held
Changes:
- task/list.go: RunTaskInBackground() copies *task before unlock,
returns the pre-made copy instead of dereferencing after unlock
## Problem
Critical race condition where task State, err, and processReturnValue fields
were written by consumer goroutine and read by concurrent accessors without
proper synchronization, causing torn reads and data races.
## Solution
Implemented single-lock model with optimal lock scope:
- Removed per-task RWMutex (unnecessary with proper lock scope)
- Removed 8 accessor methods (direct field access is simpler)
- Lock only during brief state transitions (IDLE→RUNNING, RUNNING→SUCCEEDED/FAILED)
- Release lock during task.process() execution to enable full concurrency
- Readers hold list.Lock() only during atomic struct copy
- Moved State = RUNNING before goroutine spawn for clearer semantics
## Design Principles
Lock scope matters more than lock type. When list.Lock() is held during all
task field modifications and reads, a single well-scoped lock is sufficient.
The RUNNING state is stable (not modified during execution), enabling readers
to safely copy task state without additional synchronization.
## Changes
- task/task.go: Removed sync.RWMutex field and 8 accessor methods (-80 lines)
- task/list.go: Simplified consumer and reader methods (-50 lines)
* consumer(): Set State=RUNNING before goroutine, kept brief lock scope
* GetTasks(): Hold lock through struct copy
* GetTaskByID(): Hold lock through struct copy
* DeleteTaskByID(): Hold lock for safe field access
* GetTaskReturnValueByID(): Hold lock during field read
* GetTaskErrorByID(): Hold lock during field read
* Clear(): Hold lock during field read
## Race Conditions Fixed
✅ Consumer writes State, reader reads State
✅ Consumer writes err, reader reads err
✅ Consumer writes processReturnValue, reader reads
✅ Torn reads of multiple fields
✅ Inconsistent state observations
✅ Non-atomic multi-field updates
## Performance & Concurrency
- Lock overhead: ~200ns per task (0.0007% of 30ms execution)
- Full concurrent execution: Multiple tasks run in parallel
- No lock held during task.process() execution (key for concurrency)
- Brief contention only during state transitions (~100ns)
## Safety Verification
Invariants established:
- I1: State modified only under list.Lock()
- I2: err and processReturnValue modified only under list.Lock()
- I3: When State == RUNNING, consumer doesn't modify fields
- I4: Readers hold list.Lock() when copying task
Result: No concurrent read/write, no torn reads, no deadlocks
## Testing
All existing tests pass unchanged:
go test ./task/...
Verify fix with race detector:
go test -race ./task/...
## Documentation
Comprehensive analysis in docs/:
- Task-Race-Conditions.md (original analysis of 7 race conditions)
- FINAL-DESIGN-EXPLANATION.md (design correctness proof)
- VISUAL-COMPARISON.md (before/after visualizations)
- CHANGES-DETAILED.md (line-by-line change documentation)
Total: 100+ KB of design documentation
Fixes #Issue1
Affected endpoints: apiSnapshotsCreate, apiSnapshotsUpdate, apiSnapshotsDrop,
apiSnapshotsMerge, apiSnapshotsPull.
All five endpoints shared the same architectural flaw as the previously fixed
repos and publish endpoints: operations were performed outside the task lock,
with stale DB state used inside the lock.
Issues Fixed:
1. apiSnapshotsCreate - Source snapshots loaded before task lock
Problem: snapshotCollection and collectionFactory created before task lock.
Source snapshots and destination check done with stale factory.
Concurrent creates both load pre-task state, second overwrites first.
Fix: Create fresh taskCollectionFactory inside task, fresh loads of all
sources after lock acquired, pre-task duplicate check for destination,
use fresh sources and collections for snapshot creation.
2. apiSnapshotsUpdate - Snapshot loaded before task lock
Problem: snapshot loaded outside task, duplicate check with stale factory.
Concurrent renames both load pre-task state, both pass check, second
overwrites first.
Fix: Create fresh taskCollectionFactory inside task, fresh load of snapshot
after lock acquired, fresh duplicate check inside lock, pre-task validation
of new name, atomic rename with fresh copy.
3. apiSnapshotsDrop - Collections created before task lock
Problem: snapshotCollection and publishedCollection created before task lock.
Concurrent snapshot/published modifications not detected. Can delete snapshot
that becomes published between pre-task and task.
Fix: Create fresh taskCollectionFactory inside task, fresh load of snapshot,
fresh collections for all checks (published, source dependency), all checks
inside lock.
4. apiSnapshotsMerge - Source snapshots loaded before task lock
Problem: snapshotCollection created before task lock. Source snapshots
loaded outside task, LoadComplete called on stale copies. Concurrent
merges both load pre-task state, merge result doesn't include source changes.
Fix: Create fresh taskCollectionFactory inside task, fresh load of all
sources after lock acquired, LoadComplete on fresh copies, merge using
fresh RefLists, save using fresh factory.
5. apiSnapshotsPull - Snapshots loaded before task lock
Problem: toSnapshot and sourceSnapshot loaded outside task,
collectionFactory created before task. LoadComplete called on stale copies.
Concurrent pulls load pre-task state, pull doesn't include source changes.
Fix: Create fresh taskCollectionFactory inside task, fresh load of both
snapshots after lock acquired, LoadComplete on fresh copies, all filtering
and pulling on fresh RefLists, save using fresh factory.
Root cause analysis:
The fundamental issue is the split between pre-task work and task-protected
work. Collections and objects were being loaded before lock acquisition, then
stale copies used inside the lock.
Correct pattern (from fixed publish.go and repos.go):
1. HTTP Handler (before task lock):
- Shallow load for 404 check only
- Extract resource keys
- Submit task with resources
2. Task Closure (after lock acquired):
- Create fresh collectionFactory
- Fresh load of all objects
- LoadComplete on fresh copies
- All mutations on fresh state
- All checks atomic inside lock
- Save using fresh collections
This ensures:
- Concurrent operations are serialized by task queue
- No stale DB state used for mutations
- No lost updates from concurrent modifications
- No TOCTOU races on duplicate checks
- No DB handle issues from pre-task factory capture
The pre-task validation in apiSnapshotsUpdate was incorrectly rejecting
PUT requests that set the Name to the snapshot's current name. This caused
a 409 response before creating a task, which broke the system test
SnapshotsAPITestCreateUpdate that expects a task to be created and then
fail inside the task.
The fix restores the 'b.Name != name' condition in the pre-task check so
that same-name updates pass through to the task, where the in-task
duplicate check will properly fail them (returning a failed task state
instead of a direct 409).
The SnapshotsAPITestCreateUpdate test expects that PUT /api/snapshots/:name
with the same Name in the body returns a conflict error. The previous fix
added 'b.Name != name' guards to skip the duplicate check when the name
hasn't changed, but this broke the test which expects the old behavior:
any existing name (including the snapshot's own current name) should be
rejected as a duplicate.
Remove the 'b.Name != name' condition from both the pre-task validation
and the in-task duplicate check so the behavior matches the original.
The gin context (c) may be recycled after the HTTP handler returns 202
for async tasks. Accessing c.Params.ByName() inside the task closure
returns an empty string, causing 'mirror with name not found' errors.
Capture the URL :name parameter into a local variable before the
closure so it is safely captured by value.
Affected endpoints:
- PUT /api/mirrors/:name (apiMirrorsUpdate)
- POST/DELETE /api/repos/:name/packages (apiReposPackagesAddDelete)
Affected endpoints: apiMirrorsDrop, apiMirrorsUpdate.
Both endpoints shared the same architectural flaw as the previously fixed
publish, repos, and snapshot endpoints: operations were performed outside
the task lock, with stale DB state used inside the lock.
Issues Fixed:
1. apiMirrorsDrop - Collections created before task lock
Problem: mirrorCollection and snapshotCollection created before task lock.
Snapshot dependency check done with stale factory. Concurrent drops both
load pre-task state, both see same snapshot dependencies. If snapshots
created after pre-task check, can delete mirror used by snapshots.
Fix: Create fresh taskCollectionFactory inside task, fresh load of mirror
after lock acquired, fresh snapshot check with current factory, drop using
fresh collections.
2. apiMirrorsUpdate - Mirror loaded before task lock
Problem: remote loaded outside task, rename duplicate check with stale
factory. Concurrent updates both load pre-task state, long-running update
uses stale mirror reference. TOCTOU race: rename check passes, another
creates mirror with same name, update saves with stale data.
Fix: Create fresh taskCollectionFactory inside task, fresh load of mirror
after lock acquired, pre-task rename validation, fresh rename check inside
lock, use fresh mirror and collections for all operations.
Root cause analysis:
The fundamental issue is the split between pre-task work and task-protected
work. Collections and objects were being loaded before lock acquisition, then
stale copies used inside the lock.
Correct pattern (from fixed publish.go, repos.go, and snapshot.go):
1. HTTP Handler (before task lock):
- Shallow load for 404 check only
- Extract resource keys
- Submit task with resources
2. Task Closure (after lock acquired):
- Create fresh collectionFactory
- Fresh load of all objects
- LoadComplete on fresh copies
- All mutations on fresh state
- All checks atomic inside lock
- Save using fresh collections
This ensures:
- Concurrent operations are serialized by task queue
- No stale DB state used for mutations
- No lost updates from concurrent modifications
- No TOCTOU races on duplicate checks
- No loss of mirrors used by snapshots
- No stale data in long-running updates
Affected endpoints: apiSnapshotsCreate, apiSnapshotsUpdate, apiSnapshotsDrop,
apiSnapshotsMerge, apiSnapshotsPull.
All five endpoints shared the same architectural flaw as the previously fixed
repos and publish endpoints: operations were performed outside the task lock,
with stale DB state used inside the lock.
Issues Fixed:
1. apiSnapshotsCreate - Source snapshots loaded before task lock
Problem: snapshotCollection and collectionFactory created before task lock.
Source snapshots and destination check done with stale factory.
Concurrent creates both load pre-task state, second overwrites first.
Fix: Create fresh taskCollectionFactory inside task, fresh loads of all
sources after lock acquired, pre-task duplicate check for destination,
use fresh sources and collections for snapshot creation.
2. apiSnapshotsUpdate - Snapshot loaded before task lock
Problem: snapshot loaded outside task, duplicate check with stale factory.
Concurrent renames both load pre-task state, both pass check, second
overwrites first.
Fix: Create fresh taskCollectionFactory inside task, fresh load of snapshot
after lock acquired, fresh duplicate check inside lock, pre-task validation
of new name, atomic rename with fresh copy.
3. apiSnapshotsDrop - Collections created before task lock
Problem: snapshotCollection and publishedCollection created before task lock.
Concurrent snapshot/published modifications not detected. Can delete snapshot
that becomes published between pre-task and task.
Fix: Create fresh taskCollectionFactory inside task, fresh load of snapshot,
fresh collections for all checks (published, source dependency), all checks
inside lock.
4. apiSnapshotsMerge - Source snapshots loaded before task lock
Problem: snapshotCollection created before task lock. Source snapshots
loaded outside task, LoadComplete called on stale copies. Concurrent
merges both load pre-task state, merge result doesn't include source changes.
Fix: Create fresh taskCollectionFactory inside task, fresh load of all
sources after lock acquired, LoadComplete on fresh copies, merge using
fresh RefLists, save using fresh factory.
5. apiSnapshotsPull - Snapshots loaded before task lock
Problem: toSnapshot and sourceSnapshot loaded outside task,
collectionFactory created before task. LoadComplete called on stale copies.
Concurrent pulls load pre-task state, pull doesn't include source changes.
Fix: Create fresh taskCollectionFactory inside task, fresh load of both
snapshots after lock acquired, LoadComplete on fresh copies, all filtering
and pulling on fresh RefLists, save using fresh factory.
Root cause analysis:
The fundamental issue is the split between pre-task work and task-protected
work. Collections and objects were being loaded before lock acquisition, then
stale copies used inside the lock.
Correct pattern (from fixed publish.go and repos.go):
1. HTTP Handler (before task lock):
- Shallow load for 404 check only
- Extract resource keys
- Submit task with resources
2. Task Closure (after lock acquired):
- Create fresh collectionFactory
- Fresh load of all objects
- LoadComplete on fresh copies
- All mutations on fresh state
- All checks atomic inside lock
- Save using fresh collections
This ensures:
- Concurrent operations are serialized by task queue
- No stale DB state used for mutations
- No lost updates from concurrent modifications
- No TOCTOU races on duplicate checks
- No DB handle issues from pre-task factory capture
Affected endpoints: apiReposDrop, apiReposPackagesAddDelete,
apiReposPackageFromDir, apiReposCopyPackage, apiReposIncludePackageFromDir,
apiReposEdit, apiReposCreate.
All seven endpoints shared the same architectural flaw as the previously
fixed publish endpoints: operations were performed outside the task lock,
with stale DB state used inside the lock.
Issues Fixed:
1. apiReposDrop - Collections created before task lock
Problem: snapshotCollection, publishedCollection captured from pre-task
factory. Concurrent snapshot/published modifications not detected.
Fix: Create fresh taskCollectionFactory inside task, re-read repo after
lock acquired, use fresh collections for checks.
2. apiReposPackagesAddDelete - Repo and factory stale before lock
Problem: repo loaded outside task, collectionFactory created before lock.
Concurrent add/delete operations both load same pre-task state, last
write wins, packages lost.
Fix: Create fresh taskCollectionFactory inside task, re-read repo after
lock acquired, use fresh factory for all operations.
3. apiReposPackageFromDir - Repo and factory stale before lock
Problem: repo loaded outside task, collectionFactory created before lock.
Concurrent file imports both load same pre-task state, last write wins.
Fix: Create fresh taskCollectionFactory inside task, re-read repo after
lock acquired, use fresh factory for imports.
4. apiReposCopyPackage - Both repos and factory stale before lock
Problem: dstRepo and srcRepo loaded outside task, collectionFactory
created before lock. Concurrent copy operations race on stale state.
Fix: Create fresh taskCollectionFactory inside task, re-read both repos
after lock acquired, use fresh factory for all operations.
5. apiReposIncludePackageFromDir - Repo and factory stale before lock
Problem: repo loaded outside task, collectionFactory created before lock.
Concurrent .changes file processing races on stale state.
Fix: Create fresh taskCollectionFactory inside task, use fresh factory
for import operations.
6. apiReposEdit - No serialization, concurrent modification race
Problem: Direct update without task locking. Two concurrent renames can
both pass duplicate check, second overwrites first.
Fix: Convert to async task. Duplicate check and update now atomic inside
lock, after fresh load from DB.
7. apiReposCreate - No serialization, TOCTOU on duplicate check
Problem: Duplicate check outside task lock, add outside lock. Two
concurrent creates with same name both pass check, second overwrites first.
Fix: Convert to async task. Duplicate check and add now atomic inside
lock, after fresh load from DB.
Root cause analysis:
The fundamental issue is the split between pre-task work and task-protected
work. Collections and objects were being loaded before lock acquisition, then
stale copies used inside the lock.
Correct pattern (now applied consistently across all 7 endpoints):
1. HTTP Handler (before task lock):
- Shallow load for 404 check only
- Extract resource keys
- Submit task with resources
2. Task Closure (after lock acquired):
- Create fresh collectionFactory
- Fresh load of all objects
- LoadComplete on fresh copies
- All mutations on fresh state
- All checks atomic inside lock
- Save using fresh collections
This ensures:
- Concurrent operations are serialized by task queue
- No stale DB state used for mutations
- No lost updates from concurrent modifications
- No TOCTOU races on duplicate checks
- No DB handle issues from pre-task factory capture
Affected endpoint: apiPublishUpdateSwitch (PUT /api/publish/{prefix}/{distribution}).
The handler registered only the published repo key as a task resource.
The underlying source repos (for local) or snapshots (for snapshot-based
published repos) were not locked. Concurrent updates to a source repo
or snapshot while a publish-update/switch task was running could produce
inconsistent published indexes:
Task A: apiPublishUpdateSwitch loads published, reads source repo/snapshot
Request B: modifies same source repo or snapshot (add/remove packages, etc)
Task A: Update() + Publish() reads stale/modified source -> inconsistent
published index, or partial write if source deleted mid-task.
Fix: for SourceLocalRepo, iterate published.Sources (component -> source
UUID), look up each local repo via localRepoCollection.ByUUID and append
string(repo.Key()) to resources. For SourceSnapshot, iterate b.Snapshots,
look up each snapshot via snapshotCollection.ByName and append
string(snapshot.ResourceKey()) to resources. Task queue now serialises
against both the published repo and all its sources.
When b.Distribution is empty, the pre-registered resource key
U<storage>:<prefix>>><distribution> cannot be constructed, so
concurrent POST requests to the same prefix are not serialized by the
task queue. Add a log warning so operators are aware of the gap.
Affected endpoints: apiPublishRepoOrSnapshot (POST /api/publish/{prefix}),
apiPublishDrop (DELETE /api/publish/{prefix}/{distribution}).
Both handlers used the outer-scoped collectionFactory and collection
variables inside the task closure. These were captured before the task
lock was acquired, so under concurrent load each task operated on a
stale DB view:
apiPublishRepoOrSnapshot: snapshot/localRepo LoadComplete,
NewPublishedRepo, CheckDuplicate, Publish, and collection.Add all
used the pre-lock collectionFactory/collection. Two concurrent
POST to same prefix could both pass CheckDuplicate (neither sees
the other in the stale DB view) and race on disk writes.
apiPublishDrop: collection.Remove used pre-lock collection,
potentially racing with concurrent updates/other drops.
Fix: inside the task closure create a fresh taskCollectionFactory and
taskCollection. All DB reads (LoadComplete) and writes
(CheckDuplicate, Add, Remove, Publish) now run against the authoritative
DB state after the lock is held.
Affected endpoints: apiPublishUpdateSwitch (PUT), apiPublishUpdate (POST).
Both handlers loaded the published repo and mutated scalar fields
(Label, Origin, SkipContents, SkipBz2, AcquireByHash, SignedBy,
MultiDist, Version) outside the task closure, before the lock was
acquired. Inside the task, LoadComplete only refreshed sourceItems —
it did not reload scalar fields or the Revision. Two concurrent
requests therefore each operated on a stale base:
Request A loads published (Label="old"), sets Label="A"
Request B loads published (Label="old"), sets Label="B"
Task A runs: Update() + Publish() + collection.Update() -> saves Label="A"
Task B runs: Update() on B's stale copy -> saves Label="B",
silently discarding A's Label change and potentially
reconciling a Revision built against the pre-A state.
Fix: remove all field mutations and the LoadComplete call from the HTTP
handler. Inside the task, a fresh taskCollectionFactory is created, the
published repo is re-read via ByStoragePrefixDistribution + LoadComplete
(obtaining the current DB state after the lock is held), and then all
field mutations are applied before Update / Publish / collection.Update.
Affected endpoints: apiPublishAddSource, apiPublishSetSources,
apiPublishUpdateSource, apiPublishRemoveSource, apiPublishDropChanges.
All five handlers shared the same flawed pattern: they loaded the
published repo from the DB and mutated it (ObtainRevision / DropRevision)
outside the task closure, before the task lock was acquired. Each task
closure then just wrote back the already-mutated, pre-lock object.
Because the task queue serialises tasks that share a resource key, two
concurrent requests appear safe — but each task closure holds a stale
copy of the object captured before the lock was taken:
Request A loads published: revision = {}
Request B loads published: revision = {} <- same DB state
A mutates: revision = {main: snap1}
B mutates: revision = {contrib: snap2}
Task A runs: saves {main: snap1} OK
Task B runs: saves {contrib: snap2} <- clobbers A's change
Fix: perform only a shallow ByStoragePrefixDistribution outside the task
(for the early 404 response, resource key, and task name). Inside the
task closure a dedicated taskCollectionFactory is created, the published
repo is re-read fresh from the DB (after the lock is acquired), and
LoadComplete + all mutations + Update are executed against that
authoritative copy.
apiPublishRepoOrSnapshot appended published.Key() to resources inside
the task closure, after maybeRunTaskInBackground had already been called.
The task's locked-resource set is fixed at submission time, so that append
had no effect — the published repo key was never registered as a resource.
Two concurrent POST /api/publish/{prefix} requests for the same
prefix/distribution therefore did not conflict in the task queue: both
ran in parallel, each loaded an empty PublishedRepoCollection from the DB,
both passed CheckDuplicate, and the second Add silently overwrote the first.
Fix: compute the published repo key ("U{storagePrefix}>>{distribution}")
from the already-known storage/prefix/distribution values and append it to
resources before calling maybeRunTaskInBackground, so concurrent creates
for the same destination are serialised by the task queue. The now-dead
append inside the closure is removed.
add API response wrappers with NumPackages derived from RefList length; keep show endpoint payloads unchanged for backward compatibility; add API tests for list endpoint NumPackages; update swagger response schemas for list endpoints
Both the external GPG signer (--faked-system-time) and internal Go
OpenPGP signer (signerConfig.Time) now honor SOURCE_DATE_EPOCH,
producing reproducible signatures alongside the plain Release file dates.
Adds system tests for both signer backends verifying byte-identical
Release, Release.gpg and InRelease across repeated publishes.
The signer tests (PublishRepo3[78]Test) are using an ed25519 key because
ed25519 signatures are deterministic by design. The Go openpgp library
uses a random nonce for DSA/ECDSA (see signature.go Sign calls using
config.Random() link below) so those signatures vary across runs
even with a fixed timestamp, making byte-identical verification impossible.
In addition to 49f342878a
Ref: https://github.com/aptly-dev/aptly/pull/1537
Ref: https://github.com/ProtonMail/go-crypto/blob/v1.4.0/openpgp/packet/signature.go#L945-L979
Implement support for the SOURCE_DATE_EPOCH environment variable as
specified by reproducible-builds.org. When set, this variable overrides
the current timestamp in the Release file's Date and Valid-Until fields,
enabling reproducible filesystem publishes.
- Read SOURCE_DATE_EPOCH environment variable in Publish()
- Use the epoch timestamp for both Date and Valid-Until fields
- Gracefully fallback to current time if unset or invalid
- Add comprehensive tests for valid and invalid SOURCE_DATE_EPOCH values
https://wiki.debian.org/DebianRepository/Format#Signed-By says:
> **Signed-By**
> An optional field containing a comma separated list of
> OpenPGP key fingerprints to be used for validating
> the next Release file. The fingerprints must consist
> only of hex digits and may not contain spaces.
> The fingerprint specifies either the key the Release file
> must be signed with or the key the signature key must be
> a subkey of. The later match can be disabled by appending
> an exclamation mark to the fingerprint.
>
> If the field is present, a client should only accept future updates
> to the repository that are signed with keys listed in the field.
> The field should be ignored if the Valid-Until field is not present
> or if it is expired.
For both the CLI tools and JSON, the field is taken as a string verbatim.
When specified, we must also provide `Valid-Until` field,
and i'm not sure there is an 'infinity' value for it,
so 100 years will have to do?
Fixes https://github.com/aptly-dev/aptly/issues/1497
Initially found by automated repository health checks used by Termux
in https://github.com/termux/termux-packages/issues/27472
The root problem was 4.3.5a comparing less than 4.3.5-rc1-1 by aptly
According to debian "4.3.5a" > "4.3.5-rc1-1"
This is because dpkg splits hyphen for revision at the first hyphen,
whereas aptly was splitting at the last hyphen which is different from
dpkg's behaviour.
dpkg behaviour: https://git.dpkg.org/cgit/dpkg/dpkg.git/tree/lib/dpkg/parsehelp.c#n242
Perhaps this wasn't detected as there was broken tests in the repository
since the initial commit of aptly. This also fixes those tests
Enabling coverage near-doubles the incremental build time and adds
overhead to individual tests on the order of **5-10x** or more. It's not
essential to have this for quick local system-test runs, so add an option
to disable it.
When using rootless podman, the *current user* gets mapped to uid 0,
which results in the aptly user being unable to write to the build
directory. We can instead map the current user to the corresponding uid
in the container via `PODMAN_USERNS=keep-id`, which matches up with what
docker-wrapper wants...but then that will *enter the container as the
current uid*, which messes with the ability to set permissions on
`/var/lib/aptly`. That can be fixed by explicitly passing `--user 0:0`,
which should be a no-op on docker (since the container's default user is
already root).
Additionally, this adds `--security-opt label=disable` to avoid
permission errors when running on systems with SELinux enforcing.
This fixes the race condition that happens when you call publish
concurrently. It adds a valuable test that reproduces the error almost
deterministically, it's hard to say always but I have run this in loop
100 times and it reproduces the error consistently without the patch and
after the patch it works consistently.
When aptly crashes it is possible to get a corrupt database with a dangling key reference.
This results in an error with 'key not found', eg:
ERROR: unable to load package Pall example-package 1.2.3 778cf6f877bf6e2d: key not found
This change makes `db recover` fix this situation by removing the dangling references.
sort both aptly output and gold file. output original output for
debugging on failure.
* Makefile: enable CAPTURE=1 env variable for capturing gold files
* docker-system-test: use AWS env vars for S3 tests
* fix system tests timing issue with order of gpg logs in publish tests
The fix of the -with-filter flag causes the following previously
missing source files to be downloaded, so I updated the test file.
```
rkward_0.7.5-1~bullseyecran.0.debian.tar.xz
rkward_0.7.5-1~bullseyecran.0.dsc
rkward_0.7.5.orig.tar.gz
rpy2_3.5.12-1~bullseyecran.0.debian.tar.xz
rpy2_3.5.12-1~bullseyecran.0.dsc
rpy2_3.5.12.orig.tar.gz
```
This commit allows to add, remove and update components of published repositories without the need to recreate them.
Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
Do all relevant database reading/modifying inside `maybeRunTaskInBackground`.
Notably, `LoadComplete` will load the reflist of a repo. if this is done outside of a background operation,
the data might be outdated when the background tasks runs.
This commit modifies the behavior of the publish switch method in the way, that also new components can be added to an already published repository. It is no longer necessary to drop and recreate the whole publish.
Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
this should not build release if pipeline triggered on master and master also has a version tag. avoid building same version twice and uploading to ci and release repos.
- use debian version consistently
- if the commit is not on a git tag, append ci version
- avoid double zipping
- cleanup binary builds
- replace `make release` with `make binaries`
- add missing files to archives
- use matrix build for deb packages
- allow building release version
- keep directory inside zip archives
- accept releases only on master
only tags on master will trigger a release
- cleanup namings
- keep path in zip
- add retention period for pipeline artifacts
use the new -multi-dist option to combine all distributions into one
publish point:
deb http://repo.aptly.info/ci bookworm main
or:
deb http://repo.aptly.info/release bookworm main
for the following distributions: buster, bullseye, bookworm, focal, jammy
gpgv: can't allocate lock for '/home/runner/.gnupg/aptlytest.gpg'
this forced running local system tessts in /home/runner, as it is
in the gitgub actions.
- fix make version on debian
- update gitignore
- add aptly-api and service
- install zsh completion
- add debug package
- move aptly data from orig deb package
- do not add shell for service user
- use 8080 as default port
lint: s3/public.go#L136
SA1019: config.WithEndpointResolverWithOptions is deprecated: The global endpoint resolution interface is deprecated. See deprecation docs on [WithEndpointResolver]. (staticcheck)
lint: s3/public.go#L137
SA1019: aws.Endpoint is deprecated: This structure was used with the global [EndpointResolver] interface, which has been deprecated in favor of service-specific endpoint resolution. See the deprecation docs on that interface for more information. (staticcheck)
lint: s3/public.go#L138
SA1019: aws.Endpoint is deprecated: This structure was used with the global [EndpointResolver] interface, which has been deprecated in favor of service-specific endpoint resolution. See the deprecation docs on that interface for more information. (staticcheck)
- add t13_etcd test directory
- etcd will be started for the unit tests and each system test
- etcd will load fixture DB export if requested by the test
- existing tests are reused for etcd testing
goxc fails with:
Error: database/etcddb/database.go:17:25: cannot use 2048 * 1024 * 1024 (untyped int constant 2147483648) as int value in struct literal (overflows)
Explicitly state that Python3 is supported and required.
Since aptly `v1.5.1` (with commit 035d5314b0)
the tests are ported to Python3.
With a687df2f4f the system
tests are run with `python3` per default.
With f4a152ab22 (after `v1.5.0` aptly tag)
the tests run against Python 3.11.
- log import errors for test modules
- log output only on test failure
- improve docker system test container
- use go 1.19 in docker system tests
- download go dependencies in docker container
- system tests: color failues output
- imrpove test result output
- do not install golangci-lint in system tests
If the S3 bucket used to house a repo has KMS encryption enabled then
the etag of an object may not match the MD5 of the file. This may
cause an incorrect error to be reported stating the file already
exists and is different.
A mechanism exists to work around this issue by using the MD5 stored
in object metadata. This check doesn't always cover the case where KMS
is enabled as the fallback is only used if the etag is not 32
characters long.
This commit changes the fallback mechanism so that it is used in any
case where the object's etag does not match the source MD5. This will
incur a performance penalty of an extra head request for each object
with a mismatch.
None of the commands' output is ever treated as binary, so we can just
always decode it as text.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
This adds support for storing packages directly on Azure, with no truly
"local" (on-disk) repo used. The existing Azure PublishedStorage
implementation was refactored to move the shared code to a separate
context struct, which can then be re-used by the new PackagePool. In
addition, the files package's mockChecksumStorage was made public so
that it could be used in the Azure PackagePool tests as well.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Several sections of the code *required* a LocalPackagePool, but they
could still perform their operations with a standard PackagePool.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
The contents of `os.Stat` are rather fitted towards local package pools,
but the method is in the generic PackagePool interface. This moves it to
LocalPackagePool, and the use case of simply finding a file's size is
delegated to a new, more generic PackagePool.Size() method.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Before, a "partial" URL (either "localhost:port" or an endpoint URL
*without* the account name as the subdomain) would be specified, and the
full one would automatically be inferred. Although this is somewhat
nice, it means that the endpoint string doesn't match the official Azure
syntax:
https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string
This also raises issues for the creation of functional tests for Azure,
as the code to determine the endpoint string needs to be duplicated
there as well.
Instead, it's just easiest to follow Azure's own standard, and then
sidestep the need for any custom logic in the functional tests.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
read_path() can read in binary, which the S3 tests don't support (simply
because they don't need it)...but it needs to be able to take the `mode`
argument anyway.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
This commit blocks concurrent calls to RunTaskInBackground which is
intended to fix the quirky behaviour where concurrent PUT calls to
api/publish/<prefix>/<distribution> would immedietly reuturn an error.
The solution proposed in this commit is not elegant and probaly has
unintended side-effects. The intention of this commit is to highlight
the area that actually needs to be addressed.
Ideally this patch is amended or dropped entierly in favor of a better
fixup.
This commit introduces a test which runs concurrent publishes (which
could be parallell with multiproccessing, python is fun).
The test purposly fails (at the point in history that this patch is
written) in order to make it as easy as possible to verify later patches,
which hopefully addresses concurrency problems.
The same behaviour can easily be tested outside of the system tests with
the following (or similar) shell
$ aptly serve -listen=:8080 -no-lock
$ aptly repo create create -distributions=testing local-repo
$ atply publish repo -architectures=amd64 local-repo
$ apt download aptly
$ aptly repo add local-repo ./aptly*.deb
$ for _ in $(seq 10); do curl -X PUT 127.0.0.1:8080/api/publish//testing
In the local testing perfomed (on a dual core vm) the first 1-4 jobs
would typically succeed and the rest would error out.
this change imports downloaded packages into the pool immediately after download.
in case mirroring is aborted and later resumed, already downloaded packages will not be downloaded anymore.
This change makes it possible to publish multiple distributions
with packages named the same but with different content by changing
structure of the generated pool hierarchy. The option not enabled
by default as this changes the structure of the output which could
break the expectations of other tools.
on hosts which have wildcard dns domains in their local domain search
list, builds failed because "nosuch.host" could actually be resolved.
Since ".host" isn't a recommended TLD by RFC2606, we use ".invalid" now.
And since this is not enough to fix the problem, we use now absoulte
domain names (having a '.' at the end)
The previous reflist logic would early-exit the loop body if one of the
lists was empty, but that skips the compacting logic entirely.
Instead of doing the early-exit, we can leave a list's ref as nil when
the list end is reached and then flip the comparison result, which will
essentially treat it as being greater than all others. This should
preserve the general behavior without omitting the compaction.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
The output doesn't actually depend on the reflists, and loading them for
every published repo starts to take substantial time and memory.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Reflists are basically stored as arrays of strings, which are quite
space-efficient in MessagePack. Thus, using zero-copy decoding results
in nice performance and memory savings, because the overhead of separate
allocations ends up far exceeding the overhead of the original slice.
With the included benchmark run for 20s with -benchmem, the runtime,
memory usage, and allocations go from ~740us/op, ~192KiB/op, and 4100
allocs/op to ~240us/op, ~97KiB/op, and 13 allocs/op, respectively.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
The cleanup phase needs to list out all the files in each component in
order to determine what's still in use. When there's a large number of
sources (e.g. from having many snapshots), the time spent just loading
the package information becomes substantial. However, in many cases,
most of the packages being loaded are actually shared across the
sources; if you're taking frequent snapshots, for instance, most of the
packages in each snapshot will be the same as other snapshots. In these
cases, re-reading the packages repeatedly is just a waste of time.
To improve this, we maintain a list of refs that we know were processed
for each component. When listing the refs from a source, only the ones
that have not yet been processed will be examined. Some tests were also
added specifically to check listing the files in a component.
With this change, listing the files in components on a copy of our
production database went from >10 minutes to ~10 seconds, and the newly
added benchmark went from ~300ms to ~43ms.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
When merging reflists with ignoreConflicting set to true and
overrideMatching set to false, the individual ref components are never
examined, but the refs are still split anyway. Avoiding the split when
we never use the components brings a massive speedup: on my system, the
included benchmark goes from ~1500 us/it to ~180 us/it.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
In some local tests w/ a slowed down filesystem, this massively cut down
on the time to clean up a repository by ~3x, bringing a total 'publish
update' time from ~16s to ~13s.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
- use s3 mirror instead of internet download
- reduce download verbosity
- do not use venv in docker-system-tests
- be more verbose on test output
- do not run golangci-lint in system-tests
Ubuntu has started depreciating the Debian installer in focal
and moved the installer images to a different path. In versions
after focal, they are completly removed. This basically gives
us more time to figure out how to use the new system.
When a publishing uses a publish prefix, instead of listing the contents
of the whole bucket under the storage prefix, only list the contents of
the bucket under the storage prefix and publish prefix, and cache it by
publish prefix.
This speeds up publish operations under a prefix.
instead of caching the whole s3 bucket, cache only the pool path. this
requires an additional parameter, and since this is an interface, all
implementations need to follow. might help in other backends too.
closes#1181
This will properly close the db and, more particularly, flush out any
profile files being written. Otherwise, they can end up empty or
truncated.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
1.20 changes the output format of coverage checks slightly to include
a package name on each line, followed by `coverage:`, but the current
regex assumes that the line *starts* with `coverage:`.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
The current regex runs in exponential time, which massively impacts the
runtime of the test suite, taking several seconds (~4s on my system)
just to perform a single match. By replacing the mix of re.findall + the
initial capture group with re.search + some string slicing, the time
spent matching the regex becomes nearly instant, e.g.:
$ make system-test TESTS='Config*'
goes from taking ~10s to ~1.5s.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
When trying to delete a mirror that has snapshot and not providing the
force option, the API should not return a `500
StatusInternalServerError`.
A `403 StatusForbidden` is more appropriate when the condition is
expected by the server.
A race condition for publishing packages and
mirrors at the same time was introduced in
commit 77d7c38.
The problem is that when opening a leveldb transaction
and performing another 'put' to the db
the system freezes.
Older versions go-xz didn't wait for child processes meaning for exery
unpack action a defunct xz would stick around. This got fixed in 0.1.0
Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>
This should fix some tests, as a lot of them are dependent on
deb.debian.org which no longer supports Debian 9 "Stretch".
Instead we use archive.debian.org which will continue to contain
"Stretch" packages for a long time.
If aptly.EnableDebug is active, we use Debug, otherwise we use
gin.ReleaseMode to remove the annoying nuding messages when running the
api.
fixes: #1103
While testing out Aptly, the `apt-get` client complains with the following error, since the `codename` was switched from the InRelease files that are baked out by Aptly:
```
E: Repository 'http://debianrepo.example.com/bionic testing InRelease' changed its 'Codename' value from '' to 'testing'
```
The ".../packages" endpoints for mirror, local repos and snapshots all
share the same syntax for querying. However the "/api/packages" endpoint
doesn't match this. Adjust that to allow for a bit more consistency and
allow querying the full package database.
The current endpoint functionality "/packages/:name" is kept intact and
can be used the same as now
Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>
To capture the coverage also for the integration tests,
a test only executing the cmd.Run function is used.
The test always exits with code 0 and prints the
real exit code to stdout. Otherwise no coverage
report is generated.
Those changes enable a more accurate coverage report
for future contributions.
This change lets you disable ACL when using S3 by using a configuration
value of `none`. This way we maintain backward compatibility with the
default setting being `private`.
Fixes: #1067
Seems go.mod had some modules that are no longer used since the last
version bumps? Running `make modules` or really `go mod tidy`
automagically cleans those up.
Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>
Most modern distribution use python3 for python (3). Default to that to
make it a bit simpler to run systems tests on Debian
Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>
Aptly allows create e.g. repos with a / to use those with the REST api
the router needs to allow urlencoded parameters in various places to
represent this. A specific example of this is the /api/repos/:name/packages path
Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>
dpkg-gencontrol can be called with -v flag which set binary package's
version separated from source version. When this happen, the Source
field will contain version number in addition to source package name.
This tripped Aptly's dbgsym restriction, which check for exact source
package name, which in turn prevents the dbgsym & the whole .changes
file from being imported.
From the git history, it seems like this condition is a leftover from
when Aptly filter dbgsym packages using "*-dbgsym". So, I decided to
remove it. A test case has been added to prevent regression.
It may happen that aptly retries to download data during tests (maybe because
of a network issue), but our fixtures doesn't account for it. So, we strip
those irrelevant lines before comparison.
PublishRepo26Test fails to run because something in the CI environment forces
gpg to ask for the user's password. Try to require gpg1 for the test, which
seems to run fine in other environments.
This adds a new configuration setting: AzurePublishEndpoints, similar
to the existing S3PublishEndpoints and SwiftPublishEndpoints.
For each endpoint, the following has to be defined:
- accountName
- accountKey
- container
- prefix
Azure tests require the following environment variables to be set:
- AZURE_STORAGE_ACCOUNT
- AZURE_STORAGE_ACCESS_KEY
With either of these not set, Azure-specific tests are skipped.
The S3 backend relies on ETag S3 returns being equal to the MD5 of the
object, but it’s not necessarily true. When the value returned clearly
doesn’t look like a valid MD5 hash (length isn’t exactly 32 characters),
attempt to retrieve the MD5 hash possibly stored in the metadata.
We cannot always do this since user-defined metadata isn’t returned by
the ListObjects call, so verifying it for each object is expensive as it
requires one HEAD request per each object.
This commit fixes#923.
Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>
The S3 backend relies on ETag S3 returns being equal to the MD5 of the
object, but it’s not necessarily true. For that purpose we store the MD5
object in a separate metadata field as well to make sure it isn’t lost.
From https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html:
> The entity tag is a hash of the object. The ETag reflects changes only
> to the contents of an object, not its metadata. The ETag may or may not
> be an MD5 digest of the object data. Whether or not it depends on how
> the object was created and how it is encrypted as described below:
>
> Objects created by the PUT Object, POST Object, or Copy operation,
> or through the AWS Management Console, and are encrypted by SSE-S3 or
> plaintext, have ETags that are an MD5 digest of their object data.
>
> Objects created by the PUT Object, POST Object, or Copy operation,
> or through the AWS Management Console, and are encrypted by SSE-C or
> SSE-KMS, have ETags that are not an MD5 digest of their object data.
>
> If an object is created by either the Multipart Upload or Part Copy
> operation, the ETag is not an MD5 digest, regardless of the method
> of encryption.
Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>
Specifically, I have MacGPG installed instead of upstream GPG, which
results in the version string reading
gpg (GnuPG/MacGPG2) 2.2.17
instead of the expected
gpg (GnuPG) 2.2.17
This commit closes: #145
The dependency format "pkg:arch" (e.g. "python3:any") was not well
parsed if not any version is given. This commit splits the dependency
name and architecture in all cases.
For any action which is multi-step (requires updating more than 1 DB
key), use transaction to make update atomic.
Also pack big chunks of updates (importing packages for importing and
mirror updates) into single transaction to improve aptly performance and
get some isolation.
Note that still layers up (Collections) provide some level of isolation,
so this is going to shine with the future PRs to remove collection
locks.
Spin-off of #459
This is spin-off of changes from #459.
Transactions are not being used yet, but batches are updated to work
with the new API.
`database/` package was refactored to split abstract interfaces and
implementation via goleveldb. This should make it easier to implement
new database types.
Use CDN-backed Debian mirror to make tests run faster hopefully for
everyone. Redirects might be important to know what exactly is going on
when items are being downloaded.
Apply retries as global, config-level option `downloadRetries` so that
it can be applied to any aptly command which downloads objects.
Unwrap `errors.Wrap` which is used in downloader.
Unwrap `*url.Error` which should be the actual error returned from the
HTTP client, catch more cases, be more specific around failures.
* aptly can sign and verify without issues with GnuPG 1.x and 2.x
* aptly auto-detects GnuPG version and adapts accordingly
* aptly automatically finds suitable GnuPG version
Majority of the work was to get unit-tests which can work with GnuPG 1.x & 2.x.
Locally I've verified that aptly supports GnuPG 1.4.x & 2.2.x. Travis CI
environment is based on trusty, so it runs gpg2 tests with GnuPG 2.0.x.
Configuration parameter gpgProvider now supports three values for GnuPG:
* gpg (same as before, default): use GnuPG 1.x if available (checks gpg, gpg1),
otherwise uses GnuPG 2.x; for aptly users who already have GnuPG 1.x
environment (as it was the only supported version) nothing should change; new
users might start with GnuPG 2.x if that's their installed version
* gpg1 looks for GnuPG 1.x only, fails otherwise
* gpg2 looks for GnuPG 2.x only, fails otherwise
See #765, #761
Collections were relying on keeping in-memory list of all the objects
for any kind of operation which doesn't scale well the number of
objects in the database.
With this rewrite, objects are loaded only on demand which might
be pessimization in some edge cases but should improve performance
and memory footprint signifcantly.
See #761
aptly had a concept of loading small amount of info per each object
into memory once collection is accessed for the first time.
This might have simplified some operations, but it doesn't scale well
with huge aptly databases.
This is just intermediate step towards better memory management -
list of objects is not loaded unless some method is called.
`ForEach` method (mainly used in cleanup) is reimplemented to
iterate over database without ever loading all the objects into memory.
Memory was even worse with previous approach, as for each item usually
`LoadComplete()` is called, which pulls even more data into memory
and item stays in memory till the end of the iteration as it is referenced
from `collection.list`.
For the subsequent PR: reimplement `ByUUID()` and probably other methods
to avoid loading all the items into memory, at least for all the collecitons
except for published repos. When published repository is being loaded, it
might pull source local repo which in turn would trigger loading for all the
local repos which is not acceptable.
This is not a complete fix, but the easiest first step.
During `db cleanup`, aptly is loading every repo/mirror/... into memory,
and even though each object is processed only once, collection holds
a reference to all the loaded objects, so they won't be GC'd until
process exits.
CollectionFactory.Flush() releases pointers to collection objects,
making objects egligble for GC.
This is not a complete fix, as during iteration we could have tried
to release a link to every object being GCed and that would have
helped much more.
These unit-tests cover operations via both PGP providers:
built-in `openpgp` and external `gpg`.
Next step is to run these tests for gpg1 & gpg2
as separate entities.
This test has been failing very often because of changes in nvidia
repository. As this test is not related to filtering
remove number of filtered packages from output for a more robust test.
This updates previous work in #739 to build
Debian packages and zip files for other OS.
All the build artifacts are uploaded to S3
public bucket `aptly-nightly` so that there's
archive for all the builds.
All `.deb` packages are automatically uploaded
to repo.aptly.info/nightly on build.
- new phony target build: same as install but creating aptly-$version and
putting it into a build/ subdir
- env TRAVIS_TAG in the makefile now overrides the TAG lookup, this ensures
that the tag travis is working with is actually the one being used to
construct the version number
- subdir is gitignored
- travis runs new target - lists artifacts - deploys artifacts to github
all of the above only happens on builds that are a tag and DEPLOY_BINARIES
is set to yes (which is only the case for latest stable go version)
package count chainged again -.-
I am working on a fixture set for repositories so we can stop talking to
live repositories. that's quite the undertaking though, so let's fix the
output reference to unbreak the test in the meantime
Init is actually never called and I have no clue why it is there if it is
not called.
Take this opportunity to introduce a New function which only does the
helper lookup and panics iff that fails. Panic may be a bit too aggressive,
but seems the most certain way to get out of not finding a suitable gpg1
binary.
Newer versions of debian and ubuntu come with gpg pointing to gpg2.
We can currently only handle gpg1 CLIs though. Luckily the old gpg is still
available in the package gnupg1 (providing bin/gpg1).
As a bit of a stop-gap, until #657 can be resolved properly, we'll detect
the version of bin/gpg. If it is unsuitable we'll fall back and try
bin/gpg1. If neither is found to be suitable the signer/verifier will
not work.
Same applies to gpgv/gpgv1.
1. Don't run long steps for Go versions other than 1.9 & 1.10
according to Golang Release Policy (two latest versions).
2. Switch to codecov.io, collect coverage only on Go 1.10 which
has fixes for multi-module coverage & ./... ignoring vendor.
3. Simplify Makefile.
updates with contents generation were super syscall-heavy. for each path
in a package (so at least 2-4, but ordinarily >4) we'd do a db.Put in
ContentsIndex which results in one syscall.Write. so, for every package in
a published repo we'd have to do *at least* 2 but ordinarily >4 syscalls.
this gets abysmally slow very quickly depending on the available system
specs.
instead, start a batch inside each package and finish it when we are done
with the package. this should keep the memory footprint negligible, but
reduce the write() calls from N to 1.
on one of KDE's servers I have seen update publishing of 7600 packages go
from ~28s to ~9s when using batch putting on an HDD.
on my local system the same set of packages go from ~14s to ~6s on an SSD.
(all inodes in cache in both cases)
presently there is no use case where we need this. on the other hand,
passing empty paths into any of the remove methods is indicative of a bug.
this is particularly dangerous as this can temporarily smash the publish
root but later restore it again when actually publishing. this makes
for super nasty and hard to track down problems.
to guard against this simply disallow root dir removal using empty
strings. should we find a use case for this in the future we can always
revisit this (FTR: I think very explicitly API should be used so everyone
knows what is going on and you can't accidentally run it)
the logic here was wrong.
if we managed to find the link target (the physical index file) pointed to
by our old symlink we want to remove it (this is basically "cleaning up old
index" logic).
previously we'd try to only delete it when the ReadLink came back with
error. which had two serious issues with it:
a) linkTarget was empty, so we basically called Remove("") which would
delete the storage -> root <- directory if the root is a symlink!
b) we'd leak old indexes as the cleanup logic only ran if there was en
error which would ordinarily never be
new code correctly cleans up unless there was an error.
this relates to a previous bugfix of readLink which incorrectly returned
absolute paths ultimately rendering the Remove call also broken.
previously it'd return an absolute path which makes the path absolutely
useless as all other functions of PublishedStorage need relative input
and will prepend them with the rootPath, so getting an absolute ReadLink
and then trying to remove that'd would ultimately try to remove the
absolute path `$root/AbsoluteRoot/LinkTarget` instead of `$root/LinkTarget`
add a unit test to actually verify readlink
According to https://tools.ietf.org/html/rfc7231#section-4.3.2 HEAD
must not have response body so the AWS error code NoSuchKey
cannot be received from S3 and we need to fallback to HTTP NotFound
error code.
- new publish calls can now enable AcquireByHash by right away (previously
one would have had to create a new publishing endpoint and then
explicitly switch it to AcquireByHash)
- all json marshals of PublishedRepo now contain AcquireByHash (allows
inspecting if a given endpoint has AcquireByHash enabled already; also
enables verification that a switch/update actually applied a
potential AcquireByHash change
- update all tests to reflect that default state of AcquireByHash
- update creation and switch testing to explicitly toggle AcquireByHash to
make sure state mutation works as expected
LoadComplete() modifies object, so it would cause issues if it runs
concurrently with other methods. Uprage mutex locks to write
locks when LoadComplete() is being used.
The added "aptly publish repo" option "-access-by-hash" publishes
the index files (Packages*, Sources*) also as hardlinked hashes.
Example:
/dists/yakkety/main/binary-amd64/by-hash/SHA512/31833ec39acc...
The Release files indicate this with the option "Acquire-By-Hash: yes"
This is used by apt >= 1.2.0 and prevents the "Hash sum mismatch" race
condition between a server side "aptly publish repo" and "apt-get update"
on a client.
See: http://www.chiark.greenend.org.uk/~cjwatson/blog/no-more-hash-sum-mismatch-errors.html
This implementation uses symlinks in the by-hash/*/ directory for keeping
only two versions of the index files and deleting older files
automatically.
Note: this only works with aptly.FileSystemPublishedStorage
Closes: #536
Signed-off-by: André Roth <neolynx@gmail.com>
There are two fixes here:
1. Abort package download immediately as ^C is pressed.
2. Import all the already downloaded files into package pool,
so that next time mirror is updated, aptly won't download them
once again.
DELETE requests, both for temporary files and no longer referenced
packages, lacked the configured path prefix and therefor were not
removed if a prefix is configured.
'E722 do not use bare except' wants us not to use except without type
restriction as it catches everything and the kitchen sink. Since we use
them to catch exceptions in test cases this is intentional as we implement
general purpose error handling on test failure there.
upstream switched the alignment check backend and in doing so fails to run
if the old backend is defined in the config.
also skip alignment linting on a struct we use for byte decoding as we have
no choice in its member order.
Sometimes source packages reference files already present in the pool.
Allow for those file to be omitted when importing packages either via
`repo add` or `repo include`. If file is missing, aptly would make
an attempt to look up file in the package pool (by checksum) and
use it.
Fixes: #278
When `-dep-follow-all-variants` option is enabled, dependency resolving
process shouldn't stop even if dependency is already satisfied - there
mgiht be other ways to satisfy dependency.
Also fix issue with parsing multiarch specs like
`python:any`.
When searching for packages which might satisfy given dependency,
aptly was first returning packages which `Provides` mentioned
name. By default aptly is picking up only first match (unless
follow all variants options is enabled), so `Provides:` takes
precedence over exact package name match.
Invert this logic by searching first for package name match.
Fixes: #636
Before this fix, aptly was always treating strings starting with
uppercase letter as field name, which was breaking package queries
like `VMware-Horizon-Client_4.5.0_all`.
Now aptly accepts only fields which don't contain underscore, and
everything else would be parsed as package reference.
Original PR: #621Fixes: #619
I've added unit-test to Martyn's PR.
Without this fix, if `prefix` is set on S3 publish endpoint,
aptly would incorrectly build path cache and re-upload every object
on publish.
Allow skipping unreferenced files cleanup on publish switch/update/drop
via the -skip-cleanup command line option.
Also support API SkipCleanup parameter.
Fixes#570.
While updating mirror, if package file is already in pool path,
field `PoolPath` was left as empty which results in package file
being unavailable later on while publishing.
Allow database to be initialized without opening, unify all the
open paths to retry on failure.
In API router make sure open requests are matched with acks in explicit
way.
This also enables re-open attempts in all the aptly commands, so it
should make running aptly CLI much easier now hopefully.
Fix up system tests for oldoldstable ;)
Break up URL into base part and relative path. Match checksum against relative path
and never against full URL.
This might be fixing security issue if aptly was incorrectly matching against
wrong part of Release file.
When DB fails to be open, aptly was skipping "close" phase, so that next
request considered database to be still open (while it's closed) leading
to panic.
Fixes: #431
This requires splitting up import file phase as separate step in then end,
it should be pretty fast, as it only does file move (hardlink) and
DB update for new checksums.
`PackageDownloadTask` is just a reference to file now. Whole process
was rewritten to follow pattern: download to temp location inside the pool,
verify/update checksums, import into pool as final step.
This removes a lot of edge cases when aptly internal state might be broken
if updating from rogue mirror.
Also this changes whole memory model: package list/files are kept in memory
now during the duration of `mirror update` command and saved to disk
only in the end.
Local package pool now implements more generic package pool API.
The base idea is to never expose full paths to files, so that other
kinds of package pools (e.g. package pool in S3) could be used to implement
the same interface.
Files get into the pool only using `Import` method. `Import` method is
now more smart, it supports moving files into the pool, it can detect if
files reside on the same filesystem and use hardlinking instead of copying.
This will make direct mirror downloads still as fast as they were with previous
version which was performing download directly to package pool.
New package pool doesn't have two things implemented yet:
1. New file placement according to SHA256 or other configured hash
2. Calculate at least SHA256/MD5 for each imported files.
MD5 would be required for S3/Swift publishing
* Drop multi-threaded downloader. It doesn't really belong here -
some places require it, some do not, but it's definitely not the
right place to handle it, as it's being used only when updating
mirrors
* Pass expectedChecksums as pointer, so it's easy to drive `nil` value,
and also downloader can fill back checksums (not implemented right now).
* Break down downloader and tests into more files
* Use pkg/errors instead of fmt
NB: Go `defer` order execution is reverse to the order `defer` statements
are executed.
So before the change, `Drop()` was called before `Close()`, which was no-op.
Change that to explicit order in single func, print errors if they happen.
This is related to #506
As a first step, don't pass MD5 explicitly, pass checksum info object,
so that as a next step we can choose which hash to use.
There should be no functional changes so far.
Next step: stop returning explicit paths from public package pool.
This replaces `panic` which aborts aptly execution with warning
message on console. So aptly continues publishing actions, but
`Contents` indexes might be incomplete.
Error will be printed every time contents generation is triggered.
New version format:
* for releases, `x.y.z` (follows tag without leading `v`)
* for nightly builds, `x.y.z+N+hash` (previous version, not the upcoming one)
This means that each nightly build `aptly` would report
correct version now.
Version is now complied into the aptly binary, system tests
automatically check for current version, no need to update them
anymore.
With previous version, `go install` automatically picks up
package `man` and installs `gen.go` as `main` to the $GOPATH/bin which
is not what is expected. Move man page generator to separate
private folder.
systemd has a feature called socket activation where initially systemd
manages and listens on ports/uds and only invokes a service when traffic
appears. to then hand over the involved sockets, systemd will pass the
relevant FDs into the invoked process and defines them in the environment.
use coreos/go-systemd to grab the active listeners passed by systemd and
use them to serve the api routes. only one listener may be specified right
now as we also only support one -listen argument for the binary.
this allows admins to craft a systemd socket and service file for aptly
where systemd manages the socket, its permission and its live time, and
lazy start aptly when needed.
`unix://$PATH` as listen argument will bind aptly to a unix domain socket
rather than TCP.
This allows binding the API to a UDS rather than a port.
Since aptly has no concept of authentication or any amount of high level
API hardening one needs to bottle it up in some other manner. Binding
to a localhost port is often a step in the right direction, ultimately is
still a scary insecure setup as any user on that host getting compromised
would mean that the entire archive is compromised as well.
UDS on the other hand are basically files and have their access managed
by regular file permission. As such, binding to a socket is in fact
the least insecure way to listen as you'd have to explicitly open up the
socket permissions to an access qualified group. In the most conservative
scenario that means no one but the aptly user can talk to the API, in a
more practical setup apache might get access as well and proxy the UDS
with authentication or limited to GET operations.
Using UDS allows reducing the attack surface of the API server while
preserving all the flexibility.
When api/graph.{dot,gv} is requested the raw string for dot gets returned.
This allows client-side rendering rather than server-side. It also makes
the optional dependency on graphivz for dot unnecessary to use the graph
endpoint.
Now that there's an official Go AWS SDK from Amazon, use that instead of
goamz. goamz isn't getting much love these days.
Implement support for STS credentials, as in assumed roles and EC2
instance profiles. The configuration is extended to support a session
token, though I'm not sure why anyone would put temporary credentials in
a configuration file. More likely, no credentials will be explicitly
configured at all, and they will be discovered through the standard SDK
mechanisms described at
<https://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs>.
Resolves#342.
After the first API request, the database was locked as long as the API
server is running. This prevents a user to also use the command-line
client. This commit adds a new flag `-no-lock` that will close the
database after each API request.
Closes#234
Unlocking the different elements in cache flusher was deferred to the
end of the function. Unfortunately, being a for loop wrapped in a
goroutine, deferred were never executed.
Discovered by @sobczyk
When whitespace is stripped from folded stanza fields, some fields
values are glued together without whitespace which might change its meaning.
Eliminate "hint" on missing keys which doesn't apply to .changes.
Would be good to eventually stop using GPG and start calling golang.org/x/crypto/openpgp
Debian's installer validates a mirror by downloading a Release,
and then cross-checking it based on its Codename and Suite. Without
a Suite field, the installer becomes unhappy (e.g. segfaults) and
won't continue the install.
Making the Codename and Suite the same validates with no problem.
The dependency specified when looking for conflicting packages
does not take into account the package version, as by default the
Relation field in the Dependency structure is set to VersionDontCare.
This should fix#185.
Otherwise two index files are generated (source arch, "udeb" true/false) which end up
sharing same final filename, and empty one might overwrite "real" one.
Now aptly won't strip by default, but if distribution contains slash (like 'wheezy/updates')
and component starts with last part of distribution ('updates/main'), it would be stripped.
Amazon S3 replies with a 403 when accessing files that don't exist,
while this should be fixed at Amazon, we can workaround it in aptly
for the moment.
While all the normal Debian package building tools create a
control.tar.gz archive member that contains files with a leading
"./" path element, such as "./control", dpkg and other archive tools
like reprepro are happy with omitting the "./" path element and
having files like "control". Allow for either for compatibility
with weird packages that people may want to import into aptly.
Just realised commands can not have any subcommands and therefore
consist of single words (for example serve or version). Hence I can't
assume that if len(args)==1 then the user has entered the filename.
I have created the filename flag so the user is forced to specify it
when they wish to run aptly tasks from files.
This new aptly command will allow to run multiple commands within a single
aptly command, running in a single thread. The commands can be included
in a text file or added to the aptly script run command in one line,
separated by comas ','.
If one command returns an error, then all the subsequent commands will
be skipped.
Each command output will appear on the console within coloured strings,
clearly stating which command is running and the start and the end of
the received output.
The main function - whuch runs aptly commands - has been taken out from
main.go and included to the cmd package. This is useful for the aptly
script run command, which should use that behaviour.
Users now have the choice of listing the snapshot by time as well as
name (default behaviour).
An additional flag has been added '--sort=' which controls the sort
method applied to the list produced by aptly snapshot list.
The possible values are:
--sort=name (default): sorts the snapshot list by name (lexicographic
order)
--sort=time: sorts the snapshot list in chronological order (oldest to
newest)
When performing an *aptly snapshot pull*, users might list dependency
versions that can potentially match multiple packages in the source
snapshot. However, the current implementation of the 'snapshot pull'
command only allows one package to be pulled from a snapshot at a time
for a given dependency.
The newly implemented all-matches flag allows users to pull all the
matching packages from a source snapshot, provided that they satisfy the
version requirements indicated by the dependencies.
The all-matches flag defaults to false and only produces the described
behaviour when it is explicitly set to true.
This enables us to return the latest version of a package when no version is specified rather than the oldest.
Therefore, we don't need to modify the search algorithm to return the latest version of a package.
:+1::tada: First off, thanks for taking the time to contribute! :tada::+1:
The following is a set of guidelines for contributing to [aptly](https://github.com/aptly-dev/aplty) and related repositories, which are hosted in the [aptly-dev Organization](https://github.com/aptly-dev) on GitHub.
These are just guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request.
## What should I know before I get started?
### Code of Conduct
This project adheres to the Contributor Covenant [code of conduct](CODE_OF_CONDUCT.md).
By participating, you are expected to uphold this code.
Please report unacceptable behavior on [https://github.com/aptly-dev/aptly/discussions](https://github.com/aptly-dev/aptly/discussions)
### List of Repositories
* [aptly-dev/aptly](https://github.com/aptly-dev/aptly) - aptly source code, functional tests, man page
* [aptly-dev/aptly-fixture-db](https://github.com/aptly-dev/aptly-fixture-db) & [aptly-dev/aptly-fixture-pool](https://github.com/aptly-dev/aptly-fixture-pool) provide
fixtures for aptly functional tests
## How Can I Contribute?
### Reporting Bugs
1. Please search for similar bug report in [issue tracker](https://github.com/aptly-dev/aptly/issues)
2. Please verify that bug is not fixed in latest aptly nightly ([download information](https://www.aptly.info/download/))
3. Steps to reproduce increases chances for bug to be fixed quickly. If possible, submit PR with new functional test which fails.
4. If bug is reproducible with specific package, please provide link to package file.
5. Open issue at [GitHub](https://github.com/aptly-dev/aptly/issues)
### Suggesting Enhancements
1. Please search [issue tracker](https://github.com/aptly-dev/aptly/issues) for similar feature requests.
2. Describe why enhancement is important to you.
3. Include any additional details or implementation details.
### Improving Documentation
There are two kinds of documentation:
* [aptly website](https://www.aptly.info)
* aptly `man` page
Core content is mostly the same, but website contains more information, tutorials, examples.
If you want to update `man` page, please open PR to [main aptly repo](https://github.com/aptly-dev/aptly),
details in [man page](#man-page) section.
If you want to update website, please follow steps below:
1. Install [hugo](http://gohugo.io/)
2. Fork [website source](https://github.com/aptly-dev/aptly-dev.github.io) and clone it
3. Launch hugo in development mode: `hugo -w server`
4. Navigate to `http://localhost:1313/`: you should see aptly website
5. Update documentation, most of the time editing Markdown is all you need.
6. Page in browser should reload automatically as you make changes to source files.
We're always looking for new contributions to [FAQ](https://www.aptly.info/doc/faq/), [tutorials](https://www.aptly.info/tutorial/),
general fixes, clarifications, misspellings, grammar mistakes!
### Code Contribution
Please follow [next section](#development-setup) on development process. When change is ready, please submit PR
following [PR template](.github/PULL_REQUEST_TEMPLATE.md).
Make sure that purpose of your change is clear, all the tests and checks pass, and all new code is covered with tests
if that is possible.
### Get the Source
To clone the git repo, run the following commands:
```
git clone git@github.com:aptly-dev/aptly.git
cd aptly
```
## Development Setup
Working on aptly code can be done locally on the development machine, or for convenience by using docker. The next sections describe the setup process.
### Docker Development Setup
This section describes the docker setup to start contributing to aptly.
#### Dependencies
Install the following on your development machine:
- docker
- make
- git
##### Docker installation on macOS
1. Install [Docker Desktop on Mac](https://docs.docker.com/desktop/setup/install/mac-install/) (or via [Homebrew](https://brew.sh/))
2. Allow directory sharing
- Open Docker Desktop
- Go to `Settings → Resources → File Sharing → Virtual File Shares`
- Add the aptly git repository path to the shared list (eg. /home/Users/john/aptly)
#### Create docker container
To build the development docker image, run:
```
make docker-image
```
#### Build aptly
To build the aptly in the development docker container, run:
```
make docker-build
```
#### Running aptly commands
To run aptly commands in the development docker container, run:
```
make docker-shell
```
Example:
```
$ make docker-shell
aptly@b43e8473ef81:/work/src$ aptly version
aptly version: 1.5.0+189+g0fc90dff
```
#### Running unit tests
In order to run aptly unit tests, enter the following:
```
make docker-unit-tests
```
#### Running system tests
In order to run aptly system tests, enter the following:
```
make docker-system-test
```
#### Running golangci-lint
In order to run aptly unit tests, run:
```
make docker-lint
```
#### More info
Run `make help` for more information.
### Local Development Setup
This section describes local setup to start contributing to aptly.
#### Dependencies
Building aptly requires go version 1.22.
On Debian bookworm with backports enabled, go can be installed with:
apt install -t bookworm-backports golang-go
#### Building
To build aptly, run:
make build
Run aptly:
build/aptly
To install aptly into `$GOPATH/bin`, run:
make install
#### Unit-tests
aptly has two kinds of tests: unit-tests and functional (system) tests. Functional tests are preferred way to test any
feature, but some features are much easier to test with unit-tests (e.g. algorithms, failure scenarios, ...)
aptly is using standard Go unit-test infrastructure plus [gocheck](http://labix.org/gocheck). Run the unit-tests with:
make test
#### Functional Tests
Functional tests are implemented in Python, and they use custom test runner which is similar to Python unit-test
runner. Most of the tests start with clean aptly state, run some aptly commands to prepare environment, and finally
run some aptly commands capturing output, exit code, checking any additional files being created and so on. API tests
are a bit different, as they re-use same aptly process serving API requests.
The easiest way to run functional tests is to use `make`:
make system-test
This would check all the dependencies and run all the tests. Some tests (S3, Swift) require access credentials to
be set up in the environment. For example, it needs AWS credentials to run S3 tests (they would be used to publish to S3).
If credentials are missing, tests would be skipped.
You can also run subset of tests manually:
system/run.py t04_mirror
This would run all the mirroring tests under `system/t04_mirror` folder.
Or you can run tests by test name mask:
system/run.py UpdateMirror*
Or, you can run specific test by name:
system/run.py UpdateMirror7Test
Test runner can update expected output instead of failing on mismatch (this is especially useful while
working on new tests):
system/run.py --capture <test>
Output for some tests might contain environment-specific things, e.g. your home directory. In that case
you can use `${HOME}` and similar variable expansion in expected output files.
Some tests depend on fixtures, for example pre-populated GPG trusted keys. There are also test fixtures
captured after mirror update which contain pre-build aptly database and pool contents. They're useful if you
don't want to waste time in the test on populating aptly database while you need some packages to work with.
There are some packages available under `system/files/` directory which are used to build contents of local repos.
*WARNING*: tests are running under current `$HOME` directory with aptly default settings, so they clear completely
`~/.aptly.conf` and `~/.aptly` subdirectory between the runs. So it's not wise to have non-dev aptly being used with
this default location. You can run aptly under different user or by using non-default config location with non-default
aptly root directory.
### man Page
aptly is using combination of [Go templates](http://godoc.org/text/template) and automatically generated text to build `aptly.1` man page. If either source
template [man/aptly.1.ronn.tmpl](man/aptly.1.ronn.tmpl) is changed or any command help is changed, run `make man` to regenerate
final rendered man page [man/aptly.1](man/aptly.1). In the end of the build, new man page is displayed for visual
verification.
Man page is built with small helper [\_man/gen.go](man/gen.go) which pulls in template, command-line help from [cmd/](cmd/) folder
and runs that through [forked copy](https://github.com/smira/ronn) of [ronn](https://github.com/rtomayko/ronn).
### Bash and Zsh Completion
Bash and Zsh completion for aptly reside in the same repo under in [completion.d/aptly](completion.d/aptly) and
[completion.d/\_aptly](completion.d/_aptly), respectively. It's all hand-crafted.
When new option or command is introduced, bash completion should be updated to reflect that change.
When aptly package is being built, it automatically pulls bash completion and man page into the package.
Documentation is available at `http://www.aptly.info/ <http://www.aptly.info/>`_. For support please use
open `issues <https://github.com/aptly-dev/aptly/issues>`_ or `discussions <https://github.com/aptly-dev/aptly/discussions>`_.
Aptly features:
* make mirrors of remote Debian/Ubuntu repositories, limiting by components/architectures
* take snapshots of mirrors at any point in time, fixing state of repository at some moment of time
* publish snapshot as Debian repository, ready to be consumed by apt
* controlled update of one or more packages in snapshot from upstream mirror, tracking dependencies
* merge two or more snapshots into one (+)
* filter repository by search query, pulling dependencies when required (+)
* publish self-made packages as Debian repositories (+)
* merge two or more snapshots into one
* filter repository by search query, pulling dependencies when required
* publish self-made packages as Debian repositories
* REST API for remote access
Current limitations:
Any contributions are welcome! Please see `CONTRIBUTING.md <CONTRIBUTING.md>`_.
* source packages, debian-installer and translations not supported yet
* checksums and signature are not verified while downloading
* deleting created items is not implemented
* cleaning up stale files is not implemented
Installation
=============
Currently aptly is under heavy development, so please use it with care.
Aptly can be installed on several operating systems.
Download
--------
Debian / Ubuntu
----------------
Binary executables (depends almost only on libc) are available for download from `Bintray <http://dl.bintray.com/smira/aptly/>`_.
Aptly is provided in the following debian packages:
If you have Go environment set up, you can build aptly from source by running::
***aptly**: Includes the main Aptly binary, man pages, and shell completions
***aptly-api**: A systemd service for the REST API, using the global /etc/aptly.conf
***aptly-dbg**: Debug symbols for troubleshooting
go get github.com/smira/aptly
The packages can be installed on official `Debian <https://packages.debian.org/search?keywords=aptly>`_ and `Ubuntu <https://packages.ubuntu.com/search?keywords=aptly>`_ distributions.
Upstream Debian Packages
~~~~~~~~~~~~~~~~~~~~~~~~~
If a newer version (not available in Debian/Ubuntu) of aptly is required, upstream debian packages (built from git tags) can be installed as follows:
// @Description Database cleanup removes information about unreferenced packages and deletes files in the package pool that aren’t used by packages anymore.
// @Description It is a good idea to run this command after massive deletion of mirrors, snapshots or local repos.
// @Tags Database
// @Produce json
// @Param _async query bool false "Run in background and return task object"
// @Success 200 {object} string "Output"
// @Failure 404 {object} Error "Not Found"
// @Router /api/db/cleanup [post]
funcapiDBCleanup(c*gin.Context){
resources:=[]string{string(task.AllResourcesKey)}
maybeRunTaskInBackground(c,"Clean up db",resources,func(outaptly.Progress,detail*task.Detail)(*task.ProcessReturnValue,error){
varerrerror
collectionFactory:=context.NewCollectionFactory()
// collect information about referenced packages...
existingPackageRefs:=deb.NewPackageRefList()
out.Printf("Loading mirrors, local repos, snapshots and published repos...")
return&task.ProcessReturnValue{Code:http.StatusConflict,Value:nil},fmt.Errorf("won't delete snapshot that was used as source for other snapshots, use ?force=1 to override")
// @Description **Merge several source snapshots into a new snapshot**
// @Description
// @Description Merge happens from left to right. By default, packages with the same name-architecture pair are replaced during merge (package from latest snapshot on the list wins).
// @Description
// @Description If only one snapshot is specified, merge copies source into destination.
// @Tags Snapshots
// @Consume json
// @Produce json
// @Param name path string true "Name of the snapshot to be created"
// @Param latest query int false "merge only the latest version of each package"
// @Param no-remove query int false "all versions of packages are preserved during merge"
// @Param request body snapshotsMergeParams true "Parameters"
// @Param _async query bool false "Run in background and return task object"
// @Description **Pulls new packages and dependencies from a source snapshot into a new snapshot**
// @Description
// @Description May also upgrade package versions if name snapshot already contains packages being pulled. New snapshot `Destination` is created as result of this process.
// @Description If architectures are limited (with config architectures or parameter `Architectures`, only mentioned architectures are processed, otherwise aptly will process all architectures in the snapshot.
// @Description If following dependencies by source is enabled (using dependencyFollowSource config), pulling binary packages would also pull corresponding source packages as well.
// @Description By default aptly would remove packages matching name and architecture while importing: e.g. when importing software_1.3_amd64, package software_1.2.9_amd64 would be removed.
// @Description
// @Description With flag `no-remove` both package versions would stay in the snapshot.
// @Description
// @Description Aptly pulls first package matching each of package queries, but with flag -all-matches all matching packages would be pulled.
// @Tags Snapshots
// @Param request body snapshotsPullParams true "Parameters"
// @Param name path string true "Name of the snapshot to be created"
// @Param all-matches query int false "pull all the packages that satisfy the dependency version requirements (default is to pull first matching package): 1 to enable"
// @Param dry-run query int false "don’t create destination snapshot, just show what would be pulled: 1 to enable"
// @Param no-deps query int false "don’t process dependencies, just pull listed packages: 1 to enable"
// @Param no-remove query int false "don’t remove other package versions when pulling package: 1 to enable"
// @Param _async query bool false "Run in background and return task object"
fmt.Sprintf("Pulled into '%s' with '%s' as source, pull request was: '%s'",freshToSnapshot.Name,freshSourceSnapshot.Name,strings.Join(body.Queries,", ")))
cmd.Flag.String("origin","","origin name to publish")
cmd.Flag.String("notautomatic","","set value for NotAutomatic field")
cmd.Flag.String("butautomaticupgrades","","set value for ButAutomaticUpgrades field")
cmd.Flag.String("label","","label to publish")
cmd.Flag.String("suite","","suite to publish (defaults to distribution)")
cmd.Flag.String("codename","","codename to publish (defaults to distribution)")
cmd.Flag.Bool("force-overwrite",false,"overwrite files in package pool in case of mismatch")
cmd.Flag.Bool("acquire-by-hash",false,"provide index files by hash")
cmd.Flag.String("signed-by","","an optional field containing a comma separated list of OpenPGP key fingerprints to be used for validating the next Release file")
cmd.Flag.Bool("multi-dist",false,"enable multiple packages with the same filename in different distributions")
cmd.Flag.String("version","","version of the release")
context.Progress().Printf("Warning: publishing from empty source, architectures list should be complete, it can't be changed after publishing (use -architectures flag)\n")
context.Progress().Printf("Warning: publishing from empty source, architectures list should be complete, it can't be changed after publishing (use -architectures flag)\n")
context.Progress().ColoredPrintf("@rWARNING@|: force overwrite mode enabled, aptly might corrupt other published repositories sharing the same package pool.\n")
cmd.Flag.String("origin","","overwrite origin name to publish")
cmd.Flag.String("notautomatic","","overwrite value for NotAutomatic field")
cmd.Flag.String("butautomaticupgrades","","overwrite value for ButAutomaticUpgrades field")
cmd.Flag.String("label","","label to publish")
cmd.Flag.String("suite","","suite to publish (defaults to distribution)")
cmd.Flag.String("codename","","codename to publish (defaults to distribution)")
cmd.Flag.Bool("force-overwrite",false,"overwrite files in package pool in case of mismatch")
cmd.Flag.Bool("acquire-by-hash",false,"provide index files by hash")
cmd.Flag.String("signed-by","","an optional field containing a comma separated list of OpenPGP key fingerprints to be used for validating the next Release file")
cmd.Flag.String("version","","version of the release")
cmd.Flag.Bool("multi-dist",false,"enable multiple packages with the same filename in different distributions")
cmd.Flag.String("component","","component names to update (for multi-component publishing, separate components with commas)")
cmd.Flag.Bool("force-overwrite",false,"overwrite files in package pool in case of mismatch")
cmd.Flag.String("signed-by","","an optional field containing a comma separated list of OpenPGP key fingerprints to be used for validating the next Release file")
cmd.Flag.String("version","","version of the release")
cmd.Flag.Bool("skip-cleanup",false,"don't remove unreferenced files in prefix/component")
cmd.Flag.Bool("multi-dist",false,"enable multiple packages with the same filename in different distributions")
cmd.Flag.Bool("force-overwrite",false,"overwrite files in package pool in case of mismatch")
cmd.Flag.String("signed-by","","an optional field containing a comma separated list of OpenPGP key fingerprints to be used for validating the next Release file")
cmd.Flag.Bool("skip-cleanup",false,"don't remove unreferenced files in prefix/component")
cmd.Flag.Bool("multi-dist",false,"enable multiple packages with the same filename in different distributions")
cmd.Flag.String("origin","","overwrite origin name to publish")
cmd.Flag.String("label","","overwrite label to publish")
cmd.Flag.String("version","","version of the release")
cmd.Flag.Bool("dry-run",false,"don't import, just show what would be imported")
cmd.Flag.Bool("with-deps",false,"follow dependencies when processing package-spec")
returncmd
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
André Roth
Catalin Muresan
Pierig Le Saux
dependabot[bot]
Philip Cramer
Tim Foerster
Russell Greene
tom
Zhang Xiao
Ales Bregar
Brian Witt
Yury Bushmelev
Roman Lebedev
Ato Araki
Juan Calderon-Perez
Linus Fischer
Yaksh Bariya
Christoph Fiehe
chesseed
Tobias Assarsson
Ryan Gonzalez
JupiterRider
Agustin Henze
Alejandro Guijarro Monerris
Yye847
Itay Porezky
boxjan
Silke Hofstra
Christoph Fiehe
Sébastien Delafond
Mikel Olasagasti Uranga
August Feng
Karol Swiderski
Gordian Schoenherr
Christof Warlich
iofq
Christoph Fiehe
Leigh London
Mauro Regli
5hir0kur0
Андрей Лухнов
hudeng
NeroBurner
Sylvain Nieuwlandt
Valentin BRICE
Kevin Martin
Ramón N.Rodriguez
Noa Resare
Cal Jurgella
Jens Reinsberger
Paul Cacheux
Ariel D'Alessandro
Robert LeBlanc
xinhangzhou
Ludovico Cavedon
Nic Waller
Nic Waller
Sylvain Baubeau
Crawax
guoguangwu
Benj Fassbind
Alexander Zubarev
Sjoerd Simons
Markus Muellner
Андрей Лухнов
Samuel Bachmann
Josh Bayfield
Steven Stone
Adam Bambuch
Wade Simmons
Michael Stürmer
Chuan Liu
myml
Russell Greene
Samuel Mutel
Ubuntu
Matt Bearup
Maciej Gol
Lorenzo Bolla
Luciano Lionello
Ratchanan Srirattanamet
Ximon Eighteen
Oliver Sauder
Andrej Shadura
Vítězslav Dvořák
Joshua Colson
Max Bruckner
Don Kuntz
Andrey Smirnov
Raúl Benencia
Raphael Medaer
Frank Steinborn
Nabil BENDAFI
Stephan Eicher
Shengjing Zhu
Oliver Sauder
William Manley
William Manley
Artem Smirnov
Steven! Ragnarök
aviau
Harald Sitter
tomascassidy
Harald Sitter
Strajan Sebastian Ioan
Jack Henschel
Maximilian Stein
Petr Jediný
Stefan
Stefan Felkel
Moritz Bechler
Felix Abecassis
Ringo De Smet
Ludovico Cavedon
Ludovico Cavedon
Alexey Palazhchenko
Jacob Yundt
Clemens Rabe
Raphael Medaer
Szymon Sobik
Charles Hsu
jola5
jola5
Patrick Pokatilo
Einar Forselv
Julien Fabre
Blake Kostner
Daniel Stelter-Gliese
Michael Hofer
Arup Malakar
Benoit Foucher
Geoffrey Thomas
Phil Frost
Bryan Hong
Paul Krohn
Vincent Bernat
Szymon Sobik
James Lu
Zhang, Guodong
Sean
Alexander Guy
Sylvain Baubeau
Sebastien Badia
Sebastien Badia
Andrea Bernardo Ciddio
Michael Koval
Antonio Santos
David Gillies
Chris Read
Rohan Garg
Russ Allbery
Simon Aquino
Dmitrii Kashin
Ivan Kurnosov
Vincent Batoufflet
Guilhem Lettron
Ryan Uber
Andrey Smirnov
Artem Yakimenko