Regression test for the duplicated skel payload: publishing two
components used to append each skeleton file once per component. Fails
against the previous code with doubled file contents, mirroring the
PublishRepo40Test system test at unit level.
Publish moved the skel and AppStream emission out of the per-component
loop, and codecov flags the relocated error branches as uncovered. Cover
them with deterministic failures:
- a regular file in the middle of the skel path makes the walk fail
with ENOTDIR, which is not swallowed as a not-exist error
- nested skel/AppStream names each within NAME_MAX flatten into a
temporary index file name that exceeds it, failing index creation
- an unreadable skel file (mode 0) fails the open (non-root only)
- a ~1MB filesystem as TMPDIR runs out of space mid-write, reusing the
/smallfs convention from DiskFullSuite
Also add a unit-level happy path test publishing nested skel files and
checking they land in the published tree and the Release checksums.
SkelIndex was the only index constructor not propagating acquireByHash,
so with -acquire-by-hash the Release file advertises Acquire-By-Hash: yes
while no by-hash/ entries exist for these files and apt gets a 404.
The skeleton and AppStream loops were nested inside the per-component
loop while iterating over all components themselves, so every file was
written once per component. Publishing main + contrib appended each
payload twice. Release checksums are computed after the fact, so the
duplicated content is signed and apt accepts it silently.
* remove useless resource lock
Resource locks need to be before the background task. creating same publish endpoint at the same time is unlikely...
* load data inside background tasks
This fixes a flaw in async apis, which loaded the published repo from the DB and mutated it outside the task closure, before the task lock was acquired.
Perform collection.LoadComplete inside maybeRunTaskInBackground and have tasks use a fresh copy of taskCollectionFactory, taskCollection
* lock source repos/snapshots for publish operations
Concurrent tasks were not properly locking their resources, leading to inconsistent published indexes:
SourceLocalRepo: iterate published.Sources (component -> source UUID), look up each local repo via localRepoCollection.ByUUID and append string(repo.Key()) to resources
SourceSnapshot: iterate b.Snapshots,look up each snapshot via snapshotCollection.ByName and append string(snapshot.ResourceKey()) to resources.
* lock pool on non MultiDist publish
* revert mutex on LinkFromPool
* use uuids, since names can be renamed
* add test for MultiDist change
Implement support for the SOURCE_DATE_EPOCH environment variable as
specified by reproducible-builds.org. When set, this variable overrides
the current timestamp in the Release file's Date and Valid-Until fields,
enabling reproducible filesystem publishes.
- Read SOURCE_DATE_EPOCH environment variable in Publish()
- Use the epoch timestamp for both Date and Valid-Until fields
- Gracefully fallback to current time if unset or invalid
- Add comprehensive tests for valid and invalid SOURCE_DATE_EPOCH values
This change makes it possible to publish multiple distributions
with packages named the same but with different content by changing
structure of the generated pool hierarchy. The option not enabled
by default as this changes the structure of the output which could
break the expectations of other tools.
The cleanup phase needs to list out all the files in each component in
order to determine what's still in use. When there's a large number of
sources (e.g. from having many snapshots), the time spent just loading
the package information becomes substantial. However, in many cases,
most of the packages being loaded are actually shared across the
sources; if you're taking frequent snapshots, for instance, most of the
packages in each snapshot will be the same as other snapshots. In these
cases, re-reading the packages repeatedly is just a waste of time.
To improve this, we maintain a list of refs that we know were processed
for each component. When listing the refs from a source, only the ones
that have not yet been processed will be examined. Some tests were also
added specifically to check listing the files in a component.
With this change, listing the files in components on a copy of our
production database went from >10 minutes to ~10 seconds, and the newly
added benchmark went from ~300ms to ~43ms.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
This is spin-off of changes from #459.
Transactions are not being used yet, but batches are updated to work
with the new API.
`database/` package was refactored to split abstract interfaces and
implementation via goleveldb. This should make it easier to implement
new database types.
Allow skipping unreferenced files cleanup on publish switch/update/drop
via the -skip-cleanup command line option.
Also support API SkipCleanup parameter.
Fixes#570.
Allow database to be initialized without opening, unify all the
open paths to retry on failure.
In API router make sure open requests are matched with acks in explicit
way.
This also enables re-open attempts in all the aptly commands, so it
should make running aptly CLI much easier now hopefully.
Fix up system tests for oldoldstable ;)
This is related to #506
As a first step, don't pass MD5 explicitly, pass checksum info object,
so that as a next step we can choose which hash to use.
There should be no functional changes so far.
Next step: stop returning explicit paths from public package pool.
KereMath
André Roth
Philip Cramer
Tim Foerster
Mauro Regli
Christoph Fiehe
Noa Resare
Ryan Gonzalez
Ariel D'Alessandro
Andrey Smirnov
Oliver Sauder
Ludovico Cavedon
Clemens Rabe
Chris Read