In some local tests w/ a slowed down filesystem, this massively cut down
on the time to clean up a repository by ~3x, bringing a total 'publish
update' time from ~16s to ~13s.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Ubuntu has started depreciating the Debian installer in focal
and moved the installer images to a different path. In versions
after focal, they are completly removed. This basically gives
us more time to figure out how to use the new system.
instead of caching the whole s3 bucket, cache only the pool path. this
requires an additional parameter, and since this is an interface, all
implementations need to follow. might help in other backends too.
closes#1181
A race condition for publishing packages and
mirrors at the same time was introduced in
commit 77d7c38.
The problem is that when opening a leveldb transaction
and performing another 'put' to the db
the system freezes.
While testing out Aptly, the `apt-get` client complains with the following error, since the `codename` was switched from the InRelease files that are baked out by Aptly:
```
E: Repository 'http://debianrepo.example.com/bionic testing InRelease' changed its 'Codename' value from '' to 'testing'
```
dpkg-gencontrol can be called with -v flag which set binary package's
version separated from source version. When this happen, the Source
field will contain version number in addition to source package name.
This tripped Aptly's dbgsym restriction, which check for exact source
package name, which in turn prevents the dbgsym & the whole .changes
file from being imported.
From the git history, it seems like this condition is a leftover from
when Aptly filter dbgsym packages using "*-dbgsym". So, I decided to
remove it. A test case has been added to prevent regression.
This commit closes: #145
The dependency format "pkg:arch" (e.g. "python3:any") was not well
parsed if not any version is given. This commit splits the dependency
name and architecture in all cases.
For any action which is multi-step (requires updating more than 1 DB
key), use transaction to make update atomic.
Also pack big chunks of updates (importing packages for importing and
mirror updates) into single transaction to improve aptly performance and
get some isolation.
Note that still layers up (Collections) provide some level of isolation,
so this is going to shine with the future PRs to remove collection
locks.
Spin-off of #459
This is spin-off of changes from #459.
Transactions are not being used yet, but batches are updated to work
with the new API.
`database/` package was refactored to split abstract interfaces and
implementation via goleveldb. This should make it easier to implement
new database types.
Apply retries as global, config-level option `downloadRetries` so that
it can be applied to any aptly command which downloads objects.
Unwrap `errors.Wrap` which is used in downloader.
Unwrap `*url.Error` which should be the actual error returned from the
HTTP client, catch more cases, be more specific around failures.
See #765, #761
Collections were relying on keeping in-memory list of all the objects
for any kind of operation which doesn't scale well the number of
objects in the database.
With this rewrite, objects are loaded only on demand which might
be pessimization in some edge cases but should improve performance
and memory footprint signifcantly.
Ryan Gonzalez
André Roth
Ariel D'Alessandro
Robert LeBlanc
Mauro Regli
Benj Fassbind
Alexander Zubarev
Markus Muellner
Steven Stone
Sjoerd Simons
myml
Russell Greene
Reinhold Gschweicher
Matt Bearup
Ratchanan Srirattanamet
Lorenzo Bolla
Oliver Sauder
Sylvain Baubeau
Joshua Colson
Raúl Benencia
Raphael Medaer
Andrey Smirnov
Shengjing Zhu
William Manley
William Manley
Oliver Sauder