Commit Graph

33 Commits

Author SHA1 Message Date
André Roth f7057a9517 go1.24: fix lint, unit and system tests
- development env: base on debian trixie with go1.24
- lint: run with default config
- fix lint errors
- fix unit tests
- fix system test
2025-04-26 13:29:50 +02:00
Mauro Regli 1357d246d8 rename addon files to skel files 2024-11-17 14:09:37 +01:00
Mauro Regli 2aac7baf52 add AddonIndex to index_files
I had to remove "signable: false" (line 399), since that property
doesn't exist.
2024-11-17 14:09:37 +01:00
André Roth a0af6a25aa fix lint complaints 2024-04-03 10:13:24 +02:00
Robert LeBlanc c2ee086487 Fix the installer path for Ubuntu Focal
Ubuntu has started depreciating the Debian installer in focal
and moved the installer images to a different path. In versions
after focal, they are completly removed. This basically gives
us more time to figure out how to use the new system.
2024-04-03 10:13:24 +02:00
André Roth e61a4dd53c fix golangci-lint errors 2024-02-06 20:49:35 +01:00
Sjoerd Simons f61514edaf Allow disabling bzip2 compression for index files
Using bzip2 generates smaller index files (roughly 20% smaller Packages
files) but it comes with a big performance penalty.  When publishing a
debian mirror snapshot (amd64, arm64, armhf, source) without contents
skipping bzip speeds things up around 1.8 times.

```
$ hyperfine -w 1 -L skip-bz2 true,false  -m 3 -p "aptly -config aptly.conf publish drop bullseye || true" "aptly -config aptly.conf  publish snapshot  --skip-bz2={skip-bz2} --skip-contents --skip-signing bullseye"
Benchmark 1: aptly -config aptly.conf  publish snapshot  --skip-bz2=true --skip-contents --skip-signing bullseye
  Time (mean ± σ):     35.567 s ±  0.307 s    [User: 39.366 s, System: 10.075 s]
  Range (min … max):   35.311 s … 35.907 s    3 runs

Benchmark 2: aptly -config aptly.conf  publish snapshot  --skip-bz2=false --skip-contents --skip-signing bullseye
  Time (mean ± σ):     64.740 s ±  0.135 s    [User: 68.565 s, System: 10.129 s]
  Range (min … max):   64.596 s … 64.862 s    3 runs

Summary
  'aptly -config aptly.conf  publish snapshot  --skip-bz2=true --skip-contents --skip-signing bullseye' ran
    1.82 ± 0.02 times faster than 'aptly -config aptly.conf  publish snapshot  --skip-bz2=false --skip-contents --skip-signing bullseye'
```

Allow skipping bz2 creation for setups where faster publishing is more
important then Package file size.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>
2022-06-22 11:25:45 +02:00
Oliver Sauder f09a273ad7 Add publish output progress counting remaining number of packages 2022-01-27 09:30:14 +01:00
Oliver Sauder e23e30eb44 Merge branch 'master' into with_installer 2018-09-21 13:26:15 +02:00
Andrey Smirnov 747b9752ce Keep checksum of not compressed index file even if it's not uploaded
Fixes: #756
2018-07-14 00:17:36 +03:00
Oliver Sauder b7323db31b Add detached signature to installer hashsum file 2018-07-06 15:02:37 +02:00
Oliver Sauder 108b0ea226 Add support to mirror non package installer files 2018-07-06 15:02:37 +02:00
Andrey Smirnov b8c5303fdb Fix paths after repository transfer to aptly-dev 2018-04-18 21:19:43 +03:00
Andrey Smirnov 5b85522400 Implement 'legacy' Contents indexes to match Ubuntu <=16.04
Another index is created which unifies data for all the components.

This certainly requires more resources as we have to build yet another
index.
2018-04-11 00:57:15 +03:00
Harald Sitter 00bb0ca8f3 fix a serious file leak in the by-index publishing
the logic here was wrong.
if we managed to find the link target (the physical index file) pointed to
by our old symlink we want to remove it (this is basically "cleaning up old
index" logic).
previously we'd try to only delete it when the ReadLink came back with
error. which had two serious issues with it:

a) linkTarget was empty, so we basically called Remove("") which would
   delete the storage -> root <- directory if the root is a symlink!
b) we'd leak old indexes as the cleanup logic only ran if there was en
   error which would ordinarily never be

new code correctly cleans up unless there was an error.

this relates to a previous bugfix of readLink which incorrectly returned
absolute paths ultimately rendering the Remove call also broken.
2018-02-19 17:32:06 +01:00
Harald Sitter 2d0baef3b1 make code less repetitive and more readable
by using the power of variables!
2018-02-19 17:22:36 +01:00
Oliver Sauder b2bf4f7884 Adjust FileExists to differentiate between error and actual file existence 2017-11-30 09:46:02 +01:00
Oliver Sauder 3efa1052fa Implement FileExists in files storage as simple stat to improve performance 2017-11-30 09:46:02 +01:00
Oliver Sauder 2e488608ca Simplify packaging indexing by hash and stop when there is an error 2017-11-30 09:46:02 +01:00
Oliver Sauder d6b4b795a5 Fix linting errors 2017-11-30 09:46:02 +01:00
Oliver Sauder 092a7ed8f3 Rename AccessByHash to AcquireByHash for consistency with other flags 2017-11-30 09:46:02 +01:00
Oliver Sauder 7498fd8fc8 Extend s3 storage with link and file exists methods 2017-11-30 09:46:02 +01:00
André Roth e07912770e Extend PublishedStorage interface for Acquire-By-Hash
Signed-off-by: André Roth <neolynx@gmail.com>
2017-11-30 09:46:02 +01:00
André Roth bb2db7e500 Support Acquire-By-Hash for index files
The added "aptly publish repo" option "-access-by-hash" publishes
the index files (Packages*, Sources*) also as hardlinked hashes.
Example:
 /dists/yakkety/main/binary-amd64/by-hash/SHA512/31833ec39acc...
The Release files indicate this with the option "Acquire-By-Hash: yes"

This is used by apt >= 1.2.0 and prevents the "Hash sum mismatch" race
condition between a server side "aptly publish repo" and "apt-get update"
on a client.
See: http://www.chiark.greenend.org.uk/~cjwatson/blog/no-more-hash-sum-mismatch-errors.html

This implementation uses symlinks in the by-hash/*/ directory for keeping
only two versions of the index files and deleting older files
automatically.

Note: this only works with aptly.FileSystemPublishedStorage

Closes: #536

Signed-off-by: André Roth <neolynx@gmail.com>
2017-11-30 09:46:02 +01:00
Andrey Smirnov 1be8d39105 Refactor GPG signer/verifier
Goal is to make it easier to plug in another implementation.
2017-05-23 02:54:56 +03:00
Andrey Smirnov 470165a419 Enable goconst & interfacer linters 2017-05-17 00:53:10 +03:00
Andrey Smirnov 516dd7b044 Switch to gometalinter
Only small amount of required checks is enabled,
plan is to enable more linters as issues are fixed in the code.
2017-03-23 01:51:08 +03:00
Oliver Sauder fcd4429370 only create bzip file if needed. #415 2016-10-13 13:48:28 +02:00
Andrey Smirnov 138f9f7994 Generate only .gz file for Contents index. #142 2015-04-02 00:29:49 +03:00
Andrey Smirnov 38f4fc209b Contents index support. #142 2015-03-31 00:08:23 +03:00
Andrey Smirnov eb087fd291 When generating index files, make udeb forced false for "source" architecture. #180
Otherwise two index files are generated (source arch, "udeb" true/false) which end up
sharing same final filename, and empty one might overwrite "real" one.
2015-01-26 21:16:44 +03:00
Andrey Smirnov bd34ba4088 Pregenerate all udebs indexes if at least one udeb has been discovered. #108 2014-09-30 21:11:01 +04:00
Andrey Smirnov db499f872d Major refactoring of the publishing method, now uses helper indexFile(s). #108 2014-09-27 01:39:02 +04:00