Commit Graph

33 Commits

Author SHA1 Message Date
André Roth
f7057a9517 go1.24: fix lint, unit and system tests
- development env: base on debian trixie with go1.24
- lint: run with default config
- fix lint errors
- fix unit tests
- fix system test
2025-04-26 13:29:50 +02:00
Mauro Regli
1357d246d8 rename addon files to skel files 2024-11-17 14:09:37 +01:00
Mauro Regli
2aac7baf52 add AddonIndex to index_files
I had to remove "signable: false" (line 399), since that property
doesn't exist.
2024-11-17 14:09:37 +01:00
André Roth
a0af6a25aa fix lint complaints 2024-04-03 10:13:24 +02:00
Robert LeBlanc
c2ee086487 Fix the installer path for Ubuntu Focal
Ubuntu has started depreciating the Debian installer in focal
and moved the installer images to a different path. In versions
after focal, they are completly removed. This basically gives
us more time to figure out how to use the new system.
2024-04-03 10:13:24 +02:00
André Roth
e61a4dd53c fix golangci-lint errors 2024-02-06 20:49:35 +01:00
Sjoerd Simons
f61514edaf Allow disabling bzip2 compression for index files
Using bzip2 generates smaller index files (roughly 20% smaller Packages
files) but it comes with a big performance penalty.  When publishing a
debian mirror snapshot (amd64, arm64, armhf, source) without contents
skipping bzip speeds things up around 1.8 times.

```
$ hyperfine -w 1 -L skip-bz2 true,false  -m 3 -p "aptly -config aptly.conf publish drop bullseye || true" "aptly -config aptly.conf  publish snapshot  --skip-bz2={skip-bz2} --skip-contents --skip-signing bullseye"
Benchmark 1: aptly -config aptly.conf  publish snapshot  --skip-bz2=true --skip-contents --skip-signing bullseye
  Time (mean ± σ):     35.567 s ±  0.307 s    [User: 39.366 s, System: 10.075 s]
  Range (min … max):   35.311 s … 35.907 s    3 runs

Benchmark 2: aptly -config aptly.conf  publish snapshot  --skip-bz2=false --skip-contents --skip-signing bullseye
  Time (mean ± σ):     64.740 s ±  0.135 s    [User: 68.565 s, System: 10.129 s]
  Range (min … max):   64.596 s … 64.862 s    3 runs

Summary
  'aptly -config aptly.conf  publish snapshot  --skip-bz2=true --skip-contents --skip-signing bullseye' ran
    1.82 ± 0.02 times faster than 'aptly -config aptly.conf  publish snapshot  --skip-bz2=false --skip-contents --skip-signing bullseye'
```

Allow skipping bz2 creation for setups where faster publishing is more
important then Package file size.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>
2022-06-22 11:25:45 +02:00
Oliver Sauder
f09a273ad7 Add publish output progress counting remaining number of packages 2022-01-27 09:30:14 +01:00
Oliver Sauder
e23e30eb44 Merge branch 'master' into with_installer 2018-09-21 13:26:15 +02:00
Andrey Smirnov
747b9752ce Keep checksum of not compressed index file even if it's not uploaded
Fixes: #756
2018-07-14 00:17:36 +03:00
Oliver Sauder
b7323db31b Add detached signature to installer hashsum file 2018-07-06 15:02:37 +02:00
Oliver Sauder
108b0ea226 Add support to mirror non package installer files 2018-07-06 15:02:37 +02:00
Andrey Smirnov
b8c5303fdb Fix paths after repository transfer to aptly-dev 2018-04-18 21:19:43 +03:00
Andrey Smirnov
5b85522400 Implement 'legacy' Contents indexes to match Ubuntu <=16.04
Another index is created which unifies data for all the components.

This certainly requires more resources as we have to build yet another
index.
2018-04-11 00:57:15 +03:00
Harald Sitter
00bb0ca8f3 fix a serious file leak in the by-index publishing
the logic here was wrong.
if we managed to find the link target (the physical index file) pointed to
by our old symlink we want to remove it (this is basically "cleaning up old
index" logic).
previously we'd try to only delete it when the ReadLink came back with
error. which had two serious issues with it:

a) linkTarget was empty, so we basically called Remove("") which would
   delete the storage -> root <- directory if the root is a symlink!
b) we'd leak old indexes as the cleanup logic only ran if there was en
   error which would ordinarily never be

new code correctly cleans up unless there was an error.

this relates to a previous bugfix of readLink which incorrectly returned
absolute paths ultimately rendering the Remove call also broken.
2018-02-19 17:32:06 +01:00
Harald Sitter
2d0baef3b1 make code less repetitive and more readable
by using the power of variables!
2018-02-19 17:22:36 +01:00
Oliver Sauder
b2bf4f7884 Adjust FileExists to differentiate between error and actual file existence 2017-11-30 09:46:02 +01:00
Oliver Sauder
3efa1052fa Implement FileExists in files storage as simple stat to improve performance 2017-11-30 09:46:02 +01:00
Oliver Sauder
2e488608ca Simplify packaging indexing by hash and stop when there is an error 2017-11-30 09:46:02 +01:00
Oliver Sauder
d6b4b795a5 Fix linting errors 2017-11-30 09:46:02 +01:00
Oliver Sauder
092a7ed8f3 Rename AccessByHash to AcquireByHash for consistency with other flags 2017-11-30 09:46:02 +01:00
Oliver Sauder
7498fd8fc8 Extend s3 storage with link and file exists methods 2017-11-30 09:46:02 +01:00
André Roth
e07912770e Extend PublishedStorage interface for Acquire-By-Hash
Signed-off-by: André Roth <neolynx@gmail.com>
2017-11-30 09:46:02 +01:00
André Roth
bb2db7e500 Support Acquire-By-Hash for index files
The added "aptly publish repo" option "-access-by-hash" publishes
the index files (Packages*, Sources*) also as hardlinked hashes.
Example:
 /dists/yakkety/main/binary-amd64/by-hash/SHA512/31833ec39acc...
The Release files indicate this with the option "Acquire-By-Hash: yes"

This is used by apt >= 1.2.0 and prevents the "Hash sum mismatch" race
condition between a server side "aptly publish repo" and "apt-get update"
on a client.
See: http://www.chiark.greenend.org.uk/~cjwatson/blog/no-more-hash-sum-mismatch-errors.html

This implementation uses symlinks in the by-hash/*/ directory for keeping
only two versions of the index files and deleting older files
automatically.

Note: this only works with aptly.FileSystemPublishedStorage

Closes: #536

Signed-off-by: André Roth <neolynx@gmail.com>
2017-11-30 09:46:02 +01:00
Andrey Smirnov
1be8d39105 Refactor GPG signer/verifier
Goal is to make it easier to plug in another implementation.
2017-05-23 02:54:56 +03:00
Andrey Smirnov
470165a419 Enable goconst & interfacer linters 2017-05-17 00:53:10 +03:00
Andrey Smirnov
516dd7b044 Switch to gometalinter
Only small amount of required checks is enabled,
plan is to enable more linters as issues are fixed in the code.
2017-03-23 01:51:08 +03:00
Oliver Sauder
fcd4429370 only create bzip file if needed. #415 2016-10-13 13:48:28 +02:00
Andrey Smirnov
138f9f7994 Generate only .gz file for Contents index. #142 2015-04-02 00:29:49 +03:00
Andrey Smirnov
38f4fc209b Contents index support. #142 2015-03-31 00:08:23 +03:00
Andrey Smirnov
eb087fd291 When generating index files, make udeb forced false for "source" architecture. #180
Otherwise two index files are generated (source arch, "udeb" true/false) which end up
sharing same final filename, and empty one might overwrite "real" one.
2015-01-26 21:16:44 +03:00
Andrey Smirnov
bd34ba4088 Pregenerate all udebs indexes if at least one udeb has been discovered. #108 2014-09-30 21:11:01 +04:00
Andrey Smirnov
db499f872d Major refactoring of the publishing method, now uses helper indexFile(s). #108 2014-09-27 01:39:02 +04:00