Commit Graph

163 Commits

Author SHA1 Message Date
André Roth
06cbd29d0d add storage API 2024-10-02 18:48:48 +02:00
André Roth
fb538333fa add swagger documentation 2024-10-01 01:07:09 +02:00
André Roth
1d1bd41bb8 add swagger support
- install swaggo
- add swagger config option
2024-10-01 01:07:09 +02:00
Christoph Fiehe
4195ad90bc Allow to add a new component to a published repo
This commit modifies the behavior of the publish switch method in the way, that also new components can be added to an already published repository. It is no longer necessary to drop and recreate the whole publish.

Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
2024-09-24 15:43:27 +02:00
André Roth
37a9fbe530 api: fix OOM with sync tasks
since sync API calls also use tasks internally, this lead to out of memory due to aptly never removing them.
2024-08-03 14:36:04 +02:00
André Roth
fe25414b45 api: repo copy handle package not found
and add tests for error proper handling.
2024-07-24 21:19:47 +02:00
André Roth
49184c9163 fix apiReposCopyPackage getting corrupt file name
it seems c.Params.ByName("file") should not be used
inside maybeRunTaskInBackground, as the content may be corrupted sometimes.
2024-07-24 21:19:47 +02:00
Sylvain Nieuwlandt
11401ca472 [api/copy] create system tests for new copy api endpoint 2024-07-10 16:43:03 +02:00
Valentin BRICE
66429bff45 [api/repos] Add copy API 2024-07-10 16:43:03 +02:00
Sylvain Nieuwlandt
8114786179 Declare the Copy API 2024-07-10 16:43:03 +02:00
André Roth
c1f7e5fe96 handle GpgDisableVerify and ignore-signatures consistently
and be less verbose
2024-07-03 18:08:58 +02:00
André Roth
d16110068c allow not signed mirrors without InRelease file 2024-07-03 18:08:58 +02:00
hudeng
ecc88e7a40 feat: repo and snapshots packages filter api add 'maximumVersion' query parameter support
example: `curl http://localhost:8080/api/repos/test/packages\?maximumVersion\=1`

Change-Id: Ie9ffd36146bf017bbb353737f32360f7b73d6b0a
2024-06-24 17:44:40 +02:00
André Roth
1a3cfea348 replace io/ioutil
fixes golangci-lint errors
2024-06-17 11:51:18 +02:00
André Roth
c33141d49b fix golangci-lint and compilation errors 2024-06-17 11:51:18 +02:00
Ryan Gonzalez
810df17009 Clean up temporary files when mirroring
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2024-06-17 11:51:18 +02:00
Ryan Gonzalez
19255debb9 Reduce required usage of LocalPackagePool
Several sections of the code *required* a LocalPackagePool, but they
could still perform their operations with a standard PackagePool.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2024-06-17 11:51:18 +02:00
André Roth
3e1485faf5 queue sync calls 2024-06-15 19:18:14 +02:00
André Roth
787f954833 mirror: do not download already downloaded packages
this change imports downloaded packages into the pool immediately after download.
in case mirroring is aborted and later resumed, already downloaded packages will not be downloaded anymore.
2024-06-15 16:15:23 +02:00
André Roth
e9bdb983c8 tasks: improve log level 2024-06-15 16:15:23 +02:00
Noa Resare
b4cd86aa14 Introduce option multi-dist to the publish commands
This change makes it possible to publish multiple distributions
with packages named the same but with different content by changing
structure of the generated pool hierarchy. The option not enabled
by default as this changes the structure of the output which could
break the expectations of other tools.
2024-06-15 11:27:26 +02:00
Cal Jurgella
4bd26f5977 Enable SkipArchitectureCheck and IgnoreSignatures in mirror API 2024-06-14 14:30:29 +02:00
Ryan Gonzalez
8d09c202db Skip loading reflists when listing published repos
The output doesn't actually depend on the reflists, and loading them for
every published repo starts to take substantial time and memory.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2024-04-24 17:35:44 +02:00
Ryan Gonzalez
8ab8398c50 Use github.com/saracen/walker for file walk operations
In some local tests w/ a slowed down filesystem, this massively cut down
on the time to clean up a repository by ~3x, bringing a total 'publish
update' time from ~16s to ~13s.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2024-04-24 16:46:16 +02:00
André Roth
6be4f5e8d0 gpg api: allow self signed and use default gpg version 2024-04-03 10:16:56 +02:00
André Roth
b5b0a52cbe s3 api: get publish root list 2024-04-03 10:14:01 +02:00
André Roth
72a7780054 fix golint complaints 2024-03-06 06:21:36 +01:00
André Roth
d7cc9b89d1 system tests: use repository mirrors on S3 for reproducibility
no direct internet download from apt repositories,
which over time will change or cease to exist.

also migrate to gpg2 on newer ubuntu.
2024-02-05 13:04:45 +01:00
Mauro Regli
40c242f9d1 Fix: Remove Batch from API options, set to true by default, add comments
Fixes: #1106
2023-09-14 10:34:20 +02:00
Crawax
214e9075ad Fix returncode when deleting a mirror with snapshot
When trying to delete a mirror that has snapshot and not providing the
force option, the API should not return a `500
StatusInternalServerError`.
A `403 StatusForbidden` is more appropriate when the condition is
expected by the server.
2023-08-18 14:20:23 +02:00
guoguangwu
847fd90e36 chore: unnecessary use of fmt.Sprintf 2023-07-14 11:35:08 +02:00
Markus Muellner
9c6f896666 add endpoint for listing repos while serving in api mode and add more metrics 2023-03-22 17:22:54 +01:00
Markus Muellner
0fdba29d51 make serving published repos in api mode configurable 2023-03-22 17:22:54 +01:00
Андрей Лухнов
e25ade8af3 Serve api and published repos simultaneously
refs #1017 #975
2023-03-22 17:22:54 +01:00
Mauro Regli
77e02bf7a3 Feature: Add Merge Snapshot API
Is part of Issue #176
2023-03-14 08:38:55 +01:00
Mauro Regli
dbf1ac7867 Fix: Drop Publish returned wrong status code if not found
Deleting a publish that does not exist now results in a status code 404
instead of 500.

Fixes: #1006
2023-03-07 13:46:57 +01:00
Mauro Regli
c187b0d52c Fix: Switch gin mode depending on aptly.EnableDebug
If aptly.EnableDebug is active, we use Debug, otherwise we use
gin.ReleaseMode to remove the annoying nuding messages when running the
api.

fixes: #1103
2023-03-07 13:04:12 +01:00
Markus Muellner
8e62195eb5 implement structured logging 2023-02-20 13:42:50 +01:00
Markus Muellner
ecc41f0c0f replace AbortWithError calls by custom function that sets the content type correctly 2023-01-23 10:42:57 +01:00
Samuel Bachmann
ced5ac7876 return the snapshot in apiSnapshotsCreate
In v1.4.0 it [returned the snapshot](https://github.com/aptly-dev/aptly/blob/v1.4.0/api/snapshot.go#L168), but this was removed (by accident) in v1.5.0. This adds it back.
2022-12-22 15:17:12 +01:00
Markus Muellner
2020ca9971 add ready and healthy probe endpoints 2022-12-12 13:39:07 +01:00
Markus Muellner
352f4e8772 update golangci-lint and replace deprecated calls to io/ioutil 2022-12-12 10:21:39 +01:00
Benj Fassbind
71fd730598 Return an empty array if no tasks are available
All other api endpoints also send empty arrays instead of nil.
Closes #1123
2022-11-17 10:44:35 +01:00
Sjoerd Simons
393d1a6888 api: Allow querying the packages endpoint
The ".../packages" endpoints for mirror, local repos and snapshots all
share the same syntax for querying. However the "/api/packages" endpoint
doesn't match this. Adjust that to allow for a bit more consistency and
allow querying the full package database.

The current endpoint functionality "/packages/:name" is kept intact and
can be used the same as now

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>
2022-08-29 10:28:44 +02:00
Benj Fassbind
af899149c7 Fix wrong nil check for SkipBz2 2022-08-16 09:04:16 +02:00
Benj Fassbind
1d4e6183be Capture coverage of integration tests
To capture the coverage also for the integration tests,
a test only executing the cmd.Run function is used.

The test always exits with code 0 and prints the
real exit code to stdout. Otherwise no coverage
report is generated.

Those changes enable a more accurate coverage report
for future contributions.
2022-07-13 08:33:48 +02:00
Sjoerd Simons
f61514edaf Allow disabling bzip2 compression for index files
Using bzip2 generates smaller index files (roughly 20% smaller Packages
files) but it comes with a big performance penalty.  When publishing a
debian mirror snapshot (amd64, arm64, armhf, source) without contents
skipping bzip speeds things up around 1.8 times.

```
$ hyperfine -w 1 -L skip-bz2 true,false  -m 3 -p "aptly -config aptly.conf publish drop bullseye || true" "aptly -config aptly.conf  publish snapshot  --skip-bz2={skip-bz2} --skip-contents --skip-signing bullseye"
Benchmark 1: aptly -config aptly.conf  publish snapshot  --skip-bz2=true --skip-contents --skip-signing bullseye
  Time (mean ± σ):     35.567 s ±  0.307 s    [User: 39.366 s, System: 10.075 s]
  Range (min … max):   35.311 s … 35.907 s    3 runs

Benchmark 2: aptly -config aptly.conf  publish snapshot  --skip-bz2=false --skip-contents --skip-signing bullseye
  Time (mean ± σ):     64.740 s ±  0.135 s    [User: 68.565 s, System: 10.129 s]
  Range (min … max):   64.596 s … 64.862 s    3 runs

Summary
  'aptly -config aptly.conf  publish snapshot  --skip-bz2=true --skip-contents --skip-signing bullseye' ran
    1.82 ± 0.02 times faster than 'aptly -config aptly.conf  publish snapshot  --skip-bz2=false --skip-contents --skip-signing bullseye'
```

Allow skipping bz2 creation for setups where faster publishing is more
important then Package file size.

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>
2022-06-22 11:25:45 +02:00
Sjoerd Simons
29deae6fe0 api: allow parameters with urlencoded names
Aptly allows create e.g. repos with a / to use those with the REST api
the router needs to allow urlencoded parameters in various places to
represent this. A specific example of this is the /api/repos/:name/packages path

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>
2022-06-15 17:21:15 +02:00
Markus Muellner
6539e1b856 Add metrics endpoint with http metrics using Prometheus client lib 2022-04-12 14:39:16 +02:00
Ximon Eighteen
4cf57ae84d govet: compose literal uses unkeyed fields 2022-01-27 09:30:14 +01:00