iofq
840b76228a
add shutdown context unit test
2024-11-09 15:34:35 -06:00
iofq
8436001d5b
Make HTTP server wait for tasks before shutdown
2024-11-08 14:06:23 -06:00
André Roth
f0ad0f9496
improve and test grab downloader
2024-10-04 13:37:56 +02:00
iofq
372ce3c4bc
Avoid nil panic when downloadSpeedLimit is set in api mode
2024-08-16 10:04:46 +02:00
iofq
ff8a02959c
fix throttled downloader
2024-08-11 10:20:37 +02:00
Jens Reinsberger
4bc2180eed
fix failing build on hosts with wildcard dns
...
on hosts which have wildcard dns domains in their local domain search
list, builds failed because "nosuch.host" could actually be resolved.
Since ".host" isn't a recommended TLD by RFC2606, we use ".invalid" now.
And since this is not enough to fix the problem, we use now absoulte
domain names (having a '.' at the end)
2024-05-15 16:42:37 +02:00
André Roth
9deb031c44
fix system tests
...
- use s3 mirror instead of internet download
- reduce download verbosity
- do not use venv in docker-system-tests
- be more verbose on test output
- do not run golangci-lint in system-tests
2024-04-11 10:15:02 +02:00
Mauro Regli
ae61706a34
Fix: Implement golangci-lint suggestions
2023-09-21 11:25:18 +02:00
Markus Muellner
8e62195eb5
implement structured logging
2023-02-20 13:42:50 +01:00
Markus Muellner
352f4e8772
update golangci-lint and replace deprecated calls to io/ioutil
2022-12-12 10:21:39 +01:00
Michael Stürmer
4b2efeec7a
Cope with zero-length http downloads
2022-06-20 09:47:41 +02:00
Markus Muellner
6539e1b856
Add metrics endpoint with http metrics using Prometheus client lib
2022-04-12 14:39:16 +02:00
Lorenzo Bolla
551a370c13
Basic tests for Grab downloader
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
1afcd68e01
Make downloader type configurable
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
cc30ef3ee2
Remove vendor directory
...
We use go modules now.
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
235e35a2f3
Rate limit 0 effectively disables rate limiting
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
4c54f967b7
Fix error checking
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
8925949be3
Support rate limiting in grab downloader
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
e96372c999
Implement ignore checksum mismatch
...
Also, update "pkg/errors" library.
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
e5d9d27069
Wrap errors with more context
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
853c990b6e
Handle checksums
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
86c1ffab2a
Add logs for checksum
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
952287a787
Reenable checksums
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
b5d90b7b13
Add more logging
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
3e06af8515
Disable checksum for now
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
eff2e56d0d
Specify output filename instead of directory
...
"temp" downloader uses its own naming for downloaded files.
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
eaac04ccf6
Improve logging in grab downloader
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
894192851e
Grab downloader
2022-01-31 10:32:54 +01:00
Lorenzo Bolla
3775d69a60
Fix linting errors
2022-01-27 09:30:14 +01:00
Lorenzo Bolla
6826efc723
Fix pure-go unittests
...
So they can run on e.g. LXC containers as root, or other conceivable setups.
2022-01-27 09:30:14 +01:00
Lorenzo Bolla
787cc8e3ee
Fix system tests
2022-01-27 09:30:14 +01:00
Lorenzo Bolla
fd83c1a5bf
Cap delay to sleep to avoid overflow
2022-01-27 09:30:14 +01:00
Lorenzo Bolla
de2be9b8ae
Sleep between retries to download from http
...
Fix #1
2022-01-27 09:30:14 +01:00
Lorenzo Bolla
19f7b0fe8d
mirror: increase logging for easier debugging
2022-01-27 09:30:14 +01:00
André Roth
c0528888f4
log download retries
2022-01-27 09:30:14 +01:00
Oliver Sauder
f09a273ad7
Add publish output progress counting remaining number of packages
2022-01-27 09:30:14 +01:00
Oliver Sauder
6ab5e60833
Add task api and resource locking ability
2022-01-27 09:30:14 +01:00
Oliver Sauder
25d7d7c037
Solving progress not safe issue for api
...
Progress is not safe so for api its always nil and
code needs to take care of this
2022-01-27 09:30:14 +01:00
Lorenzo Bolla
f4dc87fa44
Use a hostname more likely to be non-existent than localhost
...
Otherwise, it's possible that certain network configuration defining
*.localhost cause the tests to fail.
2021-02-08 10:52:27 +01:00
Andrey Smirnov
769e984ef4
Fix issues with progress == nil causing panics
...
Part of PR #459
This prepares for more methods to be exposed via the API.
2019-09-03 20:28:28 +04:00
Andrey Smirnov
26098f6c8d
Print redirects being followed, drop mirror.yandex.ru.
...
Use CDN-backed Debian mirror to make tests run faster hopefully for
everyone. Redirects might be important to know what exactly is going on
when items are being downloaded.
2019-08-07 21:10:04 +03:00
Andrey Smirnov
f0a370db24
Rework HTTP downloader retry logic
...
Apply retries as global, config-level option `downloadRetries` so that
it can be applied to any aptly command which downloads objects.
Unwrap `errors.Wrap` which is used in downloader.
Unwrap `*url.Error` which should be the actual error returned from the
HTTP client, catch more cases, be more specific around failures.
2019-08-07 20:23:05 +03:00
Andrey Smirnov
3b5840e248
Fix linter list and fix errors discovered by new staticcheck
2019-01-20 00:01:17 +03:00
Oliver Sauder
b1a2523ef0
Add unit test for remote and http
2018-07-06 15:02:37 +02:00
Oliver Sauder
108b0ea226
Add support to mirror non package installer files
2018-07-06 15:02:37 +02:00
Andrey Smirnov
b8c5303fdb
Fix paths after repository transfer to aptly-dev
2018-04-18 21:19:43 +03:00
Andrey Smirnov
15618c8ea8
Use Go context to abort gracefully mirror updates
...
There are two fixes here:
1. Abort package download immediately as ^C is pressed.
2. Import all the already downloaded files into package pool,
so that next time mirror is updated, aptly won't download them
once again.
2017-11-30 00:49:37 +03:00
Andrey Smirnov
cafb89f30f
Re-work the way checksum matching works against Release file
...
Break up URL into base part and relative path. Match checksum against relative path
and never against full URL.
This might be fixing security issue if aptly was incorrectly matching against
wrong part of Release file.
2017-05-23 03:00:15 +03:00
Andrey Smirnov
f0360cf2d3
Use longest suffix match to pick up checksum
2017-05-23 03:00:15 +03:00
Andrey Smirnov
51213899b7
More Go linters enabled, issues fixed
...
Ref: #528
Enables "staticcheck", "varcheck", "structcheck", "aligncheck"
2017-05-03 18:23:14 +03:00
