Lorenzo Bolla
787cc8e3ee
Fix system tests
2022-01-27 09:30:14 +01:00
Lorenzo Bolla
fd83c1a5bf
Cap delay to sleep to avoid overflow
2022-01-27 09:30:14 +01:00
Lorenzo Bolla
de2be9b8ae
Sleep between retries to download from http
...
Fix #1
2022-01-27 09:30:14 +01:00
Lorenzo Bolla
19f7b0fe8d
mirror: increase logging for easier debugging
2022-01-27 09:30:14 +01:00
André Roth
c0528888f4
log download retries
2022-01-27 09:30:14 +01:00
Oliver Sauder
6ab5e60833
Add task api and resource locking ability
2022-01-27 09:30:14 +01:00
Oliver Sauder
25d7d7c037
Solving progress not safe issue for api
...
Progress is not safe so for api its always nil and
code needs to take care of this
2022-01-27 09:30:14 +01:00
Andrey Smirnov
769e984ef4
Fix issues with progress == nil causing panics
...
Part of PR #459
This prepares for more methods to be exposed via the API.
2019-09-03 20:28:28 +04:00
Andrey Smirnov
26098f6c8d
Print redirects being followed, drop mirror.yandex.ru.
...
Use CDN-backed Debian mirror to make tests run faster hopefully for
everyone. Redirects might be important to know what exactly is going on
when items are being downloaded.
2019-08-07 21:10:04 +03:00
Andrey Smirnov
f0a370db24
Rework HTTP downloader retry logic
...
Apply retries as global, config-level option `downloadRetries` so that
it can be applied to any aptly command which downloads objects.
Unwrap `errors.Wrap` which is used in downloader.
Unwrap `*url.Error` which should be the actual error returned from the
HTTP client, catch more cases, be more specific around failures.
2019-08-07 20:23:05 +03:00
Andrey Smirnov
3b5840e248
Fix linter list and fix errors discovered by new staticcheck
2019-01-20 00:01:17 +03:00
Oliver Sauder
108b0ea226
Add support to mirror non package installer files
2018-07-06 15:02:37 +02:00
Andrey Smirnov
b8c5303fdb
Fix paths after repository transfer to aptly-dev
2018-04-18 21:19:43 +03:00
Andrey Smirnov
15618c8ea8
Use Go context to abort gracefully mirror updates
...
There are two fixes here:
1. Abort package download immediately as ^C is pressed.
2. Import all the already downloaded files into package pool,
so that next time mirror is updated, aptly won't download them
once again.
2017-11-30 00:49:37 +03:00
Andrey Smirnov
1f3cb2db5d
When downloading/importing packages, enforce all checksums
2017-04-26 23:17:04 +03:00
Andrey Smirnov
94b49818a1
Refactor HTTP downloader package
...
* Drop multi-threaded downloader. It doesn't really belong here -
some places require it, some do not, but it's definitely not the
right place to handle it, as it's being used only when updating
mirrors
* Pass expectedChecksums as pointer, so it's easy to drive `nil` value,
and also downloader can fill back checksums (not implemented right now).
* Break down downloader and tests into more files
* Use pkg/errors instead of fmt
2017-04-26 23:17:03 +03:00
Andrey Smirnov
11d828b3b1
Add govet/golint into Travis CI build
...
Fix current issues
2017-03-22 21:49:16 +03:00
Andrey Smirnov
acd8d4a6ea
Go 1.6 compatibility
2017-03-17 00:17:07 +03:00
Andrey Smirnov
c9768416ed
Fix up tests
2017-03-16 23:06:41 +03:00
Raphael Medaer
db8595711b
Added '.xz' reader in HTTP download. This fixed issue when you try to mirror a distribution with only Packages.xz (example 'debian/experimental').
2017-03-16 22:41:25 +03:00
Oliver Sauder
af71b9541c
Fixing typo
2016-11-29 09:00:16 +01:00
Oliver Sauder
6becd5a3aa
Added max-tries flag for mirror update
2016-11-28 17:02:24 +01:00
Benoit Foucher
7f5a7323a6
Fixed mkdir mode from 755 to 777
2016-03-18 09:37:18 +00:00
Andrey Smirnov
77be7b9e3b
Support for SHA-512 hashes on publishing/downloads.
2016-02-18 12:01:51 +03:00
Andrey Smirnov
7bb052ac37
Fix unit-tests. #324
2015-12-24 14:08:37 +03:00
Andrey Smirnov
631fe44c6b
Security: don't download files we don't have checksums for. #324
2015-12-22 13:52:53 +03:00
Szymon Sobik
4c51350517
fix EOF during mirror update
...
see http://stackoverflow.com/a/19006050
2015-06-19 12:36:28 +02:00
Andrey Smirnov
1845c493f4
Update mxk/flowcontrol package from Google Code to mxk/flowrate from GitHub.
2015-03-15 14:00:04 +03:00
Andrey Smirnov
1b0eb9d45a
Attempt to fix #189 and #130 : disable Amazon workaround when using proxy.
2015-02-03 21:49:55 +03:00
Andrey Smirnov
552b11e28d
Workaround for '+' escaping in Amazon S3. #130
2014-11-05 02:46:01 +03:00
Andrey Smirnov
4c1d6d1463
Fix operator precedence. #131
2014-10-26 19:53:49 +03:00
Rohan Garg
140a11c04a
Continue even when a server replies with 403
...
Amazon S3 replies with a 403 when accessing files that don't exist,
while this should be fixed at Amazon, we can workaround it in aptly
for the moment.
2014-10-25 21:41:57 +02:00
Andrey Smirnov
7be2ef8b85
Don't fallback between compression methods available unless we get strictly HTTP 404. #129 #125
...
Prior to that, some real errors could have been masked away by that fallback.
2014-10-24 08:51:04 +04:00
Andrey Smirnov
1042894123
Abort downloader on shutdown, don't wait for downloads to finish. #45 #114
2014-10-03 01:31:38 +04:00
Andrey Smirnov
91c753ad2f
Add URL to all download errrors, so that they're easier to understand. #26
2014-10-02 14:12:43 +04:00
Andrey Smirnov
7ae785f5a3
Implementation of ftp:// support for downloading. #48
2014-08-29 19:37:10 +04:00
Andrey Smirnov
94114f2c3d
Use idiomatic chan struct{} when there's nothing to send.
2014-07-24 01:25:59 +04:00
Andrey Smirnov
2906369a3b
Reuse default HTTP transport options.
2014-07-24 01:17:58 +04:00
Andrey Smirnov
83ceee1e3f
Remove debug output. #62
2014-07-13 16:10:53 +04:00
Andrey Smirnov
fb1e28b91b
Setting for downloader to limit download speed to specified level. #62
2014-07-12 23:56:32 +04:00
Andrey Smirnov
53adf39d89
Bring back automatic HTTP_PROXY setting from environment. #46
2014-05-10 13:27:01 +04:00
Andrey Smirnov
6352ce30ed
Disable compression when downloading, otherwise HTTP client may do "decompression" on its own. #33
2014-04-16 15:55:04 +04:00
Andrey Smirnov
aade09e74e
Shutdown progress bar in any case.
2014-03-12 18:13:01 +04:00
Andrey Smirnov
74c88f3ef6
Progress when downloading single files and when parsing remote mirrors.
2014-03-07 00:37:06 +04:00
Andrey Smirnov
2d589bd23d
Refactoring: new packages console, http, Progress is interface.
2014-02-19 13:08:55 +04:00
