Commit Graph

420 Commits

Author SHA1 Message Date
Andrey Smirnov
0e9f966dd1 Fix up other code to support new GPG provider structure 2017-07-21 01:01:58 +03:00
Oliver Sauder
e3f1880ad4 Added support for NotAutomatic, ButAutomaticUpgrades and Origin fields 2017-07-05 15:08:02 +02:00
Andrey Smirnov
c13eb99925 Style fixups 2017-07-05 00:36:48 +03:00
Andrey Smirnov
211ac0501f Rework the way database is open/re-open in aptly
Allow database to be initialized without opening, unify all the
open paths to retry on failure.

In API router make sure open requests are matched with acks in explicit
way.

This also enables re-open attempts in all the aptly commands, so it
should make running aptly CLI much easier now hopefully.

Fix up system tests for oldoldstable ;)
2017-07-05 00:17:48 +03:00
Andrey Smirnov
12a6b0ceb8 Merge pull request #575 from smira/pgp-refactoring
Refactor GPG signer/verifier
2017-05-24 19:24:38 +03:00
Andrey Smirnov
cafb89f30f Re-work the way checksum matching works against Release file
Break up URL into base part and relative path. Match checksum against relative path
and never against full URL.

This might be fixing security issue if aptly was incorrectly matching against
wrong part of Release file.
2017-05-23 03:00:15 +03:00
Andrey Smirnov
1be8d39105 Refactor GPG signer/verifier
Goal is to make it easier to plug in another implementation.
2017-05-23 02:54:56 +03:00
Andrey Smirnov
470165a419 Enable goconst & interfacer linters 2017-05-17 00:53:10 +03:00
Andrey Smirnov
5ce6bf8718 Enable vetshadow linter 2017-05-04 23:00:13 +03:00
Andrey Smirnov
51213899b7 More Go linters enabled, issues fixed
Ref: #528

Enables "staticcheck", "varcheck", "structcheck", "aligncheck"
2017-05-03 18:23:14 +03:00
Andrey Smirnov
bae3f949b4 Enable gosimple and ineffasign linters 2017-04-27 18:34:30 +03:00
Andrey Smirnov
186bb2dff0 Add flag to disable/enable support for legacy pool paths
Legacy pool paths are enabled by default, but for new aptly installations
(when aptly config is first generated), it would be disabled explicitly.
2017-04-26 23:37:31 +03:00
Andrey Smirnov
01512df853 Rework mirror update to support closing/reoping DB for the download duration
This requires splitting up import file phase as separate step in then end,
it should be pretty fast, as it only does file move (hardlink) and
DB update for new checksums.
2017-04-26 23:17:04 +03:00
Andrey Smirnov
b48e8425ec Fix bug with file not being updated properly 2017-04-26 23:17:04 +03:00
Andrey Smirnov
3ce8227122 Add baseName to LinkFromPool as explicit argument 2017-04-26 23:17:04 +03:00
Andrey Smirnov
5dd11a2ec2 Pull original packages when skipping existing packages 2017-04-26 23:17:04 +03:00
Andrey Smirnov
10c096fbb6 Update all other pieces for the CheckumStorage and Verify 2017-04-26 23:17:04 +03:00
Andrey Smirnov
6994e35119 ChecksumCollection implementation 2017-04-26 23:17:04 +03:00
Andrey Smirnov
4eedb62418 Fix misspelling 2017-04-26 23:17:04 +03:00
Andrey Smirnov
1f3cb2db5d When downloading/importing packages, enforce all checksums 2017-04-26 23:17:04 +03:00
Andrey Smirnov
c40025a335 Add progress bar on package saving progress 2017-04-26 23:17:03 +03:00
Andrey Smirnov
bc7903f86e Rework mirror update (download packages) implementation
`PackageDownloadTask` is just a reference to file now. Whole process
was rewritten to follow pattern: download to temp location inside the pool,
verify/update checksums, import into pool as final step.

This removes a lot of edge cases when aptly internal state might be broken
if updating from rogue mirror.

Also this changes whole memory model: package list/files are kept in memory
now during the duration of `mirror update` command and saved to disk
only in the end.
2017-04-26 23:17:03 +03:00
Andrey Smirnov
72d233b587 Final round of updates, everything except mirror download should be ready 2017-04-26 23:17:03 +03:00
Andrey Smirnov
8dc6a14766 Allow snapshot to be created from empty local repo
Fixes #288
2017-04-26 00:33:09 +03:00
Clemens Rabe
25f9c29f00 Implemented filesystem endpoint with support for hardlinks, symlinks and copy. 2017-04-13 20:25:40 +02:00
Andrey Smirnov
60800b5f25 Fix temporary contents DB being left behind after publishing
NB: Go `defer` order execution is reverse to the order `defer` statements
are executed.

So before the change, `Drop()` was called before `Close()`, which was no-op.

Change that to explicit order in single func, print errors if they happen.
2017-04-10 23:43:33 +03:00
Andrey Smirnov
50cf2b49bd Refactoring: use checksums instead of MD5 for pool/published
This is related to #506

As a first step, don't pass MD5 explicitly, pass checksum info object,
so that as a next step we can choose which hash to use.

There should be no functional changes so far.

Next step: stop returning explicit paths from public package pool.
2017-04-01 00:12:31 +03:00
Andrey Smirnov
bc469eecfb Merge branch 'master' into dep-verbose-resolve 2017-03-31 19:17:57 +03:00
Andrey Smirnov
7a5be6736d When contents generation fails, don't bail out
This replaces `panic` which aborts aptly execution with warning
message on console. So aptly continues publishing actions, but
`Contents` indexes might be incomplete.

Error will be printed every time contents generation is triggered.
2017-03-31 00:57:18 +03:00
Andrey Smirnov
85b4a8b1ae Add new option for detailed logging on dependency resolving
This adds command-line arg and config option, with option enabled
aptly is more verbose on internal depeendency resolving cycles:

```
Missing dependencies: file-rc (>= 0.8.16) [amd64], python:any (>= 2.7.1-0ubuntu2) [amd64], python3:any (>= 3.3.2-2~) [amd64], file-rc [amd64], perl (<< 5.17) [amd64], iptables-router (>= 1.2.3) [amd64], systemd [amd64], sgml-base (>= 1.26+nmu2) [amd64], sed (>= 4.1.2-8) [amd64]
Unsatisfied dependency: file-rc (>= 0.8.16) [amd64]
Unsatisfied dependency: python:any (>= 2.7.1-0ubuntu2) [amd64]
Unsatisfied dependency: python3:any (>= 3.3.2-2~) [amd64]
Unsatisfied dependency: file-rc [amd64]
Unsatisfied dependency: perl (<< 5.17) [amd64]
Unsatisfied dependency: iptables-router (>= 1.2.3) [amd64]
Unsatisfied dependency: systemd [amd64]
Injecting package: sgml-base_1.26+nmu4ubuntu1_all
Injecting package: sed_4.2.2-4ubuntu1_amd64
```
2017-03-28 22:58:07 +03:00
Andrey Smirnov
800c5c1e06 Merge branch 'master' into feature_skip_existing_packages_latest 2017-03-28 21:26:28 +03:00
Andrey Smirnov
2943422d5d Automatic versioning for aptly
New version format:

* for releases, `x.y.z` (follows tag without leading `v`)
* for nightly builds, `x.y.z+N+hash` (previous version, not the upcoming one)

This means that each nightly build `aptly` would report
correct version now.

Version is now complied into the aptly binary, system tests
automatically check for current version, no need to update them
anymore.
2017-03-25 00:18:45 +03:00
Clemens Rabe
aa16899c60 Adaption of tests. 2017-03-24 06:25:46 +01:00
Clemens Rabe
16a0d0d428 Added option --skip-existing-packages to speed up mirror update. 2017-03-23 22:01:11 +01:00
Clemens Rabe
66f51d2b17 Added option --skip-existing-packages to speed up mirror update. 2017-03-23 21:55:22 +01:00
Andrey Smirnov
516dd7b044 Switch to gometalinter
Only small amount of required checks is enabled,
plan is to enable more linters as issues are fixed in the code.
2017-03-23 01:51:08 +03:00
Andrey Smirnov
11d828b3b1 Add govet/golint into Travis CI build
Fix current issues
2017-03-22 21:49:16 +03:00
Raphael Medaer
bfb9ffad1d Added expected error on 'Packages.xz' for TestDownload[WithSources]Flat. 2017-03-16 22:41:25 +03:00
Szymon Sobik
b8e7ad9022 update changes unit test to account for dbgsym matching 2017-03-08 10:32:11 +01:00
Szymon Sobik
1b80d55ea4 since -dbgsym is for each binary package use that for PackageQuery 2017-03-08 10:31:44 +01:00
Szymon Sobik
a0832adfa5 include dbgsym packages
fixes #331
2017-03-07 17:06:59 +01:00
Andrey Smirnov
4c06e26d85 Throttle compaction on temporary DB 2017-02-23 01:01:17 +03:00
Andrey Smirnov
f2dc4eeec9 Generating contents indexes via temporary LevelDB 2017-02-21 19:09:51 +03:00
Andrey Smirnov
33d6cd8c0a Allow filter to be empty for aptly * search commands
Empty filter implies "select all packages".
2017-02-10 23:07:06 +03:00
Andrey Smirnov
c75d4c749c Add workaround for reading data.tar.gz as data.tar
It seems that in the wild there are .deb package which have
`data.tar.gz` which is actually `.tar` archive.

Add magic detection based on signature.
2017-01-24 19:30:53 +03:00
jolo
b0ab39e07f Manually undo unintended changes 2017-01-20 02:19:44 +01:00
jola5
4bf27d1dae Merge branch 'master' into master 2017-01-19 23:07:49 +01:00
Andrey Smirnov
d5df049630 Sort paths when generating checksums for Release/InRelease 2017-01-18 23:50:22 +03:00
jolo
91561b40f6 Change 'vertical' argument to a more generic 'layout', fix api 2017-01-16 22:13:13 +01:00
jolo
0e8ea6363a Support vertical graph layouts 2017-01-14 02:18:56 +01:00