- new phony target build: same as install but creating aptly-$version and
putting it into a build/ subdir
- env TRAVIS_TAG in the makefile now overrides the TAG lookup, this ensures
that the tag travis is working with is actually the one being used to
construct the version number
- subdir is gitignored
- travis runs new target - lists artifacts - deploys artifacts to github
all of the above only happens on builds that are a tag and DEPLOY_BINARIES
is set to yes (which is only the case for latest stable go version)
package count chainged again -.-
I am working on a fixture set for repositories so we can stop talking to
live repositories. that's quite the undertaking though, so let's fix the
output reference to unbreak the test in the meantime
Init is actually never called and I have no clue why it is there if it is
not called.
Take this opportunity to introduce a New function which only does the
helper lookup and panics iff that fails. Panic may be a bit too aggressive,
but seems the most certain way to get out of not finding a suitable gpg1
binary.
Newer versions of debian and ubuntu come with gpg pointing to gpg2.
We can currently only handle gpg1 CLIs though. Luckily the old gpg is still
available in the package gnupg1 (providing bin/gpg1).
As a bit of a stop-gap, until #657 can be resolved properly, we'll detect
the version of bin/gpg. If it is unsuitable we'll fall back and try
bin/gpg1. If neither is found to be suitable the signer/verifier will
not work.
Same applies to gpgv/gpgv1.
updates with contents generation were super syscall-heavy. for each path
in a package (so at least 2-4, but ordinarily >4) we'd do a db.Put in
ContentsIndex which results in one syscall.Write. so, for every package in
a published repo we'd have to do *at least* 2 but ordinarily >4 syscalls.
this gets abysmally slow very quickly depending on the available system
specs.
instead, start a batch inside each package and finish it when we are done
with the package. this should keep the memory footprint negligible, but
reduce the write() calls from N to 1.
on one of KDE's servers I have seen update publishing of 7600 packages go
from ~28s to ~9s when using batch putting on an HDD.
on my local system the same set of packages go from ~14s to ~6s on an SSD.
(all inodes in cache in both cases)
presently there is no use case where we need this. on the other hand,
passing empty paths into any of the remove methods is indicative of a bug.
this is particularly dangerous as this can temporarily smash the publish
root but later restore it again when actually publishing. this makes
for super nasty and hard to track down problems.
to guard against this simply disallow root dir removal using empty
strings. should we find a use case for this in the future we can always
revisit this (FTR: I think very explicitly API should be used so everyone
knows what is going on and you can't accidentally run it)
the logic here was wrong.
if we managed to find the link target (the physical index file) pointed to
by our old symlink we want to remove it (this is basically "cleaning up old
index" logic).
previously we'd try to only delete it when the ReadLink came back with
error. which had two serious issues with it:
a) linkTarget was empty, so we basically called Remove("") which would
delete the storage -> root <- directory if the root is a symlink!
b) we'd leak old indexes as the cleanup logic only ran if there was en
error which would ordinarily never be
new code correctly cleans up unless there was an error.
this relates to a previous bugfix of readLink which incorrectly returned
absolute paths ultimately rendering the Remove call also broken.
previously it'd return an absolute path which makes the path absolutely
useless as all other functions of PublishedStorage need relative input
and will prepend them with the rootPath, so getting an absolute ReadLink
and then trying to remove that'd would ultimately try to remove the
absolute path `$root/AbsoluteRoot/LinkTarget` instead of `$root/LinkTarget`
add a unit test to actually verify readlink
- new publish calls can now enable AcquireByHash by right away (previously
one would have had to create a new publishing endpoint and then
explicitly switch it to AcquireByHash)
- all json marshals of PublishedRepo now contain AcquireByHash (allows
inspecting if a given endpoint has AcquireByHash enabled already; also
enables verification that a switch/update actually applied a
potential AcquireByHash change
- update all tests to reflect that default state of AcquireByHash
- update creation and switch testing to explicitly toggle AcquireByHash to
make sure state mutation works as expected
'E722 do not use bare except' wants us not to use except without type
restriction as it catches everything and the kitchen sink. Since we use
them to catch exceptions in test cases this is intentional as we implement
general purpose error handling on test failure there.
upstream switched the alignment check backend and in doing so fails to run
if the old backend is defined in the config.
also skip alignment linting on a struct we use for byte decoding as we have
no choice in its member order.
systemd has a feature called socket activation where initially systemd
manages and listens on ports/uds and only invokes a service when traffic
appears. to then hand over the involved sockets, systemd will pass the
relevant FDs into the invoked process and defines them in the environment.
use coreos/go-systemd to grab the active listeners passed by systemd and
use them to serve the api routes. only one listener may be specified right
now as we also only support one -listen argument for the binary.
this allows admins to craft a systemd socket and service file for aptly
where systemd manages the socket, its permission and its live time, and
lazy start aptly when needed.
`unix://$PATH` as listen argument will bind aptly to a unix domain socket
rather than TCP.
This allows binding the API to a UDS rather than a port.
Since aptly has no concept of authentication or any amount of high level
API hardening one needs to bottle it up in some other manner. Binding
to a localhost port is often a step in the right direction, ultimately is
still a scary insecure setup as any user on that host getting compromised
would mean that the entire archive is compromised as well.
UDS on the other hand are basically files and have their access managed
by regular file permission. As such, binding to a socket is in fact
the least insecure way to listen as you'd have to explicitly open up the
socket permissions to an access qualified group. In the most conservative
scenario that means no one but the aptly user can talk to the API, in a
more practical setup apache might get access as well and proxy the UDS
with authentication or limited to GET operations.
Using UDS allows reducing the attack surface of the API server while
preserving all the flexibility.
When api/graph.{dot,gv} is requested the raw string for dot gets returned.
This allows client-side rendering rather than server-side. It also makes
the optional dependency on graphivz for dot unnecessary to use the graph
endpoint.
Harald Sitter