This fixes the race condition that happens when you call publish
concurrently. It adds a valuable test that reproduces the error almost
deterministically, it's hard to say always but I have run this in loop
100 times and it reproduces the error consistently without the patch and
after the patch it works consistently.
This adds support for storing packages directly on Azure, with no truly
"local" (on-disk) repo used. The existing Azure PublishedStorage
implementation was refactored to move the shared code to a separate
context struct, which can then be re-used by the new PackagePool. In
addition, the files package's mockChecksumStorage was made public so
that it could be used in the Azure PackagePool tests as well.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Several sections of the code *required* a LocalPackagePool, but they
could still perform their operations with a standard PackagePool.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
The contents of `os.Stat` are rather fitted towards local package pools,
but the method is in the generic PackagePool interface. This moves it to
LocalPackagePool, and the use case of simply finding a file's size is
delegated to a new, more generic PackagePool.Size() method.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
In some local tests w/ a slowed down filesystem, this massively cut down
on the time to clean up a repository by ~3x, bringing a total 'publish
update' time from ~16s to ~13s.
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
When a publishing uses a publish prefix, instead of listing the contents
of the whole bucket under the storage prefix, only list the contents of
the bucket under the storage prefix and publish prefix, and cache it by
publish prefix.
This speeds up publish operations under a prefix.
instead of caching the whole s3 bucket, cache only the pool path. this
requires an additional parameter, and since this is an interface, all
implementations need to follow. might help in other backends too.
closes#1181
presently there is no use case where we need this. on the other hand,
passing empty paths into any of the remove methods is indicative of a bug.
this is particularly dangerous as this can temporarily smash the publish
root but later restore it again when actually publishing. this makes
for super nasty and hard to track down problems.
to guard against this simply disallow root dir removal using empty
strings. should we find a use case for this in the future we can always
revisit this (FTR: I think very explicitly API should be used so everyone
knows what is going on and you can't accidentally run it)
previously it'd return an absolute path which makes the path absolutely
useless as all other functions of PublishedStorage need relative input
and will prepend them with the rootPath, so getting an absolute ReadLink
and then trying to remove that'd would ultimately try to remove the
absolute path `$root/AbsoluteRoot/LinkTarget` instead of `$root/LinkTarget`
add a unit test to actually verify readlink
While updating mirror, if package file is already in pool path,
field `PoolPath` was left as empty which results in package file
being unavailable later on while publishing.
`PackageDownloadTask` is just a reference to file now. Whole process
was rewritten to follow pattern: download to temp location inside the pool,
verify/update checksums, import into pool as final step.
This removes a lot of edge cases when aptly internal state might be broken
if updating from rogue mirror.
Also this changes whole memory model: package list/files are kept in memory
now during the duration of `mirror update` command and saved to disk
only in the end.
Local package pool now implements more generic package pool API.
The base idea is to never expose full paths to files, so that other
kinds of package pools (e.g. package pool in S3) could be used to implement
the same interface.
Files get into the pool only using `Import` method. `Import` method is
now more smart, it supports moving files into the pool, it can detect if
files reside on the same filesystem and use hardlinking instead of copying.
This will make direct mirror downloads still as fast as they were with previous
version which was performing download directly to package pool.
New package pool doesn't have two things implemented yet:
1. New file placement according to SHA256 or other configured hash
2. Calculate at least SHA256/MD5 for each imported files.
MD5 would be required for S3/Swift publishing
This is related to #506
As a first step, don't pass MD5 explicitly, pass checksum info object,
so that as a next step we can choose which hash to use.
There should be no functional changes so far.
Next step: stop returning explicit paths from public package pool.
