vincent/aptly
1
0
Fork 0
mirror of https://github.com/aptly-dev/aptly.git synced 2026-03-30 16:08:13 +00:00
Code Issues Projects Releases Wiki Activity
Files
2a07494910ff07390874448a50eb68d3499a94de
aptly/deb/remote_test.go
Ryan Gonzalez 19a705f80d Split reflists to share their contents across snapshots 
In current aptly, each repository and snapshot has its own reflist in
the database. This brings a few problems with it:

- Given a sufficiently large repositories and snapshots, these lists can
  get enormous, reaching >1MB. This is a problem for LevelDB's overall
  performance, as it tends to prefer values around the confiruged block
  size (defaults to just 4KiB).
- When you take these large repositories and snapshot them, you have a
  full, new copy of the reflist, even if only a few packages changed.
  This means that having a lot of snapshots with a few changes causes
  the database to basically be full of largely duplicate reflists.
- All the duplication also means that many of the same refs are being
  loaded repeatedly, which can cause some slowdown but, more notably,
  eats up huge amounts of memory.
- Adding on more and more new repositories and snapshots will cause the
  time and memory spent on things like cleanup and publishing to grow
  roughly linearly.

At the core, there are two problems here:

- Reflists get very big because there are just a lot of packages.
- Different reflists can tend to duplicate much of the same contents.

*Split reflists* aim at solving this by separating reflists into 64
*buckets*. Package refs are sorted into individual buckets according to
the following system:

- Take the first 3 letters of the package name, after dropping a `lib`
  prefix. (Using only the first 3 letters will cause packages with
  similar prefixes to end up in the same bucket, under the assumption
  that packages with similar names tend to be updated together.)
- Take the 64-bit xxhash of these letters. (xxhash was chosen because it
  relatively good distribution across the individual bits, which is
  important for the next step.)
- Use the first 6 bits of the hash (range [0:63]) as an index into the
  buckets.

Once refs are placed in buckets, a sha256 digest of all the refs in the
bucket is taken. These buckets are then stored in the database, split
into roughly block-sized segments, and all the repositories and
snapshots simply store an array of bucket digests.

This approach means that *repositories and snapshots can share their
reflist buckets*. If a snapshot is taken of a repository, it will have
the same contents, so its split reflist will point to the same buckets
as the base repository, and only one copy of each bucket is stored in
the database. When some packages in the repository change, only the
buckets containing those packages will be modified; all the other
buckets will remain unchanged, and thus their contents will still be
shared. Later on, when these reflists are loaded, each bucket is only
loaded once, short-cutting loaded many megabytes of data. In effect,
split reflists are essentially copy-on-write, with only the changed
buckets stored individually.

Changing the disk format means that a migration needs to take place, so
that task is moved into the database cleanup step, which will migrate
reflists over to split reflists, as well as delete any unused reflist
buckets.

All the reflist tests are also changed to additionally test out split
reflists; although the internal logic is all shared (since buckets are,
themselves, just normal reflists), some special additions are needed to
have native versions of the various reflist helper methods.

In our tests, we've observed the following improvements:

- Memory usage during publish and database cleanup, with
  `GOMEMLIMIT=2GiB`, goes down from ~3.2GiB (larger than the memory
  limit!) to ~0.7GiB, a decrease of ~4.5x.
- Database size decreases from 1.3GB to 367MB.

*In my local tests*, publish times had also decreased down to mere
seconds but the same effect wasn't observed on the server, with the
times staying around the same. My suspicions are that this is due to I/O
performance: my local system is an M1 MBP, which almost certainly has
much faster disk speeds than our DigitalOcean block volumes. Split
reflists include a side effect of requiring more random accesses from
reading all the buckets by their keys, so if your random I/O
performance is slower, it might cancel out the benefits. That being
said, even in that case, the memory usage and database size advantages
still persist.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2025-02-15 23:49:21 +01:00

961 lines
47 KiB
Go
Raw Blame History

package deb
import (
"errors"
"io"
"os"
"sort"
"github.com/aptly-dev/aptly/aptly"
"github.com/aptly-dev/aptly/console"
"github.com/aptly-dev/aptly/database"
"github.com/aptly-dev/aptly/database/goleveldb"
"github.com/aptly-dev/aptly/files"
"github.com/aptly-dev/aptly/http"
"github.com/aptly-dev/aptly/pgp"
"github.com/aptly-dev/aptly/utils"
. "gopkg.in/check.v1"
)
type NullVerifier struct {
}
func (n *NullVerifier) InitKeyring(_ bool) error {
return nil
}
func (n *NullVerifier) AddKeyring(keyring string) {
}
func (n *NullVerifier) VerifyDetachedSignature(signature, cleartext io.Reader, hint bool) error {
return nil
}
func (n *NullVerifier) VerifyClearsigned(clearsigned io.Reader, hint bool) (*pgp.KeyInfo, error) {
return nil, nil
}
func (n *NullVerifier) ExtractClearsigned(clearsigned io.Reader) (text *os.File, err error) {
text, _ = os.CreateTemp("", "aptly-test")
io.Copy(text, clearsigned)
text.Seek(0, 0)
os.Remove(text.Name())
return
}
func (n *NullVerifier) IsClearSigned(clearsign io.Reader) (bool, error) {
return false, nil
}
type PackageListMixinSuite struct {
p1, p2, p3 *Package
list *PackageList
reflist *SplitRefList
}
func (s *PackageListMixinSuite) SetUpPackages() {
s.list = NewPackageList()
s.p1 = NewPackageFromControlFile(packageStanza.Copy())
stanza := packageStanza.Copy()
stanza["Package"] = "mars-invaders"
stanza["Filename"] = "pool/contrib/m/mars-invaders/mars-invaders_7.40-2_i386.deb"
s.p2 = NewPackageFromControlFile(stanza)
stanza = packageStanza.Copy()
stanza["Package"] = "lonely-strangers"
stanza["Filename"] = "pool/contrib/l/lonely-strangers/lonely-strangers_7.40-2_i386.deb"
s.p3 = NewPackageFromControlFile(stanza)
s.list.Add(s.p1)
s.list.Add(s.p2)
s.list.Add(s.p3)
s.reflist = NewSplitRefListFromPackageList(s.list)
}
type RemoteRepoSuite struct {
PackageListMixinSuite
repo *RemoteRepo
flat *RemoteRepo
downloader *http.FakeDownloader
progress aptly.Progress
db database.Storage
collectionFactory *CollectionFactory
packagePool aptly.PackagePool
cs aptly.ChecksumStorage
}
var _ = Suite(&RemoteRepoSuite{})
func (s *RemoteRepoSuite) SetUpTest(c *C) {
s.repo, _ = NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian", "squeeze", []string{"main"}, []string{}, false, false, false)
s.flat, _ = NewRemoteRepo("exp42", "http://repos.express42.com/virool/precise/", "./", []string{}, []string{}, false, false, false)
s.downloader = http.NewFakeDownloader().ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/Release", exampleReleaseFile)
s.progress = console.NewProgress(false)
s.db, _ = goleveldb.NewOpenDB(c.MkDir())
s.collectionFactory = NewCollectionFactory(s.db)
s.packagePool = files.NewPackagePool(c.MkDir(), false)
s.cs = files.NewMockChecksumStorage()
s.SetUpPackages()
s.progress.Start()
}
func (s *RemoteRepoSuite) TearDownTest(c *C) {
s.progress.Shutdown()
s.db.Close()
}
func (s *RemoteRepoSuite) TestInvalidURL(c *C) {
_, err := NewRemoteRepo("s", "http://lolo%2", "squeeze", []string{"main"}, []string{}, false, false, false)
c.Assert(err, ErrorMatches, ".*(hexadecimal escape in host|percent-encoded characters in host|invalid URL escape).*")
}
func (s *RemoteRepoSuite) TestFlatCreation(c *C) {
c.Check(s.flat.IsFlat(), Equals, true)
c.Check(s.flat.Distribution, Equals, "./")
c.Check(s.flat.Components, IsNil)
flat2, _ := NewRemoteRepo("flat2", "http://pkg.jenkins-ci.org/debian-stable", "binary/", []string{}, []string{}, false, false, false)
c.Check(flat2.IsFlat(), Equals, true)
c.Check(flat2.Distribution, Equals, "./binary/")
_, err := NewRemoteRepo("fl", "http://some.repo/", "./", []string{"main"}, []string{}, false, false, false)
c.Check(err, ErrorMatches, "components aren't supported for flat repos")
}
func (s *RemoteRepoSuite) TestString(c *C) {
c.Check(s.repo.String(), Equals, "[yandex]: http://mirror.yandex.ru/debian/ squeeze")
c.Check(s.flat.String(), Equals, "[exp42]: http://repos.express42.com/virool/precise/ ./")
s.repo.DownloadSources = true
s.repo.DownloadUdebs = true
s.repo.DownloadInstaller = true
s.flat.DownloadSources = true
c.Check(s.repo.String(), Equals, "[yandex]: http://mirror.yandex.ru/debian/ squeeze [src] [udeb] [installer]")
c.Check(s.flat.String(), Equals, "[exp42]: http://repos.express42.com/virool/precise/ ./ [src]")
}
func (s *RemoteRepoSuite) TestNumPackages(c *C) {
c.Check(s.repo.NumPackages(), Equals, 0)
s.repo.packageRefs = s.reflist
c.Check(s.repo.NumPackages(), Equals, 3)
}
func (s *RemoteRepoSuite) TestIsFlat(c *C) {
c.Check(s.repo.IsFlat(), Equals, false)
c.Check(s.flat.IsFlat(), Equals, true)
}
func (s *RemoteRepoSuite) TestRefList(c *C) {
s.repo.packageRefs = s.reflist
c.Check(s.repo.RefList(), Equals, s.reflist)
}
func (s *RemoteRepoSuite) TestReleaseURL(c *C) {
c.Assert(s.repo.ReleaseURL("Release").String(), Equals, "http://mirror.yandex.ru/debian/dists/squeeze/Release")
c.Assert(s.repo.ReleaseURL("InRelease").String(), Equals, "http://mirror.yandex.ru/debian/dists/squeeze/InRelease")
c.Assert(s.flat.ReleaseURL("Release").String(), Equals, "http://repos.express42.com/virool/precise/Release")
}
func (s *RemoteRepoSuite) TestIndexesRootURL(c *C) {
c.Assert(s.repo.IndexesRootURL().String(), Equals, "http://mirror.yandex.ru/debian/dists/squeeze/")
c.Assert(s.flat.IndexesRootURL().String(), Equals, "http://repos.express42.com/virool/precise/")
}
func (s *RemoteRepoSuite) TestBinaryPath(c *C) {
c.Assert(s.repo.BinaryPath("main", "amd64"), Equals, "main/binary-amd64/Packages")
}
func (s *RemoteRepoSuite) TestUdebPath(c *C) {
c.Assert(s.repo.UdebPath("main", "amd64"), Equals, "main/debian-installer/binary-amd64/Packages")
}
func (s *RemoteRepoSuite) TestSourcesPath(c *C) {
c.Assert(s.repo.SourcesPath("main"), Equals, "main/source/Sources")
}
func (s *RemoteRepoSuite) TestInstallerPath(c *C) {
c.Assert(s.repo.InstallerPath("main", "amd64"), Equals, "main/installer-amd64/current/images/SHA256SUMS")
}
func (s *RemoteRepoSuite) TestFlatBinaryPath(c *C) {
c.Assert(s.flat.FlatBinaryPath(), Equals, "Packages")
}
func (s *RemoteRepoSuite) TestFlatSourcesPath(c *C) {
c.Assert(s.flat.FlatSourcesPath(), Equals, "Sources")
}
func (s *RemoteRepoSuite) TestPackageURL(c *C) {
c.Assert(s.repo.PackageURL("pool/main/0/0ad/0ad_0~r11863-2_i386.deb").String(), Equals,
"http://mirror.yandex.ru/debian/pool/main/0/0ad/0ad_0~r11863-2_i386.deb")
}
func (s *RemoteRepoSuite) TestFetch(c *C) {
err := s.repo.Fetch(s.downloader, nil, true)
c.Assert(err, IsNil)
c.Assert(s.repo.Architectures, DeepEquals, []string{"amd64", "armel", "armhf", "i386", "powerpc"})
c.Assert(s.repo.Components, DeepEquals, []string{"main"})
c.Assert(s.downloader.Empty(), Equals, true)
c.Check(s.repo.ReleaseFiles, HasLen, 39)
c.Check(s.repo.ReleaseFiles["main/binary-i386/Packages.bz2"], DeepEquals,
utils.ChecksumInfo{
Size: 734,
MD5: "7954ed80936429687122b554620c1b5b",
SHA1: "95a463a0739bf9ff622c8d68f6e4598d400f5248",
SHA256: "377890a26f99db55e117dfc691972dcbbb7d8be1630c8fc8297530c205377f2b"})
}
func (s *RemoteRepoSuite) TestFetchNullVerifier1(c *C) {
downloader := http.NewFakeDownloader()
downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/InRelease", &http.Error{Code: 404})
downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/Release", exampleReleaseFile)
downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/Release.gpg", "GPG")
err := s.repo.Fetch(downloader, &NullVerifier{}, false)
c.Assert(err, IsNil)
c.Assert(s.repo.Architectures, DeepEquals, []string{"amd64", "armel", "armhf", "i386", "powerpc"})
c.Assert(s.repo.Components, DeepEquals, []string{"main"})
c.Assert(downloader.Empty(), Equals, true)
}
func (s *RemoteRepoSuite) TestFetchNullVerifier2(c *C) {
downloader := http.NewFakeDownloader()
downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/InRelease", exampleReleaseFile)
err := s.repo.Fetch(downloader, &NullVerifier{}, false)
c.Assert(err, IsNil)
c.Assert(s.repo.Architectures, DeepEquals, []string{"amd64", "armel", "armhf", "i386", "powerpc"})
c.Assert(s.repo.Components, DeepEquals, []string{"main"})
c.Assert(downloader.Empty(), Equals, true)
}
func (s *RemoteRepoSuite) TestFetchWrongArchitecture(c *C) {
s.repo, _ = NewRemoteRepo("s", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{"xyz"}, false, false, false)
err := s.repo.Fetch(s.downloader, nil, true)
c.Assert(err, ErrorMatches, "architecture xyz not available in repo.*")
}
func (s *RemoteRepoSuite) TestFetchWrongComponent(c *C) {
s.repo, _ = NewRemoteRepo("s", "http://mirror.yandex.ru/debian/", "squeeze", []string{"xyz"}, []string{"i386"}, false, false, false)
err := s.repo.Fetch(s.downloader, nil, true)
c.Assert(err, ErrorMatches, "component xyz not available in repo.*")
}
func (s *RemoteRepoSuite) TestEncodeDecode(c *C) {
repo := &RemoteRepo{}
err := repo.Decode(s.repo.Encode())
c.Assert(err, IsNil)
c.Check(repo.Name, Equals, "yandex")
c.Check(repo.ArchiveRoot, Equals, "http://mirror.yandex.ru/debian/")
}
func (s *RemoteRepoSuite) TestKey(c *C) {
c.Assert(len(s.repo.Key()), Equals, 37)
c.Assert(s.repo.Key()[0], Equals, byte('R'))
}
func (s *RemoteRepoSuite) TestRefKey(c *C) {
c.Assert(len(s.repo.RefKey()), Equals, 37)
c.Assert(s.repo.RefKey()[0], Equals, byte('E'))
c.Assert(s.repo.RefKey()[1:], DeepEquals, s.repo.Key()[1:])
}
func (s *RemoteRepoSuite) TestDownload(c *C) {
s.repo.Architectures = []string{"i386"}
err := s.repo.Fetch(s.downloader, nil, true)
c.Assert(err, IsNil)
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.bz2", &http.Error{Code: 404})
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.gz", &http.Error{Code: 404})
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages", examplePackagesFile)
err = s.repo.DownloadPackageIndexes(s.progress, s.downloader, nil, s.collectionFactory, true, false)
c.Assert(err, IsNil)
c.Assert(s.downloader.Empty(), Equals, true)
queue, size, err := s.repo.BuildDownloadQueue(s.packagePool, s.collectionFactory.PackageCollection(), s.cs, false)
c.Assert(err, IsNil)
c.Check(size, Equals, int64(3))
c.Check(queue, HasLen, 1)
c.Check(queue[0].File.DownloadURL(), Equals, "pool/main/a/amanda/amanda-client_3.3.1-3~bpo60+1_amd64.deb")
s.repo.FinalizeDownload(s.collectionFactory, nil)
c.Assert(s.repo.packageRefs, NotNil)
pkg, err := s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Flatten().Refs[0])
c.Assert(err, IsNil)
c.Check(pkg.Name, Equals, "amanda-client")
// Next call must return an empty download list with option "skip-existing-packages"
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/Release", exampleReleaseFile)
err = s.repo.Fetch(s.downloader, nil, true)
c.Assert(err, IsNil)
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.bz2", &http.Error{Code: 404})
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.gz", &http.Error{Code: 404})
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages", examplePackagesFile)
err = s.repo.DownloadPackageIndexes(s.progress, s.downloader, nil, s.collectionFactory, true, false)
c.Assert(err, IsNil)
c.Assert(s.downloader.Empty(), Equals, true)
queue, size, err = s.repo.BuildDownloadQueue(s.packagePool, s.collectionFactory.PackageCollection(), s.cs, true)
c.Assert(err, IsNil)
c.Check(size, Equals, int64(0))
c.Check(queue, HasLen, 0)
s.repo.FinalizeDownload(s.collectionFactory, nil)
c.Assert(s.repo.packageRefs, NotNil)
// Next call must return the download list without option "skip-existing-packages"
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/Release", exampleReleaseFile)
err = s.repo.Fetch(s.downloader, nil, true)
c.Assert(err, IsNil)
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.bz2", &http.Error{Code: 404})
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.gz", &http.Error{Code: 404})
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages", examplePackagesFile)
err = s.repo.DownloadPackageIndexes(s.progress, s.downloader, nil, s.collectionFactory, true, false)
c.Assert(err, IsNil)
c.Assert(s.downloader.Empty(), Equals, true)
queue, size, err = s.repo.BuildDownloadQueue(s.packagePool, s.collectionFactory.PackageCollection(), s.cs, false)
c.Assert(err, IsNil)
c.Check(size, Equals, int64(3))
c.Check(queue, HasLen, 1)
c.Check(queue[0].File.DownloadURL(), Equals, "pool/main/a/amanda/amanda-client_3.3.1-3~bpo60+1_amd64.deb")
s.repo.FinalizeDownload(s.collectionFactory, nil)
c.Assert(s.repo.packageRefs, NotNil)
}
func (s *RemoteRepoSuite) TestDownloadWithInstaller(c *C) {
s.repo.Architectures = []string{"i386"}
s.repo.DownloadInstaller = true
err := s.repo.Fetch(s.downloader, nil, true)
c.Assert(err, IsNil)
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.bz2", &http.Error{Code: 404})
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.gz", &http.Error{Code: 404})
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages", examplePackagesFile)
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/main/installer-i386/current/images/SHA256SUMS", exampleInstallerHashSumFile)
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/main/installer-i386/current/images/MANIFEST", exampleInstallerManifestFile)
err = s.repo.DownloadPackageIndexes(s.progress, s.downloader, nil, s.collectionFactory, true, false)
c.Assert(err, IsNil)
c.Assert(s.downloader.Empty(), Equals, true)
queue, size, err := s.repo.BuildDownloadQueue(s.packagePool, s.collectionFactory.PackageCollection(), s.cs, false)
c.Assert(err, IsNil)
c.Check(size, Equals, int64(3)+int64(len(exampleInstallerManifestFile)))
c.Check(queue, HasLen, 2)
q := make([]string, 2)
for i := range q {
q[i] = queue[i].File.DownloadURL()
}
sort.Strings(q)
c.Check(q[0], Equals, "dists/squeeze/main/installer-i386/current/images/MANIFEST")
c.Check(q[1], Equals, "pool/main/a/amanda/amanda-client_3.3.1-3~bpo60+1_amd64.deb")
s.repo.FinalizeDownload(s.collectionFactory, nil)
c.Assert(s.repo.packageRefs, NotNil)
pkg, err := s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Flatten().Refs[0])
c.Assert(err, IsNil)
c.Check(pkg.Name, Equals, "amanda-client")
pkg, err = s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Flatten().Refs[1])
c.Assert(err, IsNil)
c.Check(pkg.Name, Equals, "installer")
}
func (s *RemoteRepoSuite) TestDownloadWithSources(c *C) {
s.repo.Architectures = []string{"i386"}
s.repo.DownloadSources = true
err := s.repo.Fetch(s.downloader, nil, true)
c.Assert(err, IsNil)
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.bz2", &http.Error{Code: 404})
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.gz", &http.Error{Code: 404})
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages", examplePackagesFile)
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/source/Sources.bz2", &http.Error{Code: 404})
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/source/Sources.gz", &http.Error{Code: 404})
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/main/source/Sources", exampleSourcesFile)
err = s.repo.DownloadPackageIndexes(s.progress, s.downloader, nil, s.collectionFactory, true, false)
c.Assert(err, IsNil)
c.Assert(s.downloader.Empty(), Equals, true)
queue, size, err := s.repo.BuildDownloadQueue(s.packagePool, s.collectionFactory.PackageCollection(), s.cs, false)
c.Assert(err, IsNil)
c.Check(size, Equals, int64(15))
c.Check(queue, HasLen, 4)
q := make([]string, 4)
for i := range q {
q[i] = queue[i].File.DownloadURL()
}
sort.Strings(q)
c.Check(q[3], Equals, "pool/main/a/amanda/amanda-client_3.3.1-3~bpo60+1_amd64.deb")
c.Check(q[1], Equals, "pool/main/a/access-modifier-checker/access-modifier-checker_1.0-4.dsc")
c.Check(q[2], Equals, "pool/main/a/access-modifier-checker/access-modifier-checker_1.0.orig.tar.gz")
c.Check(q[0], Equals, "pool/main/a/access-modifier-checker/access-modifier-checker_1.0-4.debian.tar.gz")
s.repo.FinalizeDownload(s.collectionFactory, nil)
c.Assert(s.repo.packageRefs, NotNil)
pkg, err := s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Flatten().Refs[0])
c.Assert(err, IsNil)
c.Check(pkg.Name, Equals, "amanda-client")
pkg, err = s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Flatten().Refs[1])
c.Assert(err, IsNil)
c.Check(pkg.Name, Equals, "access-modifier-checker")
// Next call must return an empty download list with option "skip-existing-packages"
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/Release", exampleReleaseFile)
err = s.repo.Fetch(s.downloader, nil, true)
c.Assert(err, IsNil)
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.bz2", &http.Error{Code: 404})
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.gz", &http.Error{Code: 404})
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages", examplePackagesFile)
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/source/Sources.bz2", &http.Error{Code: 404})
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/source/Sources.gz", &http.Error{Code: 404})
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/main/source/Sources", exampleSourcesFile)
err = s.repo.DownloadPackageIndexes(s.progress, s.downloader, nil, s.collectionFactory, true, false)
c.Assert(err, IsNil)
c.Assert(s.downloader.Empty(), Equals, true)
queue, size, err = s.repo.BuildDownloadQueue(s.packagePool, s.collectionFactory.PackageCollection(), s.cs, true)
c.Assert(err, IsNil)
c.Check(size, Equals, int64(0))
c.Check(queue, HasLen, 0)
s.repo.FinalizeDownload(s.collectionFactory, nil)
c.Assert(s.repo.packageRefs, NotNil)
// Next call must return the download list without option "skip-existing-packages"
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/Release", exampleReleaseFile)
err = s.repo.Fetch(s.downloader, nil, true)
c.Assert(err, IsNil)
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.bz2", &http.Error{Code: 404})
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages.gz", &http.Error{Code: 404})
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/main/binary-i386/Packages", examplePackagesFile)
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/source/Sources.bz2", &http.Error{Code: 404})
s.downloader.ExpectError("http://mirror.yandex.ru/debian/dists/squeeze/main/source/Sources.gz", &http.Error{Code: 404})
s.downloader.ExpectResponse("http://mirror.yandex.ru/debian/dists/squeeze/main/source/Sources", exampleSourcesFile)
err = s.repo.DownloadPackageIndexes(s.progress, s.downloader, nil, s.collectionFactory, true, false)
c.Assert(err, IsNil)
c.Assert(s.downloader.Empty(), Equals, true)
queue, size, err = s.repo.BuildDownloadQueue(s.packagePool, s.collectionFactory.PackageCollection(), s.cs, false)
c.Assert(err, IsNil)
c.Check(size, Equals, int64(15))
c.Check(queue, HasLen, 4)
s.repo.FinalizeDownload(s.collectionFactory, nil)
c.Assert(s.repo.packageRefs, NotNil)
}
func (s *RemoteRepoSuite) TestDownloadFlat(c *C) {
downloader := http.NewFakeDownloader()
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Release", exampleReleaseFile)
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.bz2", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.gz", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.xz", &http.Error{Code: 404})
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Packages", examplePackagesFile)
err := s.flat.Fetch(downloader, nil, true)
c.Assert(err, IsNil)
err = s.flat.DownloadPackageIndexes(s.progress, downloader, nil, s.collectionFactory, true, true)
c.Assert(err, IsNil)
c.Assert(downloader.Empty(), Equals, true)
queue, size, err := s.flat.BuildDownloadQueue(s.packagePool, s.collectionFactory.PackageCollection(), s.cs, false)
c.Assert(err, IsNil)
c.Check(size, Equals, int64(3))
c.Check(queue, HasLen, 1)
c.Check(queue[0].File.DownloadURL(), Equals, "pool/main/a/amanda/amanda-client_3.3.1-3~bpo60+1_amd64.deb")
s.flat.FinalizeDownload(s.collectionFactory, nil)
c.Assert(s.flat.packageRefs, NotNil)
pkg, err := s.collectionFactory.PackageCollection().ByKey(s.flat.packageRefs.Flatten().Refs[0])
c.Assert(err, IsNil)
c.Check(pkg.Name, Equals, "amanda-client")
// Next call must return an empty download list with option "skip-existing-packages"
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Release", exampleReleaseFile)
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.bz2", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.gz", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.xz", &http.Error{Code: 404})
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Packages", examplePackagesFile)
err = s.flat.Fetch(downloader, nil, true)
c.Assert(err, IsNil)
err = s.flat.DownloadPackageIndexes(s.progress, downloader, nil, s.collectionFactory, true, true)
c.Assert(err, IsNil)
c.Assert(downloader.Empty(), Equals, true)
queue, size, err = s.flat.BuildDownloadQueue(s.packagePool, s.collectionFactory.PackageCollection(), s.cs, true)
c.Assert(err, IsNil)
c.Check(size, Equals, int64(0))
c.Check(queue, HasLen, 0)
s.flat.FinalizeDownload(s.collectionFactory, nil)
c.Assert(s.flat.packageRefs, NotNil)
// Next call must return the download list without option "skip-existing-packages"
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Release", exampleReleaseFile)
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.bz2", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.gz", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.xz", &http.Error{Code: 404})
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Packages", examplePackagesFile)
err = s.flat.Fetch(downloader, nil, true)
c.Assert(err, IsNil)
err = s.flat.DownloadPackageIndexes(s.progress, downloader, nil, s.collectionFactory, true, true)
c.Assert(err, IsNil)
c.Assert(downloader.Empty(), Equals, true)
queue, size, err = s.flat.BuildDownloadQueue(s.packagePool, s.collectionFactory.PackageCollection(), s.cs, false)
c.Assert(err, IsNil)
c.Check(size, Equals, int64(3))
c.Check(queue, HasLen, 1)
c.Check(queue[0].File.DownloadURL(), Equals, "pool/main/a/amanda/amanda-client_3.3.1-3~bpo60+1_amd64.deb")
s.flat.FinalizeDownload(s.collectionFactory, nil)
c.Assert(s.flat.packageRefs, NotNil)
}
func (s *RemoteRepoSuite) TestDownloadWithSourcesFlat(c *C) {
s.flat.DownloadSources = true
downloader := http.NewFakeDownloader()
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Release", exampleReleaseFile)
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.bz2", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.gz", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.xz", &http.Error{Code: 404})
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Packages", examplePackagesFile)
downloader.ExpectError("http://repos.express42.com/virool/precise/Sources.bz2", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Sources.gz", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Sources.xz", &http.Error{Code: 404})
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Sources", exampleSourcesFile)
err := s.flat.Fetch(downloader, nil, true)
c.Assert(err, IsNil)
err = s.flat.DownloadPackageIndexes(s.progress, downloader, nil, s.collectionFactory, true, true)
c.Assert(err, IsNil)
c.Assert(downloader.Empty(), Equals, true)
queue, size, err := s.flat.BuildDownloadQueue(s.packagePool, s.collectionFactory.PackageCollection(), s.cs, false)
c.Assert(err, IsNil)
c.Check(size, Equals, int64(15))
c.Check(queue, HasLen, 4)
q := make([]string, 4)
for i := range q {
q[i] = queue[i].File.DownloadURL()
}
sort.Strings(q)
c.Check(q[3], Equals, "pool/main/a/amanda/amanda-client_3.3.1-3~bpo60+1_amd64.deb")
c.Check(q[1], Equals, "pool/main/a/access-modifier-checker/access-modifier-checker_1.0-4.dsc")
c.Check(q[2], Equals, "pool/main/a/access-modifier-checker/access-modifier-checker_1.0.orig.tar.gz")
c.Check(q[0], Equals, "pool/main/a/access-modifier-checker/access-modifier-checker_1.0-4.debian.tar.gz")
s.flat.FinalizeDownload(s.collectionFactory, nil)
c.Assert(s.flat.packageRefs, NotNil)
pkg, err := s.collectionFactory.PackageCollection().ByKey(s.flat.packageRefs.Flatten().Refs[0])
c.Assert(err, IsNil)
c.Check(pkg.Name, Equals, "amanda-client")
pkg, err = s.collectionFactory.PackageCollection().ByKey(s.flat.packageRefs.Flatten().Refs[1])
c.Assert(err, IsNil)
c.Check(pkg.Name, Equals, "access-modifier-checker")
// Next call must return an empty download list with option "skip-existing-packages"
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Release", exampleReleaseFile)
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.bz2", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.gz", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.xz", &http.Error{Code: 404})
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Packages", examplePackagesFile)
downloader.ExpectError("http://repos.express42.com/virool/precise/Sources.bz2", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Sources.gz", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Sources.xz", &http.Error{Code: 404})
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Sources", exampleSourcesFile)
err = s.flat.Fetch(downloader, nil, true)
c.Assert(err, IsNil)
err = s.flat.DownloadPackageIndexes(s.progress, downloader, nil, s.collectionFactory, true, true)
c.Assert(err, IsNil)
c.Assert(downloader.Empty(), Equals, true)
queue, size, err = s.flat.BuildDownloadQueue(s.packagePool, s.collectionFactory.PackageCollection(), s.cs, true)
c.Assert(err, IsNil)
c.Check(size, Equals, int64(0))
c.Check(queue, HasLen, 0)
s.flat.FinalizeDownload(s.collectionFactory, nil)
c.Assert(s.flat.packageRefs, NotNil)
// Next call must return the download list without option "skip-existing-packages"
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Release", exampleReleaseFile)
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.bz2", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.gz", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Packages.xz", &http.Error{Code: 404})
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Packages", examplePackagesFile)
downloader.ExpectError("http://repos.express42.com/virool/precise/Sources.bz2", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Sources.gz", &http.Error{Code: 404})
downloader.ExpectError("http://repos.express42.com/virool/precise/Sources.xz", &http.Error{Code: 404})
downloader.ExpectResponse("http://repos.express42.com/virool/precise/Sources", exampleSourcesFile)
err = s.flat.Fetch(downloader, nil, true)
c.Assert(err, IsNil)
err = s.flat.DownloadPackageIndexes(s.progress, downloader, nil, s.collectionFactory, true, true)
c.Assert(err, IsNil)
c.Assert(downloader.Empty(), Equals, true)
queue, size, err = s.flat.BuildDownloadQueue(s.packagePool, s.collectionFactory.PackageCollection(), s.cs, false)
c.Assert(err, IsNil)
c.Check(size, Equals, int64(15))
c.Check(queue, HasLen, 4)
s.flat.FinalizeDownload(s.collectionFactory, nil)
c.Assert(s.flat.packageRefs, NotNil)
}
type RemoteRepoCollectionSuite struct {
PackageListMixinSuite
db database.Storage
collection *RemoteRepoCollection
refListCollection *RefListCollection
}
var _ = Suite(&RemoteRepoCollectionSuite{})
func (s *RemoteRepoCollectionSuite) SetUpTest(c *C) {
s.db, _ = goleveldb.NewOpenDB(c.MkDir())
s.collection = NewRemoteRepoCollection(s.db)
s.refListCollection = NewRefListCollection(s.db)
s.SetUpPackages()
}
func (s *RemoteRepoCollectionSuite) TearDownTest(c *C) {
s.db.Close()
}
func (s *RemoteRepoCollectionSuite) TestAddByName(c *C) {
_, err := s.collection.ByName("yandex")
c.Assert(err, ErrorMatches, "*.not found")
repo, _ := NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{}, false, false, false)
c.Assert(s.collection.Add(repo, s.refListCollection), IsNil)
c.Assert(s.collection.Add(repo, s.refListCollection), ErrorMatches, ".*already exists")
r, err := s.collection.ByName("yandex")
c.Assert(err, IsNil)
c.Assert(r.String(), Equals, repo.String())
collection := NewRemoteRepoCollection(s.db)
r, err = collection.ByName("yandex")
c.Assert(err, IsNil)
c.Assert(r.String(), Equals, repo.String())
}
func (s *RemoteRepoCollectionSuite) TestByUUID(c *C) {
_, err := s.collection.ByUUID("some-uuid")
c.Assert(err, ErrorMatches, "*.not found")
repo, _ := NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{}, false, false, false)
c.Assert(s.collection.Add(repo, s.refListCollection), IsNil)
r, err := s.collection.ByUUID(repo.UUID)
c.Assert(err, IsNil)
c.Assert(r, Equals, repo)
collection := NewRemoteRepoCollection(s.db)
r, err = collection.ByUUID(repo.UUID)
c.Assert(err, IsNil)
c.Assert(r.String(), Equals, repo.String())
}
func (s *RemoteRepoCollectionSuite) TestUpdateLoadComplete(c *C) {
repo, _ := NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{}, false, false, false)
c.Assert(s.collection.Update(repo, s.refListCollection), IsNil)
collection := NewRemoteRepoCollection(s.db)
r, err := collection.ByName("yandex")
c.Assert(err, IsNil)
c.Assert(r.packageRefs, IsNil)
repo.packageRefs = s.reflist
c.Assert(s.collection.Update(repo, s.refListCollection), IsNil)
collection = NewRemoteRepoCollection(s.db)
r, err = collection.ByName("yandex")
c.Assert(err, IsNil)
c.Assert(r.packageRefs, IsNil)
c.Assert(r.NumPackages(), Equals, 0)
c.Assert(s.collection.LoadComplete(r, s.refListCollection), IsNil)
c.Assert(r.NumPackages(), Equals, 3)
}
func (s *RemoteRepoCollectionSuite) TestForEachAndLen(c *C) {
repo, _ := NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{}, false, false, false)
s.collection.Add(repo, s.refListCollection)
count := 0
err := s.collection.ForEach(func(*RemoteRepo) error {
count++
return nil
})
c.Assert(count, Equals, 1)
c.Assert(err, IsNil)
c.Check(s.collection.Len(), Equals, 1)
e := errors.New("c")
err = s.collection.ForEach(func(*RemoteRepo) error {
return e
})
c.Assert(err, Equals, e)
}
func (s *RemoteRepoCollectionSuite) TestDrop(c *C) {
repo1, _ := NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{}, false, false, false)
s.collection.Add(repo1, s.refListCollection)
repo2, _ := NewRemoteRepo("tyndex", "http://mirror.yandex.ru/debian/", "wheezy", []string{"main"}, []string{}, false, false, false)
s.collection.Add(repo2, s.refListCollection)
r1, _ := s.collection.ByUUID(repo1.UUID)
c.Check(r1, Equals, repo1)
err := s.collection.Drop(repo1)
c.Check(err, IsNil)
_, err = s.collection.ByUUID(repo1.UUID)
c.Check(err, ErrorMatches, "mirror .* not found")
collection := NewRemoteRepoCollection(s.db)
_, err = collection.ByName("yandex")
c.Check(err, ErrorMatches, "mirror .* not found")
r2, _ := collection.ByName("tyndex")
c.Check(r2.String(), Equals, repo2.String())
c.Check(s.collection.Drop(repo1), ErrorMatches, "repo not found")
}
const exampleReleaseFile = `Origin: LP-PPA-agenda-developers-daily
Label: Agenda Daily Builds
Suite: precise
Version: 12.04
Codename: precise
Date: Thu, 05 Dec 2013 8:14:32 UTC
Architectures: amd64 armel armhf i386 powerpc
Components: main
Description: Ubuntu Precise 12.04
MD5Sum:
6a5fc91b7277021999268e04a8d74d4c 134 main/binary-amd64/Release
01ff4a18aab39546fde304a35350fc2d 643 main/binary-amd64/Packages.gz
52ded91eeb8490b02016335aa3343492 1350 main/binary-amd64/Packages
5216f9ffe55d151cd7ce7b98b7a43bd7 735 main/binary-amd64/Packages.bz2
d41d8cd98f00b204e9800998ecf8427e 0 main/binary-armel/Packages
4059d198768f9f8dc9372dc1c54bc3c3 14 main/binary-armel/Packages.bz2
7a9de1fb7bf60d416a77d9c9a9716675 134 main/binary-armel/Release
9d10bb61e59bd799891ae4fbcf447ec9 29 main/binary-armel/Packages.gz
9d10bb61e59bd799891ae4fbcf447ec9 29 main/binary-armhf/Packages.gz
c63d31e8e3a5650c29a7124e541d6c23 134 main/binary-armhf/Release
4059d198768f9f8dc9372dc1c54bc3c3 14 main/binary-armhf/Packages.bz2
d41d8cd98f00b204e9800998ecf8427e 0 main/binary-armhf/Packages
c8d336856df67d509032bb54145c2f89 826 main/binary-i386/Packages
92262f0668b265401291f0467bc93763 133 main/binary-i386/Release
7954ed80936429687122b554620c1b5b 734 main/binary-i386/Packages.bz2
e2eef4fe7d285b12c511adfa3a39069e 641 main/binary-i386/Packages.gz
4059d198768f9f8dc9372dc1c54bc3c3 14 main/binary-powerpc/Packages.bz2
9d10bb61e59bd799891ae4fbcf447ec9 29 main/binary-powerpc/Packages.gz
d41d8cd98f00b204e9800998ecf8427e 0 main/binary-powerpc/Packages
b079563fd3367c11f7be049bc686dd10 136 main/binary-powerpc/Release
9d10bb61e59bd799891ae4fbcf447ec9 29 main/debian-installer/binary-amd64/Packages.gz
d41d8cd98f00b204e9800998ecf8427e 0 main/debian-installer/binary-amd64/Packages
4059d198768f9f8dc9372dc1c54bc3c3 14 main/debian-installer/binary-amd64/Packages.bz2
9d10bb61e59bd799891ae4fbcf447ec9 29 main/debian-installer/binary-armel/Packages.gz
d41d8cd98f00b204e9800998ecf8427e 0 main/debian-installer/binary-armel/Packages
4059d198768f9f8dc9372dc1c54bc3c3 14 main/debian-installer/binary-armel/Packages.bz2
9d10bb61e59bd799891ae4fbcf447ec9 29 main/debian-installer/binary-armhf/Packages.gz
d41d8cd98f00b204e9800998ecf8427e 0 main/debian-installer/binary-armhf/Packages
4059d198768f9f8dc9372dc1c54bc3c3 14 main/debian-installer/binary-armhf/Packages.bz2
d41d8cd98f00b204e9800998ecf8427e 0 main/debian-installer/binary-i386/Packages
9d10bb61e59bd799891ae4fbcf447ec9 29 main/debian-installer/binary-i386/Packages.gz
4059d198768f9f8dc9372dc1c54bc3c3 14 main/debian-installer/binary-i386/Packages.bz2
d41d8cd98f00b204e9800998ecf8427e 0 main/debian-installer/binary-powerpc/Packages
4059d198768f9f8dc9372dc1c54bc3c3 14 main/debian-installer/binary-powerpc/Packages.bz2
9d10bb61e59bd799891ae4fbcf447ec9 29 main/debian-installer/binary-powerpc/Packages.gz
3481d65651306df1596dca9078c2506a 135 main/source/Release
0459b7e4512db5479cb982bac6e2f9a1 2003 main/source/Sources
3d83a489f1bd3c04226aa6520b8a6d07 656 main/source/Sources.bz2
b062b5b77094aeeb05ca8dbb1ecf68a9 592 main/source/Sources.gz
SHA1:
fb0b7c8935623ed7d8c45044ba62225fd8cbd4ad 134 main/binary-amd64/Release
b5d62bcec4ec18b88d664255e9051645bab7bd01 643 main/binary-amd64/Packages.gz
ed47aae8926d22d529c27b40b61604aed2cb5f2f 1350 main/binary-amd64/Packages
5b9b171ffcea36e869eba31bcc0e1bfb2a6ad84f 735 main/binary-amd64/Packages.bz2
da39a3ee5e6b4b0d3255bfef95601890afd80709 0 main/binary-armel/Packages
64a543afbb5f4bf728636bdcbbe7a2ed0804adc2 14 main/binary-armel/Packages.bz2
b89234a7efb74d02f15b88e264b5cd2ae1e5dc2d 134 main/binary-armel/Release
3df6ca52b6e8ecfb4a8fac6b8e02c777e3c7960d 29 main/binary-armel/Packages.gz
3df6ca52b6e8ecfb4a8fac6b8e02c777e3c7960d 29 main/binary-armhf/Packages.gz
585a452e27c2e7e047c49d4b0a7459d8c627aa08 134 main/binary-armhf/Release
64a543afbb5f4bf728636bdcbbe7a2ed0804adc2 14 main/binary-armhf/Packages.bz2
da39a3ee5e6b4b0d3255bfef95601890afd80709 0 main/binary-armhf/Packages
1d2f0cd7a3c9e687b853eb277e241cd712b6e3b1 826 main/binary-i386/Packages
16020809662f9bda36eb516d0995658dd94d1ad5 133 main/binary-i386/Release
95a463a0739bf9ff622c8d68f6e4598d400f5248 734 main/binary-i386/Packages.bz2
bf8c0dec9665ba78311c97cae1755d4b2e60af76 641 main/binary-i386/Packages.gz
64a543afbb5f4bf728636bdcbbe7a2ed0804adc2 14 main/binary-powerpc/Packages.bz2
3df6ca52b6e8ecfb4a8fac6b8e02c777e3c7960d 29 main/binary-powerpc/Packages.gz
da39a3ee5e6b4b0d3255bfef95601890afd80709 0 main/binary-powerpc/Packages
cf2ae2d98f535d90209f2c4e5790f95b393d8c2b 136 main/binary-powerpc/Release
3df6ca52b6e8ecfb4a8fac6b8e02c777e3c7960d 29 main/debian-installer/binary-amd64/Packages.gz
da39a3ee5e6b4b0d3255bfef95601890afd80709 0 main/debian-installer/binary-amd64/Packages
64a543afbb5f4bf728636bdcbbe7a2ed0804adc2 14 main/debian-installer/binary-amd64/Packages.bz2
3df6ca52b6e8ecfb4a8fac6b8e02c777e3c7960d 29 main/debian-installer/binary-armel/Packages.gz
da39a3ee5e6b4b0d3255bfef95601890afd80709 0 main/debian-installer/binary-armel/Packages
64a543afbb5f4bf728636bdcbbe7a2ed0804adc2 14 main/debian-installer/binary-armel/Packages.bz2
3df6ca52b6e8ecfb4a8fac6b8e02c777e3c7960d 29 main/debian-installer/binary-armhf/Packages.gz
da39a3ee5e6b4b0d3255bfef95601890afd80709 0 main/debian-installer/binary-armhf/Packages
64a543afbb5f4bf728636bdcbbe7a2ed0804adc2 14 main/debian-installer/binary-armhf/Packages.bz2
da39a3ee5e6b4b0d3255bfef95601890afd80709 0 main/debian-installer/binary-i386/Packages
3df6ca52b6e8ecfb4a8fac6b8e02c777e3c7960d 29 main/debian-installer/binary-i386/Packages.gz
64a543afbb5f4bf728636bdcbbe7a2ed0804adc2 14 main/debian-installer/binary-i386/Packages.bz2
da39a3ee5e6b4b0d3255bfef95601890afd80709 0 main/debian-installer/binary-powerpc/Packages
64a543afbb5f4bf728636bdcbbe7a2ed0804adc2 14 main/debian-installer/binary-powerpc/Packages.bz2
3df6ca52b6e8ecfb4a8fac6b8e02c777e3c7960d 29 main/debian-installer/binary-powerpc/Packages.gz
49cfec0c9b1df3a25e983a3ddf29d15b0e376e02 135 main/source/Release
6b92e0fc84307226172696fde59ca5f33f380b57 2003 main/source/Sources
ecb8afea11030a5df46941cb8ec297ca24c85736 656 main/source/Sources.bz2
923e71383969c91146f12fa8cd121397f2467a2e 592 main/source/Sources.gz
SHA256:
8c0314cfb1b48a8daf47f77420330fd0d78a31897eeb46e05a51964c9f2c02df 134 main/binary-amd64/Release
81b072773d2fdd8471473e060d3bf73255e4c00d322cf387654736ea196e83b4 643 main/binary-amd64/Packages.gz
c7bb299483277bbf7bf4165042edaf547f5fa18f5782c7d2cd8407a38a327cc8 1350 main/binary-amd64/Packages
d263f735c3830caa33ae6441529bd4f8e382205af597ab2cdfcea73afdaa21ab 735 main/binary-amd64/Packages.bz2
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0 main/binary-armel/Packages
d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058 14 main/binary-armel/Packages.bz2
75ede815b020626c6aa16201d24099ed7e06f03643d0cf38ef194f1029ea648b 134 main/binary-armel/Release
825d493158fe0f50ca1acd70367aefa391170563af2e4ee9cedbcbe6796c8384 29 main/binary-armel/Packages.gz
825d493158fe0f50ca1acd70367aefa391170563af2e4ee9cedbcbe6796c8384 29 main/binary-armhf/Packages.gz
d25382b633c4a1621f8df6ce86e5c63da2e506a377e05ae9453238bb18191540 134 main/binary-armhf/Release
d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058 14 main/binary-armhf/Packages.bz2
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0 main/binary-armhf/Packages
b1bb341bb613363ca29440c2eb9c08a9289de5458209990ec502ed27711a83a2 826 main/binary-i386/Packages
e5aaceaac5ecb59143a4b4ed2bf700fe85d6cf08addd10cf2058bde697b7b219 133 main/binary-i386/Release
377890a26f99db55e117dfc691972dcbbb7d8be1630c8fc8297530c205377f2b 734 main/binary-i386/Packages.bz2
6361e8efc67d2e7c1a8db45388aec0311007c0a1bd96698623ddeb5ed0bdc914 641 main/binary-i386/Packages.gz
d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058 14 main/binary-powerpc/Packages.bz2
825d493158fe0f50ca1acd70367aefa391170563af2e4ee9cedbcbe6796c8384 29 main/binary-powerpc/Packages.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0 main/binary-powerpc/Packages
03b5c97a99aa799964eb5a77f8a62ad38a241b93a87eacac6cf75a270a6d417c 136 main/binary-powerpc/Release
825d493158fe0f50ca1acd70367aefa391170563af2e4ee9cedbcbe6796c8384 29 main/debian-installer/binary-amd64/Packages.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0 main/debian-installer/binary-amd64/Packages
d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058 14 main/debian-installer/binary-amd64/Packages.bz2
825d493158fe0f50ca1acd70367aefa391170563af2e4ee9cedbcbe6796c8384 29 main/debian-installer/binary-armel/Packages.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0 main/debian-installer/binary-armel/Packages
d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058 14 main/debian-installer/binary-armel/Packages.bz2
825d493158fe0f50ca1acd70367aefa391170563af2e4ee9cedbcbe6796c8384 29 main/debian-installer/binary-armhf/Packages.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0 main/debian-installer/binary-armhf/Packages
d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058 14 main/debian-installer/binary-armhf/Packages.bz2
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0 main/debian-installer/binary-i386/Packages
825d493158fe0f50ca1acd70367aefa391170563af2e4ee9cedbcbe6796c8384 29 main/debian-installer/binary-i386/Packages.gz
d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058 14 main/debian-installer/binary-i386/Packages.bz2
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0 main/debian-installer/binary-powerpc/Packages
d3dda84eb03b9738d118eb2be78e246106900493c0ae07819ad60815134a8058 14 main/debian-installer/binary-powerpc/Packages.bz2
825d493158fe0f50ca1acd70367aefa391170563af2e4ee9cedbcbe6796c8384 29 main/debian-installer/binary-powerpc/Packages.gz
d683102993b6f11067ce86d73111f067e36a199e9dc1f4295c8b19c274dc9ef8 135 main/source/Release
45f868fd5d9efe611d67572ffcf96a00a5b9ec38ea5102753290c38c36b8c282 2003 main/source/Sources
d178f1e310218d9f0f16c37d0780637f1cf3640a94a7fb0e24dc940c51b1e115 656 main/source/Sources.bz2
080228b550da407fb8ac73fb30b37323468fd2b2de98dd56a324ee7d701f6103 592 main/source/Sources.gz`
const examplePackagesFile = `Package: amanda-client
Source: amanda
Version: 1:3.3.1-3~bpo60+1
Installed-Size: 880
Maintainer: Bdale Garbee <bdale@gag.com>
Architecture: i386
Replaces: amanda-common (<< 1:2.5.2p1-3)
Depends: libc6 (>= 2.3), libcurl3 (>= 7.16.2-1), libglib2.0-0 (>= 2.12.0), libreadline6 (>= 6.0), libssl0.9.8 (>= 0.9.8m-1), amanda-common (= 1:3.3.1-3~bpo60+1)
Suggests: gnuplot, dump, smbclient
Conflicts: amanda, amanda-common (<< 1:2.5.2p1-3)
Description: Advanced Maryland Automatic Network Disk Archiver (Client)
Description-md5: 21af3684379a64cacc51c39152ab1062
Section: utils
Priority: optional
Filename: pool/main/a/amanda/amanda-client_3.3.1-3~bpo60+1_amd64.deb
Size: 3
MD5sum: d16fb36f0911f878998c136191af705e
SHA1: 66b27417d37e024c46526c2f6d358a754fc552f3
SHA256: 3608bca1e44ea6c4d268eb6db02260269892c0b42b86bbf1e77a6fa16c3c9282
`
const exampleInstallerHashSumFile = `82f69d557f0004d2923fb03e4fb47d18187e37768dbfd0c99756f8a6c68a6d3a ./MANIFEST
`
const exampleInstallerManifestFile = `cdrom/debian-cd_info.tar.gz -- isolinux config files for CD
cdrom/gtk/debian-cd_info.tar.gz -- isolinux help screens for CD (graphical)
cdrom/gtk/initrd.gz -- initrd for use with isolinux to build a CD (graphical)
cdrom/gtk/vmlinuz -- kernel for use with isolinux to build a CD (graphical)
cdrom/initrd.gz -- initrd for use with isolinux to build a CD
cdrom/vmlinuz -- kernel for use with isolinux to build a CD
cdrom/xen/debian.cfg -- example Xen configuration
cdrom/xen/initrd.gz -- initrd for installing under Xen
cdrom/xen/vmlinuz -- kernel image for installing under Xen
hd-media/boot.img.gz -- 1 gb image (compressed) for USB memory stick
hd-media/gtk/initrd.gz -- for use on USB memory sticks
hd-media/gtk/vmlinuz -- for use on USB memory sticks
hd-media/initrd.gz -- for use on USB memory sticks
hd-media/vmlinuz -- for use on USB memory sticks
netboot/debian-installer -- PXE boot directory for tftp server
netboot/gtk/debian-installer -- PXE boot directory for tftp server (graphical installer)
netboot/gtk/mini.iso -- not so tiny CD image that boots the graphical netboot installer
netboot/gtk/netboot.tar.gz -- tarball of PXE boot directory (graphical installer)
netboot/mini.iso -- tiny CD image that boots the netboot installer
netboot/netboot.tar.gz -- tarball of PXE boot directory
netboot/xen/debian.cfg -- example Xen configuration
netboot/xen/initrd.gz -- initrd for installing under Xen
netboot/xen/vmlinuz -- kernel image for installing under Xen`
const exampleSourcesFile = sourcePackageMeta
Reference in New Issue View Git Blame Copy Permalink