vincent/aptly
1
0
Fork 0
mirror of https://github.com/aptly-dev/aptly.git synced 2026-06-04 05:10:40 +00:00
Code Issues Projects Releases Wiki Activity

Split reflists to share their contents across snapshots

Browse Source 
In current aptly, each repository and snapshot has its own reflist in
the database. This brings a few problems with it:

- Given a sufficiently large repositories and snapshots, these lists can
  get enormous, reaching >1MB. This is a problem for LevelDB's overall
  performance, as it tends to prefer values around the confiruged block
  size (defaults to just 4KiB).
- When you take these large repositories and snapshot them, you have a
  full, new copy of the reflist, even if only a few packages changed.
  This means that having a lot of snapshots with a few changes causes
  the database to basically be full of largely duplicate reflists.
- All the duplication also means that many of the same refs are being
  loaded repeatedly, which can cause some slowdown but, more notably,
  eats up huge amounts of memory.
- Adding on more and more new repositories and snapshots will cause the
  time and memory spent on things like cleanup and publishing to grow
  roughly linearly.

At the core, there are two problems here:

- Reflists get very big because there are just a lot of packages.
- Different reflists can tend to duplicate much of the same contents.

*Split reflists* aim at solving this by separating reflists into 64
*buckets*. Package refs are sorted into individual buckets according to
the following system:

- Take the first 3 letters of the package name, after dropping a `lib`
  prefix. (Using only the first 3 letters will cause packages with
  similar prefixes to end up in the same bucket, under the assumption
  that packages with similar names tend to be updated together.)
- Take the 64-bit xxhash of these letters. (xxhash was chosen because it
  relatively good distribution across the individual bits, which is
  important for the next step.)
- Use the first 6 bits of the hash (range [0:63]) as an index into the
  buckets.

Once refs are placed in buckets, a sha256 digest of all the refs in the
bucket is taken. These buckets are then stored in the database, split
into roughly block-sized segments, and all the repositories and
snapshots simply store an array of bucket digests.

This approach means that *repositories and snapshots can share their
reflist buckets*. If a snapshot is taken of a repository, it will have
the same contents, so its split reflist will point to the same buckets
as the base repository, and only one copy of each bucket is stored in
the database. When some packages in the repository change, only the
buckets containing those packages will be modified; all the other
buckets will remain unchanged, and thus their contents will still be
shared. Later on, when these reflists are loaded, each bucket is only
loaded once, short-cutting loaded many megabytes of data. In effect,
split reflists are essentially copy-on-write, with only the changed
buckets stored individually.

Changing the disk format means that a migration needs to take place, so
that task is moved into the database cleanup step, which will migrate
reflists over to split reflists, as well as delete any unused reflist
buckets.

All the reflist tests are also changed to additionally test out split
reflists; although the internal logic is all shared (since buckets are,
themselves, just normal reflists), some special additions are needed to
have native versions of the various reflist helper methods.

In our tests, we've observed the following improvements:

- Memory usage during publish and database cleanup, with
  `GOMEMLIMIT=2GiB`, goes down from ~3.2GiB (larger than the memory
  limit!) to ~0.7GiB, a decrease of ~4.5x.
- Database size decreases from 1.3GB to 367MB.

*In my local tests*, publish times had also decreased down to mere
seconds but the same effect wasn't observed on the server, with the
times staying around the same. My suspicions are that this is due to I/O
performance: my local system is an M1 MBP, which almost certainly has
much faster disk speeds than our DigitalOcean block volumes. Split
reflists include a side effect of requiring more random accesses from
reading all the buckets by their keys, so if your random I/O
performance is slower, it might cancel out the benefits. That being
said, even in that case, the memory usage and database size advantages
still persist.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
This commit is contained in:
Ryan Gonzalez
2023-11-10 17:01:16 -06:00
committed by André Roth
parent 4be09fd407
commit 19a705f80d
69 changed files with 1976 additions and 676 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cmd/cmd.go
+1 -1
View File
@@ -21,7 +21,7 @@ const (
)
// ListPackagesRefList shows list of packages in PackageRefList
func ListPackagesRefList(reflist *deb.PackageRefList, collectionFactory *deb.CollectionFactory) (err error) {
func ListPackagesRefList(reflist deb.AnyRefList, collectionFactory *deb.CollectionFactory) (err error) {
fmt.Printf("Packages:\n")
if reflist == nil {
cmd/db_cleanup.go
+96 -34
View File
@@ -6,6 +6,7 @@ import (
"strings"
"github.com/aptly-dev/aptly/aptly"
"github.com/aptly-dev/aptly/database"
"github.com/aptly-dev/aptly/deb"
"github.com/aptly-dev/aptly/utils"
"github.com/smira/commander"
@@ -24,12 +25,20 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
dryRun := context.Flags().Lookup("dry-run").Value.Get().(bool)
collectionFactory := context.NewCollectionFactory()
// collect information about references packages...
existingPackageRefs := deb.NewPackageRefList()
// collect information about references packages and their reflistbuckets...
existingPackageRefs := deb.NewSplitRefList()
existingBuckets := deb.NewRefListDigestSet()
// used only in verbose mode to report package use source
packageRefSources := map[string][]string{}
var reflistMigration *deb.RefListMigration
if !dryRun {
reflistMigration = collectionFactory.RefListCollection().NewMigration()
} else {
reflistMigration = collectionFactory.RefListCollection().NewMigrationDryRun()
}
context.Progress().ColoredPrintf("@{w!}Loading mirrors, local repos, snapshots and published repos...@|")
if verbose {
context.Progress().ColoredPrintf("@{y}Loading mirrors:@|")
@@ -39,20 +48,21 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
context.Progress().ColoredPrintf("- @{g}%s@|", repo.Name)
}
e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
if e != nil {
sl := deb.NewSplitRefList()
e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
if e != nil && e != database.ErrNotFound {
return e
}
if repo.RefList() != nil {
existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
if verbose {
description := fmt.Sprintf("mirror %s", repo.Name)
repo.RefList().ForEach(func(key []byte) error {
packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
return nil
})
}
existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
existingBuckets.AddAllInRefList(sl)
if verbose {
description := fmt.Sprintf("mirror %s", repo.Name)
sl.ForEach(func(key []byte) error {
packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
return nil
})
}
return nil
@@ -71,21 +81,23 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
context.Progress().ColoredPrintf("- @{g}%s@|", repo.Name)
}
e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
if e != nil {
sl := deb.NewSplitRefList()
e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
if e != nil && e != database.ErrNotFound {
return e
}
if repo.RefList() != nil {
existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
existingBuckets.AddAllInRefList(sl)
if verbose {
description := fmt.Sprintf("local repo %s", repo.Name)
repo.RefList().ForEach(func(key []byte) error {
packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
return nil
})
}
existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
if verbose {
description := fmt.Sprintf("local repo %s", repo.Name)
sl.ForEach(func(key []byte) error {
packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
return nil
})
}
return nil
@@ -104,16 +116,18 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
context.Progress().ColoredPrintf("- @{g}%s@|", snapshot.Name)
}
e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
sl := deb.NewSplitRefList()
e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, snapshot.RefKey(), reflistMigration)
if e != nil {
return e
}
existingPackageRefs = existingPackageRefs.Merge(snapshot.RefList(), false, true)
existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
existingBuckets.AddAllInRefList(sl)
if verbose {
description := fmt.Sprintf("snapshot %s", snapshot.Name)
snapshot.RefList().ForEach(func(key []byte) error {
sl.ForEach(func(key []byte) error {
packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
return nil
})
@@ -136,17 +150,21 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
if published.SourceKind != deb.SourceLocalRepo {
return nil
}
e := collectionFactory.PublishedRepoCollection().LoadComplete(published, collectionFactory)
if e != nil {
return e
}
for _, component := range published.Components() {
existingPackageRefs = existingPackageRefs.Merge(published.RefList(component), false, true)
sl := deb.NewSplitRefList()
e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, published.RefKey(component), reflistMigration)
if e != nil {
return e
}
existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
existingBuckets.AddAllInRefList(sl)
if verbose {
description := fmt.Sprintf("published repository %s:%s/%s component %s",
published.Storage, published.Prefix, published.Distribution, component)
published.RefList(component).ForEach(func(key []byte) error {
sl.ForEach(func(key []byte) error {
packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
return nil
})
@@ -160,11 +178,29 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
collectionFactory.Flush()
err = reflistMigration.Flush()
if err != nil {
return err
}
if verbose {
if stats := reflistMigration.Stats(); stats.Reflists > 0 {
if !dryRun {
context.Progress().ColoredPrintf("@{w!}Split %d reflist(s) into %d bucket(s) (%d segment(s))@|",
stats.Reflists, stats.Buckets, stats.Segments)
} else {
context.Progress().ColoredPrintf(
"@{y!}Skipped splitting %d reflist(s) into %d bucket(s) (%d segment(s)), as -dry-run has been requested.@|",
stats.Reflists, stats.Buckets, stats.Segments)
}
}
}
// ... and compare it to the list of all packages
context.Progress().ColoredPrintf("@{w!}Loading list of all packages...@|")
allPackageRefs := collectionFactory.PackageCollection().AllPackageRefs()
toDelete := allPackageRefs.Subtract(existingPackageRefs)
toDelete := allPackageRefs.Subtract(existingPackageRefs.Flatten())
// delete packages that are no longer referenced
context.Progress().ColoredPrintf("@{r!}Deleting unreferenced packages (%d)...@|", toDelete.Len())
@@ -202,6 +238,32 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
}
}
bucketsToDelete, err := collectionFactory.RefListCollection().AllBucketDigests()
if err != nil {
return err
}
bucketsToDelete.RemoveAll(existingBuckets)
context.Progress().ColoredPrintf("@{r!}Deleting unreferenced reflist buckets (%d)...@|", bucketsToDelete.Len())
if bucketsToDelete.Len() > 0 {
if !dryRun {
batch := db.CreateBatch()
err := bucketsToDelete.ForEach(func(digest []byte) error {
return collectionFactory.RefListCollection().UnsafeDropBucket(digest, batch)
})
if err != nil {
return err
}
if err := batch.Write(); err != nil {
return err
}
} else {
context.Progress().ColoredPrintf("@{y!}Skipped reflist deletion, as -dry-run has been requested.@|")
}
}
collectionFactory.Flush()
// now, build a list of files that should be present in Repository (package pool)
cmd/mirror_create.go
+1 -1
View File
@@ -69,7 +69,7 @@ func aptlyMirrorCreate(cmd *commander.Command, args []string) error {
}
collectionFactory := context.NewCollectionFactory()
err = collectionFactory.RemoteRepoCollection().Add(repo)
err = collectionFactory.RemoteRepoCollection().Add(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to add mirror: %s", err)
}
cmd/mirror_edit.go
+1 -1
View File
@@ -78,7 +78,7 @@ func aptlyMirrorEdit(cmd *commander.Command, args []string) error {
}
}
err = collectionFactory.RemoteRepoCollection().Update(repo)
err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to edit: %s", err)
}
cmd/mirror_rename.go
+1 -1
View File
@@ -37,7 +37,7 @@ func aptlyMirrorRename(cmd *commander.Command, args []string) error {
}
repo.Name = newName
err = collectionFactory.RemoteRepoCollection().Update(repo)
err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to rename: %s", err)
}
cmd/mirror_show.go
+5 -4
View File
@@ -38,7 +38,7 @@ func aptlyMirrorShowTxt(_ *commander.Command, args []string) error {
return fmt.Errorf("unable to show: %s", err)
}
err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to show: %s", err)
}
@@ -98,12 +98,13 @@ func aptlyMirrorShowJSON(_ *commander.Command, args []string) error {
name := args[0]
repo, err := context.NewCollectionFactory().RemoteRepoCollection().ByName(name)
collectionFactory := context.NewCollectionFactory()
repo, err := collectionFactory.RemoteRepoCollection().ByName(name)
if err != nil {
return fmt.Errorf("unable to show: %s", err)
}
err = context.NewCollectionFactory().RemoteRepoCollection().LoadComplete(repo)
err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to show: %s", err)
}
@@ -113,7 +114,7 @@ func aptlyMirrorShowJSON(_ *commander.Command, args []string) error {
if withPackages {
if repo.RefList() != nil {
var list *deb.PackageList
list, err = deb.NewPackageListFromRefList(repo.RefList(), context.NewCollectionFactory().PackageCollection(), context.Progress())
list, err = deb.NewPackageListFromRefList(repo.RefList(), collectionFactory.PackageCollection(), context.Progress())
if err != nil {
return fmt.Errorf("unable to get package list: %s", err)
}
cmd/mirror_update.go
+4 -4
View File
@@ -29,7 +29,7 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to update: %s", err)
}
err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to update: %s", err)
}
@@ -101,12 +101,12 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
err = context.ReOpenDatabase()
if err == nil {
repo.MarkAsIdle()
collectionFactory.RemoteRepoCollection().Update(repo)
collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
}
}()
repo.MarkAsUpdating()
err = collectionFactory.RemoteRepoCollection().Update(repo)
err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to update: %s", err)
}
@@ -262,7 +262,7 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
}
repo.FinalizeDownload(collectionFactory, context.Progress())
err = collectionFactory.RemoteRepoCollection().Update(repo)
err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to update: %s", err)
}
cmd/package_show.go
+3 -3
View File
@@ -14,7 +14,7 @@ import (
func printReferencesTo(p *deb.Package, collectionFactory *deb.CollectionFactory) (err error) {
err = collectionFactory.RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
e := collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if e != nil {
return e
}
@@ -30,7 +30,7 @@ func printReferencesTo(p *deb.Package, collectionFactory *deb.CollectionFactory)
}
err = collectionFactory.LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if e != nil {
return e
}
@@ -46,7 +46,7 @@ func printReferencesTo(p *deb.Package, collectionFactory *deb.CollectionFactory)
}
err = collectionFactory.SnapshotCollection().ForEach(func(snapshot *deb.Snapshot) error {
e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
e := collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
if e != nil {
return e
}
cmd/publish_snapshot.go
+3 -3
View File
@@ -49,7 +49,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to publish: %s", err)
}
err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to publish: %s", err)
}
@@ -85,7 +85,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to publish: %s", err)
}
err = collectionFactory.LocalRepoCollection().LoadComplete(localRepo)
err = collectionFactory.LocalRepoCollection().LoadComplete(localRepo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to publish: %s", err)
}
@@ -175,7 +175,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to publish: %s", err)
}
err = collectionFactory.PublishedRepoCollection().Add(published)
err = collectionFactory.PublishedRepoCollection().Add(published, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to save to DB: %s", err)
}
cmd/publish_switch.go
+2 -2
View File
@@ -72,7 +72,7 @@ func aptlyPublishSwitch(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to switch: %s", err)
}
err = snapshotCollection.LoadComplete(snapshot)
err = snapshotCollection.LoadComplete(snapshot, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to switch: %s", err)
}
@@ -108,7 +108,7 @@ func aptlyPublishSwitch(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to publish: %s", err)
}
err = collectionFactory.PublishedRepoCollection().Update(published)
err = collectionFactory.PublishedRepoCollection().Update(published, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to save to DB: %s", err)
}
cmd/publish_update.go
+1 -1
View File
@@ -69,7 +69,7 @@ func aptlyPublishUpdate(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to publish: %s", err)
}
err = collectionFactory.PublishedRepoCollection().Update(published)
err = collectionFactory.PublishedRepoCollection().Update(published, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to save to DB: %s", err)
}
cmd/repo_add.go
+3 -3
View File
@@ -28,7 +28,7 @@ func aptlyRepoAdd(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to add: %s", err)
}
err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to add: %s", err)
}
@@ -58,9 +58,9 @@ func aptlyRepoAdd(cmd *commander.Command, args []string) error {
processedFiles = append(processedFiles, otherFiles...)
repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
err = collectionFactory.LocalRepoCollection().Update(repo)
err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to save: %s", err)
}
cmd/repo_create.go
+2 -2
View File
@@ -36,7 +36,7 @@ func aptlyRepoCreate(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to load source snapshot: %s", err)
}
err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to load source snapshot: %s", err)
}
@@ -44,7 +44,7 @@ func aptlyRepoCreate(cmd *commander.Command, args []string) error {
repo.UpdateRefList(snapshot.RefList())
}
err = collectionFactory.LocalRepoCollection().Add(repo)
err = collectionFactory.LocalRepoCollection().Add(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to add local repo: %s", err)
}
cmd/repo_edit.go
+2 -2
View File
@@ -22,7 +22,7 @@ func aptlyRepoEdit(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to edit: %s", err)
}
err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to edit: %s", err)
}
@@ -53,7 +53,7 @@ func aptlyRepoEdit(cmd *commander.Command, args []string) error {
}
}
err = collectionFactory.LocalRepoCollection().Update(repo)
err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to edit: %s", err)
}
cmd/repo_include.go
+1 -1
View File
@@ -67,7 +67,7 @@ func aptlyRepoInclude(cmd *commander.Command, args []string) error {
_, failedFiles2, err = deb.ImportChangesFiles(
changesFiles, reporter, acceptUnsigned, ignoreSignatures, forceReplace, noRemoveFiles, verifier, repoTemplate,
context.Progress(), collectionFactory.LocalRepoCollection(), collectionFactory.PackageCollection(),
context.PackagePool(), collectionFactory.ChecksumCollection,
collectionFactory.RefListCollection(), context.PackagePool(), collectionFactory.ChecksumCollection,
uploaders, query.Parse)
failedFiles = append(failedFiles, failedFiles2...)
cmd/repo_list.go
+3 -2
View File
@@ -36,7 +36,7 @@ func aptlyRepoListTxt(cmd *commander.Command, _ []string) error {
if raw {
repos[i] = repo.Name
} else {
e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if e != nil {
return e
}
@@ -77,7 +77,8 @@ func aptlyRepoListJSON(_ *commander.Command, _ []string) error {
repos := make([]*deb.LocalRepo, context.NewCollectionFactory().LocalRepoCollection().Len())
i := 0
context.NewCollectionFactory().LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
e := context.NewCollectionFactory().LocalRepoCollection().LoadComplete(repo)
collectionFactory := context.NewCollectionFactory()
e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if e != nil {
return e
}
cmd/repo_move.go
+9 -9
View File
@@ -25,13 +25,13 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to %s: %s", command, err)
}
err = collectionFactory.LocalRepoCollection().LoadComplete(dstRepo)
err = collectionFactory.LocalRepoCollection().LoadComplete(dstRepo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to %s: %s", command, err)
}
var (
srcRefList *deb.PackageRefList
srcRefList *deb.SplitRefList
srcRepo *deb.LocalRepo
)
@@ -45,7 +45,7 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to %s: source and destination are the same", command)
}
err = collectionFactory.LocalRepoCollection().LoadComplete(srcRepo)
err = collectionFactory.LocalRepoCollection().LoadComplete(srcRepo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to %s: %s", command, err)
}
@@ -59,12 +59,12 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to %s: %s", command, err)
}
err = collectionFactory.RemoteRepoCollection().LoadComplete(srcRemoteRepo)
err = collectionFactory.RemoteRepoCollection().LoadComplete(srcRemoteRepo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to %s: %s", command, err)
}
if srcRemoteRepo.RefList() == nil {
if srcRemoteRepo.RefList().Len() == 0 {
return fmt.Errorf("unable to %s: mirror not updated", command)
}
@@ -161,17 +161,17 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
if context.Flags().Lookup("dry-run").Value.Get().(bool) {
context.Progress().Printf("\nChanges not saved, as dry run has been requested.\n")
} else {
dstRepo.UpdateRefList(deb.NewPackageRefListFromPackageList(dstList))
dstRepo.UpdateRefList(deb.NewSplitRefListFromPackageList(dstList))
err = collectionFactory.LocalRepoCollection().Update(dstRepo)
err = collectionFactory.LocalRepoCollection().Update(dstRepo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to save: %s", err)
}
if command == "move" { // nolint: goconst
srcRepo.UpdateRefList(deb.NewPackageRefListFromPackageList(srcList))
srcRepo.UpdateRefList(deb.NewSplitRefListFromPackageList(srcList))
err = collectionFactory.LocalRepoCollection().Update(srcRepo)
err = collectionFactory.LocalRepoCollection().Update(srcRepo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to save: %s", err)
}
cmd/repo_remove.go
+3 -3
View File
@@ -24,7 +24,7 @@ func aptlyRepoRemove(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to remove: %s", err)
}
err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to remove: %s", err)
}
@@ -63,9 +63,9 @@ func aptlyRepoRemove(cmd *commander.Command, args []string) error {
if context.Flags().Lookup("dry-run").Value.Get().(bool) {
context.Progress().Printf("\nChanges not saved, as dry run has been requested.\n")
} else {
repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
err = collectionFactory.LocalRepoCollection().Update(repo)
err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to save: %s", err)
}
cmd/repo_rename.go
+1 -1
View File
@@ -32,7 +32,7 @@ func aptlyRepoRename(cmd *commander.Command, args []string) error {
}
repo.Name = newName
err = collectionFactory.LocalRepoCollection().Update(repo)
err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to rename: %s", err)
}
cmd/repo_show.go
+5 -4
View File
@@ -36,7 +36,7 @@ func aptlyRepoShowTxt(_ *commander.Command, args []string) error {
return fmt.Errorf("unable to show: %s", err)
}
err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to show: %s", err)
}
@@ -63,12 +63,13 @@ func aptlyRepoShowJSON(_ *commander.Command, args []string) error {
name := args[0]
repo, err := context.NewCollectionFactory().LocalRepoCollection().ByName(name)
collectionFactory := context.NewCollectionFactory()
repo, err := collectionFactory.LocalRepoCollection().ByName(name)
if err != nil {
return fmt.Errorf("unable to show: %s", err)
}
err = context.NewCollectionFactory().LocalRepoCollection().LoadComplete(repo)
err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to show: %s", err)
}
@@ -79,7 +80,7 @@ func aptlyRepoShowJSON(_ *commander.Command, args []string) error {
if withPackages {
if repo.RefList() != nil {
var list *deb.PackageList
list, err = deb.NewPackageListFromRefList(repo.RefList(), context.NewCollectionFactory().PackageCollection(), context.Progress())
list, err = deb.NewPackageListFromRefList(repo.RefList(), collectionFactory.PackageCollection(), context.Progress())
if err == nil {
packageList = list.FullNames()
}
cmd/snapshot_create.go
+3 -3
View File
@@ -30,7 +30,7 @@ func aptlySnapshotCreate(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to create snapshot: %s", err)
}
err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to create snapshot: %s", err)
}
@@ -50,7 +50,7 @@ func aptlySnapshotCreate(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to create snapshot: %s", err)
}
err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to create snapshot: %s", err)
}
@@ -71,7 +71,7 @@ func aptlySnapshotCreate(cmd *commander.Command, args []string) error {
return commander.ErrCommandError
}
err = collectionFactory.SnapshotCollection().Add(snapshot)
err = collectionFactory.SnapshotCollection().Add(snapshot, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to add snapshot: %s", err)
}
cmd/snapshot_diff.go
+3 -3
View File
@@ -23,7 +23,7 @@ func aptlySnapshotDiff(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to load snapshot A: %s", err)
}
err = collectionFactory.SnapshotCollection().LoadComplete(snapshotA)
err = collectionFactory.SnapshotCollection().LoadComplete(snapshotA, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to load snapshot A: %s", err)
}
@@ -34,13 +34,13 @@ func aptlySnapshotDiff(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to load snapshot B: %s", err)
}
err = collectionFactory.SnapshotCollection().LoadComplete(snapshotB)
err = collectionFactory.SnapshotCollection().LoadComplete(snapshotB, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to load snapshot B: %s", err)
}
// Calculate diff
diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection())
diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection(), nil)
if err != nil {
return fmt.Errorf("unable to calculate diff: %s", err)
}
cmd/snapshot_filter.go
+2 -2
View File
@@ -27,7 +27,7 @@ func aptlySnapshotFilter(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to filter: %s", err)
}
err = collectionFactory.SnapshotCollection().LoadComplete(source)
err = collectionFactory.SnapshotCollection().LoadComplete(source, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to filter: %s", err)
}
@@ -87,7 +87,7 @@ func aptlySnapshotFilter(cmd *commander.Command, args []string) error {
destination := deb.NewSnapshotFromPackageList(args[1], []*deb.Snapshot{source}, result,
fmt.Sprintf("Filtered '%s', query was: '%s'", source.Name, strings.Join(args[2:], " ")))
err = collectionFactory.SnapshotCollection().Add(destination)
err = collectionFactory.SnapshotCollection().Add(destination, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to create snapshot: %s", err)
}
cmd/snapshot_merge.go
+2 -2
View File
@@ -24,7 +24,7 @@ func aptlySnapshotMerge(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to load snapshot: %s", err)
}
err = collectionFactory.SnapshotCollection().LoadComplete(sources[i])
err = collectionFactory.SnapshotCollection().LoadComplete(sources[i], collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to load snapshot: %s", err)
}
@@ -57,7 +57,7 @@ func aptlySnapshotMerge(cmd *commander.Command, args []string) error {
destination := deb.NewSnapshotFromRefList(args[0], sources, result,
fmt.Sprintf("Merged from sources: %s", strings.Join(sourceDescription, ", ")))
err = collectionFactory.SnapshotCollection().Add(destination)
err = collectionFactory.SnapshotCollection().Add(destination, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to create snapshot: %s", err)
}
cmd/snapshot_pull.go
+3 -3
View File
@@ -29,7 +29,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to pull: %s", err)
}
err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to pull: %s", err)
}
@@ -40,7 +40,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to pull: %s", err)
}
err = collectionFactory.SnapshotCollection().LoadComplete(source)
err = collectionFactory.SnapshotCollection().LoadComplete(source, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to pull: %s", err)
}
@@ -149,7 +149,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
destination := deb.NewSnapshotFromPackageList(args[2], []*deb.Snapshot{snapshot, source}, packageList,
fmt.Sprintf("Pulled into '%s' with '%s' as source, pull request was: '%s'", snapshot.Name, source.Name, strings.Join(args[3:], " ")))
err = collectionFactory.SnapshotCollection().Add(destination)
err = collectionFactory.SnapshotCollection().Add(destination, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to create snapshot: %s", err)
}
cmd/snapshot_rename.go
+1 -1
View File
@@ -32,7 +32,7 @@ func aptlySnapshotRename(cmd *commander.Command, args []string) error {
}
snapshot.Name = newName
err = collectionFactory.SnapshotCollection().Update(snapshot)
err = collectionFactory.SnapshotCollection().Update(snapshot, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to rename: %s", err)
}
cmd/snapshot_search.go
+4 -4
View File
@@ -25,7 +25,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
command := cmd.Parent.Name()
collectionFactory := context.NewCollectionFactory()
var reflist *deb.PackageRefList
var reflist *deb.SplitRefList
if command == "snapshot" { // nolint: goconst
var snapshot *deb.Snapshot
@@ -34,7 +34,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
return fmt.Errorf("unable to search: %s", err)
}
err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to search: %s", err)
}
@@ -47,7 +47,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
return fmt.Errorf("unable to search: %s", err)
}
err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to search: %s", err)
}
@@ -60,7 +60,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
return fmt.Errorf("unable to search: %s", err)
}
err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to search: %s", err)
}
cmd/snapshot_show.go
+7 -6
View File
@@ -35,7 +35,7 @@ func aptlySnapshotShowTxt(_ *commander.Command, args []string) error {
return fmt.Errorf("unable to show: %s", err)
}
err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to show: %s", err)
}
@@ -86,16 +86,17 @@ func aptlySnapshotShowTxt(_ *commander.Command, args []string) error {
}
func aptlySnapshotShowJSON(_ *commander.Command, args []string) error {
collectionFactory := context.NewCollectionFactory()
var err error
name := args[0]
snapshot, err := context.NewCollectionFactory().SnapshotCollection().ByName(name)
snapshot, err := collectionFactory.SnapshotCollection().ByName(name)
if err != nil {
return fmt.Errorf("unable to show: %s", err)
}
err = context.NewCollectionFactory().SnapshotCollection().LoadComplete(snapshot)
err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to show: %s", err)
}
@@ -105,14 +106,14 @@ func aptlySnapshotShowJSON(_ *commander.Command, args []string) error {
for _, sourceID := range snapshot.SourceIDs {
if snapshot.SourceKind == deb.SourceSnapshot {
var source *deb.Snapshot
source, err = context.NewCollectionFactory().SnapshotCollection().ByUUID(sourceID)
source, err = collectionFactory.SnapshotCollection().ByUUID(sourceID)
if err != nil {
continue
}
snapshot.Snapshots = append(snapshot.Snapshots, source)
} else if snapshot.SourceKind == deb.SourceLocalRepo {
var source *deb.LocalRepo
source, err = context.NewCollectionFactory().LocalRepoCollection().ByUUID(sourceID)
source, err = collectionFactory.LocalRepoCollection().ByUUID(sourceID)
if err != nil {
continue
}
@@ -133,7 +134,7 @@ func aptlySnapshotShowJSON(_ *commander.Command, args []string) error {
if withPackages {
if snapshot.RefList() != nil {
var list *deb.PackageList
list, err = deb.NewPackageListFromRefList(snapshot.RefList(), context.NewCollectionFactory().PackageCollection(), context.Progress())
list, err = deb.NewPackageListFromRefList(snapshot.RefList(), collectionFactory.PackageCollection(), context.Progress())
if err != nil {
return fmt.Errorf("unable to get package list: %s", err)
}
cmd/snapshot_verify.go
+1 -1
View File
@@ -23,7 +23,7 @@ func aptlySnapshotVerify(cmd *commander.Command, args []string) error {
return fmt.Errorf("unable to verify: %s", err)
}
err = collectionFactory.SnapshotCollection().LoadComplete(snapshots[i])
err = collectionFactory.SnapshotCollection().LoadComplete(snapshots[i], collectionFactory.RefListCollection())
if err != nil {
return fmt.Errorf("unable to verify: %s", err)
}