handle packageRefs == nil

In current aptly, each repository and snapshot has its own reflist in
the database. This brings a few problems with it:

- Given a sufficiently large repositories and snapshots, these lists can
  get enormous, reaching >1MB. This is a problem for LevelDB's overall
  performance, as it tends to prefer values around the confiruged block
  size (defaults to just 4KiB).
- When you take these large repositories and snapshot them, you have a
  full, new copy of the reflist, even if only a few packages changed.
  This means that having a lot of snapshots with a few changes causes
  the database to basically be full of largely duplicate reflists.
- All the duplication also means that many of the same refs are being
  loaded repeatedly, which can cause some slowdown but, more notably,
  eats up huge amounts of memory.
- Adding on more and more new repositories and snapshots will cause the
  time and memory spent on things like cleanup and publishing to grow
  roughly linearly.

At the core, there are two problems here:

- Reflists get very big because there are just a lot of packages.
- Different reflists can tend to duplicate much of the same contents.

*Split reflists* aim at solving this by separating reflists into 64
*buckets*. Package refs are sorted into individual buckets according to
the following system:

- Take the first 3 letters of the package name, after dropping a `lib`
  prefix. (Using only the first 3 letters will cause packages with
  similar prefixes to end up in the same bucket, under the assumption
  that packages with similar names tend to be updated together.)
- Take the 64-bit xxhash of these letters. (xxhash was chosen because it
  relatively good distribution across the individual bits, which is
  important for the next step.)
- Use the first 6 bits of the hash (range [0:63]) as an index into the
  buckets.

Once refs are placed in buckets, a sha256 digest of all the refs in the
bucket is taken. These buckets are then stored in the database, split
into roughly block-sized segments, and all the repositories and
snapshots simply store an array of bucket digests.

This approach means that *repositories and snapshots can share their
reflist buckets*. If a snapshot is taken of a repository, it will have
the same contents, so its split reflist will point to the same buckets
as the base repository, and only one copy of each bucket is stored in
the database. When some packages in the repository change, only the
buckets containing those packages will be modified; all the other
buckets will remain unchanged, and thus their contents will still be
shared. Later on, when these reflists are loaded, each bucket is only
loaded once, short-cutting loaded many megabytes of data. In effect,
split reflists are essentially copy-on-write, with only the changed
buckets stored individually.

Changing the disk format means that a migration needs to take place, so
that task is moved into the database cleanup step, which will migrate
reflists over to split reflists, as well as delete any unused reflist
buckets.

All the reflist tests are also changed to additionally test out split
reflists; although the internal logic is all shared (since buckets are,
themselves, just normal reflists), some special additions are needed to
have native versions of the various reflist helper methods.

In our tests, we've observed the following improvements:

- Memory usage during publish and database cleanup, with
  `GOMEMLIMIT=2GiB`, goes down from ~3.2GiB (larger than the memory
  limit!) to ~0.7GiB, a decrease of ~4.5x.
- Database size decreases from 1.3GB to 367MB.

*In my local tests*, publish times had also decreased down to mere
seconds but the same effect wasn't observed on the server, with the
times staying around the same. My suspicions are that this is due to I/O
performance: my local system is an M1 MBP, which almost certainly has
much faster disk speeds than our DigitalOcean block volumes. Split
reflists include a side effect of requiring more random accesses from
reading all the buckets by their keys, so if your random I/O
performance is slower, it might cancel out the benefits. That being
said, even in that case, the memory usage and database size advantages
still persist.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

.github/workflows/ci.yml

@@ -30,12 +30,12 @@ jobs:
       GOPROXY: "https://proxy.golang.org"
 
     steps:
-      - name: "Install packages"
+      - name: "Install Packages"
         run: |
           sudo apt-get update
           sudo apt-get install -y --no-install-recommends graphviz gnupg2 gpgv2 git gcc make devscripts python3 python3-requests-unixsocket python3-termcolor python3-swiftclient python3-boto python3-azure-storage python3-etcd3 python3-plyvel flake8
 
-      - name: "Checkout repository"
+      - name: "Checkout Repository"
         uses: actions/checkout@v4
         with:
           # fetch the whole repo for `git describe` to work
@@ -45,7 +45,7 @@ jobs:
         run: |
           make flake8
 
-      - name: "Read go version from go.mod"
+      - name: "Read Go Version"
         run: |
           gover=$(sed -n 's/^go \(.*\)/\1/p' go.mod)
           echo "Go Version: $gover"
@@ -91,12 +91,12 @@ jobs:
           sudo mkdir -p /srv ; sudo chown runner /srv
           COVERAGE_DIR=${{ runner.temp }} make system-test
 
-      - name: "Merge code coverage"
+      - name: "Merge Code Coverage"
         run: |
           go install github.com/wadey/gocovmerge@latest
           ~/go/bin/gocovmerge unit.out ${{ runner.temp }}/*.out > coverage.txt
 
-      - name: "Upload code coverage"
+      - name: "Upload Code Coverage"
         uses: codecov/codecov-action@v2
         with:
           token: ${{ secrets.CODECOV_TOKEN }}
@@ -148,13 +148,13 @@ jobs:
                           gcc-i686-linux-gnu gcc-arm-linux-gnueabihf gcc-aarch64-linux-gnu
           git config --global --add safe.directory "$GITHUB_WORKSPACE"
 
-      - name: "Checkout repository"
+      - name: "Checkout Repository"
         uses: actions/checkout@v4
         with:
           # fetch the whole repo for `git describe` to work
           fetch-depth: 0
 
-      - name: "Read go version from go.mod"
+      - name: "Read Go Version"
         run: |
           gover=$(sed -n 's/^go \(.*\)/\1/p' go.mod)
           echo "Go Version: $gover"
@@ -221,13 +221,13 @@ jobs:
           - goos: darwin
             goarch: arm
     steps:
-      - name: "Checkout repository"
+      - name: "Checkout Repository"
         uses: actions/checkout@v4
         with:
           # fetch the whole repo for `git describe` to work
           fetch-depth: 0
 
-      - name: "Read go version from go.mod"
+      - name: "Read Go Version"
         run: |
           echo "GOVER=$(sed -n 's/^go \(.*\)/\1/p' go.mod)" >> $GITHUB_OUTPUT
         id: goversion
@@ -274,12 +274,31 @@ jobs:
     needs: ci-binary-build
     if: startsWith(github.event.ref, 'refs/tags')
     steps:
+      - name: "Checkout Repository"
+        uses: actions/checkout@v4
+
+      - name: "Get aptly version"
+        env:
+          FORCE_CI: ${{ steps.force_ci.outputs.FORCE_CI }}
+        run: |
+           aptlyver=$(make -s version)
+           echo "Aptly Version: $aptlyver"
+           echo "VERSION=$aptlyver" >> $GITHUB_OUTPUT
+        id: releaseversion
+
       - name: "Download Artifacts"
         uses: actions/download-artifact@v4
         with:
           path: out/
 
+      - name: "Create Release Notes"
+        run: |
+          echo -e "## Changes\n\n" > out/release-notes.md
+          dpkg-parsechangelog -S Changes | tail -n +4 >> out/release-notes.md
+
       - name: "Release"
         uses: softprops/action-gh-release@v2
         with:
+          name: "Aptly Release ${{ steps.releaseversion.outputs.VERSION }}"
           files: "out/**/aptly_*.zip"
+          body_path: "out/release-notes.md"

.github/workflows/scripts/upload-artifacts.sh

@@ -67,11 +67,14 @@ cleanup() {
 }
 trap cleanup EXIT
 
+sleeptime=5
+retries=60
 wait_task()
 {
     _id=$1
     _success=0
-    for t in `seq 180`
+    sleep $sleeptime
+    for t in `seq $retries`
     do
         jsonret=`curl -fsS -u $aptly_user:$aptly_password ${aptly_api}/api/tasks/$_id`
         _state=`echo $jsonret | jq .State`
@@ -84,7 +87,7 @@ wait_task()
             echo Error: task failed
             return 1
         fi
-        sleep 1
+        sleep $sleeptime
     done
     if [ "$_success" -ne 1 ]; then
         echo Error: task timeout

Makefile

@@ -140,7 +140,7 @@ dpkg: prepare swagger  ## Build debian packages
 	mkdir -p build && mv ../*.deb build/ ; \
 	cd build && ls -l *.deb
 
-binaries: prepare swagger  ## Build binary releases (FreeBSD, MacOS, Linux tar)
+binaries: prepare swagger  ## Build binary releases (FreeBSD, macOS, Linux generic)
 	# build aptly
 	GOOS=$(GOOS) GOARCH=$(GOARCH) go build -o build/tmp/aptly -ldflags='-extldflags=-static'
 	# install
@@ -195,7 +195,7 @@ docker-serve:  ## Run development server (auto recompiling) on http://localhost:
 docker-lint:  ## Run golangci-lint in docker container
 	@docker run -it --rm -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper lint
 
-docker-binaries:  ## Build binary releases (FreeBSD, MacOS, Linux tar) in docker container
+docker-binaries:  ## Build binary releases (FreeBSD, macOS, Linux generic) in docker container
 	@docker run -it --rm -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper binaries
 
 docker-man:  ## Create man page in docker container

README.rst

@@ -1,7 +1,3 @@
-=====
-aptly
-=====
-
 .. image:: https://github.com/aptly-dev/aptly/actions/workflows/ci.yml/badge.svg
     :target: https://github.com/aptly-dev/aptly/actions
 
@@ -14,15 +10,18 @@ aptly
 .. image:: https://goreportcard.com/badge/github.com/aptly-dev/aptly
     :target: https://goreportcard.com/report/aptly-dev/aptly
 
+aptly
+=====
+
 Aptly is a swiss army knife for Debian repository management.
 
 .. image:: http://www.aptly.info/img/aptly_logo.png
     :target: http://www.aptly.info/
 
 Documentation is available at `http://www.aptly.info/ <http://www.aptly.info/>`_. For support please use
-mailing list `aptly-discuss <https://groups.google.com/forum/#!forum/aptly-discuss>`_.
+open `issues <https://github.com/aptly-dev/aptly/issues>`_ or `discussions <https://github.com/aptly-dev/aptly/discussions>`_.
 
-Aptly features: ("+" means planned features)
+Aptly features:
 
 * make mirrors of remote Debian/Ubuntu repositories, limiting by components/architectures
 * take snapshots of mirrors at any point in time, fixing state of repository at some moment of time
@@ -32,48 +31,50 @@ Aptly features: ("+" means planned features)
 * filter repository by search query, pulling dependencies when required
 * publish self-made packages as Debian repositories
 * REST API for remote access
-* mirror repositories "as-is" (without resigning with user's key) (+)
-* support for yum repositories (+)
 
-Current limitations:
+Any contributions are welcome! Please see `CONTRIBUTING.md <CONTRIBUTING.md>`_.
 
-* translations are not supported yet
+Installation
+=============
 
-Install Stable Version
+Aptly can be installed on several operating systems.
------------------------
 
-To install aptly on Debian/Ubuntu, add new repository to ``/etc/apt/sources.list``::
+Debian / Ubuntu
+----------------
 
-    deb http://repo.aptly.info/ squeeze main
+Aptly is provided in the following debian packages:
 
-And import key that is used to sign the release::
+* **aptly**: Includes the main Aptly binary, man pages, and shell completions
+* **aptly-api**: A systemd service for the REST API, using the global /etc/aptly.conf
+* **aptly-dbg**: Debug symbols for troubleshooting
 
-    $ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EE727D4449467F0E
+The packages can be installed on official `Debian <https://packages.debian.org/search?keywords=aptly>`_ and `Ubuntu <https://packages.ubuntu.com/search?keywords=aptly>`_ distributions.
 
-After that you can install aptly as any other software package::
+Upstream Debian Packages
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
-    $ apt-get update
+If a newer version (not available in Debian/Ubuntu) of aptly is required, upstream debian packages (built from git tags) can be installed as follows:
-    $ apt-get install aptly
 
-Don't worry about squeeze part in repo name: aptly package should work on Debian squeeze+,
+Install the following APT key (as root)::
-Ubuntu 10.0+. Package contains aptly binary, man page and bash completion.
 
-Other Binaries
+    wget -O /etc/apt/keyrings/aptly.asc https://www.aptly.info/pubkey.txt
-~~~~~~~~~~~~~~~~~
 
-Binary executables (depends almost only on libc) are available for download from `GitHub Releases <https://github.com/aptly-dev/aptly/releases>`_.
+Define Release APT sources in ``/etc/apt/sources.list.d/aptly.list``::
 
-Install CI Version
+    deb [signed-by=/etc/apt/keyrings/aptly.asc] http://repo.aptly.info/release DIST main
---------------------
 
-More recent versions are available as CI builds (development, might be unstable).
+Where DIST is one of: ``buster``, ``bullseye``, ``bookworm``, ``focal``, ``jammy``, ``noble``
 
-Debian GNU/Linux
+Install aptly packages::
-~~~~~~~~~~~~~~~~~
 
-Install the following APT key::
+    apt-get update
+    apt-get install aptly
+    apt-get install aptly-api  # REST API systemd service
 
-    sudo wget -O /etc/apt/keyrings/aptly.asc https://www.aptly.info/pubkey.txt
+CI Builds
+~~~~~~~~~~
+
+For testing new features or bugfixes, recent builds are available as CI builds (built from master, may be unstable!) and can be installed as follows:
 
 Define CI APT sources in ``/etc/apt/sources.list.d/aptly-ci.list``::
 
@@ -81,13 +82,19 @@ Define CI APT sources in ``/etc/apt/sources.list.d/aptly-ci.list``::
 
 Where DIST is one of: ``buster``, ``bullseye``, ``bookworm``, ``focal``, ``jammy``, ``noble``
 
-Contributing
+Note: same gpg key is used as for the Upstream Debian Packages.
-------------
 
-Please follow detailed documentation in `CONTRIBUTING.md <CONTRIBUTING.md>`_.
+Other Operating Systems
+------------------------
+
+Binary executables (depends almost only on libc) are available on `GitHub Releases <https://github.com/aptly-dev/aptly/releases>`_ for:
+
+- macOS / darwin (amd64, arm64)
+- FreeBSD (amd64, arm64, 386, arm)
+- Generic Linux (amd64, arm64, 386, arm)
 
 Integrations
-------------
+=============
 
 Vagrant:
 
@@ -124,3 +131,7 @@ GUI for aptly API:
 Scala sbt:
 
 -   `sbt aptly plugin <https://github.com/amalakar/sbt-aptly>`_ by Arup Malakar
+
+Molior:
+
+-   `Molior Debian Build System <https://github.com/molior-dbs/molior>`_ by André Roth

Releasing.md

@@ -0,0 +1,16 @@
+# Creating a Release
+
+- create branch release/1.x.y
+- update debian/changelog
+- create PR, merge when approved
+- on updated master, create release:
+```
+version=$(dpkg-parsechangelog -S Version)
+echo Releasing prod version $version
+git tag -a v$version -m 'aptly: release $version'
+git push origin v$version master
+```
+- run swagger locally (`make docker-serve`)
+- copy generated docs/swagger.json to https://github.com/aptly-dev/www.aptly.info/tree/master/static/swagger/aptly_1.x.y.json
+- releae www.aptly.info
+- create release announcement on https://github.com/aptly-dev/aptly/discussions

api/api.go

@@ -229,7 +229,7 @@ func maybeRunTaskInBackground(c *gin.Context, name string, resources []string, p
 
 // Common piece of code to show list of packages,
 // with searching & details if requested
-func showPackages(c *gin.Context, reflist *deb.PackageRefList, collectionFactory *deb.CollectionFactory) {
+func showPackages(c *gin.Context, reflist deb.AnyRefList, collectionFactory *deb.CollectionFactory) {
 	result := []*deb.Package{}
 
 	list, err := deb.NewPackageListFromRefList(reflist, collectionFactory.PackageCollection(), nil)

api/db.go

@@ -5,6 +5,7 @@ import (
 	"sort"
 
 	"github.com/aptly-dev/aptly/aptly"
+	"github.com/aptly-dev/aptly/database"
 	"github.com/aptly-dev/aptly/deb"
 	"github.com/aptly-dev/aptly/task"
 	"github.com/aptly-dev/aptly/utils"
@@ -28,18 +29,22 @@ func apiDbCleanup(c *gin.Context) {
 
 		collectionFactory := context.NewCollectionFactory()
 
-		// collect information about referenced packages...
+		// collect information about referenced packages and their reflist buckets...
-		existingPackageRefs := deb.NewPackageRefList()
+		existingPackageRefs := deb.NewSplitRefList()
+		existingBuckets := deb.NewRefListDigestSet()
+
+		reflistMigration := collectionFactory.RefListCollection().NewMigration()
 
 		out.Printf("Loading mirrors, local repos, snapshots and published repos...")
 		err = collectionFactory.RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
-			e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+			sl := deb.NewSplitRefList()
-			if e != nil {
+			e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
+			if e != nil && e != database.ErrNotFound {
 				return e
 			}
-			if repo.RefList() != nil {
+
-				existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
+			existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
-			}
+			existingBuckets.AddAllInRefList(sl)
 
 			return nil
 		})
@@ -48,14 +53,14 @@ func apiDbCleanup(c *gin.Context) {
 		}
 
 		err = collectionFactory.LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
-			e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
+			sl := deb.NewSplitRefList()
-			if e != nil {
+			e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
+			if e != nil && e != database.ErrNotFound {
 				return e
 			}
 
-			if repo.RefList() != nil {
+			existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
-				existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
+			existingBuckets.AddAllInRefList(sl)
-			}
 
 			return nil
 		})
@@ -64,12 +69,14 @@ func apiDbCleanup(c *gin.Context) {
 		}
 
 		err = collectionFactory.SnapshotCollection().ForEach(func(snapshot *deb.Snapshot) error {
-			e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+			sl := deb.NewSplitRefList()
+			e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, snapshot.RefKey(), reflistMigration)
 			if e != nil {
 				return e
 			}
 
-			existingPackageRefs = existingPackageRefs.Merge(snapshot.RefList(), false, true)
+			existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+			existingBuckets.AddAllInRefList(sl)
 
 			return nil
 		})
@@ -81,13 +88,16 @@ func apiDbCleanup(c *gin.Context) {
 			if published.SourceKind != deb.SourceLocalRepo {
 				return nil
 			}
-			e := collectionFactory.PublishedRepoCollection().LoadComplete(published, collectionFactory)
+
+			for _, component := range published.Components() {
+				sl := deb.NewSplitRefList()
+				e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, published.RefKey(component), reflistMigration)
 				if e != nil {
 					return e
 				}
 
-			for _, component := range published.Components() {
+				existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
-				existingPackageRefs = existingPackageRefs.Merge(published.RefList(component), false, true)
+				existingBuckets.AddAllInRefList(sl)
 			}
 			return nil
 		})
@@ -95,11 +105,20 @@ func apiDbCleanup(c *gin.Context) {
 			return nil, err
 		}
 
+		err = reflistMigration.Flush()
+		if err != nil {
+			return nil, err
+		}
+		if stats := reflistMigration.Stats(); stats.Reflists > 0 {
+			out.Printf("Split %d reflist(s) into %d bucket(s) (%d segment(s))",
+				stats.Reflists, stats.Buckets, stats.Segments)
+		}
+
 		// ... and compare it to the list of all packages
 		out.Printf("Loading list of all packages...")
 		allPackageRefs := collectionFactory.PackageCollection().AllPackageRefs()
 
-		toDelete := allPackageRefs.Subtract(existingPackageRefs)
+		toDelete := allPackageRefs.Subtract(existingPackageRefs.Flatten())
 
 		// delete packages that are no longer referenced
 		out.Printf("Deleting unreferenced packages (%d)...", toDelete.Len())
@@ -120,6 +139,28 @@ func apiDbCleanup(c *gin.Context) {
 			}
 		}
 
+		bucketsToDelete, err := collectionFactory.RefListCollection().AllBucketDigests()
+		if err != nil {
+			return nil, err
+		}
+
+		bucketsToDelete.RemoveAll(existingBuckets)
+
+		out.Printf("Deleting unreferenced reflist buckets (%d)...", bucketsToDelete.Len())
+		if bucketsToDelete.Len() > 0 {
+			batch := db.CreateBatch()
+			err := bucketsToDelete.ForEach(func(digest []byte) error {
+				return collectionFactory.RefListCollection().UnsafeDropBucket(digest, batch)
+			})
+			if err != nil {
+				return nil, err
+			}
+
+			if err := batch.Write(); err != nil {
+				return nil, err
+			}
+		}
+
 		// now, build a list of files that should be present in Repository (package pool)
 		out.Printf("Building list of files referenced by packages...")
 		referencedFiles := make([]string, 0, existingPackageRefs.Len())

api/files.go

@@ -170,11 +170,7 @@ func apiFilesListFiles(c *gin.Context) {
 	listLock := &sync.Mutex{}
 	root := filepath.Join(context.UploadPath(), utils.SanitizePath(c.Params.ByName("dir")))
 
-	err := filepath.Walk(root, func(path string, _ os.FileInfo, err error) error {
+	err := walker.Walk(root, func(path string, _ os.FileInfo) error {
-		if err != nil {
-			return err
-		}
-
 		if path == root {
 			return nil
 		}

api/metrics.go

@@ -102,7 +102,7 @@ func countPackagesByRepos() {
 
 		components := repo.Components()
 		for _, c := range components {
-			count := float64(len(repo.RefList(c).Refs))
+			count := float64(repo.RefList(c).Len())
 			apiReposPackageCountGauge.WithLabelValues(fmt.Sprintf("%s", (repo.SourceNames())), repo.Distribution, c).Set(count)
 		}
 

api/mirror.go

@@ -150,7 +150,7 @@ func apiMirrorsCreate(c *gin.Context) {
 		return
 	}
 
-	err = collection.Add(repo)
+	err = collection.Add(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, fmt.Errorf("unable to add mirror: %s", err))
 		return
@@ -229,7 +229,7 @@ func apiMirrorsShow(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(repo)
+	err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, fmt.Errorf("unable to show: %s", err))
 	}
@@ -260,7 +260,7 @@ func apiMirrorsPackages(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(repo)
+	err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, fmt.Errorf("unable to show: %s", err))
 	}
@@ -491,12 +491,12 @@ func apiMirrorsUpdate(c *gin.Context) {
 			e := context.ReOpenDatabase()
 			if e == nil {
 				remote.MarkAsIdle()
-				collection.Update(remote)
+				collection.Update(remote, collectionFactory.RefListCollection())
 			}
 		}()
 
 		remote.MarkAsUpdating()
-		err = collection.Update(remote)
+		err = collection.Update(remote, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
 		}
@@ -654,7 +654,7 @@ func apiMirrorsUpdate(c *gin.Context) {
 
 		log.Info().Msgf("%s: Finalizing download...", b.Name)
 		remote.FinalizeDownload(collectionFactory, out)
-		err = collectionFactory.RemoteRepoCollection().Update(remote)
+		err = collectionFactory.RemoteRepoCollection().Update(remote, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
 		}

api/publish.go

@@ -297,10 +297,10 @@ func apiPublishRepoOrSnapshot(c *gin.Context) {
 			switch s := source.(type) {
 			case *deb.Snapshot:
 				snapshotCollection := collectionFactory.SnapshotCollection()
-				err = snapshotCollection.LoadComplete(s)
+				err = snapshotCollection.LoadComplete(s, collectionFactory.RefListCollection())
 			case *deb.LocalRepo:
 				localCollection := collectionFactory.LocalRepoCollection()
-				err = localCollection.LoadComplete(s)
+				err = localCollection.LoadComplete(s, collectionFactory.RefListCollection())
 			default:
 				err = fmt.Errorf("unexpected type for source: %T", source)
 			}
@@ -352,7 +352,7 @@ func apiPublishRepoOrSnapshot(c *gin.Context) {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to publish: %s", err)
 		}
 
-		err = collection.Add(published)
+		err = collection.Add(published, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
 		}
@@ -469,7 +469,7 @@ func apiPublishUpdateSwitch(c *gin.Context) {
 	resources := []string{string(published.Key())}
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
 	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		err = collection.LoadComplete(published, collectionFactory)
+		err = collection.LoadComplete(published, collectionFactory, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("Unable to update: %s", err)
 		}
@@ -495,7 +495,7 @@ func apiPublishUpdateSwitch(c *gin.Context) {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("Unable to update: %s", err)
 		}
 
-		err = collection.Update(published)
+		err = collection.Update(published, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
 		}

api/repos.go

@@ -155,6 +155,13 @@ func apiReposCreate(c *gin.Context) {
 		return
 	}
 
+	collection := collectionFactory.LocalRepoCollection()
+	err := collection.Add(repo, collectionFactory.RefListCollection())
+	if err != nil {
+		AbortWithJSONError(c, 400, err)
+		return
+	}
+
 	err := localRepoCollection.Add(repo)
 	if err != nil {
 		AbortWithJSONError(c, http.StatusInternalServerError, err)
@@ -218,7 +225,7 @@ func apiReposEdit(c *gin.Context) {
 		repo.DefaultComponent = *b.DefaultComponent
 	}
 
-	err = collection.Update(repo)
+	err = collection.Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -326,7 +333,7 @@ func apiReposPackagesShow(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(repo)
+	err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -360,7 +367,7 @@ func apiReposPackagesAddDelete(c *gin.Context, taskNamePrefix string, cb func(li
 	resources := []string{string(repo.Key())}
 
 	maybeRunTaskInBackground(c, taskNamePrefix+repo.Name, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		err = collection.LoadComplete(repo)
+		err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -389,9 +396,9 @@ func apiReposPackagesAddDelete(c *gin.Context, taskNamePrefix string, cb func(li
 			}
 		}
 
-		repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
+		repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
 
-		err = collectionFactory.LocalRepoCollection().Update(repo)
+		err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save: %s", err)
 		}
@@ -517,7 +524,7 @@ func apiReposPackageFromDir(c *gin.Context) {
 	resources := []string{string(repo.Key())}
 	resources = append(resources, sources...)
 	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		err = collection.LoadComplete(repo)
+		err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -552,9 +559,9 @@ func apiReposPackageFromDir(c *gin.Context) {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to import package files: %s", err)
 		}
 
-		repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
+		repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
 
-		err = collectionFactory.LocalRepoCollection().Update(repo)
+		err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save: %s", err)
 		}
@@ -867,7 +874,7 @@ func apiReposIncludePackageFromDir(c *gin.Context) {
 		_, failedFiles2, err = deb.ImportChangesFiles(
 			changesFiles, reporter, acceptUnsigned, ignoreSignature, forceReplace, noRemoveFiles, verifier,
 			repoTemplate, context.Progress(), collectionFactory.LocalRepoCollection(), collectionFactory.PackageCollection(),
-			context.PackagePool(), collectionFactory.ChecksumCollection, nil, query.Parse)
+			collectionFactory.RefListCollection(), context.PackagePool(), collectionFactory.ChecksumCollection, nil, query.Parse)
 		failedFiles = append(failedFiles, failedFiles2...)
 
 		if err != nil {

api/snapshot.go

@@ -93,7 +93,7 @@ func apiSnapshotsCreateFromMirror(c *gin.Context) {
 			return &task.ProcessReturnValue{Code: http.StatusConflict, Value: nil}, err
 		}
 
-		err = collection.LoadComplete(repo)
+		err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -107,7 +107,7 @@ func apiSnapshotsCreateFromMirror(c *gin.Context) {
 			snapshot.Description = b.Description
 		}
 
-		err = snapshotCollection.Add(snapshot)
+		err = snapshotCollection.Add(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, err
 		}
@@ -174,7 +174,7 @@ func apiSnapshotsCreate(c *gin.Context) {
 
 	maybeRunTaskInBackground(c, "Create snapshot "+b.Name, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
 		for i := range sources {
-			err = snapshotCollection.LoadComplete(sources[i])
+			err = snapshotCollection.LoadComplete(sources[i], collectionFactory.RefListCollection())
 			if err != nil {
 				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 			}
@@ -197,9 +197,9 @@ func apiSnapshotsCreate(c *gin.Context) {
 			}
 		}
 
-		snapshot = deb.NewSnapshotFromRefList(b.Name, sources, deb.NewPackageRefListFromPackageList(list), b.Description)
+		snapshot = deb.NewSnapshotFromRefList(b.Name, sources, deb.NewSplitRefListFromPackageList(list), b.Description)
 
-		err = snapshotCollection.Add(snapshot)
+		err = snapshotCollection.Add(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, err
 		}
@@ -255,7 +255,7 @@ func apiSnapshotsCreateFromRepository(c *gin.Context) {
 	resources := []string{string(repo.Key()), "S" + b.Name}
 	taskName := fmt.Sprintf("Create snapshot of repo %s", name)
 	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		err := collection.LoadComplete(repo)
+		err := collection.LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -269,7 +269,7 @@ func apiSnapshotsCreateFromRepository(c *gin.Context) {
 			snapshot.Description = b.Description
 		}
 
-		err = snapshotCollection.Add(snapshot)
+		err = snapshotCollection.Add(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, err
 		}
@@ -333,7 +333,7 @@ func apiSnapshotsUpdate(c *gin.Context) {
 			snapshot.Description = b.Description
 		}
 
-		err = collectionFactory.SnapshotCollection().Update(snapshot)
+		err = collectionFactory.SnapshotCollection().Update(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -360,7 +360,7 @@ func apiSnapshotsShow(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(snapshot)
+	err = collection.LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -452,20 +452,20 @@ func apiSnapshotsDiff(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(snapshotA)
+	err = collection.LoadComplete(snapshotA, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
 	}
 
-	err = collection.LoadComplete(snapshotB)
+	err = collection.LoadComplete(snapshotB, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
 	}
 
 	// Calculate diff
-	diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection())
+	diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection(), nil)
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -508,7 +508,7 @@ func apiSnapshotsSearchPackages(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(snapshot)
+	err = collection.LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -584,13 +584,13 @@ func apiSnapshotsMerge(c *gin.Context) {
 	}
 
 	maybeRunTaskInBackground(c, "Merge snapshot "+name, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		err = snapshotCollection.LoadComplete(sources[0])
+		err = snapshotCollection.LoadComplete(sources[0], collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
 		result := sources[0].RefList()
 		for i := 1; i < len(sources); i++ {
-			err = snapshotCollection.LoadComplete(sources[i])
+			err = snapshotCollection.LoadComplete(sources[i], collectionFactory.RefListCollection())
 			if err != nil {
 				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 			}
@@ -609,7 +609,7 @@ func apiSnapshotsMerge(c *gin.Context) {
 		snapshot = deb.NewSnapshotFromRefList(name, sources, result,
 			fmt.Sprintf("Merged from sources: %s", strings.Join(sourceDescription, ", ")))
 
-		err = collectionFactory.SnapshotCollection().Add(snapshot)
+		err = collectionFactory.SnapshotCollection().Add(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to create snapshot: %s", err)
 		}

azure/public.go

@@ -12,7 +12,7 @@ import (
 	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/lease"
 	"github.com/aptly-dev/aptly/aptly"
 	"github.com/aptly-dev/aptly/utils"
-	"github.com/pborman/uuid"
+	"github.com/google/uuid"
 	"github.com/pkg/errors"
 )
 
@@ -179,7 +179,7 @@ func (storage *PublishedStorage) Filelist(prefix string) ([]string, error) {
 // Internal copy or move implementation
 func (storage *PublishedStorage) internalCopyOrMoveBlob(src, dst string, metadata map[string]*string, move bool) error {
 	const leaseDuration = 30
-	leaseID := uuid.NewRandom().String()
+	leaseID := uuid.NewString()
 
 	serviceClient := storage.az.client.ServiceClient()
 	containerClient := serviceClient.NewContainerClient(storage.az.container)

cmd/cmd.go

@@ -21,7 +21,7 @@ const (
 )
 
 // ListPackagesRefList shows list of packages in PackageRefList
-func ListPackagesRefList(reflist *deb.PackageRefList, collectionFactory *deb.CollectionFactory) (err error) {
+func ListPackagesRefList(reflist deb.AnyRefList, collectionFactory *deb.CollectionFactory) (err error) {
 	fmt.Printf("Packages:\n")
 
 	if reflist == nil {

cmd/db_cleanup.go

@@ -6,6 +6,7 @@ import (
 	"strings"
 
 	"github.com/aptly-dev/aptly/aptly"
+	"github.com/aptly-dev/aptly/database"
 	"github.com/aptly-dev/aptly/deb"
 	"github.com/aptly-dev/aptly/utils"
 	"github.com/smira/commander"
@@ -24,12 +25,20 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
 	dryRun := context.Flags().Lookup("dry-run").Value.Get().(bool)
 	collectionFactory := context.NewCollectionFactory()
 
-	// collect information about references packages...
+	// collect information about references packages and their reflistbuckets...
-	existingPackageRefs := deb.NewPackageRefList()
+	existingPackageRefs := deb.NewSplitRefList()
+	existingBuckets := deb.NewRefListDigestSet()
 
 	// used only in verbose mode to report package use source
 	packageRefSources := map[string][]string{}
 
+	var reflistMigration *deb.RefListMigration
+	if !dryRun {
+		reflistMigration = collectionFactory.RefListCollection().NewMigration()
+	} else {
+		reflistMigration = collectionFactory.RefListCollection().NewMigrationDryRun()
+	}
+
 	context.Progress().ColoredPrintf("@{w!}Loading mirrors, local repos, snapshots and published repos...@|")
 	if verbose {
 		context.Progress().ColoredPrintf("@{y}Loading mirrors:@|")
@@ -39,21 +48,22 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
 			context.Progress().ColoredPrintf("- @{g}%s@|", repo.Name)
 		}
 
-		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		sl := deb.NewSplitRefList()
-		if e != nil {
+		e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
+		if e != nil && e != database.ErrNotFound {
 			return e
 		}
-		if repo.RefList() != nil {
+
-			existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
+		existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+		existingBuckets.AddAllInRefList(sl)
 
 		if verbose {
 			description := fmt.Sprintf("mirror %s", repo.Name)
-				repo.RefList().ForEach(func(key []byte) error {
+			sl.ForEach(func(key []byte) error {
 				packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
 				return nil
 			})
 		}
-		}
 
 		return nil
 	})
@@ -71,22 +81,24 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
 			context.Progress().ColoredPrintf("- @{g}%s@|", repo.Name)
 		}
 
-		e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		sl := deb.NewSplitRefList()
-		if e != nil {
+		e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
+		if e != nil && e != database.ErrNotFound {
 			return e
 		}
 
-		if repo.RefList() != nil {
+		existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
-			existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
+		existingBuckets.AddAllInRefList(sl)
+
+		existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
 
 		if verbose {
 			description := fmt.Sprintf("local repo %s", repo.Name)
-				repo.RefList().ForEach(func(key []byte) error {
+			sl.ForEach(func(key []byte) error {
 				packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
 				return nil
 			})
 		}
-		}
 
 		return nil
 	})
@@ -104,16 +116,18 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
 			context.Progress().ColoredPrintf("- @{g}%s@|", snapshot.Name)
 		}
 
-		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		sl := deb.NewSplitRefList()
+		e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, snapshot.RefKey(), reflistMigration)
 		if e != nil {
 			return e
 		}
 
-		existingPackageRefs = existingPackageRefs.Merge(snapshot.RefList(), false, true)
+		existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+		existingBuckets.AddAllInRefList(sl)
 
 		if verbose {
 			description := fmt.Sprintf("snapshot %s", snapshot.Name)
-			snapshot.RefList().ForEach(func(key []byte) error {
+			sl.ForEach(func(key []byte) error {
 				packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
 				return nil
 			})
@@ -136,17 +150,21 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
 		if published.SourceKind != deb.SourceLocalRepo {
 			return nil
 		}
-		e := collectionFactory.PublishedRepoCollection().LoadComplete(published, collectionFactory)
+
+		for _, component := range published.Components() {
+			sl := deb.NewSplitRefList()
+			e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, published.RefKey(component), reflistMigration)
 			if e != nil {
 				return e
 			}
 
-		for _, component := range published.Components() {
+			existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
-			existingPackageRefs = existingPackageRefs.Merge(published.RefList(component), false, true)
+			existingBuckets.AddAllInRefList(sl)
+
 			if verbose {
 				description := fmt.Sprintf("published repository %s:%s/%s component %s",
 					published.Storage, published.Prefix, published.Distribution, component)
-				published.RefList(component).ForEach(func(key []byte) error {
+				sl.ForEach(func(key []byte) error {
 					packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
 					return nil
 				})
@@ -160,11 +178,29 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
 
 	collectionFactory.Flush()
 
+	err = reflistMigration.Flush()
+	if err != nil {
+		return err
+	}
+
+	if verbose {
+		if stats := reflistMigration.Stats(); stats.Reflists > 0 {
+			if !dryRun {
+				context.Progress().ColoredPrintf("@{w!}Split %d reflist(s) into %d bucket(s) (%d segment(s))@|",
+					stats.Reflists, stats.Buckets, stats.Segments)
+			} else {
+				context.Progress().ColoredPrintf(
+					"@{y!}Skipped splitting %d reflist(s) into %d bucket(s) (%d segment(s)), as -dry-run has been requested.@|",
+					stats.Reflists, stats.Buckets, stats.Segments)
+			}
+		}
+	}
+
 	// ... and compare it to the list of all packages
 	context.Progress().ColoredPrintf("@{w!}Loading list of all packages...@|")
 	allPackageRefs := collectionFactory.PackageCollection().AllPackageRefs()
 
-	toDelete := allPackageRefs.Subtract(existingPackageRefs)
+	toDelete := allPackageRefs.Subtract(existingPackageRefs.Flatten())
 
 	// delete packages that are no longer referenced
 	context.Progress().ColoredPrintf("@{r!}Deleting unreferenced packages (%d)...@|", toDelete.Len())
@@ -202,6 +238,32 @@ func aptlyDbCleanup(cmd *commander.Command, args []string) error {
 		}
 	}
 
+	bucketsToDelete, err := collectionFactory.RefListCollection().AllBucketDigests()
+	if err != nil {
+		return err
+	}
+
+	bucketsToDelete.RemoveAll(existingBuckets)
+
+	context.Progress().ColoredPrintf("@{r!}Deleting unreferenced reflist buckets (%d)...@|", bucketsToDelete.Len())
+	if bucketsToDelete.Len() > 0 {
+		if !dryRun {
+			batch := db.CreateBatch()
+			err := bucketsToDelete.ForEach(func(digest []byte) error {
+				return collectionFactory.RefListCollection().UnsafeDropBucket(digest, batch)
+			})
+			if err != nil {
+				return err
+			}
+
+			if err := batch.Write(); err != nil {
+				return err
+			}
+		} else {
+			context.Progress().ColoredPrintf("@{y!}Skipped reflist deletion, as -dry-run has been requested.@|")
+		}
+	}
+
 	collectionFactory.Flush()
 
 	// now, build a list of files that should be present in Repository (package pool)

cmd/mirror_create.go

@@ -69,7 +69,7 @@ func aptlyMirrorCreate(cmd *commander.Command, args []string) error {
 	}
 
 	collectionFactory := context.NewCollectionFactory()
-	err = collectionFactory.RemoteRepoCollection().Add(repo)
+	err = collectionFactory.RemoteRepoCollection().Add(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to add mirror: %s", err)
 	}

cmd/mirror_edit.go

@@ -78,7 +78,7 @@ func aptlyMirrorEdit(cmd *commander.Command, args []string) error {
 		}
 	}
 
-	err = collectionFactory.RemoteRepoCollection().Update(repo)
+	err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to edit: %s", err)
 	}

cmd/mirror_rename.go

@@ -37,7 +37,7 @@ func aptlyMirrorRename(cmd *commander.Command, args []string) error {
 	}
 
 	repo.Name = newName
-	err = collectionFactory.RemoteRepoCollection().Update(repo)
+	err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to rename: %s", err)
 	}

cmd/mirror_show.go

@@ -38,7 +38,7 @@ func aptlyMirrorShowTxt(_ *commander.Command, args []string) error {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -98,12 +98,13 @@ func aptlyMirrorShowJSON(_ *commander.Command, args []string) error {
 
 	name := args[0]
 
-	repo, err := context.NewCollectionFactory().RemoteRepoCollection().ByName(name)
+	collectionFactory := context.NewCollectionFactory()
+	repo, err := collectionFactory.RemoteRepoCollection().ByName(name)
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = context.NewCollectionFactory().RemoteRepoCollection().LoadComplete(repo)
+	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -113,7 +114,7 @@ func aptlyMirrorShowJSON(_ *commander.Command, args []string) error {
 	if withPackages {
 		if repo.RefList() != nil {
 			var list *deb.PackageList
-			list, err = deb.NewPackageListFromRefList(repo.RefList(), context.NewCollectionFactory().PackageCollection(), context.Progress())
+			list, err = deb.NewPackageListFromRefList(repo.RefList(), collectionFactory.PackageCollection(), context.Progress())
 			if err != nil {
 				return fmt.Errorf("unable to get package list: %s", err)
 			}

cmd/mirror_update.go

@@ -29,7 +29,7 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to update: %s", err)
 	}
 
-	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to update: %s", err)
 	}
@@ -101,12 +101,12 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
 		err = context.ReOpenDatabase()
 		if err == nil {
 			repo.MarkAsIdle()
-			collectionFactory.RemoteRepoCollection().Update(repo)
+			collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 		}
 	}()
 
 	repo.MarkAsUpdating()
-	err = collectionFactory.RemoteRepoCollection().Update(repo)
+	err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to update: %s", err)
 	}
@@ -262,7 +262,7 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
 	}
 
 	repo.FinalizeDownload(collectionFactory, context.Progress())
-	err = collectionFactory.RemoteRepoCollection().Update(repo)
+	err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to update: %s", err)
 	}

cmd/package_show.go

@@ -14,7 +14,7 @@ import (
 
 func printReferencesTo(p *deb.Package, collectionFactory *deb.CollectionFactory) (err error) {
 	err = collectionFactory.RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
-		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -30,7 +30,7 @@ func printReferencesTo(p *deb.Package, collectionFactory *deb.CollectionFactory)
 	}
 
 	err = collectionFactory.LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
-		e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -46,7 +46,7 @@ func printReferencesTo(p *deb.Package, collectionFactory *deb.CollectionFactory)
 	}
 
 	err = collectionFactory.SnapshotCollection().ForEach(func(snapshot *deb.Snapshot) error {
-		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}

cmd/publish_snapshot.go

@@ -49,7 +49,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
 				return fmt.Errorf("unable to publish: %s", err)
 			}
 
-			err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+			err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 			if err != nil {
 				return fmt.Errorf("unable to publish: %s", err)
 			}
@@ -85,7 +85,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
 				return fmt.Errorf("unable to publish: %s", err)
 			}
 
-			err = collectionFactory.LocalRepoCollection().LoadComplete(localRepo)
+			err = collectionFactory.LocalRepoCollection().LoadComplete(localRepo, collectionFactory.RefListCollection())
 			if err != nil {
 				return fmt.Errorf("unable to publish: %s", err)
 			}
@@ -175,7 +175,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to publish: %s", err)
 	}
 
-	err = collectionFactory.PublishedRepoCollection().Add(published)
+	err = collectionFactory.PublishedRepoCollection().Add(published, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to save to DB: %s", err)
 	}

cmd/publish_switch.go

@@ -72,7 +72,7 @@ func aptlyPublishSwitch(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to switch: %s", err)
 		}
 
-		err = snapshotCollection.LoadComplete(snapshot)
+		err = snapshotCollection.LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to switch: %s", err)
 		}
@@ -108,7 +108,7 @@ func aptlyPublishSwitch(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to publish: %s", err)
 	}
 
-	err = collectionFactory.PublishedRepoCollection().Update(published)
+	err = collectionFactory.PublishedRepoCollection().Update(published, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to save to DB: %s", err)
 	}

cmd/publish_update.go

@@ -69,7 +69,7 @@ func aptlyPublishUpdate(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to publish: %s", err)
 	}
 
-	err = collectionFactory.PublishedRepoCollection().Update(published)
+	err = collectionFactory.PublishedRepoCollection().Update(published, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to save to DB: %s", err)
 	}

cmd/repo_add.go

@@ -28,7 +28,7 @@ func aptlyRepoAdd(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to add: %s", err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to add: %s", err)
 	}
@@ -58,9 +58,9 @@ func aptlyRepoAdd(cmd *commander.Command, args []string) error {
 
 	processedFiles = append(processedFiles, otherFiles...)
 
-	repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
+	repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
 
-	err = collectionFactory.LocalRepoCollection().Update(repo)
+	err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to save: %s", err)
 	}

cmd/repo_create.go

@@ -36,7 +36,7 @@ func aptlyRepoCreate(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to load source snapshot: %s", err)
 		}
 
-		err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to load source snapshot: %s", err)
 		}
@@ -44,7 +44,7 @@ func aptlyRepoCreate(cmd *commander.Command, args []string) error {
 		repo.UpdateRefList(snapshot.RefList())
 	}
 
-	err = collectionFactory.LocalRepoCollection().Add(repo)
+	err = collectionFactory.LocalRepoCollection().Add(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to add local repo: %s", err)
 	}

cmd/repo_edit.go

@@ -22,7 +22,7 @@ func aptlyRepoEdit(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to edit: %s", err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to edit: %s", err)
 	}
@@ -53,7 +53,7 @@ func aptlyRepoEdit(cmd *commander.Command, args []string) error {
 		}
 	}
 
-	err = collectionFactory.LocalRepoCollection().Update(repo)
+	err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to edit: %s", err)
 	}

cmd/repo_include.go

@@ -67,7 +67,7 @@ func aptlyRepoInclude(cmd *commander.Command, args []string) error {
 	_, failedFiles2, err = deb.ImportChangesFiles(
 		changesFiles, reporter, acceptUnsigned, ignoreSignatures, forceReplace, noRemoveFiles, verifier, repoTemplate,
 		context.Progress(), collectionFactory.LocalRepoCollection(), collectionFactory.PackageCollection(),
-		context.PackagePool(), collectionFactory.ChecksumCollection,
+		collectionFactory.RefListCollection(), context.PackagePool(), collectionFactory.ChecksumCollection,
 		uploaders, query.Parse)
 	failedFiles = append(failedFiles, failedFiles2...)
 

cmd/repo_list.go

@@ -36,7 +36,7 @@ func aptlyRepoListTxt(cmd *commander.Command, _ []string) error {
 		if raw {
 			repos[i] = repo.Name
 		} else {
-			e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
+			e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 			if e != nil {
 				return e
 			}
@@ -77,7 +77,8 @@ func aptlyRepoListJSON(_ *commander.Command, _ []string) error {
 	repos := make([]*deb.LocalRepo, context.NewCollectionFactory().LocalRepoCollection().Len())
 	i := 0
 	context.NewCollectionFactory().LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
-		e := context.NewCollectionFactory().LocalRepoCollection().LoadComplete(repo)
+		collectionFactory := context.NewCollectionFactory()
+		e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}

cmd/repo_move.go

@@ -25,13 +25,13 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to %s: %s", command, err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(dstRepo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(dstRepo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to %s: %s", command, err)
 	}
 
 	var (
-		srcRefList *deb.PackageRefList
+		srcRefList *deb.SplitRefList
 		srcRepo    *deb.LocalRepo
 	)
 
@@ -45,7 +45,7 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to %s: source and destination are the same", command)
 		}
 
-		err = collectionFactory.LocalRepoCollection().LoadComplete(srcRepo)
+		err = collectionFactory.LocalRepoCollection().LoadComplete(srcRepo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to %s: %s", command, err)
 		}
@@ -59,12 +59,12 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to %s: %s", command, err)
 		}
 
-		err = collectionFactory.RemoteRepoCollection().LoadComplete(srcRemoteRepo)
+		err = collectionFactory.RemoteRepoCollection().LoadComplete(srcRemoteRepo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to %s: %s", command, err)
 		}
 
-		if srcRemoteRepo.RefList() == nil {
+		if srcRemoteRepo.RefList().Len() == 0 {
 			return fmt.Errorf("unable to %s: mirror not updated", command)
 		}
 
@@ -161,17 +161,17 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
 	if context.Flags().Lookup("dry-run").Value.Get().(bool) {
 		context.Progress().Printf("\nChanges not saved, as dry run has been requested.\n")
 	} else {
-		dstRepo.UpdateRefList(deb.NewPackageRefListFromPackageList(dstList))
+		dstRepo.UpdateRefList(deb.NewSplitRefListFromPackageList(dstList))
 
-		err = collectionFactory.LocalRepoCollection().Update(dstRepo)
+		err = collectionFactory.LocalRepoCollection().Update(dstRepo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to save: %s", err)
 		}
 
 		if command == "move" { // nolint: goconst
-			srcRepo.UpdateRefList(deb.NewPackageRefListFromPackageList(srcList))
+			srcRepo.UpdateRefList(deb.NewSplitRefListFromPackageList(srcList))
 
-			err = collectionFactory.LocalRepoCollection().Update(srcRepo)
+			err = collectionFactory.LocalRepoCollection().Update(srcRepo, collectionFactory.RefListCollection())
 			if err != nil {
 				return fmt.Errorf("unable to save: %s", err)
 			}

cmd/repo_remove.go

@@ -24,7 +24,7 @@ func aptlyRepoRemove(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to remove: %s", err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to remove: %s", err)
 	}
@@ -63,9 +63,9 @@ func aptlyRepoRemove(cmd *commander.Command, args []string) error {
 	if context.Flags().Lookup("dry-run").Value.Get().(bool) {
 		context.Progress().Printf("\nChanges not saved, as dry run has been requested.\n")
 	} else {
-		repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
+		repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
 
-		err = collectionFactory.LocalRepoCollection().Update(repo)
+		err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to save: %s", err)
 		}

cmd/repo_rename.go

@@ -32,7 +32,7 @@ func aptlyRepoRename(cmd *commander.Command, args []string) error {
 	}
 
 	repo.Name = newName
-	err = collectionFactory.LocalRepoCollection().Update(repo)
+	err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to rename: %s", err)
 	}

cmd/repo_show.go

@@ -36,7 +36,7 @@ func aptlyRepoShowTxt(_ *commander.Command, args []string) error {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -63,12 +63,13 @@ func aptlyRepoShowJSON(_ *commander.Command, args []string) error {
 
 	name := args[0]
 
-	repo, err := context.NewCollectionFactory().LocalRepoCollection().ByName(name)
+	collectionFactory := context.NewCollectionFactory()
+	repo, err := collectionFactory.LocalRepoCollection().ByName(name)
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = context.NewCollectionFactory().LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -79,7 +80,7 @@ func aptlyRepoShowJSON(_ *commander.Command, args []string) error {
 	if withPackages {
 		if repo.RefList() != nil {
 			var list *deb.PackageList
-			list, err = deb.NewPackageListFromRefList(repo.RefList(), context.NewCollectionFactory().PackageCollection(), context.Progress())
+			list, err = deb.NewPackageListFromRefList(repo.RefList(), collectionFactory.PackageCollection(), context.Progress())
 			if err == nil {
 				packageList = list.FullNames()
 			}

cmd/snapshot_create.go

@@ -30,7 +30,7 @@ func aptlySnapshotCreate(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}
 
-		err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}
@@ -50,7 +50,7 @@ func aptlySnapshotCreate(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}
 
-		err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}
@@ -71,7 +71,7 @@ func aptlySnapshotCreate(cmd *commander.Command, args []string) error {
 		return commander.ErrCommandError
 	}
 
-	err = collectionFactory.SnapshotCollection().Add(snapshot)
+	err = collectionFactory.SnapshotCollection().Add(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to add snapshot: %s", err)
 	}

cmd/snapshot_diff.go

@@ -23,7 +23,7 @@ func aptlySnapshotDiff(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to load snapshot A: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(snapshotA)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshotA, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to load snapshot A: %s", err)
 	}
@@ -34,13 +34,13 @@ func aptlySnapshotDiff(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to load snapshot B: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(snapshotB)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshotB, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to load snapshot B: %s", err)
 	}
 
 	// Calculate diff
-	diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection())
+	diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection(), nil)
 	if err != nil {
 		return fmt.Errorf("unable to calculate diff: %s", err)
 	}

cmd/snapshot_filter.go

@@ -27,7 +27,7 @@ func aptlySnapshotFilter(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to filter: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(source)
+	err = collectionFactory.SnapshotCollection().LoadComplete(source, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to filter: %s", err)
 	}
@@ -87,7 +87,7 @@ func aptlySnapshotFilter(cmd *commander.Command, args []string) error {
 	destination := deb.NewSnapshotFromPackageList(args[1], []*deb.Snapshot{source}, result,
 		fmt.Sprintf("Filtered '%s', query was: '%s'", source.Name, strings.Join(args[2:], " ")))
 
-	err = collectionFactory.SnapshotCollection().Add(destination)
+	err = collectionFactory.SnapshotCollection().Add(destination, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to create snapshot: %s", err)
 	}

cmd/snapshot_merge.go

@@ -24,7 +24,7 @@ func aptlySnapshotMerge(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to load snapshot: %s", err)
 		}
 
-		err = collectionFactory.SnapshotCollection().LoadComplete(sources[i])
+		err = collectionFactory.SnapshotCollection().LoadComplete(sources[i], collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to load snapshot: %s", err)
 		}
@@ -57,7 +57,7 @@ func aptlySnapshotMerge(cmd *commander.Command, args []string) error {
 	destination := deb.NewSnapshotFromRefList(args[0], sources, result,
 		fmt.Sprintf("Merged from sources: %s", strings.Join(sourceDescription, ", ")))
 
-	err = collectionFactory.SnapshotCollection().Add(destination)
+	err = collectionFactory.SnapshotCollection().Add(destination, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to create snapshot: %s", err)
 	}

cmd/snapshot_pull.go

@@ -29,7 +29,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to pull: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to pull: %s", err)
 	}
@@ -40,7 +40,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to pull: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(source)
+	err = collectionFactory.SnapshotCollection().LoadComplete(source, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to pull: %s", err)
 	}
@@ -149,7 +149,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
 		destination := deb.NewSnapshotFromPackageList(args[2], []*deb.Snapshot{snapshot, source}, packageList,
 			fmt.Sprintf("Pulled into '%s' with '%s' as source, pull request was: '%s'", snapshot.Name, source.Name, strings.Join(args[3:], " ")))
 
-		err = collectionFactory.SnapshotCollection().Add(destination)
+		err = collectionFactory.SnapshotCollection().Add(destination, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}

cmd/snapshot_rename.go

@@ -32,7 +32,7 @@ func aptlySnapshotRename(cmd *commander.Command, args []string) error {
 	}
 
 	snapshot.Name = newName
-	err = collectionFactory.SnapshotCollection().Update(snapshot)
+	err = collectionFactory.SnapshotCollection().Update(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to rename: %s", err)
 	}

cmd/snapshot_search.go

@@ -25,7 +25,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 	command := cmd.Parent.Name()
 	collectionFactory := context.NewCollectionFactory()
 
-	var reflist *deb.PackageRefList
+	var reflist *deb.SplitRefList
 
 	if command == "snapshot" { // nolint: goconst
 		var snapshot *deb.Snapshot
@@ -34,7 +34,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 			return fmt.Errorf("unable to search: %s", err)
 		}
 
-		err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to search: %s", err)
 		}
@@ -47,7 +47,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 			return fmt.Errorf("unable to search: %s", err)
 		}
 
-		err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to search: %s", err)
 		}
@@ -60,7 +60,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 			return fmt.Errorf("unable to search: %s", err)
 		}
 
-		err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to search: %s", err)
 		}

cmd/snapshot_show.go

@@ -35,7 +35,7 @@ func aptlySnapshotShowTxt(_ *commander.Command, args []string) error {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -86,16 +86,17 @@ func aptlySnapshotShowTxt(_ *commander.Command, args []string) error {
 }
 
 func aptlySnapshotShowJSON(_ *commander.Command, args []string) error {
+	collectionFactory := context.NewCollectionFactory()
 	var err error
 
 	name := args[0]
 
-	snapshot, err := context.NewCollectionFactory().SnapshotCollection().ByName(name)
+	snapshot, err := collectionFactory.SnapshotCollection().ByName(name)
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = context.NewCollectionFactory().SnapshotCollection().LoadComplete(snapshot)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -105,14 +106,14 @@ func aptlySnapshotShowJSON(_ *commander.Command, args []string) error {
 		for _, sourceID := range snapshot.SourceIDs {
 			if snapshot.SourceKind == deb.SourceSnapshot {
 				var source *deb.Snapshot
-				source, err = context.NewCollectionFactory().SnapshotCollection().ByUUID(sourceID)
+				source, err = collectionFactory.SnapshotCollection().ByUUID(sourceID)
 				if err != nil {
 					continue
 				}
 				snapshot.Snapshots = append(snapshot.Snapshots, source)
 			} else if snapshot.SourceKind == deb.SourceLocalRepo {
 				var source *deb.LocalRepo
-				source, err = context.NewCollectionFactory().LocalRepoCollection().ByUUID(sourceID)
+				source, err = collectionFactory.LocalRepoCollection().ByUUID(sourceID)
 				if err != nil {
 					continue
 				}
@@ -133,7 +134,7 @@ func aptlySnapshotShowJSON(_ *commander.Command, args []string) error {
 	if withPackages {
 		if snapshot.RefList() != nil {
 			var list *deb.PackageList
-			list, err = deb.NewPackageListFromRefList(snapshot.RefList(), context.NewCollectionFactory().PackageCollection(), context.Progress())
+			list, err = deb.NewPackageListFromRefList(snapshot.RefList(), collectionFactory.PackageCollection(), context.Progress())
 			if err != nil {
 				return fmt.Errorf("unable to get package list: %s", err)
 			}

cmd/snapshot_verify.go

@@ -23,7 +23,7 @@ func aptlySnapshotVerify(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to verify: %s", err)
 		}
 
-		err = collectionFactory.SnapshotCollection().LoadComplete(snapshots[i])
+		err = collectionFactory.SnapshotCollection().LoadComplete(snapshots[i], collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to verify: %s", err)
 		}

database/database.go

@@ -48,6 +48,8 @@ type Storage interface {
 
 	CreateTemporary() (Storage, error)
 
+	GetRecommendedMaxKVSize() int
+
 	Open() error
 	Close() error
 	CompactDB() error

database/etcddb/storage.go

@@ -1,11 +1,11 @@
 package etcddb
 
 import (
-	"github.com/aptly-dev/aptly/database"
-	"github.com/pborman/uuid"
-	clientv3 "go.etcd.io/etcd/client/v3"
-
 	"fmt"
+
+	"github.com/aptly-dev/aptly/database"
+	"github.com/google/uuid"
+	clientv3 "go.etcd.io/etcd/client/v3"
 )
 
 type EtcDStorage struct {
@@ -16,7 +16,7 @@ type EtcDStorage struct {
 
 // CreateTemporary creates new DB of the same type in temp dir
 func (s *EtcDStorage) CreateTemporary() (database.Storage, error) {
-	tmp := uuid.NewRandom().String()
+	tmp := uuid.NewString()
 	return &EtcDStorage{
 		url:       s.url,
 		db:        s.db,

database/goleveldb/database.go

@@ -9,10 +9,13 @@ import (
 	"github.com/aptly-dev/aptly/database"
 )
 
+const blockSize = 4 * 1024
+
 func internalOpen(path string, throttleCompaction bool) (*leveldb.DB, error) {
 	o := &opt.Options{
 		Filter:                 filter.NewBloomFilter(10),
 		OpenFilesCacheCapacity: 256,
+		BlockSize:              blockSize,
 	}
 
 	if throttleCompaction {

database/goleveldb/storage.go

@@ -16,6 +16,17 @@ type storage struct {
 	db   *leveldb.DB
 }
 
+func (s *storage) GetRecommendedMaxKVSize() int {
+	// The block size configured is not actually a *set* block size, but rather a
+	// *minimum*. LevelDB only checks if a block is full after a new key/value pair is
+	// written, meaning that blocks will tend to overflow a bit.
+	// Therefore, using the default block size as the max value size will ensure
+	// that a new block will only contain a single value and that the size will
+	// only ever be as large as around double the block size (if the block was
+	// nearly full before the new items were added).
+	return blockSize
+}
+
 // CreateTemporary creates new DB of the same type in temp dir
 func (s *storage) CreateTemporary() (database.Storage, error) {
 	tempdir, err := os.MkdirTemp("", "aptly")

deb/changes.go

@@ -291,7 +291,8 @@ func CollectChangesFiles(locations []string, reporter aptly.ResultReporter) (cha
 // ImportChangesFiles imports referenced files in changes files into local repository
 func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, acceptUnsigned, ignoreSignatures, forceReplace, noRemoveFiles bool,
 	verifier pgp.Verifier, repoTemplate *template.Template, progress aptly.Progress, localRepoCollection *LocalRepoCollection, packageCollection *PackageCollection,
-	pool aptly.PackagePool, checksumStorageProvider aptly.ChecksumStorageProvider, uploaders *Uploaders, parseQuery parseQuery) (processedFiles []string, failedFiles []string, err error) {
+	reflistCollection *RefListCollection, pool aptly.PackagePool, checksumStorageProvider aptly.ChecksumStorageProvider, uploaders *Uploaders,
+	parseQuery parseQuery) (processedFiles []string, failedFiles []string, err error) {
 
 	for _, path := range changesFiles {
 		var changes *Changes
@@ -359,7 +360,7 @@ func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, ac
 			}
 		}
 
-		err = localRepoCollection.LoadComplete(repo)
+		err = localRepoCollection.LoadComplete(repo, reflistCollection)
 		if err != nil {
 			return nil, nil, fmt.Errorf("unable to load repo: %s", err)
 		}
@@ -382,9 +383,9 @@ func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, ac
 			return nil, nil, fmt.Errorf("unable to import package files: %s", err)
 		}
 
-		repo.UpdateRefList(NewPackageRefListFromPackageList(list))
+		repo.UpdateRefList(NewSplitRefListFromPackageList(list))
 
-		err = localRepoCollection.Update(repo)
+		err = localRepoCollection.Update(repo, reflistCollection)
 		if err != nil {
 			return nil, nil, fmt.Errorf("unable to save: %s", err)
 		}

deb/changes_test.go

@@ -21,6 +21,7 @@ type ChangesSuite struct {
 	db                  database.Storage
 	localRepoCollection *LocalRepoCollection
 	packageCollection   *PackageCollection
+	reflistCollection   *RefListCollection
 	packagePool         aptly.PackagePool
 	checksumStorage     aptly.ChecksumStorage
 	progress            aptly.Progress
@@ -42,6 +43,7 @@ func (s *ChangesSuite) SetUpTest(c *C) {
 	s.db, _ = goleveldb.NewOpenDB(c.MkDir())
 	s.localRepoCollection = NewLocalRepoCollection(s.db)
 	s.packageCollection = NewPackageCollection(s.db)
+	s.reflistCollection = NewRefListCollection(s.db)
 
 	s.checksumStorage = files.NewMockChecksumStorage()
 	s.packagePool = files.NewPackagePool(s.Dir, false)
@@ -88,7 +90,7 @@ func (s *ChangesSuite) TestCollectChangesFiles(c *C) {
 
 func (s *ChangesSuite) TestImportChangesFiles(c *C) {
 	repo := NewLocalRepo("test", "Test Comment")
-	c.Assert(s.localRepoCollection.Add(repo), IsNil)
+	c.Assert(s.localRepoCollection.Add(repo, s.reflistCollection), IsNil)
 
 	origFailedFiles := []string{
 		"testdata/changes/calamares.changes",
@@ -124,7 +126,8 @@ func (s *ChangesSuite) TestImportChangesFiles(c *C) {
 	processedFiles, failedFiles, err := ImportChangesFiles(
 		append(changesFiles, "testdata/changes/notexistent.changes"),
 		s.Reporter, true, true, false, false, &NullVerifier{},
-		template.Must(template.New("test").Parse("test")), s.progress, s.localRepoCollection, s.packageCollection, s.packagePool, func(database.ReaderWriter) aptly.ChecksumStorage { return s.checksumStorage },
+		template.Must(template.New("test").Parse("test")), s.progress, s.localRepoCollection, s.packageCollection, s.reflistCollection, s.packagePool,
+		func(database.ReaderWriter) aptly.ChecksumStorage { return s.checksumStorage },
 		nil, nil)
 	c.Assert(err, IsNil)
 	c.Check(failedFiles, DeepEquals, append(expectedFailedFiles, "testdata/changes/notexistent.changes"))
@@ -133,7 +136,7 @@ func (s *ChangesSuite) TestImportChangesFiles(c *C) {
 
 func (s *ChangesSuite) TestImportDbgsymWithVersionedSourceField(c *C) {
 	repo := NewLocalRepo("test", "Test Comment")
-	c.Assert(s.localRepoCollection.Add(repo), IsNil)
+	c.Assert(s.localRepoCollection.Add(repo, s.reflistCollection), IsNil)
 
 	changesFiles, failedFiles := CollectChangesFiles(
 		[]string{"testdata/dbgsym-with-source-version"}, s.Reporter)
@@ -142,7 +145,8 @@ func (s *ChangesSuite) TestImportDbgsymWithVersionedSourceField(c *C) {
 
 	_, failedFiles, err := ImportChangesFiles(
 		changesFiles, s.Reporter, true, true, false, true, &NullVerifier{},
-		template.Must(template.New("test").Parse("test")), s.progress, s.localRepoCollection, s.packageCollection, s.packagePool, func(database.ReaderWriter) aptly.ChecksumStorage { return s.checksumStorage },
+		template.Must(template.New("test").Parse("test")), s.progress, s.localRepoCollection, s.packageCollection, s.reflistCollection, s.packagePool,
+		func(database.ReaderWriter) aptly.ChecksumStorage { return s.checksumStorage },
 		nil, nil)
 	c.Assert(err, IsNil)
 	c.Check(failedFiles, IsNil)

deb/collections.go

@@ -16,6 +16,7 @@ type CollectionFactory struct {
 	snapshots      *SnapshotCollection
 	localRepos     *LocalRepoCollection
 	publishedRepos *PublishedRepoCollection
+	reflists       *RefListCollection
 	checksums      *ChecksumCollection
 }
 
@@ -91,6 +92,17 @@ func (factory *CollectionFactory) PublishedRepoCollection() *PublishedRepoCollec
 	return factory.publishedRepos
 }
 
+func (factory *CollectionFactory) RefListCollection() *RefListCollection {
+	factory.Lock()
+	defer factory.Unlock()
+
+	if factory.reflists == nil {
+		factory.reflists = NewRefListCollection(factory.db)
+	}
+
+	return factory.reflists
+}
+
 // ChecksumCollection returns (or creates) new ChecksumCollection
 func (factory *CollectionFactory) ChecksumCollection(db database.ReaderWriter) aptly.ChecksumStorage {
 	factory.Lock()

deb/contents.go

@@ -7,7 +7,7 @@ import (
 	"io"
 
 	"github.com/aptly-dev/aptly/database"
-	"github.com/pborman/uuid"
+	"github.com/google/uuid"
 )
 
 // ContentsIndex calculates mapping from files to packages, with sorting and aggregation
@@ -20,7 +20,7 @@ type ContentsIndex struct {
 func NewContentsIndex(db database.Storage) *ContentsIndex {
 	return &ContentsIndex{
 		db:     db,
-		prefix: []byte(uuid.New()),
+		prefix: []byte(uuid.NewString()),
 	}
 }
 

deb/graph.go

@@ -33,7 +33,7 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 	existingNodes := map[string]bool{}
 
 	err = collectionFactory.RemoteRepoCollection().ForEach(func(repo *RemoteRepo) error {
-		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -55,7 +55,7 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 	}
 
 	err = collectionFactory.LocalRepoCollection().ForEach(func(repo *LocalRepo) error {
-		e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -81,7 +81,7 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 	})
 
 	err = collectionFactory.SnapshotCollection().ForEach(func(snapshot *Snapshot) error {
-		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}

deb/list.go

@@ -91,7 +91,7 @@ func NewPackageListWithDuplicates(duplicates bool, capacity int) *PackageList {
 }
 
 // NewPackageListFromRefList loads packages list from PackageRefList
-func NewPackageListFromRefList(reflist *PackageRefList, collection *PackageCollection, progress aptly.Progress) (*PackageList, error) {
+func NewPackageListFromRefList(reflist AnyRefList, collection *PackageCollection, progress aptly.Progress) (*PackageList, error) {
 	// empty reflist
 	if reflist == nil {
 		return NewPackageList(), nil

deb/local.go

@@ -7,7 +7,7 @@ import (
 	"log"
 
 	"github.com/aptly-dev/aptly/database"
-	"github.com/pborman/uuid"
+	"github.com/google/uuid"
 	"github.com/ugorji/go/codec"
 )
 
@@ -26,13 +26,13 @@ type LocalRepo struct {
 	// Uploaders configuration
 	Uploaders *Uploaders `codec:"Uploaders,omitempty" json:"-"`
 	// "Snapshot" of current list of packages
-	packageRefs *PackageRefList
+	packageRefs *SplitRefList
 }
 
 // NewLocalRepo creates new instance of Debian local repository
 func NewLocalRepo(name string, comment string) *LocalRepo {
 	return &LocalRepo{
-		UUID:    uuid.New(),
+		UUID:    uuid.NewString(),
 		Name:    name,
 		Comment: comment,
 	}
@@ -55,13 +55,13 @@ func (repo *LocalRepo) NumPackages() int {
 }
 
 // RefList returns package list for repo
-func (repo *LocalRepo) RefList() *PackageRefList {
+func (repo *LocalRepo) RefList() *SplitRefList {
 	return repo.packageRefs
 }
 
 // UpdateRefList changes package list for local repo
-func (repo *LocalRepo) UpdateRefList(reflist *PackageRefList) {
+func (repo *LocalRepo) UpdateRefList(sl *SplitRefList) {
-	repo.packageRefs = reflist
+	repo.packageRefs = sl
 }
 
 // Encode does msgpack encoding of LocalRepo
@@ -140,14 +140,14 @@ func (collection *LocalRepoCollection) search(filter func(*LocalRepo) bool, uniq
 }
 
 // Add appends new repo to collection and saves it
-func (collection *LocalRepoCollection) Add(repo *LocalRepo) error {
+func (collection *LocalRepoCollection) Add(repo *LocalRepo, reflistCollection *RefListCollection) error {
 	_, err := collection.ByName(repo.Name)
 
 	if err == nil {
 		return fmt.Errorf("local repo with name %s already exists", repo.Name)
 	}
 
-	err = collection.Update(repo)
+	err = collection.Update(repo, reflistCollection)
 	if err != nil {
 		return err
 	}
@@ -157,27 +157,25 @@ func (collection *LocalRepoCollection) Add(repo *LocalRepo) error {
 }
 
 // Update stores updated information about repo in DB
-func (collection *LocalRepoCollection) Update(repo *LocalRepo) error {
+func (collection *LocalRepoCollection) Update(repo *LocalRepo, reflistCollection *RefListCollection) error {
 	batch := collection.db.CreateBatch()
 	batch.Put(repo.Key(), repo.Encode())
 	if repo.packageRefs != nil {
-		batch.Put(repo.RefKey(), repo.packageRefs.Encode())
+		bc := reflistCollection.NewBatch(batch)
+		reflistCollection.UpdateInBatch(repo.packageRefs, repo.RefKey(), bc)
 	}
 	return batch.Write()
 }
 
 // LoadComplete loads additional information for local repo
-func (collection *LocalRepoCollection) LoadComplete(repo *LocalRepo) error {
+func (collection *LocalRepoCollection) LoadComplete(repo *LocalRepo, reflistCollection *RefListCollection) error {
-	encoded, err := collection.db.Get(repo.RefKey())
+	repo.packageRefs = NewSplitRefList()
+	err := reflistCollection.LoadComplete(repo.packageRefs, repo.RefKey())
 	if err == database.ErrNotFound {
 		return nil
 	}
-	if err != nil {
-		return err
-	}
 
-	repo.packageRefs = &PackageRefList{}
+	return err
-	return repo.packageRefs.Decode(encoded)
 }
 
 // ByName looks up repository by name

deb/local_test.go

@@ -12,7 +12,7 @@ import (
 type LocalRepoSuite struct {
 	db      database.Storage
 	list    *PackageList
-	reflist *PackageRefList
+	reflist *SplitRefList
 	repo    *LocalRepo
 }
 
@@ -24,7 +24,7 @@ func (s *LocalRepoSuite) SetUpTest(c *C) {
 	s.list.Add(&Package{Name: "lib", Version: "1.7", Architecture: "i386"})
 	s.list.Add(&Package{Name: "app", Version: "1.9", Architecture: "amd64"})
 
-	s.reflist = NewPackageRefListFromPackageList(s.list)
+	s.reflist = NewSplitRefListFromPackageList(s.list)
 
 	s.repo = NewLocalRepo("lrepo", "Super repo")
 	s.repo.packageRefs = s.reflist
@@ -77,8 +77,9 @@ func (s *LocalRepoSuite) TestRefKey(c *C) {
 type LocalRepoCollectionSuite struct {
 	db                database.Storage
 	collection        *LocalRepoCollection
+	reflistCollection *RefListCollection
 	list              *PackageList
-	reflist    *PackageRefList
+	reflist           *SplitRefList
 }
 
 var _ = Suite(&LocalRepoCollectionSuite{})
@@ -86,12 +87,13 @@ var _ = Suite(&LocalRepoCollectionSuite{})
 func (s *LocalRepoCollectionSuite) SetUpTest(c *C) {
 	s.db, _ = goleveldb.NewOpenDB(c.MkDir())
 	s.collection = NewLocalRepoCollection(s.db)
+	s.reflistCollection = NewRefListCollection(s.db)
 
 	s.list = NewPackageList()
 	s.list.Add(&Package{Name: "lib", Version: "1.7", Architecture: "i386"})
 	s.list.Add(&Package{Name: "app", Version: "1.9", Architecture: "amd64"})
 
-	s.reflist = NewPackageRefListFromPackageList(s.list)
+	s.reflist = NewSplitRefListFromRefList(NewPackageRefListFromPackageList(s.list))
 }
 
 func (s *LocalRepoCollectionSuite) TearDownTest(c *C) {
@@ -103,8 +105,8 @@ func (s *LocalRepoCollectionSuite) TestAddByName(c *C) {
 	c.Assert(err, ErrorMatches, "*.not found")
 
 	repo := NewLocalRepo("local1", "Comment 1")
-	c.Assert(s.collection.Add(repo), IsNil)
+	c.Assert(s.collection.Add(repo, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Add(repo), ErrorMatches, ".*already exists")
+	c.Assert(s.collection.Add(repo, s.reflistCollection), ErrorMatches, ".*already exists")
 
 	r, err := s.collection.ByName("local1")
 	c.Assert(err, IsNil)
@@ -121,7 +123,7 @@ func (s *LocalRepoCollectionSuite) TestByUUID(c *C) {
 	c.Assert(err, ErrorMatches, "*.not found")
 
 	repo := NewLocalRepo("local1", "Comment 1")
-	c.Assert(s.collection.Add(repo), IsNil)
+	c.Assert(s.collection.Add(repo, s.reflistCollection), IsNil)
 
 	r, err := s.collection.ByUUID(repo.UUID)
 	c.Assert(err, IsNil)
@@ -135,7 +137,7 @@ func (s *LocalRepoCollectionSuite) TestByUUID(c *C) {
 
 func (s *LocalRepoCollectionSuite) TestUpdateLoadComplete(c *C) {
 	repo := NewLocalRepo("local1", "Comment 1")
-	c.Assert(s.collection.Update(repo), IsNil)
+	c.Assert(s.collection.Update(repo, s.reflistCollection), IsNil)
 
 	collection := NewLocalRepoCollection(s.db)
 	r, err := collection.ByName("local1")
@@ -143,20 +145,20 @@ func (s *LocalRepoCollectionSuite) TestUpdateLoadComplete(c *C) {
 	c.Assert(r.packageRefs, IsNil)
 
 	repo.packageRefs = s.reflist
-	c.Assert(s.collection.Update(repo), IsNil)
+	c.Assert(s.collection.Update(repo, s.reflistCollection), IsNil)
 
 	collection = NewLocalRepoCollection(s.db)
 	r, err = collection.ByName("local1")
 	c.Assert(err, IsNil)
 	c.Assert(r.packageRefs, IsNil)
 	c.Assert(r.NumPackages(), Equals, 0)
-	c.Assert(s.collection.LoadComplete(r), IsNil)
+	c.Assert(s.collection.LoadComplete(r, s.reflistCollection), IsNil)
 	c.Assert(r.NumPackages(), Equals, 2)
 }
 
 func (s *LocalRepoCollectionSuite) TestForEachAndLen(c *C) {
 	repo := NewLocalRepo("local1", "Comment 1")
-	s.collection.Add(repo)
+	s.collection.Add(repo, s.reflistCollection)
 
 	count := 0
 	err := s.collection.ForEach(func(*LocalRepo) error {
@@ -178,10 +180,10 @@ func (s *LocalRepoCollectionSuite) TestForEachAndLen(c *C) {
 
 func (s *LocalRepoCollectionSuite) TestDrop(c *C) {
 	repo1 := NewLocalRepo("local1", "Comment 1")
-	s.collection.Add(repo1)
+	s.collection.Add(repo1, s.reflistCollection)
 
 	repo2 := NewLocalRepo("local2", "Comment 2")
-	s.collection.Add(repo2)
+	s.collection.Add(repo2, s.reflistCollection)
 
 	r1, _ := s.collection.ByUUID(repo1.UUID)
 	c.Check(r1, Equals, repo1)

deb/publish.go

@@ -12,7 +12,7 @@ import (
 	"strings"
 	"time"
 
-	"github.com/pborman/uuid"
+	"github.com/google/uuid"
 	"github.com/ugorji/go/codec"
 
 	"github.com/aptly-dev/aptly/aptly"
@@ -37,7 +37,7 @@ type repoSourceItem struct {
 	// Pointer to local repo if SourceKind == "local"
 	localRepo *LocalRepo
 	// Package references is SourceKind == "local"
-	packageRefs *PackageRefList
+	packageRefs *SplitRefList
 }
 
 // PublishedRepo is a published for http/ftp representation of snapshot as Debian repository
@@ -354,7 +354,7 @@ func walkUpTree(source interface{}, collectionFactory *CollectionFactory) (rootD
 func NewPublishedRepo(storage, prefix, distribution string, architectures []string,
 	components []string, sources []interface{}, collectionFactory *CollectionFactory, multiDist bool) (*PublishedRepo, error) {
 	result := &PublishedRepo{
-		UUID:          uuid.New(),
+		UUID:          uuid.NewString(),
 		Storage:       storage,
 		Architectures: architectures,
 		Sources:       make(map[string]string),
@@ -609,7 +609,7 @@ func (p *PublishedRepo) RefKey(component string) []byte {
 }
 
 // RefList returns list of package refs in local repo
-func (p *PublishedRepo) RefList(component string) *PackageRefList {
+func (p *PublishedRepo) RefList(component string) *SplitRefList {
 	item := p.sourceItems[component]
 	if p.SourceKind == SourceLocalRepo {
 		return item.packageRefs
@@ -1243,14 +1243,14 @@ func (collection *PublishedRepoCollection) loadList() {
 }
 
 // Add appends new repo to collection and saves it
-func (collection *PublishedRepoCollection) Add(repo *PublishedRepo) error {
+func (collection *PublishedRepoCollection) Add(repo *PublishedRepo, reflistCollection *RefListCollection) error {
 	collection.loadList()
 
 	if collection.CheckDuplicate(repo) != nil {
 		return fmt.Errorf("published repo with storage/prefix/distribution %s/%s/%s already exists", repo.Storage, repo.Prefix, repo.Distribution)
 	}
 
-	err := collection.Update(repo)
+	err := collection.Update(repo, reflistCollection)
 	if err != nil {
 		return err
 	}
@@ -1273,13 +1273,14 @@ func (collection *PublishedRepoCollection) CheckDuplicate(repo *PublishedRepo) *
 }
 
 // Update stores updated information about repo in DB
-func (collection *PublishedRepoCollection) Update(repo *PublishedRepo) error {
+func (collection *PublishedRepoCollection) Update(repo *PublishedRepo, reflistCollection *RefListCollection) error {
 	batch := collection.db.CreateBatch()
 	batch.Put(repo.Key(), repo.Encode())
 
 	if repo.SourceKind == SourceLocalRepo {
+		rb := reflistCollection.NewBatch(batch)
 		for component, item := range repo.sourceItems {
-			batch.Put(repo.RefKey(component), item.packageRefs.Encode())
+			reflistCollection.UpdateInBatch(item.packageRefs, repo.RefKey(component), rb)
 		}
 	}
 	return batch.Write()
@@ -1312,7 +1313,7 @@ func (collection *PublishedRepoCollection) LoadShallow(repo *PublishedRepo, coll
 				return
 			}
 
-			item.packageRefs = &PackageRefList{}
+			item.packageRefs = NewSplitRefList()
 			repo.sourceItems[component] = item
 		}
 	} else {
@@ -1328,35 +1329,29 @@ func (collection *PublishedRepoCollection) LoadComplete(repo *PublishedRepo, col
 
 	if repo.SourceKind == SourceSnapshot {
 		for _, item := range repo.sourceItems {
-			err = collectionFactory.SnapshotCollection().LoadComplete(item.snapshot)
+			err = collectionFactory.SnapshotCollection().LoadComplete(item.snapshot, collectionFactory.RefListCollection())
 			if err != nil {
 				return
 			}
 		}
 	} else if repo.SourceKind == SourceLocalRepo {
 		for component, item := range repo.sourceItems {
-			err = collectionFactory.LocalRepoCollection().LoadComplete(item.localRepo)
+			err = collectionFactory.LocalRepoCollection().LoadComplete(item.localRepo, collectionFactory.RefListCollection())
 			if err != nil {
 				return
 			}
 
-			var encoded []byte
+			err = collectionFactory.RefListCollection().LoadComplete(item.packageRefs, repo.RefKey(component))
-			encoded, err = collection.db.Get(repo.RefKey(component))
 			if err != nil {
 				// < 0.6 saving w/o component name
 				if err == database.ErrNotFound && len(repo.Sources) == 1 {
-					encoded, err = collection.db.Get(repo.RefKey(""))
+					err = collectionFactory.RefListCollection().LoadComplete(item.packageRefs, repo.RefKey(""))
 				}
 
 				if err != nil {
 					return
 				}
 			}
-
-			err = item.packageRefs.Decode(encoded)
-			if err != nil {
-				return
-			}
 		}
 	} else {
 		panic("unknown SourceKind")
@@ -1461,6 +1456,11 @@ func (collection *PublishedRepoCollection) listReferencedFilesByComponent(prefix
 	referencedFiles := map[string][]string{}
 	processedComponentRefs := map[string]*PackageRefList{}
 
+	processedComponentBuckets := map[string]*RefListDigestSet{}
+	for _, component := range components {
+		processedComponentBuckets[component] = NewRefListDigestSet()
+	}
+
 	for _, r := range collection.list {
 		if r.Prefix == prefix && !r.MultiDist {
 			matches := false
@@ -1484,7 +1484,15 @@ func (collection *PublishedRepoCollection) listReferencedFilesByComponent(prefix
 
 			for _, component := range components {
 				if utils.StrSliceHasItem(repoComponents, component) {
-					unseenRefs := r.RefList(component)
+					processedBuckets := processedComponentBuckets[component]
+
+					err := r.RefList(component).ForEachBucket(func(digest []byte, bucket *PackageRefList) error {
+						if processedBuckets.Has(digest) {
+							return nil
+						}
+						processedBuckets.Add(digest)
+
+						unseenRefs := bucket
 						processedRefs := processedComponentRefs[component]
 						if processedRefs != nil {
 							unseenRefs = unseenRefs.Subtract(processedRefs)
@@ -1493,13 +1501,13 @@ func (collection *PublishedRepoCollection) listReferencedFilesByComponent(prefix
 						}
 
 						if unseenRefs.Len() == 0 {
-						continue
+							return nil
 						}
 						processedComponentRefs[component] = processedRefs.Merge(unseenRefs, false, true)
 
 						packageList, err := NewPackageListFromRefList(unseenRefs, collectionFactory.PackageCollection(), progress)
 						if err != nil {
-						return nil, err
+							return err
 						}
 
 						packageList.ForEach(func(p *Package) error {
@@ -1514,6 +1522,13 @@ func (collection *PublishedRepoCollection) listReferencedFilesByComponent(prefix
 
 							return nil
 						})
+
+						return nil
+					})
+
+					if err != nil {
+						return nil, err
+					}
 				}
 			}
 		}

deb/publish_bench_test.go

@@ -31,6 +31,7 @@ func BenchmarkListReferencedFiles(b *testing.B) {
 	packageCollection := factory.PackageCollection()
 	repoCollection := factory.LocalRepoCollection()
 	publishCollection := factory.PublishedRepoCollection()
+	reflistCollection := factory.RefListCollection()
 
 	sharedRefs := NewPackageRefList()
 	{
@@ -91,14 +92,14 @@ func BenchmarkListReferencedFiles(b *testing.B) {
 		repo := NewLocalRepo(fmt.Sprintf("repo%d", repoIndex), "comment")
 		repo.DefaultDistribution = fmt.Sprintf("dist%d", repoIndex)
 		repo.DefaultComponent = defaultComponent
-		repo.UpdateRefList(refs.Merge(sharedRefs, false, true))
+		repo.UpdateRefList(NewSplitRefListFromRefList(refs.Merge(sharedRefs, false, true)))
-		repoCollection.Add(repo)
+		repoCollection.Add(repo, reflistCollection)
 
 		publish, err := NewPublishedRepo("", "test", "", nil, []string{defaultComponent}, []interface{}{repo}, factory, false)
 		if err != nil {
 			b.Fatal(err)
 		}
-		publishCollection.Add(publish)
+		publishCollection.Add(publish, reflistCollection)
 	}
 
 	db.CompactDB()

deb/publish_test.go

@@ -83,6 +83,7 @@ type PublishedRepoSuite struct {
 	db                                  database.Storage
 	factory                             *CollectionFactory
 	packageCollection                   *PackageCollection
+	reflistCollection                   *RefListCollection
 }
 
 var _ = Suite(&PublishedRepoSuite{})
@@ -114,21 +115,22 @@ func (s *PublishedRepoSuite) SetUpTest(c *C) {
 	s.p2.UpdateFiles(s.p1.Files())
 	s.p3.UpdateFiles(s.p1.Files())
 
-	s.reflist = NewPackageRefListFromPackageList(s.list)
+	s.reflist = NewSplitRefListFromPackageList(s.list)
+	s.reflistCollection = s.factory.RefListCollection()
 
 	repo, _ := NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{}, false, false, false)
 	repo.packageRefs = s.reflist
-	s.factory.RemoteRepoCollection().Add(repo)
+	s.factory.RemoteRepoCollection().Add(repo, s.reflistCollection)
 
 	s.localRepo = NewLocalRepo("local1", "comment1")
 	s.localRepo.packageRefs = s.reflist
-	s.factory.LocalRepoCollection().Add(s.localRepo)
+	s.factory.LocalRepoCollection().Add(s.localRepo, s.reflistCollection)
 
 	s.snapshot, _ = NewSnapshotFromRepository("snap", repo)
-	s.factory.SnapshotCollection().Add(s.snapshot)
+	s.factory.SnapshotCollection().Add(s.snapshot, s.reflistCollection)
 
 	s.snapshot2, _ = NewSnapshotFromRepository("snap", repo)
-	s.factory.SnapshotCollection().Add(s.snapshot2)
+	s.factory.SnapshotCollection().Add(s.snapshot2, s.reflistCollection)
 
 	s.packageCollection = s.factory.PackageCollection()
 	s.packageCollection.Update(s.p1)
@@ -337,7 +339,7 @@ func (s *PublishedRepoSuite) TestDistributionComponentGuessing(c *C) {
 
 	s.localRepo.DefaultDistribution = "precise"
 	s.localRepo.DefaultComponent = "contrib"
-	s.factory.LocalRepoCollection().Update(s.localRepo)
+	s.factory.LocalRepoCollection().Update(s.localRepo, s.reflistCollection)
 
 	repo, err = NewPublishedRepo("", "ppa", "", nil, []string{""}, []interface{}{s.localRepo}, s.factory, false)
 	c.Check(err, IsNil)
@@ -538,6 +540,7 @@ type PublishedRepoCollectionSuite struct {
 	db                                database.Storage
 	factory                           *CollectionFactory
 	snapshotCollection                *SnapshotCollection
+	reflistCollection                 *RefListCollection
 	collection                        *PublishedRepoCollection
 	snap1, snap2                      *Snapshot
 	localRepo                         *LocalRepo
@@ -553,22 +556,23 @@ func (s *PublishedRepoCollectionSuite) SetUpTest(c *C) {
 	s.factory = NewCollectionFactory(s.db)
 
 	s.snapshotCollection = s.factory.SnapshotCollection()
+	s.reflistCollection = s.factory.RefListCollection()
 
 	snap1Refs := NewPackageRefList()
 	snap1Refs.Refs = [][]byte{s.p1.Key(""), s.p2.Key("")}
 	sort.Sort(snap1Refs)
-	s.snap1 = NewSnapshotFromRefList("snap1", []*Snapshot{}, snap1Refs, "desc1")
+	s.snap1 = NewSnapshotFromRefList("snap1", []*Snapshot{}, NewSplitRefListFromRefList(snap1Refs), "desc1")
 
 	snap2Refs := NewPackageRefList()
 	snap2Refs.Refs = [][]byte{s.p3.Key("")}
 	sort.Sort(snap2Refs)
-	s.snap2 = NewSnapshotFromRefList("snap2", []*Snapshot{}, snap2Refs, "desc2")
+	s.snap2 = NewSnapshotFromRefList("snap2", []*Snapshot{}, NewSplitRefListFromRefList(snap2Refs), "desc2")
 
-	s.snapshotCollection.Add(s.snap1)
+	s.snapshotCollection.Add(s.snap1, s.reflistCollection)
-	s.snapshotCollection.Add(s.snap2)
+	s.snapshotCollection.Add(s.snap2, s.reflistCollection)
 
 	s.localRepo = NewLocalRepo("local1", "comment1")
-	s.factory.LocalRepoCollection().Add(s.localRepo)
+	s.factory.LocalRepoCollection().Add(s.localRepo, s.reflistCollection)
 
 	s.repo1, _ = NewPublishedRepo("", "ppa", "anaconda", []string{}, []string{"main"}, []interface{}{s.snap1}, s.factory, false)
 	s.repo2, _ = NewPublishedRepo("", "", "anaconda", []string{}, []string{"main", "contrib"}, []interface{}{s.snap2, s.snap1}, s.factory, false)
@@ -587,14 +591,14 @@ func (s *PublishedRepoCollectionSuite) TestAddByStoragePrefixDistribution(c *C)
 	_, err := s.collection.ByStoragePrefixDistribution("", "ppa", "anaconda")
 	c.Assert(err, ErrorMatches, "*.not found")
 
-	c.Assert(s.collection.Add(s.repo1), IsNil)
+	c.Assert(s.collection.Add(s.repo1, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Add(s.repo1), ErrorMatches, ".*already exists")
+	c.Assert(s.collection.Add(s.repo1, s.reflistCollection), ErrorMatches, ".*already exists")
 	c.Assert(s.collection.CheckDuplicate(s.repo2), IsNil)
-	c.Assert(s.collection.Add(s.repo2), IsNil)
+	c.Assert(s.collection.Add(s.repo2, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Add(s.repo3), ErrorMatches, ".*already exists")
+	c.Assert(s.collection.Add(s.repo3, s.reflistCollection), ErrorMatches, ".*already exists")
 	c.Assert(s.collection.CheckDuplicate(s.repo3), Equals, s.repo1)
-	c.Assert(s.collection.Add(s.repo4), IsNil)
+	c.Assert(s.collection.Add(s.repo4, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Add(s.repo5), IsNil)
+	c.Assert(s.collection.Add(s.repo5, s.reflistCollection), IsNil)
 
 	r, err := s.collection.ByStoragePrefixDistribution("", "ppa", "anaconda")
 	c.Assert(err, IsNil)
@@ -620,7 +624,7 @@ func (s *PublishedRepoCollectionSuite) TestByUUID(c *C) {
 	_, err := s.collection.ByUUID(s.repo1.UUID)
 	c.Assert(err, ErrorMatches, "*.not found")
 
-	c.Assert(s.collection.Add(s.repo1), IsNil)
+	c.Assert(s.collection.Add(s.repo1, s.reflistCollection), IsNil)
 
 	r, err := s.collection.ByUUID(s.repo1.UUID)
 	c.Assert(err, IsNil)
@@ -631,8 +635,8 @@ func (s *PublishedRepoCollectionSuite) TestByUUID(c *C) {
 }
 
 func (s *PublishedRepoCollectionSuite) TestUpdateLoadComplete(c *C) {
-	c.Assert(s.collection.Update(s.repo1), IsNil)
+	c.Assert(s.collection.Update(s.repo1, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Update(s.repo4), IsNil)
+	c.Assert(s.collection.Update(s.repo4, s.reflistCollection), IsNil)
 
 	collection := NewPublishedRepoCollection(s.db)
 	r, err := collection.ByStoragePrefixDistribution("", "ppa", "anaconda")
@@ -680,7 +684,7 @@ func (s *PublishedRepoCollectionSuite) TestLoadPre0_6(c *C) {
 	encoder.Encode(&old)
 
 	c.Assert(s.db.Put(s.repo1.Key(), buf.Bytes()), IsNil)
-	c.Assert(s.db.Put(s.repo1.RefKey(""), s.localRepo.RefList().Encode()), IsNil)
+	c.Assert(s.db.Put(s.repo1.RefKey(""), NewPackageRefList().Encode()), IsNil)
 
 	collection := NewPublishedRepoCollection(s.db)
 	repo, err := collection.ByStoragePrefixDistribution("", "ppa", "anaconda")
@@ -695,7 +699,7 @@ func (s *PublishedRepoCollectionSuite) TestLoadPre0_6(c *C) {
 }
 
 func (s *PublishedRepoCollectionSuite) TestForEachAndLen(c *C) {
-	s.collection.Add(s.repo1)
+	s.collection.Add(s.repo1, s.reflistCollection)
 
 	count := 0
 	err := s.collection.ForEach(func(*PublishedRepo) error {
@@ -716,17 +720,17 @@ func (s *PublishedRepoCollectionSuite) TestForEachAndLen(c *C) {
 }
 
 func (s *PublishedRepoCollectionSuite) TestBySnapshot(c *C) {
-	c.Check(s.collection.Add(s.repo1), IsNil)
+	c.Check(s.collection.Add(s.repo1, s.reflistCollection), IsNil)
-	c.Check(s.collection.Add(s.repo2), IsNil)
+	c.Check(s.collection.Add(s.repo2, s.reflistCollection), IsNil)
 
 	c.Check(s.collection.BySnapshot(s.snap1), DeepEquals, []*PublishedRepo{s.repo1, s.repo2})
 	c.Check(s.collection.BySnapshot(s.snap2), DeepEquals, []*PublishedRepo{s.repo2})
 }
 
 func (s *PublishedRepoCollectionSuite) TestByLocalRepo(c *C) {
-	c.Check(s.collection.Add(s.repo1), IsNil)
+	c.Check(s.collection.Add(s.repo1, s.reflistCollection), IsNil)
-	c.Check(s.collection.Add(s.repo4), IsNil)
+	c.Check(s.collection.Add(s.repo4, s.reflistCollection), IsNil)
-	c.Check(s.collection.Add(s.repo5), IsNil)
+	c.Check(s.collection.Add(s.repo5, s.reflistCollection), IsNil)
 
 	c.Check(s.collection.ByLocalRepo(s.localRepo), DeepEquals, []*PublishedRepo{s.repo4, s.repo5})
 }
@@ -736,10 +740,10 @@ func (s *PublishedRepoCollectionSuite) TestListReferencedFiles(c *C) {
 	c.Check(s.factory.PackageCollection().Update(s.p2), IsNil)
 	c.Check(s.factory.PackageCollection().Update(s.p3), IsNil)
 
-	c.Check(s.collection.Add(s.repo1), IsNil)
+	c.Check(s.collection.Add(s.repo1, s.reflistCollection), IsNil)
-	c.Check(s.collection.Add(s.repo2), IsNil)
+	c.Check(s.collection.Add(s.repo2, s.reflistCollection), IsNil)
-	c.Check(s.collection.Add(s.repo4), IsNil)
+	c.Check(s.collection.Add(s.repo4, s.reflistCollection), IsNil)
-	c.Check(s.collection.Add(s.repo5), IsNil)
+	c.Check(s.collection.Add(s.repo5, s.reflistCollection), IsNil)
 
 	files, err := s.collection.listReferencedFilesByComponent(".", []string{"main", "contrib"}, s.factory, nil)
 	c.Assert(err, IsNil)
@@ -755,12 +759,12 @@ func (s *PublishedRepoCollectionSuite) TestListReferencedFiles(c *C) {
 	})
 
 	snap3 := NewSnapshotFromRefList("snap3", []*Snapshot{}, s.snap2.RefList(), "desc3")
-	s.snapshotCollection.Add(snap3)
+	s.snapshotCollection.Add(snap3, s.reflistCollection)
 
 	// Ensure that adding a second publish point with matching files doesn't give duplicate results.
 	repo3, err := NewPublishedRepo("", "", "anaconda-2", []string{}, []string{"main"}, []interface{}{snap3}, s.factory, false)
 	c.Check(err, IsNil)
-	c.Check(s.collection.Add(repo3), IsNil)
+	c.Check(s.collection.Add(repo3, s.reflistCollection), IsNil)
 
 	files, err = s.collection.listReferencedFilesByComponent(".", []string{"main", "contrib"}, s.factory, nil)
 	c.Assert(err, IsNil)
@@ -781,6 +785,7 @@ type PublishedRepoRemoveSuite struct {
 	db                                  database.Storage
 	factory                             *CollectionFactory
 	snapshotCollection                  *SnapshotCollection
+	reflistCollection                   *RefListCollection
 	collection                          *PublishedRepoCollection
 	root, root2                         string
 	provider                            *FakeStorageProvider
@@ -796,10 +801,11 @@ func (s *PublishedRepoRemoveSuite) SetUpTest(c *C) {
 	s.factory = NewCollectionFactory(s.db)
 
 	s.snapshotCollection = s.factory.SnapshotCollection()
+	s.reflistCollection = s.factory.RefListCollection()
 
 	s.snap1 = NewSnapshotFromPackageList("snap1", []*Snapshot{}, NewPackageList(), "desc1")
 
-	s.snapshotCollection.Add(s.snap1)
+	s.snapshotCollection.Add(s.snap1, s.reflistCollection)
 
 	s.repo1, _ = NewPublishedRepo("", "ppa", "anaconda", []string{}, []string{"main"}, []interface{}{s.snap1}, s.factory, false)
 	s.repo2, _ = NewPublishedRepo("", "", "anaconda", []string{}, []string{"main"}, []interface{}{s.snap1}, s.factory, false)
@@ -808,11 +814,11 @@ func (s *PublishedRepoRemoveSuite) SetUpTest(c *C) {
 	s.repo5, _ = NewPublishedRepo("files:other", "ppa", "osminog", []string{}, []string{"contrib"}, []interface{}{s.snap1}, s.factory, false)
 
 	s.collection = s.factory.PublishedRepoCollection()
-	s.collection.Add(s.repo1)
+	s.collection.Add(s.repo1, s.reflistCollection)
-	s.collection.Add(s.repo2)
+	s.collection.Add(s.repo2, s.reflistCollection)
-	s.collection.Add(s.repo3)
+	s.collection.Add(s.repo3, s.reflistCollection)
-	s.collection.Add(s.repo4)
+	s.collection.Add(s.repo4, s.reflistCollection)
-	s.collection.Add(s.repo5)
+	s.collection.Add(s.repo5, s.reflistCollection)
 
 	s.root = c.MkDir()
 	s.publishedStorage = files.NewPublishedStorage(s.root, "", "")

deb/reflist.go

@@ -2,10 +2,15 @@ package deb
 
 import (
 	"bytes"
+	"crypto/sha256"
+	"encoding/base64"
 	"encoding/json"
+	"fmt"
 	"sort"
 
 	"github.com/AlekSi/pointer"
+	"github.com/aptly-dev/aptly/database"
+	"github.com/cespare/xxhash/v2"
 	"github.com/ugorji/go/codec"
 )
 
@@ -44,6 +49,13 @@ func NewPackageRefListFromPackageList(list *PackageList) *PackageRefList {
 	return reflist
 }
 
+func (l *PackageRefList) Clone() *PackageRefList {
+	clone := &PackageRefList{}
+	clone.Refs = make([][]byte, l.Len())
+	copy(clone.Refs, l.Refs)
+	return clone
+}
+
 // Len returns number of refs
 func (l *PackageRefList) Len() int {
 	return len(l.Refs)
@@ -184,8 +196,12 @@ func (d PackageDiff) MarshalJSON() ([]byte, error) {
 type PackageDiffs []PackageDiff
 
 // Diff calculates difference between two reflists
-func (l *PackageRefList) Diff(r *PackageRefList, packageCollection *PackageCollection) (result PackageDiffs, err error) {
+func (l *PackageRefList) Diff(r *PackageRefList, packageCollection *PackageCollection, result PackageDiffs) (PackageDiffs, error) {
+	var err error
+
+	if result == nil {
 		result = make(PackageDiffs, 0, 128)
+	}
 
 	// pointer to left and right reflists
 	il, ir := 0, 0
@@ -258,7 +274,7 @@ func (l *PackageRefList) Diff(r *PackageRefList, packageCollection *PackageColle
 		}
 	}
 
-	return
+	return result, nil
 }
 
 // Merge merges reflist r into current reflist. If overrideMatching, merge
@@ -391,3 +407,773 @@ func (l *PackageRefList) FilterLatestRefs() {
 		lastArch, lastName, lastVer = arch, name, ver
 	}
 }
+
+const (
+	reflistBucketCount = 1 << 6
+	reflistBucketMask  = reflistBucketCount - 1
+)
+
+type reflistDigestArray [sha256.Size]byte
+
+func bucketRefPrefix(ref []byte) []byte {
+	const maxPrefixLen = 3
+
+	// Cut out the arch, leaving behind the package name and subsequent info.
+	_, ref, _ = bytes.Cut(ref, []byte{' '})
+
+	// Strip off the lib prefix, so that "libxyz" and "xyz", which are likely
+	// to be updated together, go in the same bucket.
+	libPrefix := []byte("lib")
+	if bytes.HasPrefix(ref, libPrefix) {
+		ref = ref[len(libPrefix):]
+	}
+
+	prefixLen := len(ref)
+	if maxPrefixLen < prefixLen {
+		prefixLen = maxPrefixLen
+	}
+	prefix, _, _ := bytes.Cut(ref[:prefixLen], []byte{' '})
+	return prefix
+}
+
+func bucketIdxForRef(ref []byte) int {
+	return int(xxhash.Sum64(bucketRefPrefix(ref))) & reflistBucketMask
+}
+
+// SplitRefList is a list of package refs, similar to a PackageRefList. However,
+// instead of storing a linear array of refs, SplitRefList splits the refs into
+// PackageRefList "buckets", based on a hash of the package name inside the ref.
+// Each bucket has a digest of its contents that serves as its key in the database.
+//
+// When serialized, a SplitRefList just becomes an array of bucket digests, and
+// the buckets themselves are stored separately. Because the buckets are then
+// referenced by their digests, multiple independent reflists can share buckets,
+// if their buckets have matching digests.
+//
+// Buckets themselves may not be confirmed to a single database value; instead,
+// they're split into "segments", based on the database's preferred maximum
+// value size. This prevents large buckets from slowing down the database.
+type SplitRefList struct {
+	Buckets [][]byte
+
+	bucketRefs []*PackageRefList
+}
+
+// NewSplitRefList creates empty SplitRefList
+func NewSplitRefList() *SplitRefList {
+	sl := &SplitRefList{}
+	sl.reset()
+	return sl
+}
+
+// NewSplitRefListFromRefList creates SplitRefList from PackageRefList
+func NewSplitRefListFromRefList(reflist *PackageRefList) *SplitRefList {
+	sl := NewSplitRefList()
+	sl.Replace(reflist)
+	return sl
+}
+
+// NewSplitRefListFromRefList creates SplitRefList from PackageList
+func NewSplitRefListFromPackageList(list *PackageList) *SplitRefList {
+	return NewSplitRefListFromRefList(NewPackageRefListFromPackageList(list))
+}
+
+func (sl *SplitRefList) reset() {
+	sl.Buckets = make([][]byte, reflistBucketCount)
+	sl.bucketRefs = make([]*PackageRefList, reflistBucketCount)
+}
+
+// Has checks whether package is part of reflist
+func (sl *SplitRefList) Has(p *Package) bool {
+	idx := bucketIdxForRef(p.Key(""))
+	if bucket := sl.bucketRefs[idx]; bucket != nil {
+		return bucket.Has(p)
+	}
+	return false
+}
+
+// Len returns number of refs
+func (sl *SplitRefList) Len() int {
+	total := 0
+	for _, bucket := range sl.bucketRefs {
+		if bucket != nil {
+			total += bucket.Len()
+		}
+	}
+	return total
+}
+
+func reflistDigest(l *PackageRefList) []byte {
+	// Different algorithms on PackageRefLists will sometimes return a nil slice
+	// of refs and other times return an empty slice. Regardless, they should
+	// both be treated identically and be given an empty digest.
+	if len(l.Refs) == 0 {
+		return nil
+	}
+
+	h := sha256.New()
+	for _, ref := range l.Refs {
+		h.Write(ref)
+		h.Write([]byte{0})
+	}
+	return h.Sum(nil)
+}
+
+// Removes all the refs inside and replaces them with those in the given reflist
+func (sl *SplitRefList) Replace(reflist *PackageRefList) {
+	sl.reset()
+
+	for _, ref := range reflist.Refs {
+		idx := bucketIdxForRef(ref)
+
+		bucket := sl.bucketRefs[idx]
+		if bucket == nil {
+			bucket = NewPackageRefList()
+			sl.bucketRefs[idx] = bucket
+		}
+
+		bucket.Refs = append(bucket.Refs, ref)
+	}
+
+	for idx, bucket := range sl.bucketRefs {
+		if bucket != nil {
+			sort.Sort(bucket)
+			sl.Buckets[idx] = reflistDigest(bucket)
+		}
+	}
+}
+
+// Merge merges reflist r into current reflist (see PackageRefList.Merge)
+func (sl *SplitRefList) Merge(r *SplitRefList, overrideMatching, ignoreConflicting bool) (result *SplitRefList) {
+	result = NewSplitRefList()
+
+	var empty PackageRefList
+	for idx, lbucket := range sl.bucketRefs {
+		rbucket := r.bucketRefs[idx]
+		if lbucket == nil && rbucket == nil {
+			continue
+		}
+
+		if lbucket == nil {
+			lbucket = &empty
+		} else if rbucket == nil {
+			rbucket = &empty
+		}
+
+		result.bucketRefs[idx] = lbucket.Merge(rbucket, overrideMatching, ignoreConflicting)
+		result.Buckets[idx] = reflistDigest(result.bucketRefs[idx])
+	}
+
+	return
+}
+
+// Subtract returns all packages in l that are not in r
+func (sl *SplitRefList) Subtract(r *SplitRefList) (result *SplitRefList) {
+	result = NewSplitRefList()
+
+	for idx, lbucket := range sl.bucketRefs {
+		rbucket := r.bucketRefs[idx]
+		if lbucket != nil {
+			if rbucket != nil {
+				result.bucketRefs[idx] = lbucket.Subtract(rbucket)
+				result.Buckets[idx] = reflistDigest(result.bucketRefs[idx])
+			} else {
+				result.bucketRefs[idx] = lbucket.Clone()
+				result.Buckets[idx] = sl.Buckets[idx]
+			}
+		}
+	}
+
+	return
+}
+
+// Diff calculates difference between two reflists
+func (sl *SplitRefList) Diff(r *SplitRefList, packageCollection *PackageCollection, result PackageDiffs) (PackageDiffs, error) {
+	var err error
+
+	if result == nil {
+		result = make(PackageDiffs, 0, 128)
+	}
+
+	var empty PackageRefList
+	for idx, lbucket := range sl.bucketRefs {
+		rbucket := r.bucketRefs[idx]
+		if lbucket != nil {
+			if rbucket != nil {
+				result, err = lbucket.Diff(rbucket, packageCollection, result)
+			} else {
+				result, err = lbucket.Diff(&empty, packageCollection, result)
+			}
+		} else if rbucket != nil {
+			result, err = empty.Diff(rbucket, packageCollection, result)
+		}
+
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	sort.Slice(result, func(i, j int) bool {
+		var ri, rj []byte
+		if result[i].Left != nil {
+			ri = result[i].Left.Key("")
+		} else {
+			ri = result[i].Right.Key("")
+		}
+		if result[j].Left != nil {
+			rj = result[j].Left.Key("")
+		} else {
+			rj = result[j].Right.Key("")
+		}
+
+		return bytes.Compare(ri, rj) < 0
+	})
+
+	return result, nil
+}
+
+// FilterLatestRefs reduces a reflist to the latest of each package (see PackageRefList.FilterLatestRefs)
+func (sl *SplitRefList) FilterLatestRefs() {
+	for idx, bucket := range sl.bucketRefs {
+		if bucket != nil {
+			bucket.FilterLatestRefs()
+			sl.Buckets[idx] = reflistDigest(bucket)
+		}
+	}
+}
+
+// Flatten creates a flat PackageRefList containing all the refs in this reflist
+func (sl *SplitRefList) Flatten() *PackageRefList {
+	reflist := NewPackageRefList()
+	sl.ForEach(func(ref []byte) error {
+		reflist.Refs = append(reflist.Refs, ref)
+		return nil
+	})
+	sort.Sort(reflist)
+	return reflist
+}
+
+// ForEachBucket calls handler for each bucket in list
+func (sl *SplitRefList) ForEachBucket(handler func(digest []byte, bucket *PackageRefList) error) error {
+	for idx, digest := range sl.Buckets {
+		if len(digest) == 0 {
+			continue
+		}
+
+		bucket := sl.bucketRefs[idx]
+		if bucket != nil {
+			if err := handler(digest, bucket); err != nil {
+				return err
+			}
+		}
+	}
+
+	return nil
+}
+
+// ForEach calls handler for each package ref in list
+//
+// IMPORTANT: unlike PackageRefList.ForEach, the order of handler invocations
+// is *not* guaranteed to be sorted.
+func (sl *SplitRefList) ForEach(handler func([]byte) error) error {
+	for idx, digest := range sl.Buckets {
+		if len(digest) == 0 {
+			continue
+		}
+
+		bucket := sl.bucketRefs[idx]
+		if bucket != nil {
+			if err := bucket.ForEach(handler); err != nil {
+				return err
+			}
+		}
+	}
+
+	return nil
+}
+
+// RefListDigestSet is a set of SplitRefList bucket digests
+type RefListDigestSet struct {
+	items map[reflistDigestArray]struct{}
+}
+
+// NewRefListDigestSet creates empty RefListDigestSet
+func NewRefListDigestSet() *RefListDigestSet {
+	return &RefListDigestSet{items: map[reflistDigestArray]struct{}{}}
+}
+
+// Len returns number of digests in the set
+func (set *RefListDigestSet) Len() int {
+	return len(set.items)
+}
+
+// ForEach calls handler for each digest in the set
+func (set *RefListDigestSet) ForEach(handler func(digest []byte) error) error {
+	for digest := range set.items {
+		if err := handler(digest[:]); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+// workaround for: conversion of slices to arrays requires go1.20 or later
+func newRefListArray(digest []byte) reflistDigestArray {
+	var array reflistDigestArray
+	copy(array[:], digest)
+	return array
+}
+
+// Add adds digest to set, doing nothing if the digest was already present
+func (set *RefListDigestSet) Add(digest []byte) {
+	set.items[newRefListArray(digest)] = struct{}{}
+}
+
+// AddAllInRefList adds all the bucket digests in a SplitRefList to the set
+func (set *RefListDigestSet) AddAllInRefList(sl *SplitRefList) {
+	for _, digest := range sl.Buckets {
+		if len(digest) > 0 {
+			set.Add(digest)
+		}
+	}
+}
+
+// Has checks whether a digest is part of set
+func (set *RefListDigestSet) Has(digest []byte) bool {
+	_, ok := set.items[newRefListArray(digest)]
+	return ok
+}
+
+// Remove removes a digest from set
+func (set *RefListDigestSet) Remove(digest []byte) {
+	delete(set.items, newRefListArray(digest))
+}
+
+// RemoveAll removes all the digests in other from the current set
+func (set *RefListDigestSet) RemoveAll(other *RefListDigestSet) {
+	for digest := range other.items {
+		delete(set.items, digest)
+	}
+}
+
+// RefListCollection does listing, updating/adding/deleting of SplitRefLists
+type RefListCollection struct {
+	db database.Storage
+
+	cache map[reflistDigestArray]*PackageRefList
+}
+
+// NewRefListCollection creates a RefListCollection
+func NewRefListCollection(db database.Storage) *RefListCollection {
+	return &RefListCollection{db: db, cache: make(map[reflistDigestArray]*PackageRefList)}
+}
+
+type reflistStorageFormat int
+
+const (
+	// (legacy format) all the refs are stored inline in a single value
+	reflistStorageFormatInline reflistStorageFormat = iota
+	// the refs are split into buckets that are stored externally from the value
+	reflistStorageFormatSplit
+)
+
+// NoPadding is used because all digests are the same length, so the padding
+// is useless and only serves to muddy the output.
+var bucketDigestEncoding = base64.StdEncoding.WithPadding(base64.NoPadding)
+
+func segmentPrefix(encodedDigest string) []byte {
+	return []byte(fmt.Sprintf("F%s-", encodedDigest))
+}
+
+// workaround for go 1.19 instead of bytes.Clone
+func cloneBytes(b []byte) []byte {
+	if b == nil {
+		return nil
+	}
+	cloned := make([]byte, len(b))
+	copy(cloned, b)
+	return cloned
+}
+
+func segmentIndexKey(prefix []byte, idx int) []byte {
+	// Assume most buckets won't have more than 0xFFFF = ~65k segments (which
+	// would be an extremely large bucket!).
+	return append(cloneBytes(prefix), []byte(fmt.Sprintf("%04x", idx))...)
+}
+
+// AllBucketDigests returns a set of all the bucket digests in the database
+func (collection *RefListCollection) AllBucketDigests() (*RefListDigestSet, error) {
+	digests := NewRefListDigestSet()
+
+	err := collection.db.ProcessByPrefix([]byte("F"), func(key []byte, _ []byte) error {
+		if !bytes.HasSuffix(key, []byte("-0000")) {
+			// Ignore additional segments for the same digest.
+			return nil
+		}
+
+		encodedDigest, _, foundDash := bytes.Cut(key[1:], []byte("-"))
+		if !foundDash {
+			return fmt.Errorf("invalid key: %s", string(key))
+		}
+		digest := make([]byte, bucketDigestEncoding.DecodedLen(len(encodedDigest)))
+		if _, err := bucketDigestEncoding.Decode(digest, encodedDigest); err != nil {
+			return fmt.Errorf("decoding key %s: %w", string(key), err)
+		}
+
+		digests.Add(digest)
+		return nil
+	})
+
+	if err != nil {
+		return nil, err
+	}
+	return digests, nil
+}
+
+// UnsafeDropBucket drops the bucket associated with digest from the database,
+// doing so inside batch
+//
+// This is considered "unsafe" because no checks are performed to ensure that
+// the bucket is no longer referenced by any saved reflists.
+func (collection *RefListCollection) UnsafeDropBucket(digest []byte, batch database.Batch) error {
+	prefix := segmentPrefix(bucketDigestEncoding.EncodeToString(digest))
+	return collection.db.ProcessByPrefix(prefix, func(key []byte, _ []byte) error {
+		return batch.Delete(key)
+	})
+}
+
+func (collection *RefListCollection) load(sl *SplitRefList, key []byte) (reflistStorageFormat, error) {
+	sl.reset()
+
+	data, err := collection.db.Get(key)
+	if err != nil {
+		return 0, err
+	}
+
+	var splitOrInlineRefList struct {
+		*SplitRefList
+		*PackageRefList
+	}
+	handle := &codec.MsgpackHandle{}
+	handle.ZeroCopy = true
+	decoder := codec.NewDecoderBytes(data, handle)
+	if err := decoder.Decode(&splitOrInlineRefList); err != nil {
+		return 0, err
+	}
+
+	if splitOrInlineRefList.SplitRefList != nil {
+		sl.Buckets = splitOrInlineRefList.Buckets
+	} else if splitOrInlineRefList.PackageRefList != nil {
+		sl.Replace(splitOrInlineRefList.PackageRefList)
+		return reflistStorageFormatInline, nil
+	}
+
+	return reflistStorageFormatSplit, nil
+}
+
+func (collection *RefListCollection) loadBuckets(sl *SplitRefList) error {
+	for idx := range sl.Buckets {
+		if sl.bucketRefs[idx] != nil {
+			continue
+		}
+
+		var bucket *PackageRefList
+
+		if digest := sl.Buckets[idx]; len(digest) > 0 {
+			cacheKey := newRefListArray(digest)
+			bucket = collection.cache[cacheKey]
+			if bucket == nil {
+				bucket = NewPackageRefList()
+				prefix := segmentPrefix(bucketDigestEncoding.EncodeToString(digest))
+				err := collection.db.ProcessByPrefix(prefix, func(_ []byte, value []byte) error {
+					var l PackageRefList
+					if err := l.Decode(append([]byte{}, value...)); err != nil {
+						return err
+					}
+
+					bucket.Refs = append(bucket.Refs, l.Refs...)
+					return nil
+				})
+
+				if err != nil {
+					return err
+				}
+
+				// The segments may not have been iterated in order, so make sure to re-sort
+				// here.
+				sort.Sort(bucket)
+				collection.cache[cacheKey] = bucket
+			}
+
+			actualDigest := reflistDigest(bucket)
+			if !bytes.Equal(actualDigest, digest) {
+				return fmt.Errorf("corrupt reflist bucket %d: expected digest %s, got %s",
+					idx,
+					bucketDigestEncoding.EncodeToString(digest),
+					bucketDigestEncoding.EncodeToString(actualDigest))
+			}
+		}
+
+		sl.bucketRefs[idx] = bucket
+	}
+
+	return nil
+}
+
+// LoadComplete loads the reflist stored at the given key, as well as all the
+// buckets referenced by a split reflist
+func (collection *RefListCollection) LoadComplete(sl *SplitRefList, key []byte) error {
+	if _, err := collection.load(sl, key); err != nil {
+		return err
+	}
+
+	return collection.loadBuckets(sl)
+}
+
+// RefListBatch is a wrapper over a database.Batch that tracks already-written
+// reflists to avoid writing them multiple times
+//
+// It is *not* safe to use the same underlying database.Batch that has already
+// been given to UnsafeDropBucket.
+type RefListBatch struct {
+	batch database.Batch
+
+	alreadyWritten *RefListDigestSet
+}
+
+// NewBatch creates a new RefListBatch wrapping the given database.Batch
+func (collection *RefListCollection) NewBatch(batch database.Batch) *RefListBatch {
+	return &RefListBatch{
+		batch:          batch,
+		alreadyWritten: NewRefListDigestSet(),
+	}
+}
+
+type reflistUpdateContext struct {
+	rb    *RefListBatch
+	stats *RefListMigrationStats
+}
+
+func clearSegmentRefs(reflist *PackageRefList, recommendedMaxKVSize int) {
+	avgRefsInSegment := recommendedMaxKVSize / 70
+	reflist.Refs = make([][]byte, 0, avgRefsInSegment)
+}
+
+func flushSegmentRefs(uctx *reflistUpdateContext, prefix []byte, segment int, reflist *PackageRefList) error {
+	encoded := reflist.Encode()
+	err := uctx.rb.batch.Put(segmentIndexKey(prefix, segment), encoded)
+	if err == nil && uctx.stats != nil {
+		uctx.stats.Segments++
+	}
+	return err
+}
+
+func (collection *RefListCollection) updateWithContext(sl *SplitRefList, key []byte, uctx *reflistUpdateContext) error {
+	if sl != nil {
+		recommendedMaxKVSize := collection.db.GetRecommendedMaxKVSize()
+
+		for idx, digest := range sl.Buckets {
+			if len(digest) == 0 {
+				continue
+			}
+
+			if uctx.rb.alreadyWritten.Has(digest) {
+				continue
+			}
+
+			prefix := segmentPrefix(bucketDigestEncoding.EncodeToString(digest))
+			if collection.db.HasPrefix(prefix) {
+				continue
+			}
+
+			// All the sizing information taken from the msgpack spec:
+			// https://github.com/msgpack/msgpack/blob/master/spec.md
+
+			// Assume that a segment will have [16,2^16) elements, which would
+			// fit into an array 16 and thus have 3 bytes of overhead.
+			// (A database would need a massive recommendedMaxKVSize to pass
+			// that limit.)
+			size := len(segmentIndexKey(prefix, 0)) + 3
+			segment := 0
+
+			var reflist PackageRefList
+			clearSegmentRefs(&reflist, recommendedMaxKVSize)
+			for _, ref := range sl.bucketRefs[idx].Refs {
+				// In order to determine the size of the ref in the database,
+				// we need to know how much overhead will be added with by msgpack
+				// encoding.
+				requiredSize := len(ref)
+				if requiredSize < 1<<5 {
+					requiredSize++
+				} else if requiredSize < 1<<8 {
+					requiredSize += 2
+				} else if requiredSize < 1<<16 {
+					requiredSize += 3
+				} else {
+					requiredSize += 4
+				}
+				if size+requiredSize > recommendedMaxKVSize {
+					if err := flushSegmentRefs(uctx, prefix, segment, &reflist); err != nil {
+						return err
+					}
+					clearSegmentRefs(&reflist, recommendedMaxKVSize)
+					segment++
+				}
+
+				reflist.Refs = append(reflist.Refs, ref)
+				size += requiredSize
+			}
+
+			if len(reflist.Refs) > 0 {
+				if err := flushSegmentRefs(uctx, prefix, segment, &reflist); err != nil {
+					return err
+				}
+			}
+
+			uctx.rb.alreadyWritten.Add(digest)
+			if uctx.stats != nil {
+				uctx.stats.Buckets++
+			}
+		}
+	}
+
+	var buf bytes.Buffer
+	encoder := codec.NewEncoder(&buf, &codec.MsgpackHandle{})
+	encoder.Encode(sl)
+	err := uctx.rb.batch.Put(key, buf.Bytes())
+	if err == nil && uctx.stats != nil {
+		uctx.stats.Reflists++
+	}
+	return err
+}
+
+// UpdateInBatch will save or update the SplitRefList at key, as well as save the buckets inside,
+// as part of the given batch
+func (collection *RefListCollection) UpdateInBatch(sl *SplitRefList, key []byte, batch *RefListBatch) error {
+	return collection.updateWithContext(sl, key, &reflistUpdateContext{rb: batch})
+}
+
+// Update will save or update the SplitRefList at key, as well as save the buckets inside
+func (collection *RefListCollection) Update(sl *SplitRefList, key []byte) error {
+	rb := collection.NewBatch(collection.db.CreateBatch())
+	err := collection.UpdateInBatch(sl, key, rb)
+	if err == nil {
+		err = rb.batch.Write()
+	}
+	return err
+}
+
+// RefListMigrationStats counts a number of reflists, buckets, and segments
+type RefListMigrationStats struct {
+	Reflists, Buckets, Segments int
+}
+
+// RefListMigration wraps a RefListBatch for the purpose of migrating inline format
+// reflists to split reflists
+//
+// Once the batch gets too large, it will automatically be flushed to the database,
+// and a new batch will be created in its place.
+type RefListMigration struct {
+	rb *RefListBatch
+
+	dryRun bool
+
+	// current number of reflists/buckets/segments queued in the current, unwritten batch
+	batchStats RefListMigrationStats
+	flushStats RefListMigrationStats
+}
+
+// NewMigration creates an empty RefListMigration
+func (collection *RefListCollection) NewMigration() *RefListMigration {
+	return &RefListMigration{}
+}
+
+// NewMigrationDryRun creates an empty RefListMigration that will track the
+// changes to make as usual but avoid actually writing to the db
+func (collection *RefListCollection) NewMigrationDryRun() *RefListMigration {
+	return &RefListMigration{dryRun: true}
+}
+
+// Stats returns statistics on the written values in the current migration
+func (migration *RefListMigration) Stats() RefListMigrationStats {
+	return migration.flushStats
+}
+
+// Flush will flush the current batch in the migration to the database
+func (migration *RefListMigration) Flush() error {
+	if migration.batchStats.Segments > 0 {
+		if !migration.dryRun {
+			if err := migration.rb.batch.Write(); err != nil {
+				return err
+			}
+
+			// It's important that we don't clear the batch on dry runs, because
+			// the batch is what contains the list of already-written buckets.
+			// If we're not writing to the database, and we clear that list,
+			// duplicate "writes" will occur.
+			migration.rb = nil
+		}
+
+		migration.flushStats.Reflists += migration.batchStats.Reflists
+		migration.flushStats.Buckets += migration.batchStats.Buckets
+		migration.flushStats.Segments += migration.batchStats.Segments
+		migration.batchStats = RefListMigrationStats{}
+	}
+
+	return nil
+}
+
+// LoadCompleteAndMigrate will load the reflist and its buckets as RefListCollection.LoadComplete,
+// migrating any inline reflists to split ones along the way
+func (collection *RefListCollection) LoadCompleteAndMigrate(sl *SplitRefList, key []byte, migration *RefListMigration) error {
+	// Given enough reflists, the memory used by a batch starts to become massive, so
+	// make sure to flush the written segments periodically. Note that this is only
+	// checked *after* a migration of a full bucket (and all the segments inside)
+	// takes place, as splitting a single bucket write into multiple batches would
+	// be unsafe if an interruption occurs midway.
+	const maxMigratorBatch = 50000
+
+	format, err := collection.load(sl, key)
+	if err != nil {
+		return err
+	}
+
+	switch format {
+	case reflistStorageFormatInline:
+		if migration.rb == nil {
+			migration.rb = collection.NewBatch(collection.db.CreateBatch())
+		}
+
+		collection.updateWithContext(sl, key, &reflistUpdateContext{
+			rb:    migration.rb,
+			stats: &migration.batchStats,
+		})
+
+		if migration.batchStats.Segments > maxMigratorBatch {
+			if err := migration.Flush(); err != nil {
+				return err
+			}
+		}
+
+		return nil
+	case reflistStorageFormatSplit:
+		return collection.loadBuckets(sl)
+	default:
+		panic(fmt.Sprintf("unexpected format %v", format))
+	}
+}
+
+// AnyRefList is implemented by both PackageRefList and SplitRefList
+type AnyRefList interface {
+	Has(p *Package) bool
+	Len() int
+	ForEach(handler func([]byte) error) error
+	FilterLatestRefs()
+}
+
+// Check interface
+var (
+	_ AnyRefList = (*PackageRefList)(nil)
+	_ AnyRefList = (*SplitRefList)(nil)
+)

deb/reflist_bench_test.go

@@ -45,3 +45,41 @@ func BenchmarkReflistDecode(b *testing.B) {
 		(&PackageRefList{}).Decode(data)
 	}
 }
+
+func BenchmarkSplitRefListCreationSmall(b *testing.B) {
+	const count = 400
+
+	l := NewPackageRefList()
+
+	for i := 0; i < count; i++ {
+		l.Refs = append(l.Refs, []byte(fmt.Sprintf("Pamd64 %x %d", i, i)))
+	}
+
+	sort.Sort(l)
+
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		for j := 0; j < 8; j++ {
+			NewSplitRefListFromRefList(l)
+		}
+	}
+}
+
+func BenchmarkSplitRefListCreationLarge(b *testing.B) {
+	const count = 4096
+
+	l := NewPackageRefList()
+
+	for i := 0; i < count; i++ {
+		l.Refs = append(l.Refs, []byte(fmt.Sprintf("Pamd64 %x %d", i, i)))
+	}
+
+	sort.Sort(l)
+
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		for j := 0; j < 8; j++ {
+			NewSplitRefListFromRefList(l)
+		}
+	}
+}

deb/reflist_test.go

@@ -1,7 +1,10 @@
 package deb
 
 import (
+	"bytes"
+	"encoding/hex"
 	"errors"
+	"fmt"
 
 	"github.com/aptly-dev/aptly/database/goleveldb"
 
@@ -9,24 +12,83 @@ import (
 )
 
 type PackageRefListSuite struct {
-	// Simple list with "real" packages from stanzas
-	list                   *PackageList
 	p1, p2, p3, p4, p5, p6 *Package
 }
 
 var _ = Suite(&PackageRefListSuite{})
 
-func toStrSlice(reflist *PackageRefList) (result []string) {
+func verifyRefListIntegrity(c *C, rl AnyRefList) AnyRefList {
+	if rl, ok := rl.(*SplitRefList); ok {
+		for idx, bucket := range rl.bucketRefs {
+			if bucket == nil {
+				bucket = NewPackageRefList()
+			}
+			c.Check(rl.Buckets[idx], DeepEquals, reflistDigest(bucket))
+		}
+	}
+
+	return rl
+}
+
+func getRefs(rl AnyRefList) (refs [][]byte) {
+	switch rl := rl.(type) {
+	case *PackageRefList:
+		refs = rl.Refs
+	case *SplitRefList:
+		refs = rl.Flatten().Refs
+	default:
+		panic(fmt.Sprintf("unexpected reflist type %t", rl))
+	}
+
+	// Hack so that passing getRefs-returned slices to DeepEquals won't fail given a nil
+	// slice and an empty slice.
+	if len(refs) == 0 {
+		refs = nil
+	}
+	return
+}
+
+func toStrSlice(reflist AnyRefList) (result []string) {
 	result = make([]string, reflist.Len())
-	for i, r := range reflist.Refs {
+	for i, r := range getRefs(reflist) {
 		result[i] = string(r)
 	}
 	return
 }
 
-func (s *PackageRefListSuite) SetUpTest(c *C) {
+type reflistFactory struct {
-	s.list = NewPackageList()
+	new                func() AnyRefList
+	newFromRefs        func(refs ...[]byte) AnyRefList
+	newFromPackageList func(list *PackageList) AnyRefList
+}
 
+func forEachRefList(test func(f reflistFactory)) {
+	test(reflistFactory{
+		new: func() AnyRefList {
+			return NewPackageRefList()
+		},
+		newFromRefs: func(refs ...[]byte) AnyRefList {
+			return &PackageRefList{Refs: refs}
+		},
+		newFromPackageList: func(list *PackageList) AnyRefList {
+			return NewPackageRefListFromPackageList(list)
+		},
+	})
+
+	test(reflistFactory{
+		new: func() AnyRefList {
+			return NewSplitRefList()
+		},
+		newFromRefs: func(refs ...[]byte) AnyRefList {
+			return NewSplitRefListFromRefList(&PackageRefList{Refs: refs})
+		},
+		newFromPackageList: func(list *PackageList) AnyRefList {
+			return NewSplitRefListFromPackageList(list)
+		},
+	})
+}
+
+func (s *PackageRefListSuite) SetUpTest(c *C) {
 	s.p1 = NewPackageFromControlFile(packageStanza.Copy())
 	s.p2 = NewPackageFromControlFile(packageStanza.Copy())
 	stanza := packageStanza.Copy()
@@ -44,17 +106,20 @@ func (s *PackageRefListSuite) SetUpTest(c *C) {
 }
 
 func (s *PackageRefListSuite) TestNewPackageListFromRefList(c *C) {
+	forEachRefList(func(f reflistFactory) {
+		list := NewPackageList()
+
 		db, _ := goleveldb.NewOpenDB(c.MkDir())
 		coll := NewPackageCollection(db)
 		coll.Update(s.p1)
 		coll.Update(s.p3)
 
-	s.list.Add(s.p1)
+		list.Add(s.p1)
-	s.list.Add(s.p3)
+		list.Add(s.p3)
-	s.list.Add(s.p5)
+		list.Add(s.p5)
-	s.list.Add(s.p6)
+		list.Add(s.p6)
 
-	reflist := NewPackageRefListFromPackageList(s.list)
+		reflist := f.newFromPackageList(list)
 
 		_, err := NewPackageListFromRefList(reflist, coll, nil)
 		c.Assert(err, ErrorMatches, "unable to load package with key.*")
@@ -62,7 +127,7 @@ func (s *PackageRefListSuite) TestNewPackageListFromRefList(c *C) {
 		coll.Update(s.p5)
 		coll.Update(s.p6)
 
-	list, err := NewPackageListFromRefList(reflist, coll, nil)
+		list, err = NewPackageListFromRefList(reflist, coll, nil)
 		c.Assert(err, IsNil)
 		c.Check(list.Len(), Equals, 4)
 		c.Check(list.Add(s.p4), ErrorMatches, "package already exists and is different: .*")
@@ -70,47 +135,40 @@ func (s *PackageRefListSuite) TestNewPackageListFromRefList(c *C) {
 		list, err = NewPackageListFromRefList(nil, coll, nil)
 		c.Assert(err, IsNil)
 		c.Check(list.Len(), Equals, 0)
+	})
 }
 
 func (s *PackageRefListSuite) TestNewPackageRefList(c *C) {
-	s.list.Add(s.p1)
+	forEachRefList(func(f reflistFactory) {
-	s.list.Add(s.p3)
+		list := NewPackageList()
-	s.list.Add(s.p5)
+		list.Add(s.p1)
-	s.list.Add(s.p6)
+		list.Add(s.p3)
+		list.Add(s.p5)
+		list.Add(s.p6)
 
-	reflist := NewPackageRefListFromPackageList(s.list)
+		reflist := f.newFromPackageList(list)
+		verifyRefListIntegrity(c, reflist)
 		c.Assert(reflist.Len(), Equals, 4)
-	c.Check(reflist.Refs[0], DeepEquals, []byte(s.p1.Key("")))
+		refs := getRefs(reflist)
-	c.Check(reflist.Refs[1], DeepEquals, []byte(s.p6.Key("")))
+		c.Check(refs[0], DeepEquals, []byte(s.p1.Key("")))
-	c.Check(reflist.Refs[2], DeepEquals, []byte(s.p5.Key("")))
+		c.Check(refs[1], DeepEquals, []byte(s.p6.Key("")))
-	c.Check(reflist.Refs[3], DeepEquals, []byte(s.p3.Key("")))
+		c.Check(refs[2], DeepEquals, []byte(s.p5.Key("")))
+		c.Check(refs[3], DeepEquals, []byte(s.p3.Key("")))
 
-	reflist = NewPackageRefList()
+		reflist = f.new()
 		c.Check(reflist.Len(), Equals, 0)
-}
+	})
-
-func (s *PackageRefListSuite) TestPackageRefListEncodeDecode(c *C) {
-	s.list.Add(s.p1)
-	s.list.Add(s.p3)
-	s.list.Add(s.p5)
-	s.list.Add(s.p6)
-
-	reflist := NewPackageRefListFromPackageList(s.list)
-
-	reflist2 := &PackageRefList{}
-	err := reflist2.Decode(reflist.Encode())
-	c.Assert(err, IsNil)
-	c.Check(reflist2.Len(), Equals, reflist.Len())
-	c.Check(reflist2.Refs, DeepEquals, reflist.Refs)
 }
 
 func (s *PackageRefListSuite) TestPackageRefListForeach(c *C) {
-	s.list.Add(s.p1)
+	forEachRefList(func(f reflistFactory) {
-	s.list.Add(s.p3)
+		list := NewPackageList()
-	s.list.Add(s.p5)
+		list.Add(s.p1)
-	s.list.Add(s.p6)
+		list.Add(s.p3)
+		list.Add(s.p5)
+		list.Add(s.p6)
 
-	reflist := NewPackageRefListFromPackageList(s.list)
+		reflist := f.newFromPackageList(list)
 
 		Len := 0
 		err := reflist.ForEach(func([]byte) error {
@@ -128,44 +186,73 @@ func (s *PackageRefListSuite) TestPackageRefListForeach(c *C) {
 		})
 
 		c.Check(err, Equals, e)
+	})
 }
 
 func (s *PackageRefListSuite) TestHas(c *C) {
-	s.list.Add(s.p1)
+	forEachRefList(func(f reflistFactory) {
-	s.list.Add(s.p3)
+		list := NewPackageList()
-	s.list.Add(s.p5)
+		list.Add(s.p1)
-	reflist := NewPackageRefListFromPackageList(s.list)
+		list.Add(s.p3)
+		list.Add(s.p5)
+		reflist := f.newFromPackageList(list)
 
 		c.Check(reflist.Has(s.p1), Equals, true)
 		c.Check(reflist.Has(s.p3), Equals, true)
 		c.Check(reflist.Has(s.p5), Equals, true)
 		c.Check(reflist.Has(s.p2), Equals, true)
 		c.Check(reflist.Has(s.p6), Equals, false)
+	})
 }
 
-func (s *PackageRefListSuite) TestSubstract(c *C) {
+func subtractRefLists(l, r AnyRefList) AnyRefList {
-	r1 := []byte("r1")
+	switch l := l.(type) {
-	r2 := []byte("r2")
+	case *PackageRefList:
-	r3 := []byte("r3")
+		return l.Subtract(r.(*PackageRefList))
-	r4 := []byte("r4")
+	case *SplitRefList:
-	r5 := []byte("r5")
+		return l.Subtract(r.(*SplitRefList))
+	default:
+		panic(fmt.Sprintf("unexpected reflist type %t", l))
+	}
+}
 
-	empty := &PackageRefList{Refs: [][]byte{}}
+func (s *PackageRefListSuite) TestSubtract(c *C) {
-	l1 := &PackageRefList{Refs: [][]byte{r1, r2, r3, r4}}
+	forEachRefList(func(f reflistFactory) {
-	l2 := &PackageRefList{Refs: [][]byte{r1, r3}}
+		r1 := []byte("Pall r1")
-	l3 := &PackageRefList{Refs: [][]byte{r2, r4}}
+		r2 := []byte("Pall r2")
-	l4 := &PackageRefList{Refs: [][]byte{r4, r5}}
+		r3 := []byte("Pall r3")
-	l5 := &PackageRefList{Refs: [][]byte{r1, r2, r3}}
+		r4 := []byte("Pall r4")
+		r5 := []byte("Pall r5")
 
-	c.Check(l1.Subtract(empty), DeepEquals, l1)
+		empty := f.newFromRefs()
-	c.Check(l1.Subtract(l2), DeepEquals, l3)
+		l1 := f.newFromRefs(r1, r2, r3, r4)
-	c.Check(l1.Subtract(l3), DeepEquals, l2)
+		l2 := f.newFromRefs(r1, r3)
-	c.Check(l1.Subtract(l4), DeepEquals, l5)
+		l3 := f.newFromRefs(r2, r4)
-	c.Check(empty.Subtract(l1), DeepEquals, empty)
+		l4 := f.newFromRefs(r4, r5)
-	c.Check(l2.Subtract(l3), DeepEquals, l2)
+		l5 := f.newFromRefs(r1, r2, r3)
+
+		c.Check(getRefs(subtractRefLists(l1, empty)), DeepEquals, getRefs(l1))
+		c.Check(getRefs(subtractRefLists(l1, l2)), DeepEquals, getRefs(l3))
+		c.Check(getRefs(subtractRefLists(l1, l3)), DeepEquals, getRefs(l2))
+		c.Check(getRefs(subtractRefLists(l1, l4)), DeepEquals, getRefs(l5))
+		c.Check(getRefs(subtractRefLists(empty, l1)), DeepEquals, getRefs(empty))
+		c.Check(getRefs(subtractRefLists(l2, l3)), DeepEquals, getRefs(l2))
+	})
+}
+
+func diffRefLists(l, r AnyRefList, packageCollection *PackageCollection) (PackageDiffs, error) {
+	switch l := l.(type) {
+	case *PackageRefList:
+		return l.Diff(r.(*PackageRefList), packageCollection, nil)
+	case *SplitRefList:
+		return l.Diff(r.(*SplitRefList), packageCollection, nil)
+	default:
+		panic(fmt.Sprintf("unexpected reflist type %t", l))
+	}
 }
 
 func (s *PackageRefListSuite) TestDiff(c *C) {
+	forEachRefList(func(f reflistFactory) {
 		db, _ := goleveldb.NewOpenDB(c.MkDir())
 		coll := NewPackageCollection(db)
 
@@ -196,14 +283,14 @@ func (s *PackageRefListSuite) TestDiff(c *C) {
 		listB.Add(packages[4])
 		listB.Add(packages[5])
 
-	reflistA := NewPackageRefListFromPackageList(listA)
+		reflistA := f.newFromPackageList(listA)
-	reflistB := NewPackageRefListFromPackageList(listB)
+		reflistB := f.newFromPackageList(listB)
 
-	diffAA, err := reflistA.Diff(reflistA, coll)
+		diffAA, err := diffRefLists(reflistA, reflistA, coll)
 		c.Check(err, IsNil)
 		c.Check(diffAA, HasLen, 0)
 
-	diffAB, err := reflistA.Diff(reflistB, coll)
+		diffAB, err := diffRefLists(reflistA, reflistB, coll)
 		c.Check(err, IsNil)
 		c.Check(diffAB, HasLen, 4)
 
@@ -219,7 +306,7 @@ func (s *PackageRefListSuite) TestDiff(c *C) {
 		c.Check(diffAB[3].Left.String(), Equals, "xyz_3.0_sparc")
 		c.Check(diffAB[3].Right, IsNil)
 
-	diffBA, err := reflistB.Diff(reflistA, coll)
+		diffBA, err := diffRefLists(reflistB, reflistA, coll)
 		c.Check(err, IsNil)
 		c.Check(diffBA, HasLen, 4)
 
@@ -234,10 +321,11 @@ func (s *PackageRefListSuite) TestDiff(c *C) {
 
 		c.Check(diffBA[3].Right.String(), Equals, "xyz_3.0_sparc")
 		c.Check(diffBA[3].Left, IsNil)
-
+	})
 }
 
 func (s *PackageRefListSuite) TestDiffCompactsAtEnd(c *C) {
+	forEachRefList(func(f reflistFactory) {
 		db, _ := goleveldb.NewOpenDB(c.MkDir())
 		coll := NewPackageCollection(db)
 
@@ -258,10 +346,10 @@ func (s *PackageRefListSuite) TestDiffCompactsAtEnd(c *C) {
 		listB.Add(packages[1])
 		listB.Add(packages[2])
 
-	reflistA := NewPackageRefListFromPackageList(listA)
+		reflistA := f.newFromPackageList(listA)
-	reflistB := NewPackageRefListFromPackageList(listB)
+		reflistB := f.newFromPackageList(listB)
 
-	diffAB, err := reflistA.Diff(reflistB, coll)
+		diffAB, err := diffRefLists(reflistA, reflistB, coll)
 		c.Check(err, IsNil)
 		c.Check(diffAB, HasLen, 2)
 
@@ -270,9 +358,22 @@ func (s *PackageRefListSuite) TestDiffCompactsAtEnd(c *C) {
 
 		c.Check(diffAB[1].Left.String(), Equals, "app_1.1~bp1_i386")
 		c.Check(diffAB[1].Right.String(), Equals, "app_1.1~bp2_i386")
+	})
+}
+
+func mergeRefLists(l, r AnyRefList, overrideMatching, ignoreConflicting bool) AnyRefList {
+	switch l := l.(type) {
+	case *PackageRefList:
+		return l.Merge(r.(*PackageRefList), overrideMatching, ignoreConflicting)
+	case *SplitRefList:
+		return l.Merge(r.(*SplitRefList), overrideMatching, ignoreConflicting)
+	default:
+		panic(fmt.Sprintf("unexpected reflist type %t", l))
+	}
 }
 
 func (s *PackageRefListSuite) TestMerge(c *C) {
+	forEachRefList(func(f reflistFactory) {
 		db, _ := goleveldb.NewOpenDB(c.MkDir())
 		coll := NewPackageCollection(db)
 
@@ -313,15 +414,21 @@ func (s *PackageRefListSuite) TestMerge(c *C) {
 		listC.Add(packages[8])
 		listC.Add(packages[9])
 
-	reflistA := NewPackageRefListFromPackageList(listA)
+		reflistA := f.newFromPackageList(listA)
-	reflistB := NewPackageRefListFromPackageList(listB)
+		reflistB := f.newFromPackageList(listB)
-	reflistC := NewPackageRefListFromPackageList(listC)
+		reflistC := f.newFromPackageList(listC)
 
-	mergeAB := reflistA.Merge(reflistB, true, false)
+		mergeAB := mergeRefLists(reflistA, reflistB, true, false)
-	mergeBA := reflistB.Merge(reflistA, true, false)
+		mergeBA := mergeRefLists(reflistB, reflistA, true, false)
-	mergeAC := reflistA.Merge(reflistC, true, false)
+		mergeAC := mergeRefLists(reflistA, reflistC, true, false)
-	mergeBC := reflistB.Merge(reflistC, true, false)
+		mergeBC := mergeRefLists(reflistB, reflistC, true, false)
-	mergeCB := reflistC.Merge(reflistB, true, false)
+		mergeCB := mergeRefLists(reflistC, reflistB, true, false)
+
+		verifyRefListIntegrity(c, mergeAB)
+		verifyRefListIntegrity(c, mergeBA)
+		verifyRefListIntegrity(c, mergeAC)
+		verifyRefListIntegrity(c, mergeBC)
+		verifyRefListIntegrity(c, mergeCB)
 
 		c.Check(toStrSlice(mergeAB), DeepEquals,
 			[]string{"Pall data 1.1~bp1 00000000", "Pamd64 app 1.1~bp2 00000000", "Pi386 app 1.1~bp2 00000000", "Pi386 dpkg 1.0 00000000", "Pi386 lib 1.0 00000000", "Psparc xyz 1.0 00000000"})
@@ -334,11 +441,17 @@ func (s *PackageRefListSuite) TestMerge(c *C) {
 		c.Check(toStrSlice(mergeCB), DeepEquals,
 			[]string{"Pall data 1.1~bp1 00000000", "Pamd64 app 1.1~bp2 00000000", "Pi386 app 1.1~bp2 00000000", "Pi386 dpkg 1.0 00000000", "Pi386 lib 1.0 00000000"})
 
-	mergeABall := reflistA.Merge(reflistB, false, false)
+		mergeABall := mergeRefLists(reflistA, reflistB, false, false)
-	mergeBAall := reflistB.Merge(reflistA, false, false)
+		mergeBAall := mergeRefLists(reflistB, reflistA, false, false)
-	mergeACall := reflistA.Merge(reflistC, false, false)
+		mergeACall := mergeRefLists(reflistA, reflistC, false, false)
-	mergeBCall := reflistB.Merge(reflistC, false, false)
+		mergeBCall := mergeRefLists(reflistB, reflistC, false, false)
-	mergeCBall := reflistC.Merge(reflistB, false, false)
+		mergeCBall := mergeRefLists(reflistC, reflistB, false, false)
+
+		verifyRefListIntegrity(c, mergeABall)
+		verifyRefListIntegrity(c, mergeBAall)
+		verifyRefListIntegrity(c, mergeACall)
+		verifyRefListIntegrity(c, mergeBCall)
+		verifyRefListIntegrity(c, mergeCBall)
 
 		c.Check(mergeABall, DeepEquals, mergeBAall)
 		c.Check(toStrSlice(mergeBAall), DeepEquals,
@@ -353,13 +466,15 @@ func (s *PackageRefListSuite) TestMerge(c *C) {
 			[]string{"Pall data 1.1~bp1 00000000", "Pamd64 app 1.1~bp2 00000000", "Pi386 app 1.1~bp2 00000044", "Pi386 dpkg 1.0 00034445",
 				"Pi386 lib 1.0 00000000"})
 
-	mergeBCwithConflicts := reflistB.Merge(reflistC, false, true)
+		mergeBCwithConflicts := mergeRefLists(reflistB, reflistC, false, true)
 		c.Check(toStrSlice(mergeBCwithConflicts), DeepEquals,
 			[]string{"Pall data 1.1~bp1 00000000", "Pamd64 app 1.1~bp2 00000000", "Pi386 app 1.1~bp2 00000000", "Pi386 app 1.1~bp2 00000044",
 				"Pi386 dpkg 1.0 00000000", "Pi386 dpkg 1.0 00034445", "Pi386 lib 1.0 00000000"})
+	})
 }
 
 func (s *PackageRefListSuite) TestFilterLatestRefs(c *C) {
+	forEachRefList(func(f reflistFactory) {
 		packages := []*Package{
 			{Name: "lib", Version: "1.0", Architecture: "i386"},
 			{Name: "lib", Version: "1.2~bp1", Architecture: "i386"},
@@ -381,9 +496,210 @@ func (s *PackageRefListSuite) TestFilterLatestRefs(c *C) {
 		rl.Add(packages[6])
 		rl.Add(packages[7])
 
-	result := NewPackageRefListFromPackageList(rl)
+		result := f.newFromPackageList(rl)
 		result.FilterLatestRefs()
 
+		verifyRefListIntegrity(c, result)
 		c.Check(toStrSlice(result), DeepEquals,
 			[]string{"Pi386 dpkg 1.6", "Pi386 lib 1.2"})
+	})
+}
+
+func (s *PackageRefListSuite) TestPackageRefListEncodeDecode(c *C) {
+	list := NewPackageList()
+	list.Add(s.p1)
+	list.Add(s.p3)
+	list.Add(s.p5)
+	list.Add(s.p6)
+
+	reflist := NewPackageRefListFromPackageList(list)
+
+	reflist2 := &PackageRefList{}
+	err := reflist2.Decode(reflist.Encode())
+	c.Assert(err, IsNil)
+	c.Check(reflist2.Len(), Equals, reflist.Len())
+	c.Check(reflist2.Refs, DeepEquals, reflist.Refs)
+}
+
+func (s *PackageRefListSuite) TestRefListBucketPrefix(c *C) {
+	c.Check(bucketRefPrefix([]byte("Pall abcd 1.0")), DeepEquals, []byte("abc"))
+	c.Check(bucketRefPrefix([]byte("Pall libabcd 1.0")), DeepEquals, []byte("abc"))
+	c.Check(bucketRefPrefix([]byte("Pamd64 xy 1.0")), DeepEquals, []byte("xy"))
+}
+
+func (s *PackageRefListSuite) TestRefListBucketIdx(c *C) {
+	c.Check(bucketIdxForRef(s.p1.Key("")), Equals, 46)
+	c.Check(bucketIdxForRef(s.p2.Key("")), Equals, 46)
+	c.Check(bucketIdxForRef(s.p3.Key("")), Equals, 26)
+	c.Check(bucketIdxForRef(s.p4.Key("")), Equals, 46)
+	c.Check(bucketIdxForRef(s.p5.Key("")), Equals, 4)
+	c.Check(bucketIdxForRef(s.p6.Key("")), Equals, 46)
+}
+
+func (s *PackageRefListSuite) TestSplitRefListBuckets(c *C) {
+	list := NewPackageList()
+	list.Add(s.p1)
+	list.Add(s.p3)
+	list.Add(s.p4)
+	list.Add(s.p5)
+	list.Add(s.p6)
+
+	sl := NewSplitRefListFromPackageList(list)
+	verifyRefListIntegrity(c, sl)
+
+	c.Check(hex.EncodeToString(sl.Buckets[4]), Equals, "7287aed32daad5d1aab4e89533bde135381d932e60548cfc00b882ca8858ae07")
+	c.Check(toStrSlice(sl.bucketRefs[4]), DeepEquals, []string{string(s.p5.Key(""))})
+	c.Check(hex.EncodeToString(sl.Buckets[26]), Equals, "f31fc28e82368b63c8be47eefc64b8e217e2e5349c7e3827b98f80536b956f6e")
+	c.Check(toStrSlice(sl.bucketRefs[26]), DeepEquals, []string{string(s.p3.Key(""))})
+	c.Check(hex.EncodeToString(sl.Buckets[46]), Equals, "55e70286393afc5da5046d68c632d35f98bec24781ae433bd1a1069b52853367")
+	c.Check(toStrSlice(sl.bucketRefs[46]), DeepEquals, []string{string(s.p1.Key("")), string(s.p6.Key(""))})
+}
+
+func (s *PackageRefListSuite) TestRefListDigestSet(c *C) {
+	list := NewPackageList()
+	list.Add(s.p1)
+	list.Add(s.p3)
+	list.Add(s.p4)
+	list.Add(s.p5)
+	list.Add(s.p6)
+
+	sl := NewSplitRefListFromPackageList(list)
+
+	set := NewRefListDigestSet()
+	c.Check(set.Len(), Equals, 0)
+
+	err := sl.ForEachBucket(func(digest []byte, bucket *PackageRefList) error {
+		c.Check(set.Has(digest), Equals, false)
+		return nil
+	})
+	c.Assert(err, IsNil)
+
+	set.AddAllInRefList(sl)
+	c.Check(set.Len(), Equals, 3)
+
+	err = sl.ForEachBucket(func(digest []byte, bucket *PackageRefList) error {
+		c.Check(set.Has(digest), Equals, true)
+		return nil
+	})
+	c.Assert(err, IsNil)
+
+	firstDigest := sl.Buckets[bucketIdxForRef(s.p1.Key(""))]
+	set.Remove(firstDigest)
+	c.Check(set.Len(), Equals, 2)
+
+	err = sl.ForEachBucket(func(digest []byte, bucket *PackageRefList) error {
+		c.Check(set.Has(digest), Equals, !bytes.Equal(digest, firstDigest))
+		return nil
+	})
+	c.Assert(err, IsNil)
+
+	set2 := NewRefListDigestSet()
+	set2.AddAllInRefList(sl)
+	set2.RemoveAll(set)
+
+	err = sl.ForEachBucket(func(digest []byte, bucket *PackageRefList) error {
+		c.Check(set2.Has(digest), Equals, bytes.Equal(digest, firstDigest))
+		return nil
+	})
+	c.Assert(err, IsNil)
+}
+
+func (s *PackageRefListSuite) TestRefListCollectionLoadSave(c *C) {
+	db, _ := goleveldb.NewOpenDB(c.MkDir())
+	reflistCollection := NewRefListCollection(db)
+	packageCollection := NewPackageCollection(db)
+
+	packageCollection.Update(s.p1)
+	packageCollection.Update(s.p2)
+	packageCollection.Update(s.p3)
+	packageCollection.Update(s.p4)
+	packageCollection.Update(s.p5)
+	packageCollection.Update(s.p6)
+
+	list := NewPackageList()
+	list.Add(s.p1)
+	list.Add(s.p2)
+	list.Add(s.p3)
+	list.Add(s.p4)
+	list.Add(s.p5)
+
+	key := []byte("test")
+
+	reflist := NewPackageRefListFromPackageList(list)
+	db.Put(key, reflist.Encode())
+
+	sl := NewSplitRefList()
+	err := reflistCollection.LoadComplete(sl, key)
+	c.Assert(err, IsNil)
+	verifyRefListIntegrity(c, sl)
+	c.Check(toStrSlice(sl), DeepEquals, toStrSlice(reflist))
+
+	list.Add(s.p6)
+	sl = NewSplitRefListFromPackageList(list)
+	err = reflistCollection.Update(sl, key)
+	c.Assert(err, IsNil)
+
+	sl = NewSplitRefList()
+	err = reflistCollection.LoadComplete(sl, key)
+	c.Assert(err, IsNil)
+	verifyRefListIntegrity(c, sl)
+	c.Check(toStrSlice(sl), DeepEquals, toStrSlice(NewPackageRefListFromPackageList(list)))
+}
+
+func (s *PackageRefListSuite) TestRefListCollectionMigrate(c *C) {
+	db, _ := goleveldb.NewOpenDB(c.MkDir())
+	reflistCollection := NewRefListCollection(db)
+	packageCollection := NewPackageCollection(db)
+
+	packageCollection.Update(s.p1)
+	packageCollection.Update(s.p2)
+	packageCollection.Update(s.p3)
+	packageCollection.Update(s.p4)
+	packageCollection.Update(s.p5)
+	packageCollection.Update(s.p6)
+
+	list := NewPackageList()
+	list.Add(s.p1)
+	list.Add(s.p2)
+	list.Add(s.p3)
+	list.Add(s.p4)
+	list.Add(s.p5)
+
+	key := []byte("test")
+
+	reflist := NewPackageRefListFromPackageList(list)
+	db.Put(key, reflist.Encode())
+
+	sl := NewSplitRefList()
+	format, err := reflistCollection.load(sl, key)
+	c.Assert(err, IsNil)
+	c.Check(format, Equals, reflistStorageFormatInline)
+
+	migrator := reflistCollection.NewMigration()
+	err = reflistCollection.LoadCompleteAndMigrate(sl, key, migrator)
+	c.Assert(err, IsNil)
+	verifyRefListIntegrity(c, sl)
+	c.Check(toStrSlice(sl), DeepEquals, toStrSlice(NewPackageRefListFromPackageList(list)))
+
+	stats := migrator.Stats()
+	c.Check(stats.Reflists, Equals, 0)
+	c.Check(stats.Buckets, Equals, 0)
+	c.Check(stats.Segments, Equals, 0)
+
+	err = migrator.Flush()
+	c.Assert(err, IsNil)
+	stats = migrator.Stats()
+	c.Check(stats.Reflists, Equals, 1)
+	c.Check(stats.Buckets, Not(Equals), 0)
+	c.Check(stats.Segments, Equals, stats.Segments)
+
+	sl = NewSplitRefList()
+	err = reflistCollection.LoadComplete(sl, key)
+	c.Assert(err, IsNil)
+	verifyRefListIntegrity(c, sl)
+	c.Check(toStrSlice(sl), DeepEquals, toStrSlice(NewPackageRefListFromPackageList(list)))
+
+	format, err = reflistCollection.load(sl, key)
+	c.Assert(err, IsNil)
+	c.Check(format, Equals, reflistStorageFormatSplit)
 }

deb/remote.go

@@ -20,7 +20,7 @@ import (
 	"github.com/aptly-dev/aptly/http"
 	"github.com/aptly-dev/aptly/pgp"
 	"github.com/aptly-dev/aptly/utils"
-	"github.com/pborman/uuid"
+	"github.com/google/uuid"
 	"github.com/ugorji/go/codec"
 )
 
@@ -73,7 +73,7 @@ type RemoteRepo struct {
 	// Packages for json output
 	Packages []string `codec:"-" json:",omitempty"`
 	// "Snapshot" of current list of packages
-	packageRefs *PackageRefList
+	packageRefs *SplitRefList
 	// Parsed archived root
 	archiveRootURL *url.URL
 	// Current list of packages (filled while updating mirror)
@@ -84,7 +84,7 @@ type RemoteRepo struct {
 func NewRemoteRepo(name string, archiveRoot string, distribution string, components []string,
 	architectures []string, downloadSources bool, downloadUdebs bool, downloadInstaller bool) (*RemoteRepo, error) {
 	result := &RemoteRepo{
-		UUID:              uuid.New(),
+		UUID:              uuid.NewString(),
 		Name:              name,
 		ArchiveRoot:       archiveRoot,
 		Distribution:      distribution,
@@ -170,7 +170,7 @@ func (repo *RemoteRepo) NumPackages() int {
 }
 
 // RefList returns package list for repo
-func (repo *RemoteRepo) RefList() *PackageRefList {
+func (repo *RemoteRepo) RefList() *SplitRefList {
 	return repo.packageRefs
 }
 
@@ -686,7 +686,7 @@ func (repo *RemoteRepo) FinalizeDownload(collectionFactory *CollectionFactory, p
 	})
 
 	if err == nil {
-		repo.packageRefs = NewPackageRefListFromPackageList(repo.packageList)
+		repo.packageRefs = NewSplitRefListFromPackageList(repo.packageList)
 		repo.packageList = nil
 	}
 
@@ -828,14 +828,14 @@ func (collection *RemoteRepoCollection) search(filter func(*RemoteRepo) bool, un
 }
 
 // Add appends new repo to collection and saves it
-func (collection *RemoteRepoCollection) Add(repo *RemoteRepo) error {
+func (collection *RemoteRepoCollection) Add(repo *RemoteRepo, reflistCollection *RefListCollection) error {
 	_, err := collection.ByName(repo.Name)
 
 	if err == nil {
 		return fmt.Errorf("mirror with name %s already exists", repo.Name)
 	}
 
-	err = collection.Update(repo)
+	err = collection.Update(repo, reflistCollection)
 	if err != nil {
 		return err
 	}
@@ -845,28 +845,26 @@ func (collection *RemoteRepoCollection) Add(repo *RemoteRepo) error {
 }
 
 // Update stores updated information about repo in DB
-func (collection *RemoteRepoCollection) Update(repo *RemoteRepo) error {
+func (collection *RemoteRepoCollection) Update(repo *RemoteRepo, reflistCollection *RefListCollection) error {
 	batch := collection.db.CreateBatch()
 
 	batch.Put(repo.Key(), repo.Encode())
 	if repo.packageRefs != nil {
-		batch.Put(repo.RefKey(), repo.packageRefs.Encode())
+		rb := reflistCollection.NewBatch(batch)
+		reflistCollection.UpdateInBatch(repo.packageRefs, repo.RefKey(), rb)
 	}
 	return batch.Write()
 }
 
 // LoadComplete loads additional information for remote repo
-func (collection *RemoteRepoCollection) LoadComplete(repo *RemoteRepo) error {
+func (collection *RemoteRepoCollection) LoadComplete(repo *RemoteRepo, reflistCollection *RefListCollection) error {
-	encoded, err := collection.db.Get(repo.RefKey())
+	repo.packageRefs = NewSplitRefList()
+	err := reflistCollection.LoadComplete(repo.packageRefs, repo.RefKey())
 	if err == database.ErrNotFound {
 		return nil
 	}
-	if err != nil {
-		return err
-	}
 
-	repo.packageRefs = &PackageRefList{}
+	return err
-	return repo.packageRefs.Decode(encoded)
 }
 
 // ByName looks up repository by name

deb/remote_test.go

@@ -52,7 +52,7 @@ func (n *NullVerifier) IsClearSigned(clearsign io.Reader) (bool, error) {
 type PackageListMixinSuite struct {
 	p1, p2, p3 *Package
 	list       *PackageList
-	reflist    *PackageRefList
+	reflist    *SplitRefList
 }
 
 func (s *PackageListMixinSuite) SetUpPackages() {
@@ -72,7 +72,7 @@ func (s *PackageListMixinSuite) SetUpPackages() {
 	s.list.Add(s.p2)
 	s.list.Add(s.p3)
 
-	s.reflist = NewPackageRefListFromPackageList(s.list)
+	s.reflist = NewSplitRefListFromPackageList(s.list)
 }
 
 type RemoteRepoSuite struct {
@@ -290,7 +290,7 @@ func (s *RemoteRepoSuite) TestDownload(c *C) {
 	s.repo.FinalizeDownload(s.collectionFactory, nil)
 	c.Assert(s.repo.packageRefs, NotNil)
 
-	pkg, err := s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Refs[0])
+	pkg, err := s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Flatten().Refs[0])
 	c.Assert(err, IsNil)
 
 	c.Check(pkg.Name, Equals, "amanda-client")
@@ -372,12 +372,12 @@ func (s *RemoteRepoSuite) TestDownloadWithInstaller(c *C) {
 	s.repo.FinalizeDownload(s.collectionFactory, nil)
 	c.Assert(s.repo.packageRefs, NotNil)
 
-	pkg, err := s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Refs[0])
+	pkg, err := s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Flatten().Refs[0])
 	c.Assert(err, IsNil)
 
 	c.Check(pkg.Name, Equals, "amanda-client")
 
-	pkg, err = s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Refs[1])
+	pkg, err = s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Flatten().Refs[1])
 	c.Assert(err, IsNil)
 	c.Check(pkg.Name, Equals, "installer")
 }
@@ -418,12 +418,12 @@ func (s *RemoteRepoSuite) TestDownloadWithSources(c *C) {
 	s.repo.FinalizeDownload(s.collectionFactory, nil)
 	c.Assert(s.repo.packageRefs, NotNil)
 
-	pkg, err := s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Refs[0])
+	pkg, err := s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Flatten().Refs[0])
 	c.Assert(err, IsNil)
 
 	c.Check(pkg.Name, Equals, "amanda-client")
 
-	pkg, err = s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Refs[1])
+	pkg, err = s.collectionFactory.PackageCollection().ByKey(s.repo.packageRefs.Flatten().Refs[1])
 	c.Assert(err, IsNil)
 	c.Check(pkg.Name, Equals, "access-modifier-checker")
 
@@ -502,7 +502,7 @@ func (s *RemoteRepoSuite) TestDownloadFlat(c *C) {
 	s.flat.FinalizeDownload(s.collectionFactory, nil)
 	c.Assert(s.flat.packageRefs, NotNil)
 
-	pkg, err := s.collectionFactory.PackageCollection().ByKey(s.flat.packageRefs.Refs[0])
+	pkg, err := s.collectionFactory.PackageCollection().ByKey(s.flat.packageRefs.Flatten().Refs[0])
 	c.Assert(err, IsNil)
 
 	c.Check(pkg.Name, Equals, "amanda-client")
@@ -592,12 +592,12 @@ func (s *RemoteRepoSuite) TestDownloadWithSourcesFlat(c *C) {
 	s.flat.FinalizeDownload(s.collectionFactory, nil)
 	c.Assert(s.flat.packageRefs, NotNil)
 
-	pkg, err := s.collectionFactory.PackageCollection().ByKey(s.flat.packageRefs.Refs[0])
+	pkg, err := s.collectionFactory.PackageCollection().ByKey(s.flat.packageRefs.Flatten().Refs[0])
 	c.Assert(err, IsNil)
 
 	c.Check(pkg.Name, Equals, "amanda-client")
 
-	pkg, err = s.collectionFactory.PackageCollection().ByKey(s.flat.packageRefs.Refs[1])
+	pkg, err = s.collectionFactory.PackageCollection().ByKey(s.flat.packageRefs.Flatten().Refs[1])
 	c.Assert(err, IsNil)
 
 	c.Check(pkg.Name, Equals, "access-modifier-checker")
@@ -659,6 +659,7 @@ type RemoteRepoCollectionSuite struct {
 	PackageListMixinSuite
 	db                database.Storage
 	collection        *RemoteRepoCollection
+	refListCollection *RefListCollection
 }
 
 var _ = Suite(&RemoteRepoCollectionSuite{})
@@ -666,6 +667,7 @@ var _ = Suite(&RemoteRepoCollectionSuite{})
 func (s *RemoteRepoCollectionSuite) SetUpTest(c *C) {
 	s.db, _ = goleveldb.NewOpenDB(c.MkDir())
 	s.collection = NewRemoteRepoCollection(s.db)
+	s.refListCollection = NewRefListCollection(s.db)
 	s.SetUpPackages()
 }
 
@@ -678,8 +680,8 @@ func (s *RemoteRepoCollectionSuite) TestAddByName(c *C) {
 	c.Assert(err, ErrorMatches, "*.not found")
 
 	repo, _ := NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{}, false, false, false)
-	c.Assert(s.collection.Add(repo), IsNil)
+	c.Assert(s.collection.Add(repo, s.refListCollection), IsNil)
-	c.Assert(s.collection.Add(repo), ErrorMatches, ".*already exists")
+	c.Assert(s.collection.Add(repo, s.refListCollection), ErrorMatches, ".*already exists")
 
 	r, err := s.collection.ByName("yandex")
 	c.Assert(err, IsNil)
@@ -696,7 +698,7 @@ func (s *RemoteRepoCollectionSuite) TestByUUID(c *C) {
 	c.Assert(err, ErrorMatches, "*.not found")
 
 	repo, _ := NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{}, false, false, false)
-	c.Assert(s.collection.Add(repo), IsNil)
+	c.Assert(s.collection.Add(repo, s.refListCollection), IsNil)
 
 	r, err := s.collection.ByUUID(repo.UUID)
 	c.Assert(err, IsNil)
@@ -710,7 +712,7 @@ func (s *RemoteRepoCollectionSuite) TestByUUID(c *C) {
 
 func (s *RemoteRepoCollectionSuite) TestUpdateLoadComplete(c *C) {
 	repo, _ := NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{}, false, false, false)
-	c.Assert(s.collection.Update(repo), IsNil)
+	c.Assert(s.collection.Update(repo, s.refListCollection), IsNil)
 
 	collection := NewRemoteRepoCollection(s.db)
 	r, err := collection.ByName("yandex")
@@ -718,20 +720,20 @@ func (s *RemoteRepoCollectionSuite) TestUpdateLoadComplete(c *C) {
 	c.Assert(r.packageRefs, IsNil)
 
 	repo.packageRefs = s.reflist
-	c.Assert(s.collection.Update(repo), IsNil)
+	c.Assert(s.collection.Update(repo, s.refListCollection), IsNil)
 
 	collection = NewRemoteRepoCollection(s.db)
 	r, err = collection.ByName("yandex")
 	c.Assert(err, IsNil)
 	c.Assert(r.packageRefs, IsNil)
 	c.Assert(r.NumPackages(), Equals, 0)
-	c.Assert(s.collection.LoadComplete(r), IsNil)
+	c.Assert(s.collection.LoadComplete(r, s.refListCollection), IsNil)
 	c.Assert(r.NumPackages(), Equals, 3)
 }
 
 func (s *RemoteRepoCollectionSuite) TestForEachAndLen(c *C) {
 	repo, _ := NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{}, false, false, false)
-	s.collection.Add(repo)
+	s.collection.Add(repo, s.refListCollection)
 
 	count := 0
 	err := s.collection.ForEach(func(*RemoteRepo) error {
@@ -753,10 +755,10 @@ func (s *RemoteRepoCollectionSuite) TestForEachAndLen(c *C) {
 
 func (s *RemoteRepoCollectionSuite) TestDrop(c *C) {
 	repo1, _ := NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{}, false, false, false)
-	s.collection.Add(repo1)
+	s.collection.Add(repo1, s.refListCollection)
 
 	repo2, _ := NewRemoteRepo("tyndex", "http://mirror.yandex.ru/debian/", "wheezy", []string{"main"}, []string{}, false, false, false)
-	s.collection.Add(repo2)
+	s.collection.Add(repo2, s.refListCollection)
 
 	r1, _ := s.collection.ByUUID(repo1.UUID)
 	c.Check(r1, Equals, repo1)

deb/snapshot.go

@@ -11,7 +11,7 @@ import (
 
 	"github.com/aptly-dev/aptly/database"
 	"github.com/aptly-dev/aptly/utils"
-	"github.com/pborman/uuid"
+	"github.com/google/uuid"
 	"github.com/ugorji/go/codec"
 )
 
@@ -40,17 +40,17 @@ type Snapshot struct {
 	NotAutomatic         string
 	ButAutomaticUpgrades string
 
-	packageRefs *PackageRefList
+	packageRefs *SplitRefList
 }
 
 // NewSnapshotFromRepository creates snapshot from current state of repository
 func NewSnapshotFromRepository(name string, repo *RemoteRepo) (*Snapshot, error) {
-	if repo.packageRefs == nil || repo.packageRefs.Len() == 0 {
+	if repo.packageRefs == nil {
 		return nil, errors.New("mirror not updated")
 	}
 
 	return &Snapshot{
-		UUID:                 uuid.New(),
+		UUID:                 uuid.NewString(),
 		Name:                 name,
 		CreatedAt:            time.Now(),
 		SourceKind:           SourceRemoteRepo,
@@ -66,7 +66,7 @@ func NewSnapshotFromRepository(name string, repo *RemoteRepo) (*Snapshot, error)
 // NewSnapshotFromLocalRepo creates snapshot from current state of local repository
 func NewSnapshotFromLocalRepo(name string, repo *LocalRepo) (*Snapshot, error) {
 	snap := &Snapshot{
-		UUID:        uuid.New(),
+		UUID:        uuid.NewString(),
 		Name:        name,
 		CreatedAt:   time.Now(),
 		SourceKind:  SourceLocalRepo,
@@ -76,7 +76,7 @@ func NewSnapshotFromLocalRepo(name string, repo *LocalRepo) (*Snapshot, error) {
 	}
 
 	if snap.packageRefs == nil {
-		snap.packageRefs = NewPackageRefList()
+		snap.packageRefs = NewSplitRefList()
 	}
 
 	return snap, nil
@@ -84,18 +84,20 @@ func NewSnapshotFromLocalRepo(name string, repo *LocalRepo) (*Snapshot, error) {
 
 // NewSnapshotFromPackageList creates snapshot from PackageList
 func NewSnapshotFromPackageList(name string, sources []*Snapshot, list *PackageList, description string) *Snapshot {
-	return NewSnapshotFromRefList(name, sources, NewPackageRefListFromPackageList(list), description)
+	sl := NewSplitRefList()
+	sl.Replace(NewPackageRefListFromPackageList(list))
+	return NewSnapshotFromRefList(name, sources, sl, description)
 }
 
-// NewSnapshotFromRefList creates snapshot from PackageRefList
+// NewSnapshotFromRefList creates snapshot from SplitRefList
-func NewSnapshotFromRefList(name string, sources []*Snapshot, list *PackageRefList, description string) *Snapshot {
+func NewSnapshotFromRefList(name string, sources []*Snapshot, list *SplitRefList, description string) *Snapshot {
 	sourceUUIDs := make([]string, len(sources))
 	for i := range sources {
 		sourceUUIDs[i] = sources[i].UUID
 	}
 
 	return &Snapshot{
-		UUID:        uuid.New(),
+		UUID:        uuid.NewString(),
 		Name:        name,
 		CreatedAt:   time.Now(),
 		SourceKind:  "snapshot",
@@ -112,11 +114,14 @@ func (s *Snapshot) String() string {
 
 // NumPackages returns number of packages in snapshot
 func (s *Snapshot) NumPackages() int {
+	if s.packageRefs == nil {
+		return 0
+	}
 	return s.packageRefs.Len()
 }
 
 // RefList returns list of package refs in snapshot
-func (s *Snapshot) RefList() *PackageRefList {
+func (s *Snapshot) RefList() *SplitRefList {
 	return s.packageRefs
 }
 
@@ -209,13 +214,13 @@ func NewSnapshotCollection(db database.Storage) *SnapshotCollection {
 }
 
 // Add appends new repo to collection and saves it
-func (collection *SnapshotCollection) Add(snapshot *Snapshot) error {
+func (collection *SnapshotCollection) Add(snapshot *Snapshot, reflistCollection *RefListCollection) error {
 	_, err := collection.ByName(snapshot.Name)
 	if err == nil {
 		return fmt.Errorf("snapshot with name %s already exists", snapshot.Name)
 	}
 
-	err = collection.Update(snapshot)
+	err = collection.Update(snapshot, reflistCollection)
 	if err != nil {
 		return err
 	}
@@ -225,26 +230,22 @@ func (collection *SnapshotCollection) Add(snapshot *Snapshot) error {
 }
 
 // Update stores updated information about snapshot in DB
-func (collection *SnapshotCollection) Update(snapshot *Snapshot) error {
+func (collection *SnapshotCollection) Update(snapshot *Snapshot, reflistCollection *RefListCollection) error {
 	batch := collection.db.CreateBatch()
 
 	batch.Put(snapshot.Key(), snapshot.Encode())
 	if snapshot.packageRefs != nil {
-		batch.Put(snapshot.RefKey(), snapshot.packageRefs.Encode())
+		rb := reflistCollection.NewBatch(batch)
+		reflistCollection.UpdateInBatch(snapshot.packageRefs, snapshot.RefKey(), rb)
 	}
 
 	return batch.Write()
 }
 
 // LoadComplete loads additional information about snapshot
-func (collection *SnapshotCollection) LoadComplete(snapshot *Snapshot) error {
+func (collection *SnapshotCollection) LoadComplete(snapshot *Snapshot, reflistCollection *RefListCollection) error {
-	encoded, err := collection.db.Get(snapshot.RefKey())
+	snapshot.packageRefs = NewSplitRefList()
-	if err != nil {
+	return reflistCollection.LoadComplete(snapshot.packageRefs, snapshot.RefKey())
-		return err
-	}
-
-	snapshot.packageRefs = &PackageRefList{}
-	return snapshot.packageRefs.Decode(encoded)
 }
 
 func (collection *SnapshotCollection) search(filter func(*Snapshot) bool, unique bool) []*Snapshot {

deb/snapshot_bench_test.go

@@ -18,10 +18,11 @@ func BenchmarkSnapshotCollectionForEach(b *testing.B) {
 	defer db.Close()
 
 	collection := NewSnapshotCollection(db)
+	reflistCollection := NewRefListCollection(db)
 
 	for i := 0; i < count; i++ {
-		snapshot := NewSnapshotFromRefList(fmt.Sprintf("snapshot%d", i), nil, NewPackageRefList(), fmt.Sprintf("Snapshot number %d", i))
+		snapshot := NewSnapshotFromRefList(fmt.Sprintf("snapshot%d", i), nil, NewSplitRefList(), fmt.Sprintf("Snapshot number %d", i))
-		if collection.Add(snapshot) != nil {
+		if collection.Add(snapshot, reflistCollection) != nil {
 			b.FailNow()
 		}
 	}
@@ -47,11 +48,12 @@ func BenchmarkSnapshotCollectionByUUID(b *testing.B) {
 	defer db.Close()
 
 	collection := NewSnapshotCollection(db)
+	reflistCollection := NewRefListCollection(db)
 
 	uuids := []string{}
 	for i := 0; i < count; i++ {
-		snapshot := NewSnapshotFromRefList(fmt.Sprintf("snapshot%d", i), nil, NewPackageRefList(), fmt.Sprintf("Snapshot number %d", i))
+		snapshot := NewSnapshotFromRefList(fmt.Sprintf("snapshot%d", i), nil, NewSplitRefList(), fmt.Sprintf("Snapshot number %d", i))
-		if collection.Add(snapshot) != nil {
+		if collection.Add(snapshot, reflistCollection) != nil {
 			b.FailNow()
 		}
 		uuids = append(uuids, snapshot.UUID)
@@ -78,10 +80,11 @@ func BenchmarkSnapshotCollectionByName(b *testing.B) {
 	defer db.Close()
 
 	collection := NewSnapshotCollection(db)
+	reflistCollection := NewRefListCollection(db)
 
 	for i := 0; i < count; i++ {
-		snapshot := NewSnapshotFromRefList(fmt.Sprintf("snapshot%d", i), nil, NewPackageRefList(), fmt.Sprintf("Snapshot number %d", i))
+		snapshot := NewSnapshotFromRefList(fmt.Sprintf("snapshot%d", i), nil, NewSplitRefList(), fmt.Sprintf("Snapshot number %d", i))
-		if collection.Add(snapshot) != nil {
+		if collection.Add(snapshot, reflistCollection) != nil {
 			b.FailNow()
 		}
 	}

deb/snapshot_test.go

@@ -109,6 +109,7 @@ type SnapshotCollectionSuite struct {
 	snapshot1, snapshot2 *Snapshot
 	snapshot3, snapshot4 *Snapshot
 	collection           *SnapshotCollection
+	reflistCollection    *RefListCollection
 }
 
 var _ = Suite(&SnapshotCollectionSuite{})
@@ -116,6 +117,7 @@ var _ = Suite(&SnapshotCollectionSuite{})
 func (s *SnapshotCollectionSuite) SetUpTest(c *C) {
 	s.db, _ = goleveldb.NewOpenDB(c.MkDir())
 	s.collection = NewSnapshotCollection(s.db)
+	s.reflistCollection = NewRefListCollection(s.db)
 	s.SetUpPackages()
 
 	s.repo1, _ = NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{}, false, false, false)
@@ -143,10 +145,10 @@ func (s *SnapshotCollectionSuite) TestAddByNameByUUID(c *C) {
 	_, err := s.collection.ByName("snap1")
 	c.Assert(err, ErrorMatches, "*.not found")
 
-	c.Assert(s.collection.Add(s.snapshot1), IsNil)
+	c.Assert(s.collection.Add(s.snapshot1, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Add(s.snapshot1), ErrorMatches, ".*already exists")
+	c.Assert(s.collection.Add(s.snapshot1, s.reflistCollection), ErrorMatches, ".*already exists")
 
-	c.Assert(s.collection.Add(s.snapshot2), IsNil)
+	c.Assert(s.collection.Add(s.snapshot2, s.reflistCollection), IsNil)
 
 	snapshot, err := s.collection.ByName("snap1")
 	c.Assert(err, IsNil)
@@ -167,20 +169,20 @@ func (s *SnapshotCollectionSuite) TestAddByNameByUUID(c *C) {
 }
 
 func (s *SnapshotCollectionSuite) TestUpdateLoadComplete(c *C) {
-	c.Assert(s.collection.Update(s.snapshot1), IsNil)
+	c.Assert(s.collection.Update(s.snapshot1, s.reflistCollection), IsNil)
 
 	collection := NewSnapshotCollection(s.db)
 	snapshot, err := collection.ByName("snap1")
 	c.Assert(err, IsNil)
 	c.Assert(snapshot.packageRefs, IsNil)
 
-	c.Assert(s.collection.LoadComplete(snapshot), IsNil)
+	c.Assert(s.collection.LoadComplete(snapshot, s.reflistCollection), IsNil)
 	c.Assert(snapshot.NumPackages(), Equals, 3)
 }
 
 func (s *SnapshotCollectionSuite) TestForEachAndLen(c *C) {
-	s.collection.Add(s.snapshot1)
+	s.collection.Add(s.snapshot1, s.reflistCollection)
-	s.collection.Add(s.snapshot2)
+	s.collection.Add(s.snapshot2, s.reflistCollection)
 
 	count := 0
 	err := s.collection.ForEach(func(*Snapshot) error {
@@ -200,10 +202,10 @@ func (s *SnapshotCollectionSuite) TestForEachAndLen(c *C) {
 }
 
 func (s *SnapshotCollectionSuite) TestForEachSorted(c *C) {
-	s.collection.Add(s.snapshot2)
+	s.collection.Add(s.snapshot2, s.reflistCollection)
-	s.collection.Add(s.snapshot1)
+	s.collection.Add(s.snapshot1, s.reflistCollection)
-	s.collection.Add(s.snapshot4)
+	s.collection.Add(s.snapshot4, s.reflistCollection)
-	s.collection.Add(s.snapshot3)
+	s.collection.Add(s.snapshot3, s.reflistCollection)
 
 	names := []string{}
 
@@ -217,8 +219,8 @@ func (s *SnapshotCollectionSuite) TestForEachSorted(c *C) {
 }
 
 func (s *SnapshotCollectionSuite) TestFindByRemoteRepoSource(c *C) {
-	c.Assert(s.collection.Add(s.snapshot1), IsNil)
+	c.Assert(s.collection.Add(s.snapshot1, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Add(s.snapshot2), IsNil)
+	c.Assert(s.collection.Add(s.snapshot2, s.reflistCollection), IsNil)
 
 	c.Check(s.collection.ByRemoteRepoSource(s.repo1), DeepEquals, []*Snapshot{s.snapshot1})
 	c.Check(s.collection.ByRemoteRepoSource(s.repo2), DeepEquals, []*Snapshot{s.snapshot2})
@@ -229,10 +231,10 @@ func (s *SnapshotCollectionSuite) TestFindByRemoteRepoSource(c *C) {
 }
 
 func (s *SnapshotCollectionSuite) TestFindByLocalRepoSource(c *C) {
-	c.Assert(s.collection.Add(s.snapshot1), IsNil)
+	c.Assert(s.collection.Add(s.snapshot1, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Add(s.snapshot2), IsNil)
+	c.Assert(s.collection.Add(s.snapshot2, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Add(s.snapshot3), IsNil)
+	c.Assert(s.collection.Add(s.snapshot3, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Add(s.snapshot4), IsNil)
+	c.Assert(s.collection.Add(s.snapshot4, s.reflistCollection), IsNil)
 
 	c.Check(s.collection.ByLocalRepoSource(s.lrepo1), DeepEquals, []*Snapshot{s.snapshot3})
 	c.Check(s.collection.ByLocalRepoSource(s.lrepo2), DeepEquals, []*Snapshot{s.snapshot4})
@@ -247,11 +249,11 @@ func (s *SnapshotCollectionSuite) TestFindSnapshotSource(c *C) {
 	snapshot4 := NewSnapshotFromRefList("snap4", []*Snapshot{s.snapshot1}, s.reflist, "desc2")
 	snapshot5 := NewSnapshotFromRefList("snap5", []*Snapshot{snapshot3}, s.reflist, "desc3")
 
-	c.Assert(s.collection.Add(s.snapshot1), IsNil)
+	c.Assert(s.collection.Add(s.snapshot1, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Add(s.snapshot2), IsNil)
+	c.Assert(s.collection.Add(s.snapshot2, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Add(snapshot3), IsNil)
+	c.Assert(s.collection.Add(snapshot3, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Add(snapshot4), IsNil)
+	c.Assert(s.collection.Add(snapshot4, s.reflistCollection), IsNil)
-	c.Assert(s.collection.Add(snapshot5), IsNil)
+	c.Assert(s.collection.Add(snapshot5, s.reflistCollection), IsNil)
 
 	list := s.collection.BySnapshotSource(s.snapshot1)
 	sorter, _ := newSnapshotSorter("name", list)
@@ -263,8 +265,8 @@ func (s *SnapshotCollectionSuite) TestFindSnapshotSource(c *C) {
 }
 
 func (s *SnapshotCollectionSuite) TestDrop(c *C) {
-	s.collection.Add(s.snapshot1)
+	s.collection.Add(s.snapshot1, s.reflistCollection)
-	s.collection.Add(s.snapshot2)
+	s.collection.Add(s.snapshot2, s.reflistCollection)
 
 	snap, _ := s.collection.ByUUID(s.snapshot1.UUID)
 	c.Check(snap, Equals, s.snapshot1)

debian/aptly-api.postrm

@@ -0,0 +1,24 @@
+#!/bin/sh
+
+set -e
+
+# source debconf library
+. /usr/share/debconf/confmodule
+
+case "$1" in
+    purge|remove|upgrade|failed-upgrade|abort-install|abort-upgrade|disappear)
+	# only remove aptly-api user and its homedir on purge
+	if [ "${1}" = "purge" ] ; then
+	    userdel -r aptly-api
+	fi
+    ;;
+
+    *)
+        echo "postrm called with unknown argument \`$1'" >&2
+        exit 1
+    ;;
+esac
+
+#DEBHELPER#
+
+exit 0

debian/aptly.conf

@@ -228,7 +228,7 @@ s3_publish_endpoints:
     #     # Encryption Method (optional)
     #     # Server-side encryption method, defaults to none. Currently
     #     # the only available encryption method is `AES256`
-    #     encryption_method: none
+    #     encryption_method: ""
     #     # Plus Workaround (optional)
     #     # Workaround misbehavior in apt and Amazon S3 for files with `+` in filename by
     #     # creating two copies of package files with `+` in filename: one original

debian/changelog

@@ -1,3 +1,13 @@
+aptly (1.6.1) stable; urgency=medium
+
+  * update golang-github-syndtr-goleveldb-dev dependency (v1.0.1-0.20220721030215-126854af5e6d) to fix segfault on arm64
+    (bug in golang-github-golang-snappy-dev)
+  * allow snapshotting empty mirrors again (regression)
+  * debian compliance: add postrm (note: `apt purge aptly-api` will remove all data in ~aptly-api/)
+  * update other dependencies (x/net 0.33.0, gin-gonic/gin 1.9.1)
+
+ -- André Roth <neolynx@gmail.com>  Sat, 15 Feb 2025 13:03:16 +0100
+
 aptly (1.6.0) stable; urgency=medium
 
   * support reading filters from file or stdin

debian/control

@@ -45,7 +45,6 @@ Build-Depends: bash-completion,
                golang-github-munnerz-goautoneg-dev,
                golang-github-mxk-go-flowrate-dev,
                golang-github-ncw-swift-dev,
-               golang-github-pborman-uuid-dev,
                golang-github-pelletier-go-toml,
                golang-github-pkg-errors-dev,
                golang-github-prometheus-client-golang-dev,

files/package_pool.go

@@ -9,7 +9,7 @@ import (
 	"sync"
 	"syscall"
 
-	"github.com/pborman/uuid"
+	"github.com/google/uuid"
 	"github.com/saracen/walker"
 
 	"github.com/aptly-dev/aptly/aptly"
@@ -417,7 +417,7 @@ func (pool *PackagePool) FullPath(path string) string {
 
 // GenerateTempPath generates temporary path for download (which is fast to import into package pool later on)
 func (pool *PackagePool) GenerateTempPath(filename string) (string, error) {
-	random := uuid.NewRandom().String()
+	random := uuid.NewString()
 
 	return filepath.Join(pool.rootPath, random[0:2], random[2:4], random[4:]+filename), nil
 }

go.mod

@@ -8,8 +8,8 @@ require (
 	github.com/awalterschulze/gographviz v2.0.1+incompatible
 	github.com/cavaliergopher/grab/v3 v3.0.1
 	github.com/cheggaaa/pb v1.0.25
-	github.com/gin-gonic/gin v1.9.0
+	github.com/gin-gonic/gin v1.9.1
-	github.com/go-playground/validator/v10 v10.11.2 // indirect
+	github.com/go-playground/validator/v10 v10.14.0 // indirect
 	github.com/h2non/filetype v1.1.3
 	github.com/jlaffaye/ftp v0.2.0 // indirect
 	github.com/kjk/lzma v0.0.0-20120628231508-2a7c55cad4a2
@@ -21,7 +21,6 @@ require (
 	github.com/mkrautz/goar v0.0.0-20150919110319-282caa8bd9da
 	github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f
 	github.com/ncw/swift v1.0.53
-	github.com/pborman/uuid v1.2.0
 	github.com/pkg/errors v0.9.1
 	github.com/prometheus/client_golang v1.20.0
 	github.com/rs/zerolog v1.29.1
@@ -30,7 +29,7 @@ require (
 	github.com/smira/flag v0.0.0-20170926215700-695ea5e84e76
 	github.com/smira/go-ftp-protocol v0.0.0-20140829150050-066b75c2b70d
 	github.com/smira/go-xz v0.1.0
-	github.com/syndtr/goleveldb v1.0.1-0.20200815110645-5c35d600f0ca
+	github.com/syndtr/goleveldb v1.0.1-0.20220721030215-126854af5e6d
 	github.com/ugorji/go/codec v1.2.11
 	github.com/wsxiaoys/terminal v0.0.0-20160513160801-0940f3fc43a0
 	golang.org/x/crypto v0.31.0 // indirect
@@ -60,13 +59,14 @@ require (
 	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.5 // indirect
 	github.com/aws/aws-sdk-go-v2/service/sts v1.33.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
-	github.com/bytedance/sonic v1.8.0 // indirect
+	github.com/bytedance/sonic v1.9.1 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
 	github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 // indirect
 	github.com/cloudflare/circl v1.4.0 // indirect
 	github.com/coreos/go-semver v0.3.0 // indirect
 	github.com/coreos/go-systemd/v22 v22.5.0 // indirect
 	github.com/fatih/color v1.17.0 // indirect
+	github.com/gabriel-vasile/mimetype v1.4.2 // indirect
 	github.com/gin-contrib/sse v0.1.0 // indirect
 	github.com/go-openapi/jsonpointer v0.19.5 // indirect
 	github.com/go-openapi/jsonreference v0.19.6 // indirect
@@ -77,16 +77,15 @@ require (
 	github.com/goccy/go-json v0.10.2 // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/golang/protobuf v1.5.4 // indirect
-	github.com/golang/snappy v0.0.2 // indirect
+	github.com/golang/snappy v0.0.4 // indirect
-	github.com/google/uuid v1.6.0 // indirect
 	github.com/hashicorp/errwrap v1.1.0 // indirect
 	github.com/hashicorp/go-multierror v1.1.1 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
-	github.com/klauspost/cpuid/v2 v2.0.9 // indirect
+	github.com/klauspost/cpuid/v2 v2.2.4 // indirect
 	github.com/kr/pretty v0.3.1 // indirect
 	github.com/kr/text v0.2.0 // indirect
-	github.com/leodido/go-urn v1.2.1 // indirect
+	github.com/leodido/go-urn v1.2.4 // indirect
 	github.com/mailru/easyjson v0.7.6 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
@@ -103,8 +102,8 @@ require (
 	go.etcd.io/etcd/client/pkg/v3 v3.5.15 // indirect
 	go.uber.org/multierr v1.10.0 // indirect
 	go.uber.org/zap v1.26.0 // indirect
-	golang.org/x/arch v0.0.0-20210923205945-b76863e36670 // indirect
+	golang.org/x/arch v0.3.0 // indirect
-	golang.org/x/net v0.28.0 // indirect
+	golang.org/x/net v0.33.0 // indirect
 	golang.org/x/sync v0.10.0 // indirect
 	golang.org/x/text v0.21.0 // indirect
 	golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
@@ -124,6 +123,7 @@ require (
 	github.com/aws/aws-sdk-go-v2/credentials v1.17.46
 	github.com/aws/aws-sdk-go-v2/service/s3 v1.67.1
 	github.com/aws/smithy-go v1.22.1
+	github.com/google/uuid v1.6.0
 	github.com/swaggo/files v1.0.1
 	github.com/swaggo/gin-swagger v1.6.0
 	github.com/swaggo/swag v1.16.3

go.sum

@@ -64,8 +64,8 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/bwesterb/go-ristretto v1.2.3/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0=
 github.com/bytedance/sonic v1.5.0/go.mod h1:ED5hyg4y6t3/9Ku1R6dU/4KyJ48DZ4jPhfY1O2AihPM=
-github.com/bytedance/sonic v1.8.0 h1:ea0Xadu+sHlu7x5O3gKhRpQ1IKiMrSiHttPF0ybECuA=
+github.com/bytedance/sonic v1.9.1 h1:6iJ6NqdoxCDr6mbY8h18oSO+cShGSMRGCEo7F2h0x8s=
-github.com/bytedance/sonic v1.8.0/go.mod h1:i736AoUSYt75HyZLoJW9ERYxcy6eaN6h4BZXU064P/U=
+github.com/bytedance/sonic v1.9.1/go.mod h1:i736AoUSYt75HyZLoJW9ERYxcy6eaN6h4BZXU064P/U=
 github.com/cavaliergopher/grab/v3 v3.0.1 h1:4z7TkBfmPjmLAAmkkAZNX/6QJ1nNFdv3SdIHXju0Fr4=
 github.com/cavaliergopher/grab/v3 v3.0.1/go.mod h1:1U/KNnD+Ft6JJiYoYBAimKH2XrYptb8Kl3DFGmsjpq4=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
@@ -75,6 +75,9 @@ github.com/cheggaaa/pb v1.0.25/go.mod h1:pQciLPpbU0oxA0h+VJYYLxO+XeDQb5pZijXscXH
 github.com/chenzhuoyu/base64x v0.0.0-20211019084208-fb5309c8db06/go.mod h1:DH46F32mSOjUmXrMHnKwZdA8wcEefY7UVqBKYGjpdQY=
 github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 h1:qSGYFH7+jGhDF8vLC+iwCD4WpbV1EBDSzWkJODFLams=
 github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311/go.mod h1:b583jCggY9gE99b6G5LEC39OIiVsWj+R97kbl5odCEk=
+github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
+github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
+github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
 github.com/cloudflare/circl v1.3.3/go.mod h1:5XYMA4rFBvNIrhs50XuiBJ15vF2pZn4nnUKZrLbUZFA=
 github.com/cloudflare/circl v1.4.0 h1:BV7h5MgrktNzytKmWjpOtdYrf0lkkbF8YMlBGPhJQrY=
 github.com/cloudflare/circl v1.4.0/go.mod h1:PDRU+oXvdD7KCtgKxW95M5Z8BpSCJXQORiZFnBQS5QU=
@@ -89,14 +92,17 @@ github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
 github.com/fatih/color v1.17.0 h1:GlRw1BRJxkpqUCBKzKOw098ed57fEsKeNjpTe3cSjK4=
 github.com/fatih/color v1.17.0/go.mod h1:YZ7TlrGPkiz6ku9fK3TLD/pl3CpsiFyu8N92HLgmosI=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
-github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
 github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
+github.com/fsnotify/fsnotify v1.5.4 h1:jRbGcIw6P2Meqdwuo0H1p6JVLbL5DHKAKlYndzMwVZI=
+github.com/fsnotify/fsnotify v1.5.4/go.mod h1:OVB6XrOHzAwXMpEM7uPOzcehqUV2UqJxmVXmkdnm1bU=
+github.com/gabriel-vasile/mimetype v1.4.2 h1:w5qFW6JKBz9Y393Y4q372O9A7cUSequkh1Q7OhCmWKU=
+github.com/gabriel-vasile/mimetype v1.4.2/go.mod h1:zApsH/mKG4w07erKIaJPFiX0Tsq9BFQgN3qGY5GnNgA=
 github.com/gin-contrib/gzip v0.0.6 h1:NjcunTcGAj5CO1gn4N8jHOSIeRFHIbn51z6K+xaN4d4=
 github.com/gin-contrib/gzip v0.0.6/go.mod h1:QOJlmV2xmayAjkNS2Y8NQsMneuRShOU/kjovCXNuzzk=
 github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE=
 github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI=
-github.com/gin-gonic/gin v1.9.0 h1:OjyFBKICoexlu99ctXNR2gg+c5pKrKMuyjgARg9qeY8=
+github.com/gin-gonic/gin v1.9.1 h1:4idEAncQnU5cB7BeOkPtxjfCSye0AAm1R0RVIqJ+Jmg=
-github.com/gin-gonic/gin v1.9.0/go.mod h1:W1Me9+hsUSyj3CePGrd1/QrKJMSJ1Tu/0hFEH89961k=
+github.com/gin-gonic/gin v1.9.1/go.mod h1:hPrL7YrpYKXt5YId3A/Tnip5kqbEAP+KLuI3SUcPTeU=
 github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
 github.com/go-openapi/jsonpointer v0.19.5 h1:gZr+CIYByUqjcgeLXnQu2gHYQC9o73G2XUeOFYEICuY=
 github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
@@ -113,8 +119,9 @@ github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/o
 github.com/go-playground/locales v0.14.1/go.mod h1:hxrqLVvrK65+Rwrd5Fc6F2O76J/NuW9t0sjnWqG1slY=
 github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJnYK9S473LQFuzCbDbfSFY=
 github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
-github.com/go-playground/validator/v10 v10.11.2 h1:q3SHpufmypg+erIExEKUmsgmhDTyhcJ38oeKGACXohU=
+github.com/go-playground/validator/v10 v10.14.0 h1:vgvQWe3XCz3gIeFDm/HnTIbj6UGmg/+t63MyGU2n5js=
-github.com/go-playground/validator/v10 v10.11.2/go.mod h1:NieE624vt4SCTJtD87arVLvdmjPAeV8BQlHtMnw9D7s=
+github.com/go-playground/validator/v10 v10.14.0/go.mod h1:9iXMNT7sEkjXb0I+enO7QXmzG6QCsPWY4zveKFVRSyU=
+github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0/go.mod h1:fyg7847qk6SyHyPtNmDHnmrv/HOrqktSC+C9fM+CJOE=
 github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
 github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
 github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
@@ -129,18 +136,20 @@ github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrU
 github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
 github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
 github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
+github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
+github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
 github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
 github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
-github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
+github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=
-github.com/golang/snappy v0.0.2 h1:aeE13tS0IiQgFjYdoL8qN3K1N2bXXtI6Vi51/y7BpMw=
+github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
-github.com/golang/snappy v0.0.2/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
 github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
 github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
 github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
 github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
-github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/h2non/filetype v1.1.3 h1:FKkx9QbD7HR/zjK1Ia5XiBsq9zdLi5Kf3zGyFTAFkGg=
@@ -151,6 +160,7 @@ github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brv
 github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
 github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
 github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
+github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
 github.com/jlaffaye/ftp v0.2.0 h1:lXNvW7cBu7R/68bknOX3MrRIIqZ61zELs1P2RAiA3lg=
 github.com/jlaffaye/ftp v0.2.0/go.mod h1:is2Ds5qkhceAPy2xD6RLI6hmp/qysSoymZ+Z2uTnspI=
 github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
@@ -163,8 +173,9 @@ github.com/kjk/lzma v0.0.0-20120628231508-2a7c55cad4a2 h1:TVZQgMi+I83S3rCuE65Hnm
 github.com/kjk/lzma v0.0.0-20120628231508-2a7c55cad4a2/go.mod h1:phT/jsRPBAEqjAibu1BurrabCBNTYiVI+zbmyCZJY6Q=
 github.com/klauspost/compress v1.17.9 h1:6KIumPrER1LHsvBVuDa0r5xaG0Es51mhhB9BQB2qeMA=
 github.com/klauspost/compress v1.17.9/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=
-github.com/klauspost/cpuid/v2 v2.0.9 h1:lgaqFMSdTdQYdZ04uHyN2d/eKdOMyi2YLSvlQIBFYa4=
 github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
+github.com/klauspost/cpuid/v2 v2.2.4 h1:acbojRNwl3o09bUq+yDCtZFc1aiwaAAxtcn8YkZXnvk=
+github.com/klauspost/cpuid/v2 v2.2.4/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY=
 github.com/klauspost/pgzip v1.2.5 h1:qnWYvvKqedOF2ulHpMG72XQol4ILEJ8k2wwRl/Km8oE=
 github.com/klauspost/pgzip v1.2.5/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
@@ -177,8 +188,8 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
 github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
 github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
-github.com/leodido/go-urn v1.2.1 h1:BqpAaACuzVSgi/VLzGZIobT2z4v53pjosyNd9Yv6n/w=
+github.com/leodido/go-urn v1.2.4 h1:XlAE/cm/ms7TE/VMVoduSpNBoyc2dOxHs5MZSwAN63Q=
-github.com/leodido/go-urn v1.2.1/go.mod h1:zt4jvISO2HfUBqxjfIshjdMTYS56ZS/qv49ictyFfxY=
+github.com/leodido/go-urn v1.2.4/go.mod h1:7ZrI8mTSeBSHl/UaRyKQW1qZeMgak41ANeCNaVckg+4=
 github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
 github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
 github.com/mailru/easyjson v0.7.6 h1:8yTIVnZgCoiM1TgqoeTl+LfU5Jg6/xL3QhGQnimLYnA=
@@ -208,17 +219,20 @@ github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+
 github.com/ncw/swift v1.0.53 h1:luHjjTNtekIEvHg5KdAFIBaH7bWfNkefwFnpDffSIks=
 github.com/ncw/swift v1.0.53/go.mod h1:23YIA4yWVnGwv2dQlN4bB7egfYX6YLn0Yo/S6zZO/ZM=
 github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
-github.com/nxadm/tail v1.4.4 h1:DQuhQpB1tVlglWS2hLQ5OV6B5r8aGxSrPc5Qo6uTN78=
 github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
+github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
+github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU=
 github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
-github.com/onsi/ginkgo v1.14.0 h1:2mOpI4JVVPBN+WQRa0WKH2eXR+Ey+uK4n7Zj0aYpIQA=
+github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0=
-github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
+github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
+github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU=
+github.com/onsi/ginkgo/v2 v2.1.3/go.mod h1:vw5CSIxN1JObi/U8gcbwft7ZxR2dgaR70JSE3/PpL4c=
 github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
-github.com/onsi/gomega v1.10.1 h1:o0+MgICZLuZ7xjH7Vx6zS/zcu93/BEp1VwkIW1mEXCE=
 github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
-github.com/pborman/uuid v1.2.0 h1:J7Q5mO4ysT1dv8hyrUGHb9+ooztCXu1D8MY8DZYsu3g=
+github.com/onsi/gomega v1.17.0/go.mod h1:HnhC7FXeEQY45zxNK3PPoIUhzk/80Xly9PcubAlGdZY=
-github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
+github.com/onsi/gomega v1.19.0 h1:4ieX6qQjPP/BfC3mpsAtIGGlxTWPeA3Inl/7DtXw1tw=
+github.com/onsi/gomega v1.19.0/go.mod h1:LY+I3pBVzYsTBU1AnDwOSxaYi9WoWiqgwooUqq9yPro=
 github.com/pelletier/go-toml/v2 v2.1.0 h1:FnwAJ4oYMvbT/34k9zzHuZNrhlz48GB3/s6at6/MHO4=
 github.com/pelletier/go-toml/v2 v2.1.0/go.mod h1:tJU2Z3ZkXwnxa4DPO899bsyIoywizdUvyaeZurnPPDc=
 github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
@@ -259,11 +273,14 @@ github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+
 github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
 github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
 github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals=
 github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
 github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
+github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
 github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
 github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
 github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
@@ -273,8 +290,8 @@ github.com/swaggo/gin-swagger v1.6.0 h1:y8sxvQ3E20/RCyrXeFfg60r6H0Z+SwpTjMYsMm+z
 github.com/swaggo/gin-swagger v1.6.0/go.mod h1:BG00cCEy294xtVpyIAHG6+e2Qzj/xKlRdOqDkvq0uzo=
 github.com/swaggo/swag v1.16.3 h1:PnCYjPCah8FK4I26l2F/KQ4yz3sILcVUN3cTlBFA9Pg=
 github.com/swaggo/swag v1.16.3/go.mod h1:DImHIuOFXKpMFAQjcC7FG4m3Dg4+QuUgUzJmKjI/gRk=
-github.com/syndtr/goleveldb v1.0.1-0.20200815110645-5c35d600f0ca h1:Ld/zXl5t4+D69SiV4JoN7kkfvJdOWlPpfxrzxpLMoUk=
+github.com/syndtr/goleveldb v1.0.1-0.20220721030215-126854af5e6d h1:vfofYNRScrDdvS342BElfbETmL1Aiz3i2t0zfRj16Hs=
-github.com/syndtr/goleveldb v1.0.1-0.20200815110645-5c35d600f0ca/go.mod h1:u2MKkTVTVJWe5D1rCvame8WqhBd88EuIwODJZ1VHCPM=
+github.com/syndtr/goleveldb v1.0.1-0.20220721030215-126854af5e6d/go.mod h1:RRCYJbIwD5jmqPI9XoAFR0OcDxqUctll6zUj/+B4S48=
 github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS4MhqMhdFk5YI=
 github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08=
 github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4dU=
@@ -296,8 +313,9 @@ go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
 go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
 go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
 go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
-golang.org/x/arch v0.0.0-20210923205945-b76863e36670 h1:18EFjUmQOcUvxNYSkA6jO9VAiXCnxFY6NyDX0bHDmkU=
 golang.org/x/arch v0.0.0-20210923205945-b76863e36670/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8=
+golang.org/x/arch v0.3.0 h1:02VY4/ZcO/gBOH6PUaoiptASxtXU10jazRCP865E97k=
+golang.org/x/arch v0.3.0/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
@@ -317,17 +335,19 @@ golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
-golang.org/x/net v0.0.0-20200813134508-3edf25e44fcc/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20210421230115-4e50805a0758/go.mod h1:72T/g9IO56b78aLF+1Kcs5dz7/ng1VjMUvfKvpfy+jM=
+golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk=
+golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
+golang.org/x/net v0.0.0-20220607020251-c690dde0001d/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY=
 golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
 golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
 golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
-golang.org/x/net v0.28.0 h1:a9JDOJc5GMUJ0+UDqmLT86WiEy7iWyIhz8gz8E4e5hE=
+golang.org/x/net v0.33.0 h1:74SYHlV8BIgHIFC/LrYkOGIwL19eTYXQ5wc6TBuO36I=
-golang.org/x/net v0.28.0/go.mod h1:yqtgsTWOOnlGLG9GFRrK3++bGOUEkNBoHZc8MEDWPNg=
+golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -343,16 +363,20 @@ golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200814200057-3d37ad5750ed/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210420072515-93ed5bcd2bfe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -369,7 +393,6 @@ golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
 golang.org/x/term v0.27.0 h1:WP60Sv1nlK1T6SupCHbXzSaN0b9wUmsPoRS9b61A23Q=
 golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
-golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
@@ -383,6 +406,7 @@ golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
+golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
 golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
 golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
 golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
@@ -391,8 +415,8 @@ golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxb
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20220517211312-f3a8303e98df/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8=
 google.golang.org/genproto/googleapis/api v0.0.0-20240318140521-94a12d6c2237 h1:RFiFrvy37/mpSpdySBDrUdipW/dHwsRwh3J3+A9VgT4=
 google.golang.org/genproto/googleapis/api v0.0.0-20240318140521-94a12d6c2237/go.mod h1:Z5Iiy3jtmioajWHDGFk7CeugTyHtPvMHA4UTmUkyalE=
 google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237 h1:NnYq6UN9ReLM9/Y01KWNOWyI5xQ9kbIms5GGJVwS/Yc=
@@ -405,6 +429,8 @@ google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQ
 google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
 google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
 google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
+google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
+google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
 google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
 google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

system/t02_config/CreateConfigTest_gold

@@ -228,7 +228,7 @@ s3_publish_endpoints:
     #     # Encryption Method (optional)
     #     # Server-side encryption method, defaults to none. Currently
     #     # the only available encryption method is `AES256`
-    #     encryption_method: none
+    #     encryption_method: ""
     #     # Plus Workaround (optional)
     #     # Workaround misbehavior in apt and Amazon S3 for files with `+` in filename by
     #     # creating two copies of package files with `+` in filename: one original

system/t05_snapshot/CreateSnapshot10Test_gold

@@ -0,0 +1,3 @@
+
+Snapshot oh-snap successfully created.
+You can run 'aptly publish snapshot oh-snap' to publish snapshot as Debian repository.

system/t05_snapshot/create.py

@@ -106,3 +106,14 @@ class CreateSnapshot9Test(BaseTest):
         "aptly repo create local-repo",
     ]
     runCmd = "aptly snapshot create snap9 from repo local-repo"
+
+
+class CreateSnapshot10Test(BaseTest):
+    """
+    create snapshot: from empty mirror
+    """
+    fixtureCmds = [
+        "aptly mirror create -ignore-signatures non-free http://repo.aptly.info/system-tests/security.debian.org/debian-security bookworm-security updates/non-free",
+        "aptly mirror update -ignore-signatures non-free"
+    ]
+    runCmd = "aptly snapshot create oh-snap from mirror non-free"

system/t08_db/CleanupDB10Test_gold

@@ -1,6 +1,7 @@
 Loading mirrors, local repos, snapshots and published repos...
 Loading list of all packages...
 Deleting unreferenced packages (0)...
+Deleting unreferenced reflist buckets (0)...
 Building list of files referenced by packages...
 Building list of files in package pool...
 Deleting unreferenced files (0)...

system/t08_db/CleanupDB11Test_gold

@@ -14,6 +14,7 @@ Loading mirrors:
 Loading local repos:
 Loading snapshots:
 Loading published repositories:
+Split 11 reflist(s) into 510 bucket(s) (123181 segment(s))
 Loading list of all packages...
 Deleting unreferenced packages (7)...
 List of package keys to delete:
@@ -24,6 +25,7 @@ List of package keys to delete:
  - Pi386 gnuplot-nox 4.6.1-1~maverick2 17785995cf0f815
  - Pi386 gnuplot-x11 4.6.1-1~maverick2 d42e1d0d2f23740
  - Psource gnuplot 4.6.1-1~maverick2 b8cd36358f5db41f
+Deleting unreferenced reflist buckets (0)...
 Building list of files referenced by packages...
 Building list of files in package pool...
 Deleting unreferenced files (9)...

system/t08_db/CleanupDB12Test_gold

@@ -14,6 +14,7 @@ Loading mirrors:
 Loading local repos:
 Loading snapshots:
 Loading published repositories:
+Skipped splitting 11 reflist(s) into 510 bucket(s) (123181 segment(s)), as -dry-run has been requested.
 Loading list of all packages...
 Deleting unreferenced packages (7)...
 List of package keys to delete:
@@ -25,6 +26,7 @@ List of package keys to delete:
  - Pi386 gnuplot-x11 4.6.1-1~maverick2 d42e1d0d2f23740
  - Psource gnuplot 4.6.1-1~maverick2 b8cd36358f5db41f
 Skipped deletion, as -dry-run has been requested.
+Deleting unreferenced reflist buckets (0)...
 Building list of files referenced by packages...
 Building list of files in package pool...
 Deleting unreferenced files (9)...

system/t08_db/CleanupDB1Test_gold

@@ -1,6 +1,7 @@
 Loading mirrors, local repos, snapshots and published repos...
 Loading list of all packages...
 Deleting unreferenced packages (0)...
+Deleting unreferenced reflist buckets (0)...
 Building list of files referenced by packages...
 Building list of files in package pool...
 Deleting unreferenced files (0)...

system/t08_db/CleanupDB2Test_gold

@@ -1,6 +1,7 @@
 Loading mirrors, local repos, snapshots and published repos...
 Loading list of all packages...
 Deleting unreferenced packages (73270)...
+Deleting unreferenced reflist buckets (0)...
 Building list of files referenced by packages...
 Building list of files in package pool...
 Deleting unreferenced files (0)...

system/t08_db/CleanupDB3Test_gold

@@ -1,6 +1,7 @@
 Loading mirrors, local repos, snapshots and published repos...
 Loading list of all packages...
 Deleting unreferenced packages (7)...
+Deleting unreferenced reflist buckets (0)...
 Building list of files referenced by packages...
 Building list of files in package pool...
 Deleting unreferenced files (9)...

system/t08_db/CleanupDB4Test_gold

@@ -1,6 +1,7 @@
 Loading mirrors, local repos, snapshots and published repos...
 Loading list of all packages...
 Deleting unreferenced packages (0)...
+Deleting unreferenced reflist buckets (0)...
 Building list of files referenced by packages...
 Building list of files in package pool...
 Deleting unreferenced files (0)...

system/t08_db/CleanupDB5Test_gold

@@ -1,6 +1,7 @@
 Loading mirrors, local repos, snapshots and published repos...
 Loading list of all packages...
 Deleting unreferenced packages (7)...
+Deleting unreferenced reflist buckets (1)...
 Building list of files referenced by packages...
 Building list of files in package pool...
 Deleting unreferenced files (9)...

system/t08_db/CleanupDB6Test_gold

@@ -1,6 +1,7 @@
 Loading mirrors, local repos, snapshots and published repos...
 Loading list of all packages...
 Deleting unreferenced packages (0)...
+Deleting unreferenced reflist buckets (0)...
 Building list of files referenced by packages...
 Building list of files in package pool...
 Deleting unreferenced files (0)...

system/t08_db/CleanupDB7Test_gold

@@ -1,6 +1,7 @@
 Loading mirrors, local repos, snapshots and published repos...
 Loading list of all packages...
 Deleting unreferenced packages (0)...
+Deleting unreferenced reflist buckets (0)...
 Building list of files referenced by packages...
 Building list of files in package pool...
 Deleting unreferenced files (0)...

system/t08_db/CleanupDB8Test_gold

@@ -1,6 +1,7 @@
 Loading mirrors, local repos, snapshots and published repos...
 Loading list of all packages...
 Deleting unreferenced packages (4)...
+Deleting unreferenced reflist buckets (1)...
 Building list of files referenced by packages...
 Building list of files in package pool...
 Deleting unreferenced files (6)...

system/t08_db/CleanupDB9Test_gold

@@ -1,6 +1,7 @@
 Loading mirrors, local repos, snapshots and published repos...
 Loading list of all packages...
 Deleting unreferenced packages (0)...
+Deleting unreferenced reflist buckets (0)...
 Building list of files referenced by packages...
 Building list of files in package pool...
 Deleting unreferenced files (0)...