handle packageRefs == nil

In current aptly, each repository and snapshot has its own reflist in
the database. This brings a few problems with it:

- Given a sufficiently large repositories and snapshots, these lists can
  get enormous, reaching >1MB. This is a problem for LevelDB's overall
  performance, as it tends to prefer values around the confiruged block
  size (defaults to just 4KiB).
- When you take these large repositories and snapshot them, you have a
  full, new copy of the reflist, even if only a few packages changed.
  This means that having a lot of snapshots with a few changes causes
  the database to basically be full of largely duplicate reflists.
- All the duplication also means that many of the same refs are being
  loaded repeatedly, which can cause some slowdown but, more notably,
  eats up huge amounts of memory.
- Adding on more and more new repositories and snapshots will cause the
  time and memory spent on things like cleanup and publishing to grow
  roughly linearly.

At the core, there are two problems here:

- Reflists get very big because there are just a lot of packages.
- Different reflists can tend to duplicate much of the same contents.

*Split reflists* aim at solving this by separating reflists into 64
*buckets*. Package refs are sorted into individual buckets according to
the following system:

- Take the first 3 letters of the package name, after dropping a `lib`
  prefix. (Using only the first 3 letters will cause packages with
  similar prefixes to end up in the same bucket, under the assumption
  that packages with similar names tend to be updated together.)
- Take the 64-bit xxhash of these letters. (xxhash was chosen because it
  relatively good distribution across the individual bits, which is
  important for the next step.)
- Use the first 6 bits of the hash (range [0:63]) as an index into the
  buckets.

Once refs are placed in buckets, a sha256 digest of all the refs in the
bucket is taken. These buckets are then stored in the database, split
into roughly block-sized segments, and all the repositories and
snapshots simply store an array of bucket digests.

This approach means that *repositories and snapshots can share their
reflist buckets*. If a snapshot is taken of a repository, it will have
the same contents, so its split reflist will point to the same buckets
as the base repository, and only one copy of each bucket is stored in
the database. When some packages in the repository change, only the
buckets containing those packages will be modified; all the other
buckets will remain unchanged, and thus their contents will still be
shared. Later on, when these reflists are loaded, each bucket is only
loaded once, short-cutting loaded many megabytes of data. In effect,
split reflists are essentially copy-on-write, with only the changed
buckets stored individually.

Changing the disk format means that a migration needs to take place, so
that task is moved into the database cleanup step, which will migrate
reflists over to split reflists, as well as delete any unused reflist
buckets.

All the reflist tests are also changed to additionally test out split
reflists; although the internal logic is all shared (since buckets are,
themselves, just normal reflists), some special additions are needed to
have native versions of the various reflist helper methods.

In our tests, we've observed the following improvements:

- Memory usage during publish and database cleanup, with
  `GOMEMLIMIT=2GiB`, goes down from ~3.2GiB (larger than the memory
  limit!) to ~0.7GiB, a decrease of ~4.5x.
- Database size decreases from 1.3GB to 367MB.

*In my local tests*, publish times had also decreased down to mere
seconds but the same effect wasn't observed on the server, with the
times staying around the same. My suspicions are that this is due to I/O
performance: my local system is an M1 MBP, which almost certainly has
much faster disk speeds than our DigitalOcean block volumes. Split
reflists include a side effect of requiring more random accesses from
reading all the buckets by their keys, so if your random I/O
performance is slower, it might cancel out the benefits. That being
said, even in that case, the memory usage and database size advantages
still persist.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

.github/workflows/ci.yml

@@ -30,10 +30,10 @@ jobs:
       GOPROXY: "https://proxy.golang.org"
 
     steps:
-      - name: "Install Test Packages"
+      - name: "Install Packages"
         run: |
           sudo apt-get update
-          sudo apt-get install -y --no-install-recommends graphviz gnupg2 gpgv2 git gcc make devscripts python3 python3-requests-unixsocket python3-termcolor python3-swiftclient python3-boto python3-azure-storage python3-etcd3 python3-plyvel flake8 faketime
+          sudo apt-get install -y --no-install-recommends graphviz gnupg2 gpgv2 git gcc make devscripts python3 python3-requests-unixsocket python3-termcolor python3-swiftclient python3-boto python3-azure-storage python3-etcd3 python3-plyvel flake8
 
       - name: "Checkout Repository"
         uses: actions/checkout@v4
@@ -93,7 +93,7 @@ jobs:
 
       - name: "Merge Code Coverage"
         run: |
-          go install github.com/wadey/gocovmerge@v0.0.0-20160331181800-b5bfa59ec0ad
+          go install github.com/wadey/gocovmerge@latest
           ~/go/bin/gocovmerge unit.out ${{ runner.temp }}/*.out > coverage.txt
 
       - name: "Upload Code Coverage"
@@ -139,7 +139,7 @@ jobs:
         APT_LISTCHANGES_FRONTEND: none
         DEBIAN_FRONTEND: noninteractive
     steps:
-      - name: "Install Build Packages"
+      - name: "Install packages"
         run: |
           apt-get update
           apt-get install -y --no-install-recommends make ca-certificates git curl build-essential devscripts dh-golang jq bash-completion lintian \
@@ -207,23 +207,6 @@ jobs:
         run: |
           .github/workflows/scripts/upload-artifacts.sh release ${{ matrix.suite }}
 
-      - name: "Get aptly version"
-        env:
-          FORCE_CI: ${{ steps.force_ci.outputs.FORCE_CI }}
-        run: |
-           aptlyver=$(make -s version)
-           echo "Aptly Version: $aptlyver"
-           echo "VERSION=$aptlyver" >> $GITHUB_OUTPUT
-        id: releaseversion
-
-      - name: "Upload CI Artifacts"
-        if: github.ref != 'refs/heads/master' && !startsWith(github.event.ref, 'refs/tags')
-        uses: actions/upload-artifact@v4
-        with:
-          name: aptly_${{ steps.releaseversion.outputs.VERSION }}_${{ matrix.suite }}_${{ matrix.arch }}
-          path: build/
-          retention-days: 7
-
   ci-binary-build:
     name: "Build"
     needs: test
@@ -284,15 +267,6 @@ jobs:
           path: build/aptly_${{ steps.releaseversion.outputs.VERSION }}_${{ matrix.goos }}_${{ matrix.goarch }}.zip
           compression-level: 0 # no compression
 
-      - name: "Upload CI Artifacts"
-        uses: actions/upload-artifact@v4
-        if: "!startsWith(github.event.ref, 'refs/tags')"
-        with:
-          name: aptly_${{ steps.releaseversion.outputs.VERSION }}_${{ matrix.goos }}_${{ matrix.goarch }}
-          path: build/aptly_${{ steps.releaseversion.outputs.VERSION }}_${{ matrix.goos }}_${{ matrix.goarch }}.zip
-          compression-level: 0 # no compression
-          retention-days: 7
-
   gh-release:
     name: "Github Release"
     runs-on: ubuntu-latest

.github/workflows/golangci-lint.yml

@@ -44,7 +44,7 @@ jobs:
           # Require: The version of golangci-lint to use.
           # When `install-mode` is `binary` (default) the value can be v1.2 or v1.2.3 or `latest` to use the latest version.
           # When `install-mode` is `goinstall` the value can be v1.2.3, `latest`, or the hash of a commit.
-          version: v1.64.5
+          version: v1.54.1
 
           # Optional: working directory, useful for monorepos
           # working-directory: somedir

.golangci.yml

@@ -1,11 +1,16 @@
-version: "2"
+run:
+  tests: false
+
+
 linters:
-  settings:
-    staticcheck:
-      checks:
-      - "all"
-      - "-QF1004"  # could use strings.ReplaceAll instead
-      - "-QF1012"  # Use fmt.Fprintf(...) instead of WriteString(fmt.Sprintf(...))
-      - "-QF1003"  # could use tagged switch
-      - "-ST1000"  # at least one file in a package should have a package comment
-      - "-QF1001"  # could apply De Morgan's law
+  disable-all: true
+  enable:
+   - goconst
+   - gofmt
+   - goimports
+   - govet
+   - ineffassign
+   - misspell
+   - revive
+   - staticcheck
+   - vetshadow

AUTHORS

@@ -68,4 +68,3 @@ List of contributors, in chronological order:
 * Blake Kostner (https://github.com/btkostner)
 * Leigh London (https://github.com/leighlondon)
 * Gordian Schoenherr (https://github.com/schoenherrg)
-* Silke Hofstra (https://github.com/silkeh)

Makefile

@@ -2,18 +2,13 @@ GOPATH=$(shell go env GOPATH)
 VERSION=$(shell make -s version)
 PYTHON?=python3
 BINPATH?=$(GOPATH)/bin
-GOLANGCI_LINT_VERSION=v2.0.2  # version supporting go 1.24
+GOLANGCI_LINT_VERSION=v1.54.1  # version supporting go 1.19
 COVERAGE_DIR?=$(shell mktemp -d)
 GOOS=$(shell go env GOHOSTOS)
 GOARCH=$(shell go env GOHOSTARCH)
 
-# Unit Tests and some sysmte tests rely on expired certificates, turn back the time
-export TEST_FAKETIME := 2025-01-02 03:04:05
-
-# export CAPUTRE=1 for regenrating test gold files
-ifeq ($(CAPTURE),1)
-CAPTURE_ARG := --capture
-endif
+# Uncomment to update system test gold files
+# CAPTURE := "--capture"
 
 help:  ## Print this help
 	@grep -E '^[a-zA-Z][a-zA-Z0-9_-]*:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
@@ -55,7 +50,7 @@ swagger-install:
 	echo "// @version $(VERSION)" >> docs/swagger.conf
 
 azurite-start:
-	azurite -l /tmp/aptly-azurite > ~/.azurite.log 2>&1 & \
+	azurite -l /tmp/aptly-azurite & \
 	echo $$! > ~/.azurite.pid
 
 azurite-stop:
@@ -74,9 +69,9 @@ flake8:  ## run flake8 on system test python files
 
 lint: prepare
 	# Install golangci-lint
-	@test -f $(BINPATH)/golangci-lint || go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@$(GOLANGCI_LINT_VERSION)
+	@test -f $(BINPATH)/golangci-lint || go install github.com/golangci/golangci-lint/cmd/golangci-lint@$(GOLANGCI_LINT_VERSION)
 	# Running lint
-	@NO_COLOR=true PATH=$(BINPATH)/:$(PATH) golangci-lint run --max-issues-per-linter=0 --max-same-issues=0
+	@PATH=$(BINPATH)/:$(PATH) golangci-lint run
 
 
 build: prepare swagger  ## Build aptly
@@ -89,11 +84,11 @@ install:
 	# go install -v
 	@out=`mktemp`; if ! go install -v > $$out 2>&1; then cat $$out; rm -f $$out; echo "\nBuild failed\n"; exit 1; else rm -f $$out; fi
 
-test: prepare swagger etcd-install  ## Run unit tests (add TEST=regex to specify which tests to run)
+test: prepare swagger etcd-install  ## Run unit tests
 	@echo "\e[33m\e[1mStarting etcd ...\e[0m"
 	@mkdir -p /tmp/aptly-etcd-data; system/t13_etcd/start-etcd.sh > /tmp/aptly-etcd-data/etcd.log 2>&1 &
 	@echo "\e[33m\e[1mRunning go test ...\e[0m"
-	faketime "$(TEST_FAKETIME)" go test -v ./... -gocheck.v=true -check.f "$(TEST)" -coverprofile=unit.out; echo $$? > .unit-test.ret
+	go test -v ./... -gocheck.v=true -coverprofile=unit.out; echo $$? > .unit-test.ret
 	@echo "\e[33m\e[1mStopping etcd ...\e[0m"
 	@pid=`cat /tmp/etcd.pid`; kill $$pid
 	@rm -f /tmp/aptly-etcd-data/etcd.log
@@ -107,7 +102,7 @@ system-test: prepare swagger etcd-install  ## Run system tests
 	if [ ! -e ~/aptly-fixture-pool ]; then git clone https://github.com/aptly-dev/aptly-fixture-pool.git ~/aptly-fixture-pool/; fi
 	test -f ~/etcd.db || (curl -o ~/etcd.db.xz http://repo.aptly.info/system-tests/etcd.db.xz && xz -d ~/etcd.db.xz)
 	# Run system tests
-	PATH=$(BINPATH)/:$(PATH) FORCE_COLOR=1 $(PYTHON) system/run.py --long --coverage-dir $(COVERAGE_DIR) $(CAPTURE_ARG) $(TEST)
+	PATH=$(BINPATH)/:$(PATH) && FORCE_COLOR=1 $(PYTHON) system/run.py --long --coverage-dir $(COVERAGE_DIR) $(CAPTURE) $(TEST)
 
 bench:
 	@echo "\e[33m\e[1mRunning benchmark ...\e[0m"
@@ -167,9 +162,6 @@ binaries: prepare swagger  ## Build binary releases (FreeBSD, macOS, Linux gener
 docker-image:  ## Build aptly-dev docker image
 	@docker build -f system/Dockerfile . -t aptly-dev
 
-docker-image-no-cache:  ## Build aptly-dev docker image (no cache)
-	@docker build --no-cache -f system/Dockerfile . -t aptly-dev
-
 docker-build:  ## Build aptly in docker container
 	@docker run -it --rm -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper build
 
@@ -179,13 +171,13 @@ docker-shell:  ## Run aptly and other commands in docker container
 docker-deb:  ## Build debian packages in docker container
 	@docker run -it --rm -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper dpkg DEBARCH=amd64
 
-docker-unit-test:  ## Run unit tests in docker container (add TEST=regex to specify which tests to run)
+docker-unit-test:  ## Run unit tests in docker container
 	@docker run -it --rm -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper \
 		azurite-start \
 		AZURE_STORAGE_ENDPOINT=http://127.0.0.1:10000/devstoreaccount1 \
 		AZURE_STORAGE_ACCOUNT=devstoreaccount1 \
 		AZURE_STORAGE_ACCESS_KEY="Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==" \
-		test TEST=$(TEST) \
+		test \
 		azurite-stop
 
 docker-system-test:  ## Run system tests in docker container (add TEST=t04_mirror or TEST=UpdateMirror26Test to run only specific tests)
@@ -194,8 +186,6 @@ docker-system-test:  ## Run system tests in docker container (add TEST=t04_mirro
 		AZURE_STORAGE_ENDPOINT=http://127.0.0.1:10000/devstoreaccount1 \
 		AZURE_STORAGE_ACCOUNT=devstoreaccount1 \
 		AZURE_STORAGE_ACCESS_KEY="Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==" \
-		AWS_ACCESS_KEY_ID=$(AWS_ACCESS_KEY_ID) \
-		AWS_SECRET_ACCESS_KEY=$(AWS_SECRET_ACCESS_KEY) \
 		system-test TEST=$(TEST) \
 		azurite-stop
 

Releasing.md

@@ -12,6 +12,5 @@ git push origin v$version master
 ```
 - run swagger locally (`make docker-serve`)
 - copy generated docs/swagger.json to https://github.com/aptly-dev/www.aptly.info/tree/master/static/swagger/aptly_1.x.y.json
-- add new version to select tag in content/doc/api/swagger.md line 48
-- push commit to master
+- releae www.aptly.info
 - create release announcement on https://github.com/aptly-dev/aptly/discussions

api/api.go

@@ -41,10 +41,7 @@ type aptlyVersion struct {
 // @Success 200 {object} aptlyVersion
 // @Router /api/version [get]
 func apiVersion(c *gin.Context) {
-	version := aptlyVersion{
-		Version: aptly.Version,
-	}
-	c.JSON(200, version)
+	c.JSON(200, gin.H{"Version": aptly.Version})
 }
 
 type aptlyStatus struct {
@@ -70,8 +67,7 @@ func apiReady(isReady *atomic.Value) func(*gin.Context) {
 			return
 		}
 
-                status := aptlyStatus{Status: "Aptly is ready"}
-		c.JSON(200, status)
+		c.JSON(200, gin.H{"Status": "Aptly is ready"})
 	}
 }
 
@@ -169,7 +165,7 @@ func runTaskInBackground(name string, resources []string, proc task.Process) (ta
 			return nil, err
 		}
 
-		defer func() { _ = releaseDatabaseConnection() }()
+		defer releaseDatabaseConnection()
 		return proc(out, detail)
 	})
 }
@@ -178,18 +174,18 @@ func truthy(value interface{}) bool {
 	if value == nil {
 		return false
 	}
-        switch v := value.(type) {
+	switch value.(type) {
 	case string:
-		switch strings.ToLower(v) {
+		switch strings.ToLower(value.(string)) {
 		case "n", "no", "f", "false", "0", "off":
 			return false
 		default:
 			return true
 		}
 	case int:
-		return v != 0
+		return !(value.(int) == 0)
 	case bool:
-		return v
+		return value.(bool)
 	}
 	return true
 }
@@ -214,11 +210,11 @@ func maybeRunTaskInBackground(c *gin.Context, name string, resources []string, p
 		}
 
 		// wait for task to finish
-		_, _ = context.TaskList().WaitForTaskByID(task.ID)
+		context.TaskList().WaitForTaskByID(task.ID)
 
 		retValue, _ := context.TaskList().GetTaskReturnValueByID(task.ID)
 		err, _ := context.TaskList().GetTaskErrorByID(task.ID)
-		_, _ = context.TaskList().DeleteTaskByID(task.ID)
+		context.TaskList().DeleteTaskByID(task.ID)
 		if err != nil {
 			AbortWithJSONError(c, retValue.Code, err)
 			return
@@ -233,7 +229,7 @@ func maybeRunTaskInBackground(c *gin.Context, name string, resources []string, p
 
 // Common piece of code to show list of packages,
 // with searching & details if requested
-func showPackages(c *gin.Context, reflist *deb.PackageRefList, collectionFactory *deb.CollectionFactory) {
+func showPackages(c *gin.Context, reflist deb.AnyRefList, collectionFactory *deb.CollectionFactory) {
 	result := []*deb.Package{}
 
 	list, err := deb.NewPackageListFromRefList(reflist, collectionFactory.PackageCollection(), nil)
@@ -286,11 +282,11 @@ func showPackages(c *gin.Context, reflist *deb.PackageRefList, collectionFactory
 	// filter packages by version
 	if c.Request.URL.Query().Get("maximumVersion") == "1" {
 		list.PrepareIndex()
-		_ = list.ForEach(func(p *deb.Package) error {
+		list.ForEach(func(p *deb.Package) error {
 			versionQ, err := query.Parse(fmt.Sprintf("Name (%s), $Version (<= %s)", p.Name, p.Version))
 			if err != nil {
 				fmt.Println("filter packages by version, query string parse err: ", err)
-				_ = c.AbortWithError(500, fmt.Errorf("unable to parse %s maximum version query string: %s", p.Name, err))
+				c.AbortWithError(500, fmt.Errorf("unable to parse %s maximum version query string: %s", p.Name, err))
 			} else {
 				tmpList, err := list.Filter(deb.FilterOptions{
 					Queries: []deb.PackageQuery{versionQ},
@@ -298,15 +294,15 @@ func showPackages(c *gin.Context, reflist *deb.PackageRefList, collectionFactory
 
 				if err == nil {
 					if tmpList.Len() > 0 {
-						_ = tmpList.ForEach(func(tp *deb.Package) error {
+						tmpList.ForEach(func(tp *deb.Package) error {
 							list.Remove(tp)
 							return nil
 						})
-						_ = list.Add(p)
+						list.Add(p)
 					}
 				} else {
 					fmt.Println("filter packages by version, filter err: ", err)
-					_ = c.AbortWithError(500, fmt.Errorf("unable to get %s maximum version: %s", p.Name, err))
+					c.AbortWithError(500, fmt.Errorf("unable to get %s maximum version: %s", p.Name, err))
 				}
 			}
 
@@ -315,7 +311,7 @@ func showPackages(c *gin.Context, reflist *deb.PackageRefList, collectionFactory
 	}
 
 	if c.Request.URL.Query().Get("format") == "details" {
-		_ = list.ForEach(func(p *deb.Package) error {
+		list.ForEach(func(p *deb.Package) error {
 			result = append(result, p)
 			return nil
 		})
@@ -326,7 +322,7 @@ func showPackages(c *gin.Context, reflist *deb.PackageRefList, collectionFactory
 	}
 }
 
-func AbortWithJSONError(c *gin.Context, code int, err error) {
+func AbortWithJSONError(c *gin.Context, code int, err error) *gin.Error {
 	c.Writer.Header().Set("Content-Type", "application/json; charset=utf-8")
-	_ = c.AbortWithError(code, err)
+	return c.AbortWithError(code, err)
 }

api/api_test.go

@@ -24,14 +24,14 @@ func Test(t *testing.T) {
 	TestingT(t)
 }
 
-type APISuite struct {
+type ApiSuite struct {
 	context    *ctx.AptlyContext
 	flags      *flag.FlagSet
 	configFile *os.File
 	router     http.Handler
 }
 
-var _ = Suite(&APISuite{})
+var _ = Suite(&ApiSuite{})
 
 func createTestConfig() *os.File {
 	file, err := os.CreateTemp("", "aptly")
@@ -45,11 +45,11 @@ func createTestConfig() *os.File {
 	if err != nil {
 		return nil
 	}
-	_, _ = file.Write(jsonString)
+	file.Write(jsonString)
 	return file
 }
 
-func (s *APISuite) setupContext() error {
+func (s *ApiSuite) setupContext() error {
 	aptly.Version = "testVersion"
 	file := createTestConfig()
 	if nil == file {
@@ -75,23 +75,23 @@ func (s *APISuite) setupContext() error {
 	return nil
 }
 
-func (s *APISuite) SetUpSuite(c *C) {
+func (s *ApiSuite) SetUpSuite(c *C) {
 	err := s.setupContext()
 	c.Assert(err, IsNil)
 }
 
-func (s *APISuite) TearDownSuite(c *C) {
-	_ = os.Remove(s.configFile.Name())
+func (s *ApiSuite) TearDownSuite(c *C) {
+	os.Remove(s.configFile.Name())
 	s.context.Shutdown()
 }
 
-func (s *APISuite) SetUpTest(c *C) {
+func (s *ApiSuite) SetUpTest(c *C) {
 }
 
-func (s *APISuite) TearDownTest(c *C) {
+func (s *ApiSuite) TearDownTest(c *C) {
 }
 
-func (s *APISuite) HTTPRequest(method string, url string, body io.Reader) (*httptest.ResponseRecorder, error) {
+func (s *ApiSuite) HTTPRequest(method string, url string, body io.Reader) (*httptest.ResponseRecorder, error) {
 	w := httptest.NewRecorder()
 	req, err := http.NewRequest(method, url, body)
 	if err != nil {
@@ -102,32 +102,32 @@ func (s *APISuite) HTTPRequest(method string, url string, body io.Reader) (*http
 	return w, nil
 }
 
-func (s *APISuite) TestGinRunsInReleaseMode(c *C) {
+func (s *ApiSuite) TestGinRunsInReleaseMode(c *C) {
 	c.Check(gin.Mode(), Equals, gin.ReleaseMode)
 }
 
-func (s *APISuite) TestGetVersion(c *C) {
+func (s *ApiSuite) TestGetVersion(c *C) {
 	response, err := s.HTTPRequest("GET", "/api/version", nil)
 	c.Assert(err, IsNil)
 	c.Check(response.Code, Equals, 200)
 	c.Check(response.Body.String(), Matches, "{\"Version\":\""+aptly.Version+"\"}")
 }
 
-func (s *APISuite) TestGetReadiness(c *C) {
+func (s *ApiSuite) TestGetReadiness(c *C) {
 	response, err := s.HTTPRequest("GET", "/api/ready", nil)
 	c.Assert(err, IsNil)
 	c.Check(response.Code, Equals, 200)
 	c.Check(response.Body.String(), Matches, "{\"Status\":\"Aptly is ready\"}")
 }
 
-func (s *APISuite) TestGetHealthiness(c *C) {
+func (s *ApiSuite) TestGetHealthiness(c *C) {
 	response, err := s.HTTPRequest("GET", "/api/healthy", nil)
 	c.Assert(err, IsNil)
 	c.Check(response.Code, Equals, 200)
 	c.Check(response.Body.String(), Matches, "{\"Status\":\"Aptly is healthy\"}")
 }
 
-func (s *APISuite) TestGetMetrics(c *C) {
+func (s *ApiSuite) TestGetMetrics(c *C) {
 	response, err := s.HTTPRequest("GET", "/api/metrics", nil)
 	c.Assert(err, IsNil)
 	c.Check(response.Code, Equals, 200)
@@ -141,7 +141,7 @@ func (s *APISuite) TestGetMetrics(c *C) {
 	c.Check(b, Matches, ".*aptly_build_info.*version=\"testVersion\".*")
 }
 
-func (s *APISuite) TestRepoCreate(c *C) {
+func (s *ApiSuite) TestRepoCreate(c *C) {
 	body, err := json.Marshal(gin.H{
 		"Name": "dummy",
 	})
@@ -150,7 +150,7 @@ func (s *APISuite) TestRepoCreate(c *C) {
 	c.Assert(err, IsNil)
 }
 
-func (s *APISuite) TestTruthy(c *C) {
+func (s *ApiSuite) TestTruthy(c *C) {
 	c.Check(truthy("no"), Equals, false)
 	c.Check(truthy("n"), Equals, false)
 	c.Check(truthy("off"), Equals, false)

api/db.go

@@ -5,6 +5,7 @@ import (
 	"sort"
 
 	"github.com/aptly-dev/aptly/aptly"
+	"github.com/aptly-dev/aptly/database"
 	"github.com/aptly-dev/aptly/deb"
 	"github.com/aptly-dev/aptly/task"
 	"github.com/aptly-dev/aptly/utils"
@@ -21,25 +22,29 @@ import (
 // @Success 200 {object} string "Output"
 // @Failure 404 {object} Error "Not Found"
 // @Router /api/db/cleanup [post]
-func apiDBCleanup(c *gin.Context) {
+func apiDbCleanup(c *gin.Context) {
 	resources := []string{string(task.AllResourcesKey)}
 	maybeRunTaskInBackground(c, "Clean up db", resources, func(out aptly.Progress, detail *task.Detail) (*task.ProcessReturnValue, error) {
 		var err error
 
 		collectionFactory := context.NewCollectionFactory()
 
-		// collect information about referenced packages...
-		existingPackageRefs := deb.NewPackageRefList()
+		// collect information about referenced packages and their reflist buckets...
+		existingPackageRefs := deb.NewSplitRefList()
+		existingBuckets := deb.NewRefListDigestSet()
+
+		reflistMigration := collectionFactory.RefListCollection().NewMigration()
 
 		out.Printf("Loading mirrors, local repos, snapshots and published repos...")
 		err = collectionFactory.RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
-			e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
-			if e != nil {
+			sl := deb.NewSplitRefList()
+			e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
+			if e != nil && e != database.ErrNotFound {
 				return e
 			}
-			if repo.RefList() != nil {
-				existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
-			}
+
+			existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+			existingBuckets.AddAllInRefList(sl)
 
 			return nil
 		})
@@ -48,14 +53,14 @@ func apiDBCleanup(c *gin.Context) {
 		}
 
 		err = collectionFactory.LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
-			e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
-			if e != nil {
+			sl := deb.NewSplitRefList()
+			e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
+			if e != nil && e != database.ErrNotFound {
 				return e
 			}
 
-			if repo.RefList() != nil {
-				existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
-			}
+			existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+			existingBuckets.AddAllInRefList(sl)
 
 			return nil
 		})
@@ -64,12 +69,14 @@ func apiDBCleanup(c *gin.Context) {
 		}
 
 		err = collectionFactory.SnapshotCollection().ForEach(func(snapshot *deb.Snapshot) error {
-			e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+			sl := deb.NewSplitRefList()
+			e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, snapshot.RefKey(), reflistMigration)
 			if e != nil {
 				return e
 			}
 
-			existingPackageRefs = existingPackageRefs.Merge(snapshot.RefList(), false, true)
+			existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+			existingBuckets.AddAllInRefList(sl)
 
 			return nil
 		})
@@ -81,13 +88,16 @@ func apiDBCleanup(c *gin.Context) {
 			if published.SourceKind != deb.SourceLocalRepo {
 				return nil
 			}
-			e := collectionFactory.PublishedRepoCollection().LoadComplete(published, collectionFactory)
+
+			for _, component := range published.Components() {
+				sl := deb.NewSplitRefList()
+				e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, published.RefKey(component), reflistMigration)
 				if e != nil {
 					return e
 				}
 
-			for _, component := range published.Components() {
-				existingPackageRefs = existingPackageRefs.Merge(published.RefList(component), false, true)
+				existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+				existingBuckets.AddAllInRefList(sl)
 			}
 			return nil
 		})
@@ -95,11 +105,20 @@ func apiDBCleanup(c *gin.Context) {
 			return nil, err
 		}
 
+		err = reflistMigration.Flush()
+		if err != nil {
+			return nil, err
+		}
+		if stats := reflistMigration.Stats(); stats.Reflists > 0 {
+			out.Printf("Split %d reflist(s) into %d bucket(s) (%d segment(s))",
+				stats.Reflists, stats.Buckets, stats.Segments)
+		}
+
 		// ... and compare it to the list of all packages
 		out.Printf("Loading list of all packages...")
 		allPackageRefs := collectionFactory.PackageCollection().AllPackageRefs()
 
-		toDelete := allPackageRefs.Subtract(existingPackageRefs)
+		toDelete := allPackageRefs.Subtract(existingPackageRefs.Flatten())
 
 		// delete packages that are no longer referenced
 		out.Printf("Deleting unreferenced packages (%d)...", toDelete.Len())
@@ -109,8 +128,8 @@ func apiDBCleanup(c *gin.Context) {
 
 		if toDelete.Len() > 0 {
 			batch := db.CreateBatch()
-			_ = toDelete.ForEach(func(ref []byte) error {
-				_ = collectionFactory.PackageCollection().DeleteByKey(ref, batch)
+			toDelete.ForEach(func(ref []byte) error {
+				collectionFactory.PackageCollection().DeleteByKey(ref, batch)
 				return nil
 			})
 
@@ -120,6 +139,28 @@ func apiDBCleanup(c *gin.Context) {
 			}
 		}
 
+		bucketsToDelete, err := collectionFactory.RefListCollection().AllBucketDigests()
+		if err != nil {
+			return nil, err
+		}
+
+		bucketsToDelete.RemoveAll(existingBuckets)
+
+		out.Printf("Deleting unreferenced reflist buckets (%d)...", bucketsToDelete.Len())
+		if bucketsToDelete.Len() > 0 {
+			batch := db.CreateBatch()
+			err := bucketsToDelete.ForEach(func(digest []byte) error {
+				return collectionFactory.RefListCollection().UnsafeDropBucket(digest, batch)
+			})
+			if err != nil {
+				return nil, err
+			}
+
+			if err := batch.Write(); err != nil {
+				return nil, err
+			}
+		}
+
 		// now, build a list of files that should be present in Repository (package pool)
 		out.Printf("Building list of files referenced by packages...")
 		referencedFiles := make([]string, 0, existingPackageRefs.Len())

api/files.go

@@ -122,7 +122,7 @@ func apiFilesUpload(c *gin.Context) {
 				AbortWithJSONError(c, 500, err)
 				return
 			}
-			defer func() { _ = src.Close() }()
+			defer src.Close()
 
 			destPath := filepath.Join(path, filepath.Base(file.Filename))
 			dst, err := os.Create(destPath)
@@ -130,7 +130,7 @@ func apiFilesUpload(c *gin.Context) {
 				AbortWithJSONError(c, 500, err)
 				return
 			}
-			defer func() { _ = dst.Close() }()
+			defer dst.Close()
 
 			_, err = io.Copy(dst, src)
 			if err != nil {
@@ -170,11 +170,7 @@ func apiFilesListFiles(c *gin.Context) {
 	listLock := &sync.Mutex{}
 	root := filepath.Join(context.UploadPath(), utils.SanitizePath(c.Params.ByName("dir")))
 
-	err := filepath.Walk(root, func(path string, _ os.FileInfo, err error) error {
-		if err != nil {
-			return err
-		}
-
+	err := walker.Walk(root, func(path string, _ os.FileInfo) error {
 		if path == root {
 			return nil
 		}

api/gpg.go

@@ -13,34 +13,26 @@ import (
 )
 
 type gpgAddKeyParams struct {
+	// Keyserver, when downloading GpgKeyIDs
+	Keyserver string `json:"Keyserver"       example:"hkp://keyserver.ubuntu.com:80"`
+	// GpgKeyIDs to download from Keyserver, comma separated list
+	GpgKeyID string `json:"GpgKeyID"        example:"EF0F382A1A7B6500,8B48AD6246925553"`
+	// Armored gpg public ket, instead of downloading from keyserver
+	GpgKeyArmor string `json:"GpgKeyArmor"     example:""`
 	// Keyring for adding the keys (default: trustedkeys.gpg)
 	Keyring string `json:"Keyring"         example:"trustedkeys.gpg"`
-
-	// Add ASCII armored gpg public key, do not download from keyserver
-	GpgKeyArmor string `json:"GpgKeyArmor"     example:""`
-
-	// Keyserver to download keys provided in `GpgKeyID`
-	Keyserver string `json:"Keyserver"       example:"hkp://keyserver.ubuntu.com:80"`
-	// Keys do download from `Keyserver`, separated by space
-	GpgKeyID string `json:"GpgKeyID"        example:"EF0F382A1A7B6500 8B48AD6246925553"`
 }
 
 // @Summary Add GPG Keys
 // @Description **Adds GPG keys to aptly keyring**
 // @Description
 // @Description Add GPG public keys for veryfing remote repositories for mirroring.
-// @Description
-// @Description Keys can be added in two ways:
-// @Description * By providing the ASCII armord key in `GpgKeyArmor` (leave Keyserver and GpgKeyID empty)
-// @Description * By providing a `Keyserver` and one or more key IDs in `GpgKeyID`, separated by space (leave GpgKeyArmor empty)
-// @Description
 // @Tags Mirrors
-// @Consume  json
-// @Param request body gpgAddKeyParams true "Parameters"
 // @Produce json
 // @Success 200 {object} string "OK"
 // @Failure 400 {object} Error "Bad Request"
-// @Router /api/gpg/key [post]
+// @Failure 404 {object} Error "Not Found"
+// @Router /api/gpg [post]
 func apiGPGAddKey(c *gin.Context) {
 	b := gpgAddKeyParams{}
 	if c.Bind(&b) != nil {
@@ -68,7 +60,7 @@ func apiGPGAddKey(c *gin.Context) {
 			AbortWithJSONError(c, 400, err)
 			return
 		}
-		defer func() { _ = os.RemoveAll(tempdir) }()
+		defer os.RemoveAll(tempdir)
 
 		keypath := filepath.Join(tempdir, "key")
 		keyfile, e := os.Create(keypath)

api/metrics.go

@@ -102,7 +102,7 @@ func countPackagesByRepos() {
 
 		components := repo.Components()
 		for _, c := range components {
-			count := float64(len(repo.RefList(c).Refs))
+			count := float64(repo.RefList(c).Len())
 			apiReposPackageCountGauge.WithLabelValues(fmt.Sprintf("%s", (repo.SourceNames())), repo.Distribution, c).Set(count)
 		}
 

api/middleware_test.go

@@ -67,17 +67,17 @@ func (s *MiddlewareSuite) TestJSONMiddleware4xx(c *C) {
 	outC := make(chan string)
 	go func() {
 		var buf bytes.Buffer
-		_, _ = io.Copy(&buf, s.logReader)
+		io.Copy(&buf, s.logReader)
 		fmt.Println(buf.String())
 		outC <- buf.String()
 	}()
 
 	s.HTTPRequest(http.MethodGet, "/", nil)
-	_ = s.logWriter.Close()
+	s.logWriter.Close()
 	capturedOutput := <-outC
 
 	var jsonMap map[string]interface{}
-	_ = json.Unmarshal([]byte(capturedOutput), &jsonMap)
+	json.Unmarshal([]byte(capturedOutput), &jsonMap)
 
 	if val, ok := jsonMap["level"]; ok {
 		c.Check(val, Equals, "warn")
@@ -130,17 +130,17 @@ func (s *MiddlewareSuite) TestJSONMiddleware2xx(c *C) {
 	outC := make(chan string)
 	go func() {
 		var buf bytes.Buffer
-		_, _ = io.Copy(&buf, s.logReader)
+		io.Copy(&buf, s.logReader)
 		fmt.Println(buf.String())
 		outC <- buf.String()
 	}()
 
 	s.HTTPRequest(http.MethodGet, "/api/healthy", nil)
-	_ = s.logWriter.Close()
+	s.logWriter.Close()
 	capturedOutput := <-outC
 
 	var jsonMap map[string]interface{}
-	_ = json.Unmarshal([]byte(capturedOutput), &jsonMap)
+	json.Unmarshal([]byte(capturedOutput), &jsonMap)
 
 	if val, ok := jsonMap["level"]; ok {
 		c.Check(val, Equals, "info")
@@ -153,17 +153,17 @@ func (s *MiddlewareSuite) TestJSONMiddleware5xx(c *C) {
 	outC := make(chan string)
 	go func() {
 		var buf bytes.Buffer
-		_, _ = io.Copy(&buf, s.logReader)
+		io.Copy(&buf, s.logReader)
 		fmt.Println(buf.String())
 		outC <- buf.String()
 	}()
 
 	s.HTTPRequest(http.MethodGet, "/api/ready", nil)
-	_ = s.logWriter.Close()
+	s.logWriter.Close()
 	capturedOutput := <-outC
 
 	var jsonMap map[string]interface{}
-	_ = json.Unmarshal([]byte(capturedOutput), &jsonMap)
+	json.Unmarshal([]byte(capturedOutput), &jsonMap)
 
 	if val, ok := jsonMap["level"]; ok {
 		c.Check(val, Equals, "error")
@@ -176,17 +176,17 @@ func (s *MiddlewareSuite) TestJSONMiddlewareRaw(c *C) {
 	outC := make(chan string)
 	go func() {
 		var buf bytes.Buffer
-		_, _ = io.Copy(&buf, s.logReader)
+		io.Copy(&buf, s.logReader)
 		fmt.Println(buf.String())
 		outC <- buf.String()
 	}()
 
 	s.HTTPRequest(http.MethodGet, "/api/healthy?test=raw", nil)
-	_ = s.logWriter.Close()
+	s.logWriter.Close()
 	capturedOutput := <-outC
 
 	var jsonMap map[string]interface{}
-	_ = json.Unmarshal([]byte(capturedOutput), &jsonMap)
+	json.Unmarshal([]byte(capturedOutput), &jsonMap)
 
 	fmt.Println(capturedOutput)
 

api/mirror.go

@@ -43,7 +43,7 @@ func apiMirrorsList(c *gin.Context) {
 	collection := collectionFactory.RemoteRepoCollection()
 
 	result := []*deb.RemoteRepo{}
-	_ = collection.ForEach(func(repo *deb.RemoteRepo) error {
+	collection.ForEach(func(repo *deb.RemoteRepo) error {
 		result = append(result, repo)
 		return nil
 	})
@@ -150,7 +150,7 @@ func apiMirrorsCreate(c *gin.Context) {
 		return
 	}
 
-	err = collection.Add(repo)
+	err = collection.Add(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, fmt.Errorf("unable to add mirror: %s", err))
 		return
@@ -229,7 +229,7 @@ func apiMirrorsShow(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(repo)
+	err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, fmt.Errorf("unable to show: %s", err))
 	}
@@ -260,7 +260,7 @@ func apiMirrorsPackages(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(repo)
+	err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, fmt.Errorf("unable to show: %s", err))
 	}
@@ -319,7 +319,7 @@ func apiMirrorsPackages(c *gin.Context) {
 	}
 
 	if c.Request.URL.Query().Get("format") == "details" {
-		_ = list.ForEach(func(p *deb.Package) error {
+		list.ForEach(func(p *deb.Package) error {
 			result = append(result, p)
 			return nil
 		})
@@ -491,12 +491,12 @@ func apiMirrorsUpdate(c *gin.Context) {
 			e := context.ReOpenDatabase()
 			if e == nil {
 				remote.MarkAsIdle()
-				_ = collection.Update(remote)
+				collection.Update(remote, collectionFactory.RefListCollection())
 			}
 		}()
 
 		remote.MarkAsUpdating()
-		err = collection.Update(remote)
+		err = collection.Update(remote, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
 		}
@@ -579,7 +579,7 @@ func apiMirrorsUpdate(c *gin.Context) {
 							file, e = os.CreateTemp("", task.File.Filename)
 							if e == nil {
 								task.TempDownPath = file.Name()
-								_ = file.Close()
+								file.Close()
 							}
 						}
 						if e != nil {
@@ -653,8 +653,8 @@ func apiMirrorsUpdate(c *gin.Context) {
 		}
 
 		log.Info().Msgf("%s: Finalizing download...", b.Name)
-		_ = remote.FinalizeDownload(collectionFactory, out)
-		err = collectionFactory.RemoteRepoCollection().Update(remote)
+		remote.FinalizeDownload(collectionFactory, out)
+		err = collectionFactory.RemoteRepoCollection().Update(remote, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
 		}

api/mirror_test.go

@@ -9,7 +9,7 @@ import (
 )
 
 type MirrorSuite struct {
-	APISuite
+	ApiSuite
 }
 
 var _ = Suite(&MirrorSuite{})

api/packages_test.go

@@ -5,7 +5,7 @@ import (
 )
 
 type PackagesSuite struct {
-	APISuite
+	ApiSuite
 }
 
 var _ = Suite(&PackagesSuite{})

api/publish.go

@@ -297,10 +297,10 @@ func apiPublishRepoOrSnapshot(c *gin.Context) {
 			switch s := source.(type) {
 			case *deb.Snapshot:
 				snapshotCollection := collectionFactory.SnapshotCollection()
-				err = snapshotCollection.LoadComplete(s)
+				err = snapshotCollection.LoadComplete(s, collectionFactory.RefListCollection())
 			case *deb.LocalRepo:
 				localCollection := collectionFactory.LocalRepoCollection()
-				err = localCollection.LoadComplete(s)
+				err = localCollection.LoadComplete(s, collectionFactory.RefListCollection())
 			default:
 				err = fmt.Errorf("unexpected type for source: %T", source)
 			}
@@ -343,7 +343,7 @@ func apiPublishRepoOrSnapshot(c *gin.Context) {
 
 		duplicate := collection.CheckDuplicate(published)
 		if duplicate != nil {
-			_ = collectionFactory.PublishedRepoCollection().LoadComplete(duplicate, collectionFactory)
+			collectionFactory.PublishedRepoCollection().LoadComplete(duplicate, collectionFactory)
 			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, fmt.Errorf("prefix/distribution already used by another published repo: %s", duplicate)
 		}
 
@@ -352,7 +352,7 @@ func apiPublishRepoOrSnapshot(c *gin.Context) {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to publish: %s", err)
 		}
 
-		err = collection.Add(published)
+		err = collection.Add(published, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
 		}
@@ -469,9 +469,9 @@ func apiPublishUpdateSwitch(c *gin.Context) {
 	resources := []string{string(published.Key())}
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
 	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		err = collection.LoadComplete(published, collectionFactory)
+		err = collection.LoadComplete(published, collectionFactory, collectionFactory.RefListCollection())
 		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
+			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("Unable to update: %s", err)
 		}
 
 		revision := published.ObtainRevision()
@@ -487,15 +487,15 @@ func apiPublishUpdateSwitch(c *gin.Context) {
 
 		result, err := published.Update(collectionFactory, out)
 		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
+			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("Unable to update: %s", err)
 		}
 
 		err = published.Publish(context.PackagePool(), context, collectionFactory, signer, out, b.ForceOverwrite, context.SkelPath())
 		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
+			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("Unable to update: %s", err)
 		}
 
-		err = collection.Update(published)
+		err = collection.Update(published, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
 		}
@@ -625,7 +625,7 @@ func apiPublishAddSource(c *gin.Context) {
 
 	resources := []string{string(published.Key())}
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
-	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
+	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
 		err = collection.Update(published)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
@@ -739,7 +739,7 @@ func apiPublishSetSources(c *gin.Context) {
 
 	resources := []string{string(published.Key())}
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
-	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
+	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
 		err = collection.Update(published)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
@@ -789,7 +789,7 @@ func apiPublishDropChanges(c *gin.Context) {
 
 	resources := []string{string(published.Key())}
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
-	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
+	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
 		err = collection.Update(published)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
@@ -869,7 +869,7 @@ func apiPublishUpdateSource(c *gin.Context) {
 
 	resources := []string{string(published.Key())}
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
-	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
+	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
 		err = collection.Update(published)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
@@ -932,7 +932,7 @@ func apiPublishRemoveSource(c *gin.Context) {
 
 	resources := []string{string(published.Key())}
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
-	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
+	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
 		err = collection.Update(published)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)

api/repos.go

@@ -29,14 +29,14 @@ func reposListInAPIMode(localRepos map[string]utils.FileSystemPublishRoot) gin.H
 	return func(c *gin.Context) {
 		c.Writer.Header().Set("Content-Type", "text/html; charset=utf-8")
 		c.Writer.Flush()
-		_, _ = c.Writer.WriteString("<pre>\n")
+		c.Writer.WriteString("<pre>\n")
 		if len(localRepos) == 0 {
-			_, _ = c.Writer.WriteString("<a href=\"-/\">default</a>\n")
+			c.Writer.WriteString("<a href=\"-/\">default</a>\n")
 		}
 		for publishPrefix := range localRepos {
-			_, _ = c.Writer.WriteString(fmt.Sprintf("<a href=\"%[1]s/\">%[1]s</a>\n", publishPrefix))
+			c.Writer.WriteString(fmt.Sprintf("<a href=\"%[1]s/\">%[1]s</a>\n", publishPrefix))
 		}
-		_, _ = c.Writer.WriteString("</pre>")
+		c.Writer.WriteString("</pre>")
 		c.Writer.Flush()
 	}
 }
@@ -76,7 +76,7 @@ func apiReposList(c *gin.Context) {
 
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.LocalRepoCollection()
-	_ = collection.ForEach(func(r *deb.LocalRepo) error {
+	collection.ForEach(func(r *deb.LocalRepo) error {
 		result = append(result, r)
 		return nil
 	})
@@ -107,9 +107,9 @@ type repoCreateParams struct {
 // @Description {"Name":"aptly-repo","Comment":"","DefaultDistribution":"","DefaultComponent":""}
 // @Description ```
 // @Tags Repos
+// @Produce  json
 // @Consume  json
 // @Param request body repoCreateParams true "Parameters"
-// @Produce  json
 // @Success 201 {object} deb.LocalRepo
 // @Failure 404 {object} Error "Source snapshot not found"
 // @Failure 409 {object} Error "Local repo already exists"
@@ -155,6 +155,13 @@ func apiReposCreate(c *gin.Context) {
 		return
 	}
 
+	collection := collectionFactory.LocalRepoCollection()
+	err := collection.Add(repo, collectionFactory.RefListCollection())
+	if err != nil {
+		AbortWithJSONError(c, 400, err)
+		return
+	}
+
 	err := localRepoCollection.Add(repo)
 	if err != nil {
 		AbortWithJSONError(c, http.StatusInternalServerError, err)
@@ -178,10 +185,8 @@ type reposEditParams struct {
 // @Summary Update Repository
 // @Description **Update local repository meta information**
 // @Tags Repos
-// @Param name path string true "Repository name"
-// @Consume  json
-// @Param request body reposEditParams true "Parameters"
 // @Produce json
+// @Param request body reposEditParams true "Parameters"
 // @Success 200 {object} deb.LocalRepo "msg"
 // @Failure 404 {object} Error "Not Found"
 // @Failure 500 {object} Error "Internal Server Error"
@@ -220,7 +225,7 @@ func apiReposEdit(c *gin.Context) {
 		repo.DefaultComponent = *b.DefaultComponent
 	}
 
-	err = collection.Update(repo)
+	err = collection.Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -233,8 +238,8 @@ func apiReposEdit(c *gin.Context) {
 // @Summary Get Repository Info
 // @Description Returns basic information about local repository.
 // @Tags Repos
-// @Param name path string true "Repository name"
 // @Produce  json
+// @Param name path string true "Repository name"
 // @Success 200 {object} deb.LocalRepo
 // @Failure 404 {object} Error "Repository not found"
 // @Router /api/repos/{name} [get]
@@ -256,10 +261,9 @@ func apiReposShow(c *gin.Context) {
 // @Description Cannot drop repos that are published.
 // @Description Needs force=1 to drop repos used as source by other repos.
 // @Tags Repos
-// @Param name path string true "Repository name"
+// @Produce json
 // @Param _async query bool false "Run in background and return task object"
 // @Param force query int false "force: 1 to enable"
-// @Produce json
 // @Success 200 {object} task.ProcessReturnValue "Repo object"
 // @Failure 404 {object} Error "Not Found"
 // @Failure 404 {object} Error "Repo Conflict"
@@ -309,12 +313,12 @@ func apiReposDrop(c *gin.Context) {
 // @Description ["Pi386 aptly 0.8 966561016b44ed80"]
 // @Description ```
 // @Tags Repos
-// @Param name path string true "Repository name"
+// @Produce json
+// @Param name path string true "Snapshot to search"
 // @Param q query string true "Package query (e.g Name%20(~%20matlab))"
 // @Param withDeps query string true "Set to 1 to include dependencies when evaluating package query"
 // @Param format query string true "Set to 'details' to return extra info about each package"
 // @Param maximumVersion query string true "Set to 1 to only return the highest version for each package name"
-// @Produce json
 // @Success 200 {object} string "msg"
 // @Failure 404 {object} Error "Not Found"
 // @Failure 404 {object} Error "Internal Server Error"
@@ -329,7 +333,7 @@ func apiReposPackagesShow(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(repo)
+	err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -363,7 +367,7 @@ func apiReposPackagesAddDelete(c *gin.Context, taskNamePrefix string, cb func(li
 	resources := []string{string(repo.Key())}
 
 	maybeRunTaskInBackground(c, taskNamePrefix+repo.Name, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		err = collection.LoadComplete(repo)
+		err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -392,9 +396,9 @@ func apiReposPackagesAddDelete(c *gin.Context, taskNamePrefix string, cb func(li
 			}
 		}
 
-		repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
+		repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
 
-		err = collectionFactory.LocalRepoCollection().Update(repo)
+		err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save: %s", err)
 		}
@@ -409,10 +413,9 @@ func apiReposPackagesAddDelete(c *gin.Context, taskNamePrefix string, cb func(li
 // @Description
 // @Description API verifies that packages actually exist in aptly database and checks constraint that conflicting packages can’t be part of the same local repository.
 // @Tags Repos
-// @Param name path string true "Repository name"
+// @Produce json
 // @Param request body reposPackagesAddDeleteParams true "Parameters"
 // @Param _async query bool false "Run in background and return task object"
-// @Produce json
 // @Success 200 {object} string "msg"
 // @Failure 400 {object} Error "Bad Request"
 // @Failure 404 {object} Error "Not Found"
@@ -430,11 +433,9 @@ func apiReposPackagesAdd(c *gin.Context) {
 // @Description
 // @Description Any package(s) can be removed from a local repository. Package references from a local repository can be retrieved with GET /api/repos/:name/packages.
 // @Tags Repos
-// @Param name path string true "Repository name"
-// @Param _async query bool false "Run in background and return task object"
-// @Consume  json
-// @Param request body reposPackagesAddDeleteParams true "Parameters"
 // @Produce json
+// @Param request body reposPackagesAddDeleteParams true "Parameters"
+// @Param _async query bool false "Run in background and return task object"
 // @Success 200 {object} string "msg"
 // @Failure 400 {object} Error "Bad Request"
 // @Failure 404 {object} Error "Not Found"
@@ -523,7 +524,7 @@ func apiReposPackageFromDir(c *gin.Context) {
 	resources := []string{string(repo.Key())}
 	resources = append(resources, sources...)
 	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		err = collection.LoadComplete(repo)
+		err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -558,9 +559,9 @@ func apiReposPackageFromDir(c *gin.Context) {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to import package files: %s", err)
 		}
 
-		repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
+		repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
 
-		err = collectionFactory.LocalRepoCollection().Update(repo)
+		err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save: %s", err)
 		}
@@ -576,7 +577,7 @@ func apiReposPackageFromDir(c *gin.Context) {
 			}
 
 			// atempt to remove dir, if it fails, that's fine: probably it's not empty
-			_ = os.Remove(filepath.Join(context.UploadPath(), dirParam))
+			os.Remove(filepath.Join(context.UploadPath(), dirParam))
 		}
 
 		if failedFiles == nil {
@@ -614,8 +615,8 @@ type reposCopyPackageParams struct {
 // @Description Copies a package from a source to destination repository
 // @Tags Repos
 // @Produce json
-// @Param name path string true "Destination repo"
-// @Param src path string true "Source repo"
+// @Param name path string true "Source repo"
+// @Param src path string true "Destination repo"
 // @Param file path string true "File/packages to copy"
 // @Param _async query bool false "Run in background and return task object"
 // @Success 200 {object} task.ProcessReturnValue "msg"
@@ -768,15 +769,12 @@ func apiReposCopyPackage(c *gin.Context) {
 // @Summary Include File from Directory
 // @Description Allows automatic processing of .changes file controlling package upload (uploaded using File Upload API) to the local repository. i.e. Exposes repo include command in api.
 // @Tags Repos
-// @Param name path string true "Repository name"
-// @Param dir path string true "Directory of packages"
-// @Param file path string true "File/packages to include"
+// @Produce json
 // @Param forceReplace query int false "when value is set to 1, when adding package that conflicts with existing package, remove existing package"
 // @Param noRemoveFiles query int false "when value is set to 1, don’t remove files that have been imported successfully into repository"
 // @Param acceptUnsigned query int false "when value is set to 1, accept unsigned .changes files"
 // @Param ignoreSignature query int false "when value is set to 1 disable verification of .changes file signature"
 // @Param _async query bool false "Run in background and return task object"
-// @Produce json
 // @Success 200 {object} string "msg"
 // @Failure 404 {object} Error "Not Found"
 // @Router /api/repos/{name}/include/{dir}/{file} [post]
@@ -785,22 +783,26 @@ func apiReposIncludePackageFromFile(c *gin.Context) {
 	apiReposIncludePackageFromDir(c)
 }
 
+type reposIncludePackageFromDirReport struct {
+	Warnings []string
+	Added    []string
+	Deleted  []string
+}
+
 type reposIncludePackageFromDirResponse struct {
-	Report      *aptly.RecordingResultReporter
+	Report      reposIncludePackageFromDirReport
 	FailedFiles []string
 }
 
 // @Summary Include Directory
 // @Description Allows automatic processing of .changes file controlling package upload (uploaded using File Upload API) to the local repository. i.e. Exposes repo include command in api.
 // @Tags Repos
-// @Param name path string true "Repository name"
-// @Param dir path string true "Directory of packages"
+// @Produce json
 // @Param forceReplace query int false "when value is set to 1, when adding package that conflicts with existing package, remove existing package"
 // @Param noRemoveFiles query int false "when value is set to 1, don’t remove files that have been imported successfully into repository"
 // @Param acceptUnsigned query int false "when value is set to 1, accept unsigned .changes files"
 // @Param ignoreSignature query int false "when value is set to 1 disable verification of .changes file signature"
 // @Param _async query bool false "Run in background and return task object"
-// @Produce json
 // @Success 200 {object} reposIncludePackageFromDirResponse "Response"
 // @Failure 404 {object} Error "Not Found"
 // @Router /api/repos/{name}/include/{dir} [post]
@@ -841,7 +843,7 @@ func apiReposIncludePackageFromDir(c *gin.Context) {
 	}
 
 	var resources []string
-	if len(repoTemplate.Root.Nodes) > 1 {
+	if len(repoTemplate.Tree.Root.Nodes) > 1 {
 		resources = append(resources, task.AllLocalReposResourcesKey)
 	} else {
 		// repo template string is simple text so only use resource key of specific repository
@@ -872,7 +874,7 @@ func apiReposIncludePackageFromDir(c *gin.Context) {
 		_, failedFiles2, err = deb.ImportChangesFiles(
 			changesFiles, reporter, acceptUnsigned, ignoreSignature, forceReplace, noRemoveFiles, verifier,
 			repoTemplate, context.Progress(), collectionFactory.LocalRepoCollection(), collectionFactory.PackageCollection(),
-			context.PackagePool(), collectionFactory.ChecksumCollection, nil, query.Parse)
+			collectionFactory.RefListCollection(), context.PackagePool(), collectionFactory.ChecksumCollection, nil, query.Parse)
 		failedFiles = append(failedFiles, failedFiles2...)
 
 		if err != nil {
@@ -881,7 +883,7 @@ func apiReposIncludePackageFromDir(c *gin.Context) {
 
 		if !noRemoveFiles {
 			// atempt to remove dir, if it fails, that's fine: probably it's not empty
-			_ = os.Remove(filepath.Join(context.UploadPath(), dirParam))
+			os.Remove(filepath.Join(context.UploadPath(), dirParam))
 		}
 
 		if failedFiles == nil {
@@ -901,10 +903,9 @@ func apiReposIncludePackageFromDir(c *gin.Context) {
 			out.Printf("Failed files: %s\n", strings.Join(failedFiles, ", "))
 		}
 
-                ret := reposIncludePackageFromDirResponse{
-			Report:      reporter,
-			FailedFiles: failedFiles,
-                    }
-		return &task.ProcessReturnValue{Code: http.StatusOK, Value: ret}, nil
+		return &task.ProcessReturnValue{Code: http.StatusOK, Value: gin.H{
+			"Report":      reporter,
+			"FailedFiles": failedFiles,
+		}}, nil
 	})
 }

api/router.go

@@ -2,6 +2,7 @@ package api
 
 import (
 	"net/http"
+	"os"
 	"sync/atomic"
 
 	"github.com/aptly-dev/aptly/aptly"
@@ -55,9 +56,13 @@ func Router(c *ctx.AptlyContext) http.Handler {
 	router.UseRawPath = true
 
 	if c.Config().LogFormat == "json" {
+		c.StructuredLogging(true)
+		utils.SetupJSONLogger(c.Config().LogLevel, os.Stdout)
 		gin.DefaultWriter = utils.LogWriter{Logger: log.Logger}
 		router.Use(JSONLogger())
 	} else {
+		c.StructuredLogging(false)
+		utils.SetupDefaultLogger(c.Config().LogLevel)
 		router.Use(gin.Logger())
 	}
 
@@ -215,7 +220,7 @@ func Router(c *ctx.AptlyContext) http.Handler {
 		api.GET("/graph.:ext", apiGraph)
 	}
 	{
-		api.POST("/db/cleanup", apiDBCleanup)
+		api.POST("/db/cleanup", apiDbCleanup)
 	}
 	{
 		api.GET("/tasks", apiTasksList)

api/snapshot.go

@@ -33,7 +33,7 @@ func apiSnapshotsList(c *gin.Context) {
 	}
 
 	result := []*deb.Snapshot{}
-	_ = collection.ForEachSorted(SortMethodString, func(snapshot *deb.Snapshot) error {
+	collection.ForEachSorted(SortMethodString, func(snapshot *deb.Snapshot) error {
 		result = append(result, snapshot)
 		return nil
 	})
@@ -93,7 +93,7 @@ func apiSnapshotsCreateFromMirror(c *gin.Context) {
 			return &task.ProcessReturnValue{Code: http.StatusConflict, Value: nil}, err
 		}
 
-		err = collection.LoadComplete(repo)
+		err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -107,7 +107,7 @@ func apiSnapshotsCreateFromMirror(c *gin.Context) {
 			snapshot.Description = b.Description
 		}
 
-		err = snapshotCollection.Add(snapshot)
+		err = snapshotCollection.Add(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, err
 		}
@@ -174,7 +174,7 @@ func apiSnapshotsCreate(c *gin.Context) {
 
 	maybeRunTaskInBackground(c, "Create snapshot "+b.Name, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
 		for i := range sources {
-			err = snapshotCollection.LoadComplete(sources[i])
+			err = snapshotCollection.LoadComplete(sources[i], collectionFactory.RefListCollection())
 			if err != nil {
 				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 			}
@@ -197,9 +197,9 @@ func apiSnapshotsCreate(c *gin.Context) {
 			}
 		}
 
-		snapshot = deb.NewSnapshotFromRefList(b.Name, sources, deb.NewPackageRefListFromPackageList(list), b.Description)
+		snapshot = deb.NewSnapshotFromRefList(b.Name, sources, deb.NewSplitRefListFromPackageList(list), b.Description)
 
-		err = snapshotCollection.Add(snapshot)
+		err = snapshotCollection.Add(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, err
 		}
@@ -255,7 +255,7 @@ func apiSnapshotsCreateFromRepository(c *gin.Context) {
 	resources := []string{string(repo.Key()), "S" + b.Name}
 	taskName := fmt.Sprintf("Create snapshot of repo %s", name)
 	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		err := collection.LoadComplete(repo)
+		err := collection.LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -269,7 +269,7 @@ func apiSnapshotsCreateFromRepository(c *gin.Context) {
 			snapshot.Description = b.Description
 		}
 
-		err = snapshotCollection.Add(snapshot)
+		err = snapshotCollection.Add(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, err
 		}
@@ -333,7 +333,7 @@ func apiSnapshotsUpdate(c *gin.Context) {
 			snapshot.Description = b.Description
 		}
 
-		err = collectionFactory.SnapshotCollection().Update(snapshot)
+		err = collectionFactory.SnapshotCollection().Update(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -360,7 +360,7 @@ func apiSnapshotsShow(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(snapshot)
+	err = collection.LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -452,20 +452,20 @@ func apiSnapshotsDiff(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(snapshotA)
+	err = collection.LoadComplete(snapshotA, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
 	}
 
-	err = collection.LoadComplete(snapshotB)
+	err = collection.LoadComplete(snapshotB, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
 	}
 
 	// Calculate diff
-	diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection())
+	diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection(), nil)
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -508,7 +508,7 @@ func apiSnapshotsSearchPackages(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(snapshot)
+	err = collection.LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -555,7 +555,7 @@ func apiSnapshotsMerge(c *gin.Context) {
 	}
 
 	if len(body.Sources) < 1 {
-		AbortWithJSONError(c, http.StatusBadRequest, fmt.Errorf("minimum one source snapshot is required"))
+		AbortWithJSONError(c, http.StatusBadRequest, fmt.Errorf("At least one source snapshot is required"))
 		return
 	}
 
@@ -584,13 +584,13 @@ func apiSnapshotsMerge(c *gin.Context) {
 	}
 
 	maybeRunTaskInBackground(c, "Merge snapshot "+name, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		err = snapshotCollection.LoadComplete(sources[0])
+		err = snapshotCollection.LoadComplete(sources[0], collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
 		result := sources[0].RefList()
 		for i := 1; i < len(sources); i++ {
-			err = snapshotCollection.LoadComplete(sources[i])
+			err = snapshotCollection.LoadComplete(sources[i], collectionFactory.RefListCollection())
 			if err != nil {
 				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 			}
@@ -609,7 +609,7 @@ func apiSnapshotsMerge(c *gin.Context) {
 		snapshot = deb.NewSnapshotFromRefList(name, sources, result,
 			fmt.Sprintf("Merged from sources: %s", strings.Join(sourceDescription, ", ")))
 
-		err = collectionFactory.SnapshotCollection().Add(snapshot)
+		err = collectionFactory.SnapshotCollection().Add(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to create snapshot: %s", err)
 		}
@@ -765,7 +765,7 @@ func apiSnapshotsPull(c *gin.Context) {
 		addedPackages := []string{}
 		alreadySeen := map[string]bool{}
 
-		_ = destinationPackageList.ForEachIndexed(func(pkg *deb.Package) error {
+		destinationPackageList.ForEachIndexed(func(pkg *deb.Package) error {
 			key := pkg.Architecture + "_" + pkg.Name
 			_, seen := alreadySeen[key]
 
@@ -781,7 +781,7 @@ func apiSnapshotsPull(c *gin.Context) {
 
 			// If !allMatches, add only first matching name-arch package
 			if !seen || allMatches {
-				_ = toPackageList.Add(pkg)
+				toPackageList.Add(pkg)
 				addedPackages = append(addedPackages, pkg.String())
 			}
 

aptly/conf.go

@@ -1,4 +1,4 @@
 package aptly
 
-// AptlyConf holds the default aptly.conf (filled in at link time)
+// Default aptly.conf (filled in at link time)
 var AptlyConf []byte

azure/package_pool.go

@@ -29,7 +29,7 @@ func NewPackagePool(accountName, accountKey, container, prefix, endpoint string)
 	return &PackagePool{az: azctx}, nil
 }
 
-// String returns the storage as string
+// String
 func (pool *PackagePool) String() string {
 	return pool.az.String()
 }
@@ -104,7 +104,7 @@ func (pool *PackagePool) Open(path string) (aptly.ReadSeekerCloser, error) {
 	if err != nil {
 		return nil, errors.Wrapf(err, "error creating tempfile for %s", path)
 	}
-	defer func () { _ = os.Remove(temp.Name()) }()
+	defer os.Remove(temp.Name())
 
 	_, err = pool.az.client.DownloadFile(context.TODO(), pool.az.container, path, temp, nil)
 	if err != nil {
@@ -156,7 +156,7 @@ func (pool *PackagePool) Import(srcPath, basename string, checksums *utils.Check
 	if err != nil {
 		return "", err
 	}
-	defer func() { _ = source.Close() }()
+	defer source.Close()
 
 	err = pool.az.putFile(path, source, checksums.MD5)
 	if err != nil {

azure/package_pool_test.go

@@ -2,7 +2,7 @@ package azure
 
 import (
 	"context"
-	"io"
+	"io/ioutil"
 	"os"
 	"path/filepath"
 	"runtime"
@@ -69,8 +69,8 @@ func (s *PackagePoolSuite) TestFilepathList(c *C) {
 	c.Check(err, IsNil)
 	c.Check(list, DeepEquals, []string{})
 
-	_, _ = s.pool.Import(s.debFile, "a.deb", &utils.ChecksumInfo{}, false, s.cs)
-	_, _ = s.pool.Import(s.debFile, "b.deb", &utils.ChecksumInfo{}, false, s.cs)
+	s.pool.Import(s.debFile, "a.deb", &utils.ChecksumInfo{}, false, s.cs)
+	s.pool.Import(s.debFile, "b.deb", &utils.ChecksumInfo{}, false, s.cs)
 
 	list, err = s.pool.FilepathList(nil)
 	c.Check(err, IsNil)
@@ -81,8 +81,8 @@ func (s *PackagePoolSuite) TestFilepathList(c *C) {
 }
 
 func (s *PackagePoolSuite) TestRemove(c *C) {
-	_, _ = s.pool.Import(s.debFile, "a.deb", &utils.ChecksumInfo{}, false, s.cs)
-	_, _ = s.pool.Import(s.debFile, "b.deb", &utils.ChecksumInfo{}, false, s.cs)
+	s.pool.Import(s.debFile, "a.deb", &utils.ChecksumInfo{}, false, s.cs)
+	s.pool.Import(s.debFile, "b.deb", &utils.ChecksumInfo{}, false, s.cs)
 
 	size, err := s.pool.Remove("c7/6b/4bd12fd92e4dfe1b55b18a67a669_a.deb")
 	c.Check(err, IsNil)
@@ -247,7 +247,7 @@ func (s *PackagePoolSuite) TestOpen(c *C) {
 
 	f, err := s.pool.Open(path)
 	c.Assert(err, IsNil)
-	contents, err := io.ReadAll(f)
+	contents, err := ioutil.ReadAll(f)
 	c.Assert(err, IsNil)
 	c.Check(len(contents), Equals, 2738)
 	c.Check(f.Close(), IsNil)

azure/public.go

@@ -18,7 +18,7 @@ import (
 
 // PublishedStorage abstract file system with published files (actually hosted on Azure)
 type PublishedStorage struct {
-	// FIXME: unused ???? prefix    string
+	prefix    string
 	az        *azContext
 	pathCache map[string]map[string]string
 }
@@ -38,7 +38,7 @@ func NewPublishedStorage(accountName, accountKey, container, prefix, endpoint st
 	return &PublishedStorage{az: azctx}, nil
 }
 
-// String returns the storage as string
+// String
 func (storage *PublishedStorage) String() string {
 	return storage.az.String()
 }
@@ -65,7 +65,7 @@ func (storage *PublishedStorage) PutFile(path string, sourceFilename string) err
 	if err != nil {
 		return err
 	}
-	defer func() { _ = source.Close() }()
+	defer source.Close()
 
 	err = storage.az.putFile(path, source, sourceMD5)
 	if err != nil {
@@ -158,7 +158,7 @@ func (storage *PublishedStorage) LinkFromPool(publishedPrefix, publishedRelPath,
 	if err != nil {
 		return err
 	}
-	defer func() { _ = source.Close() }()
+	defer source.Close()
 
 	err = storage.az.putFile(relFilePath, source, sourceMD5)
 	if err == nil {
@@ -193,9 +193,7 @@ func (storage *PublishedStorage) internalCopyOrMoveBlob(src, dst string, metadat
 	if err != nil {
 		return fmt.Errorf("error acquiring lease on source blob %s", src)
 	}
-	defer func() {
-		_, _ = blobLeaseClient.BreakLease(context.Background(), &lease.BlobBreakOptions{BreakPeriod: to.Ptr(int32(60))})
-	}()
+	defer blobLeaseClient.BreakLease(context.Background(), &lease.BlobBreakOptions{BreakPeriod: to.Ptr(int32(60))})
 
 	dstBlobClient := containerClient.NewBlobClient(dst)
 	copyResp, err := dstBlobClient.StartCopyFromURL(context.Background(), srcBlobClient.URL(), &blob.StartCopyFromURLOptions{

azure/public_test.go

@@ -4,7 +4,7 @@ import (
 	"context"
 	"crypto/md5"
 	"crypto/rand"
-	"io"
+	"io/ioutil"
 	"os"
 	"path/filepath"
         "bytes"
@@ -36,7 +36,7 @@ func randString(n int) string {
 	}
 	const alphanum = "0123456789abcdefghijklmnopqrstuvwxyz"
 	var bytes = make([]byte, n)
-	_, _ = rand.Read(bytes)
+	rand.Read(bytes)
 	for i, b := range bytes {
 		bytes[i] = alphanum[b%byte(len(alphanum))]
 	}
@@ -87,7 +87,7 @@ func (s *PublishedStorageSuite) TearDownTest(c *C) {
 func (s *PublishedStorageSuite) GetFile(c *C, path string) []byte {
         resp, err := s.storage.az.client.DownloadStream(context.Background(), s.storage.az.container, path, nil)
 	c.Assert(err, IsNil)
-	data, err := io.ReadAll(resp.Body)
+	data, err := ioutil.ReadAll(resp.Body)
 	c.Assert(err, IsNil)
 	return data
 }
@@ -121,7 +121,7 @@ func (s *PublishedStorageSuite) TestPutFile(c *C) {
 	filename := "a/b.txt"
 
 	dir := c.MkDir()
-	err := os.WriteFile(filepath.Join(dir, "a"), content, 0644)
+	err := ioutil.WriteFile(filepath.Join(dir, "a"), content, 0644)
 	c.Assert(err, IsNil)
 
 	err = s.storage.PutFile(filename, filepath.Join(dir, "a"))
@@ -140,7 +140,7 @@ func (s *PublishedStorageSuite) TestPutFilePlus(c *C) {
 	filename := "a/b+c.txt"
 
 	dir := c.MkDir()
-	err := os.WriteFile(filepath.Join(dir, "a"), content, 0644)
+	err := ioutil.WriteFile(filepath.Join(dir, "a"), content, 0644)
 	c.Assert(err, IsNil)
 
 	err = s.storage.PutFile(filename, filepath.Join(dir, "a"))
@@ -258,7 +258,7 @@ func (s *PublishedStorageSuite) TestRemoveDirsPlus(c *C) {
 
 func (s *PublishedStorageSuite) TestRenameFile(c *C) {
 	dir := c.MkDir()
-	err := os.WriteFile(filepath.Join(dir, "a"), []byte("Welcome to Azure!"), 0644)
+	err := ioutil.WriteFile(filepath.Join(dir, "a"), []byte("Welcome to Azure!"), 0644)
 	c.Assert(err, IsNil)
 
 	err = s.storage.PutFile("source.txt", filepath.Join(dir, "a"))
@@ -280,18 +280,18 @@ func (s *PublishedStorageSuite) TestLinkFromPool(c *C) {
 	cs := files.NewMockChecksumStorage()
 
 	tmpFile1 := filepath.Join(c.MkDir(), "mars-invaders_1.03.deb")
-	err := os.WriteFile(tmpFile1, []byte("Contents"), 0644)
+	err := ioutil.WriteFile(tmpFile1, []byte("Contents"), 0644)
 	c.Assert(err, IsNil)
 	cksum1 := utils.ChecksumInfo{MD5: "c1df1da7a1ce305a3b60af9d5733ac1d"}
 
 	tmpFile2 := filepath.Join(c.MkDir(), "mars-invaders_1.03.deb")
-	err = os.WriteFile(tmpFile2, []byte("Spam"), 0644)
+	err = ioutil.WriteFile(tmpFile2, []byte("Spam"), 0644)
 	c.Assert(err, IsNil)
 	cksum2 := utils.ChecksumInfo{MD5: "e9dfd31cc505d51fc26975250750deab"}
 
 	tmpFile3 := filepath.Join(c.MkDir(), "netboot/boot.img.gz")
-	_ = os.MkdirAll(filepath.Dir(tmpFile3), 0777)
-	err = os.WriteFile(tmpFile3, []byte("Contents"), 0644)
+	os.MkdirAll(filepath.Dir(tmpFile3), 0777)
+	err = ioutil.WriteFile(tmpFile3, []byte("Contents"), 0644)
 	c.Assert(err, IsNil)
 	cksum3 := utils.ChecksumInfo{MD5: "c1df1da7a1ce305a3b60af9d5733ac1d"}
 

cmd/api_serve.go

@@ -46,7 +46,7 @@ func aptlyAPIServe(cmd *commander.Command, args []string) error {
 	}
 	if err == nil && len(listeners) == 1 {
 		listener := listeners[0]
-		defer func() { _ = listener.Close() }()
+		defer listener.Close()
 		fmt.Printf("\nTaking over web server at: %s (press Ctrl+C to quit)...\n", listener.Addr().String())
 		err = http.Serve(listener, api.Router(context))
 		if err != nil {
@@ -67,7 +67,7 @@ func aptlyAPIServe(cmd *commander.Command, args []string) error {
 		if _, ok := <-sigchan; ok {
 			fmt.Printf("\nShutdown signal received, waiting for background tasks...\n")
 			context.TaskList().Wait()
-			_ = server.Shutdown(stdcontext.Background())
+			server.Shutdown(stdcontext.Background())
 		}
 	})()
 	defer close(sigchan)
@@ -75,14 +75,14 @@ func aptlyAPIServe(cmd *commander.Command, args []string) error {
 	listenURL, err := url.Parse(listen)
 	if err == nil && listenURL.Scheme == "unix" {
 		file := listenURL.Path
-		_ = os.Remove(file)
+		os.Remove(file)
 
 		var listener net.Listener
 		listener, err = net.Listen("unix", file)
 		if err != nil {
 			return fmt.Errorf("failed to listen on: %s\n%s", file, err)
 		}
-		defer func() { _ = listener.Close() }()
+		defer listener.Close()
 
 		err = server.Serve(listener)
 	} else {

cmd/cmd.go

@@ -21,7 +21,7 @@ const (
 )
 
 // ListPackagesRefList shows list of packages in PackageRefList
-func ListPackagesRefList(reflist *deb.PackageRefList, collectionFactory *deb.CollectionFactory) (err error) {
+func ListPackagesRefList(reflist deb.AnyRefList, collectionFactory *deb.CollectionFactory) (err error) {
 	fmt.Printf("Packages:\n")
 
 	if reflist == nil {
@@ -97,7 +97,7 @@ package environment to new version.`,
 		Flag: *flag.NewFlagSet("aptly", flag.ExitOnError),
 		Subcommands: []*commander.Command{
 			makeCmdConfig(),
-			makeCmdDB(),
+			makeCmdDb(),
 			makeCmdGraph(),
 			makeCmdMirror(),
 			makeCmdRepo(),

cmd/config_show.go

@@ -5,7 +5,7 @@ import (
 	"fmt"
 
 	"github.com/smira/commander"
-	yaml "gopkg.in/yaml.v3"
+	"gopkg.in/yaml.v3"
 )
 
 func aptlyConfigShow(_ *commander.Command, _ []string) error {

cmd/db.go

@@ -4,13 +4,13 @@ import (
 	"github.com/smira/commander"
 )
 
-func makeCmdDB() *commander.Command {
+func makeCmdDb() *commander.Command {
 	return &commander.Command{
 		UsageLine: "db",
 		Short:     "manage aptly's internal database and package pool",
 		Subcommands: []*commander.Command{
-			makeCmdDBCleanup(),
-			makeCmdDBRecover(),
+			makeCmdDbCleanup(),
+			makeCmdDbRecover(),
 		},
 	}
 }

cmd/db_cleanup.go

@@ -6,13 +6,14 @@ import (
 	"strings"
 
 	"github.com/aptly-dev/aptly/aptly"
+	"github.com/aptly-dev/aptly/database"
 	"github.com/aptly-dev/aptly/deb"
 	"github.com/aptly-dev/aptly/utils"
 	"github.com/smira/commander"
 )
 
 // aptly db cleanup
-func aptlyDBCleanup(cmd *commander.Command, args []string) error {
+func aptlyDbCleanup(cmd *commander.Command, args []string) error {
 	var err error
 
 	if len(args) != 0 {
@@ -24,12 +25,20 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 	dryRun := context.Flags().Lookup("dry-run").Value.Get().(bool)
 	collectionFactory := context.NewCollectionFactory()
 
-	// collect information about references packages...
-	existingPackageRefs := deb.NewPackageRefList()
+	// collect information about references packages and their reflistbuckets...
+	existingPackageRefs := deb.NewSplitRefList()
+	existingBuckets := deb.NewRefListDigestSet()
 
 	// used only in verbose mode to report package use source
 	packageRefSources := map[string][]string{}
 
+	var reflistMigration *deb.RefListMigration
+	if !dryRun {
+		reflistMigration = collectionFactory.RefListCollection().NewMigration()
+	} else {
+		reflistMigration = collectionFactory.RefListCollection().NewMigrationDryRun()
+	}
+
 	context.Progress().ColoredPrintf("@{w!}Loading mirrors, local repos, snapshots and published repos...@|")
 	if verbose {
 		context.Progress().ColoredPrintf("@{y}Loading mirrors:@|")
@@ -39,21 +48,22 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 			context.Progress().ColoredPrintf("- @{g}%s@|", repo.Name)
 		}
 
-		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
-		if e != nil {
+		sl := deb.NewSplitRefList()
+		e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
+		if e != nil && e != database.ErrNotFound {
 			return e
 		}
-		if repo.RefList() != nil {
-			existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
+
+		existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+		existingBuckets.AddAllInRefList(sl)
 
 		if verbose {
 			description := fmt.Sprintf("mirror %s", repo.Name)
-				_ = repo.RefList().ForEach(func(key []byte) error {
+			sl.ForEach(func(key []byte) error {
 				packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
 				return nil
 			})
 		}
-		}
 
 		return nil
 	})
@@ -71,22 +81,24 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 			context.Progress().ColoredPrintf("- @{g}%s@|", repo.Name)
 		}
 
-		e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
-		if e != nil {
+		sl := deb.NewSplitRefList()
+		e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
+		if e != nil && e != database.ErrNotFound {
 			return e
 		}
 
-		if repo.RefList() != nil {
-			existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
+		existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+		existingBuckets.AddAllInRefList(sl)
+
+		existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
 
 		if verbose {
 			description := fmt.Sprintf("local repo %s", repo.Name)
-				_ = repo.RefList().ForEach(func(key []byte) error {
+			sl.ForEach(func(key []byte) error {
 				packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
 				return nil
 			})
 		}
-		}
 
 		return nil
 	})
@@ -104,16 +116,18 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 			context.Progress().ColoredPrintf("- @{g}%s@|", snapshot.Name)
 		}
 
-		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		sl := deb.NewSplitRefList()
+		e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, snapshot.RefKey(), reflistMigration)
 		if e != nil {
 			return e
 		}
 
-		existingPackageRefs = existingPackageRefs.Merge(snapshot.RefList(), false, true)
+		existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+		existingBuckets.AddAllInRefList(sl)
 
 		if verbose {
 			description := fmt.Sprintf("snapshot %s", snapshot.Name)
-			_ = snapshot.RefList().ForEach(func(key []byte) error {
+			sl.ForEach(func(key []byte) error {
 				packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
 				return nil
 			})
@@ -136,17 +150,21 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 		if published.SourceKind != deb.SourceLocalRepo {
 			return nil
 		}
-		e := collectionFactory.PublishedRepoCollection().LoadComplete(published, collectionFactory)
+
+		for _, component := range published.Components() {
+			sl := deb.NewSplitRefList()
+			e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, published.RefKey(component), reflistMigration)
 			if e != nil {
 				return e
 			}
 
-		for _, component := range published.Components() {
-			existingPackageRefs = existingPackageRefs.Merge(published.RefList(component), false, true)
+			existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+			existingBuckets.AddAllInRefList(sl)
+
 			if verbose {
 				description := fmt.Sprintf("published repository %s:%s/%s component %s",
 					published.Storage, published.Prefix, published.Distribution, component)
-				_ = published.RefList(component).ForEach(func(key []byte) error {
+				sl.ForEach(func(key []byte) error {
 					packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
 					return nil
 				})
@@ -160,11 +178,29 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 
 	collectionFactory.Flush()
 
+	err = reflistMigration.Flush()
+	if err != nil {
+		return err
+	}
+
+	if verbose {
+		if stats := reflistMigration.Stats(); stats.Reflists > 0 {
+			if !dryRun {
+				context.Progress().ColoredPrintf("@{w!}Split %d reflist(s) into %d bucket(s) (%d segment(s))@|",
+					stats.Reflists, stats.Buckets, stats.Segments)
+			} else {
+				context.Progress().ColoredPrintf(
+					"@{y!}Skipped splitting %d reflist(s) into %d bucket(s) (%d segment(s)), as -dry-run has been requested.@|",
+					stats.Reflists, stats.Buckets, stats.Segments)
+			}
+		}
+	}
+
 	// ... and compare it to the list of all packages
 	context.Progress().ColoredPrintf("@{w!}Loading list of all packages...@|")
 	allPackageRefs := collectionFactory.PackageCollection().AllPackageRefs()
 
-	toDelete := allPackageRefs.Subtract(existingPackageRefs)
+	toDelete := allPackageRefs.Subtract(existingPackageRefs.Flatten())
 
 	// delete packages that are no longer referenced
 	context.Progress().ColoredPrintf("@{r!}Deleting unreferenced packages (%d)...@|", toDelete.Len())
@@ -202,6 +238,32 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 		}
 	}
 
+	bucketsToDelete, err := collectionFactory.RefListCollection().AllBucketDigests()
+	if err != nil {
+		return err
+	}
+
+	bucketsToDelete.RemoveAll(existingBuckets)
+
+	context.Progress().ColoredPrintf("@{r!}Deleting unreferenced reflist buckets (%d)...@|", bucketsToDelete.Len())
+	if bucketsToDelete.Len() > 0 {
+		if !dryRun {
+			batch := db.CreateBatch()
+			err := bucketsToDelete.ForEach(func(digest []byte) error {
+				return collectionFactory.RefListCollection().UnsafeDropBucket(digest, batch)
+			})
+			if err != nil {
+				return err
+			}
+
+			if err := batch.Write(); err != nil {
+				return err
+			}
+		} else {
+			context.Progress().ColoredPrintf("@{y!}Skipped reflist deletion, as -dry-run has been requested.@|")
+		}
+	}
+
 	collectionFactory.Flush()
 
 	// now, build a list of files that should be present in Repository (package pool)
@@ -291,9 +353,9 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 	return err
 }
 
-func makeCmdDBCleanup() *commander.Command {
+func makeCmdDbCleanup() *commander.Command {
 	cmd := &commander.Command{
-		Run:       aptlyDBCleanup,
+		Run:       aptlyDbCleanup,
 		UsageLine: "cleanup",
 		Short:     "cleanup DB and package pool",
 		Long: `

cmd/db_recover.go

@@ -1,16 +1,13 @@
 package cmd
 
 import (
-	"fmt"
-
-	"github.com/aptly-dev/aptly/deb"
 	"github.com/smira/commander"
 
 	"github.com/aptly-dev/aptly/database/goleveldb"
 )
 
 // aptly db recover
-func aptlyDBRecover(cmd *commander.Command, args []string) error {
+func aptlyDbRecover(cmd *commander.Command, args []string) error {
 	var err error
 
 	if len(args) != 0 {
@@ -19,19 +16,14 @@ func aptlyDBRecover(cmd *commander.Command, args []string) error {
 	}
 
 	context.Progress().Printf("Recovering database...\n")
-	if err = goleveldb.RecoverDB(context.DBPath()); err != nil {
-		return err
-	}
-
-	context.Progress().Printf("Checking database integrity...\n")
-	err = checkIntegrity()
+	err = goleveldb.RecoverDB(context.DBPath())
 
 	return err
 }
 
-func makeCmdDBRecover() *commander.Command {
+func makeCmdDbRecover() *commander.Command {
 	cmd := &commander.Command{
-		Run:       aptlyDBRecover,
+		Run:       aptlyDbRecover,
 		UsageLine: "recover",
 		Short:     "recover DB after crash",
 		Long: `
@@ -46,36 +38,3 @@ Example:
 
 	return cmd
 }
-
-func checkIntegrity() error {
-	return context.NewCollectionFactory().LocalRepoCollection().ForEach(checkRepo)
-}
-
-func checkRepo(repo *deb.LocalRepo) error {
-	collectionFactory := context.NewCollectionFactory()
-	repos := collectionFactory.LocalRepoCollection()
-
-	err := repos.LoadComplete(repo)
-	if err != nil {
-		return fmt.Errorf("load complete repo %q: %s", repo.Name, err)
-	}
-
-	dangling, err := deb.FindDanglingReferences(repo.RefList(), collectionFactory.PackageCollection())
-	if err != nil {
-		return fmt.Errorf("find dangling references: %w", err)
-	}
-
-	if len(dangling.Refs) > 0 {
-		for _, ref := range dangling.Refs {
-			context.Progress().Printf("Removing dangling database reference %q\n", ref)
-		}
-
-		repo.UpdateRefList(repo.RefList().Subtract(dangling))
-
-		if err = repos.Update(repo); err != nil {
-			return fmt.Errorf("update repo: %w", err)
-		}
-	}
-
-	return nil
-}

cmd/graph.go

@@ -38,8 +38,8 @@ func aptlyGraph(cmd *commander.Command, args []string) error {
 	if err != nil {
 		return err
 	}
-	_ = tempfile.Close()
-	_ = os.Remove(tempfile.Name())
+	tempfile.Close()
+	os.Remove(tempfile.Name())
 
 	format := context.Flags().Lookup("format").Value.String()
 	output := context.Flags().Lookup("output").Value.String()

cmd/mirror.go

@@ -20,7 +20,7 @@ func getVerifier(flags *flag.FlagSet) (pgp.Verifier, error) {
 		verifier.AddKeyring(keyRing)
 	}
 
-	err := verifier.InitKeyring(!ignoreSignatures) // be verbose only if verifying signatures is requested
+	err := verifier.InitKeyring(ignoreSignatures == false) // be verbose only if verifying signatures is requested
 	if err != nil {
 		return nil, err
 	}

cmd/mirror_create.go

@@ -69,7 +69,7 @@ func aptlyMirrorCreate(cmd *commander.Command, args []string) error {
 	}
 
 	collectionFactory := context.NewCollectionFactory()
-	err = collectionFactory.RemoteRepoCollection().Add(repo)
+	err = collectionFactory.RemoteRepoCollection().Add(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to add mirror: %s", err)
 	}

cmd/mirror_edit.go

@@ -78,7 +78,7 @@ func aptlyMirrorEdit(cmd *commander.Command, args []string) error {
 		}
 	}
 
-	err = collectionFactory.RemoteRepoCollection().Update(repo)
+	err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to edit: %s", err)
 	}

cmd/mirror_list.go

@@ -32,7 +32,7 @@ func aptlyMirrorListTxt(cmd *commander.Command, _ []string) error {
 
 	repos := make([]string, collectionFactory.RemoteRepoCollection().Len())
 	i := 0
-	_ = collectionFactory.RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
+	collectionFactory.RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
 		if raw {
 			repos[i] = repo.Name
 		} else {
@@ -42,7 +42,7 @@ func aptlyMirrorListTxt(cmd *commander.Command, _ []string) error {
 		return nil
 	})
 
-	_ = context.CloseDatabase()
+	context.CloseDatabase()
 
 	sort.Strings(repos)
 
@@ -70,13 +70,13 @@ func aptlyMirrorListJSON(_ *commander.Command, _ []string) error {
 
 	repos := make([]*deb.RemoteRepo, context.NewCollectionFactory().RemoteRepoCollection().Len())
 	i := 0
-	_ = context.NewCollectionFactory().RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
+	context.NewCollectionFactory().RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
 		repos[i] = repo
 		i++
 		return nil
 	})
 
-	_ = context.CloseDatabase()
+	context.CloseDatabase()
 
 	sort.Slice(repos, func(i, j int) bool {
 		return repos[i].Name < repos[j].Name

cmd/mirror_rename.go

@@ -37,7 +37,7 @@ func aptlyMirrorRename(cmd *commander.Command, args []string) error {
 	}
 
 	repo.Name = newName
-	err = collectionFactory.RemoteRepoCollection().Update(repo)
+	err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to rename: %s", err)
 	}

cmd/mirror_show.go

@@ -38,7 +38,7 @@ func aptlyMirrorShowTxt(_ *commander.Command, args []string) error {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -86,7 +86,7 @@ func aptlyMirrorShowTxt(_ *commander.Command, args []string) error {
 		if repo.LastDownloadDate.IsZero() {
 			fmt.Printf("Unable to show package list, mirror hasn't been downloaded yet.\n")
 		} else {
-			_ = ListPackagesRefList(repo.RefList(), collectionFactory)
+			ListPackagesRefList(repo.RefList(), collectionFactory)
 		}
 	}
 
@@ -98,12 +98,13 @@ func aptlyMirrorShowJSON(_ *commander.Command, args []string) error {
 
 	name := args[0]
 
-	repo, err := context.NewCollectionFactory().RemoteRepoCollection().ByName(name)
+	collectionFactory := context.NewCollectionFactory()
+	repo, err := collectionFactory.RemoteRepoCollection().ByName(name)
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = context.NewCollectionFactory().RemoteRepoCollection().LoadComplete(repo)
+	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -113,13 +114,13 @@ func aptlyMirrorShowJSON(_ *commander.Command, args []string) error {
 	if withPackages {
 		if repo.RefList() != nil {
 			var list *deb.PackageList
-			list, err = deb.NewPackageListFromRefList(repo.RefList(), context.NewCollectionFactory().PackageCollection(), context.Progress())
+			list, err = deb.NewPackageListFromRefList(repo.RefList(), collectionFactory.PackageCollection(), context.Progress())
 			if err != nil {
 				return fmt.Errorf("unable to get package list: %s", err)
 			}
 
 			list.PrepareIndex()
-			_ = list.ForEachIndexed(func(p *deb.Package) error {
+			list.ForEachIndexed(func(p *deb.Package) error {
 				repo.Packages = append(repo.Packages, p.GetFullName())
 				return nil
 			})

cmd/mirror_update.go

@@ -29,7 +29,7 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to update: %s", err)
 	}
 
-	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to update: %s", err)
 	}
@@ -101,12 +101,12 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
 		err = context.ReOpenDatabase()
 		if err == nil {
 			repo.MarkAsIdle()
-			_ = collectionFactory.RemoteRepoCollection().Update(repo)
+			collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 		}
 	}()
 
 	repo.MarkAsUpdating()
-	err = collectionFactory.RemoteRepoCollection().Update(repo)
+	err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to update: %s", err)
 	}
@@ -173,7 +173,7 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
 						file, e = os.CreateTemp("", task.File.Filename)
 						if e == nil {
 							task.TempDownPath = file.Name()
-							_ = file.Close()
+							file.Close()
 						}
 					}
 					if e != nil {
@@ -261,8 +261,8 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to update: download errors:\n  %s", strings.Join(errors, "\n  "))
 	}
 
-	_ = repo.FinalizeDownload(collectionFactory, context.Progress())
-	err = collectionFactory.RemoteRepoCollection().Update(repo)
+	repo.FinalizeDownload(collectionFactory, context.Progress())
+	err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to update: %s", err)
 	}

cmd/package_search.go

@@ -40,7 +40,7 @@ func aptlyPackageSearch(cmd *commander.Command, args []string) error {
 	}
 
 	format := context.Flags().Lookup("format").Value.String()
-	_ = PrintPackageList(result, format, "")
+	PrintPackageList(result, format, "")
 
 	return err
 }

cmd/package_show.go

@@ -14,7 +14,7 @@ import (
 
 func printReferencesTo(p *deb.Package, collectionFactory *deb.CollectionFactory) (err error) {
 	err = collectionFactory.RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
-		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -30,7 +30,7 @@ func printReferencesTo(p *deb.Package, collectionFactory *deb.CollectionFactory)
 	}
 
 	err = collectionFactory.LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
-		e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -46,7 +46,7 @@ func printReferencesTo(p *deb.Package, collectionFactory *deb.CollectionFactory)
 	}
 
 	err = collectionFactory.SnapshotCollection().ForEach(func(snapshot *deb.Snapshot) error {
-		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -84,8 +84,8 @@ func aptlyPackageShow(cmd *commander.Command, args []string) error {
 	result := q.Query(collectionFactory.PackageCollection())
 
 	err = result.ForEach(func(p *deb.Package) error {
-		_ = p.Stanza().WriteTo(w, p.IsSource, false, false)
-		_ = w.Flush()
+		p.Stanza().WriteTo(w, p.IsSource, false, false)
+		w.Flush()
 		fmt.Printf("\n")
 
 		if withFiles {
@@ -109,7 +109,7 @@ func aptlyPackageShow(cmd *commander.Command, args []string) error {
 
 		if withReferences {
 			fmt.Printf("References to package:\n")
-			_ = printReferencesTo(p, collectionFactory)
+			printReferencesTo(p, collectionFactory)
 			fmt.Printf("\n")
 		}
 

cmd/publish_list.go

@@ -53,7 +53,7 @@ func aptlyPublishListTxt(cmd *commander.Command, _ []string) error {
 		return fmt.Errorf("unable to load list of repos: %s", err)
 	}
 
-	_ = context.CloseDatabase()
+	context.CloseDatabase()
 
 	sort.Strings(published)
 
@@ -99,7 +99,7 @@ func aptlyPublishListJSON(_ *commander.Command, _ []string) error {
 		return fmt.Errorf("unable to load list of repos: %s", err)
 	}
 
-	_ = context.CloseDatabase()
+	context.CloseDatabase()
 
 	sort.Slice(repos, func(i, j int) bool {
 		return repos[i].GetPath() < repos[j].GetPath()

cmd/publish_snapshot.go

@@ -49,7 +49,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
 				return fmt.Errorf("unable to publish: %s", err)
 			}
 
-			err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+			err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 			if err != nil {
 				return fmt.Errorf("unable to publish: %s", err)
 			}
@@ -85,7 +85,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
 				return fmt.Errorf("unable to publish: %s", err)
 			}
 
-			err = collectionFactory.LocalRepoCollection().LoadComplete(localRepo)
+			err = collectionFactory.LocalRepoCollection().LoadComplete(localRepo, collectionFactory.RefListCollection())
 			if err != nil {
 				return fmt.Errorf("unable to publish: %s", err)
 			}
@@ -156,7 +156,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
 
 	duplicate := collectionFactory.PublishedRepoCollection().CheckDuplicate(published)
 	if duplicate != nil {
-		_ = collectionFactory.PublishedRepoCollection().LoadComplete(duplicate, collectionFactory)
+		collectionFactory.PublishedRepoCollection().LoadComplete(duplicate, collectionFactory)
 		return fmt.Errorf("prefix/distribution already used by another published repo: %s", duplicate)
 	}
 
@@ -175,7 +175,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to publish: %s", err)
 	}
 
-	err = collectionFactory.PublishedRepoCollection().Add(published)
+	err = collectionFactory.PublishedRepoCollection().Add(published, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to save to DB: %s", err)
 	}

cmd/publish_switch.go

@@ -72,7 +72,7 @@ func aptlyPublishSwitch(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to switch: %s", err)
 		}
 
-		err = snapshotCollection.LoadComplete(snapshot)
+		err = snapshotCollection.LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to switch: %s", err)
 		}
@@ -108,7 +108,7 @@ func aptlyPublishSwitch(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to publish: %s", err)
 	}
 
-	err = collectionFactory.PublishedRepoCollection().Update(published)
+	err = collectionFactory.PublishedRepoCollection().Update(published, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to save to DB: %s", err)
 	}

cmd/publish_update.go

@@ -69,7 +69,7 @@ func aptlyPublishUpdate(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to publish: %s", err)
 	}
 
-	err = collectionFactory.PublishedRepoCollection().Update(published)
+	err = collectionFactory.PublishedRepoCollection().Update(published, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to save to DB: %s", err)
 	}

cmd/repo_add.go

@@ -28,7 +28,7 @@ func aptlyRepoAdd(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to add: %s", err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to add: %s", err)
 	}
@@ -58,9 +58,9 @@ func aptlyRepoAdd(cmd *commander.Command, args []string) error {
 
 	processedFiles = append(processedFiles, otherFiles...)
 
-	repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
+	repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
 
-	err = collectionFactory.LocalRepoCollection().Update(repo)
+	err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to save: %s", err)
 	}

cmd/repo_create.go

@@ -36,7 +36,7 @@ func aptlyRepoCreate(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to load source snapshot: %s", err)
 		}
 
-		err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to load source snapshot: %s", err)
 		}
@@ -44,7 +44,7 @@ func aptlyRepoCreate(cmd *commander.Command, args []string) error {
 		repo.UpdateRefList(snapshot.RefList())
 	}
 
-	err = collectionFactory.LocalRepoCollection().Add(repo)
+	err = collectionFactory.LocalRepoCollection().Add(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to add local repo: %s", err)
 	}

cmd/repo_edit.go

@@ -22,7 +22,7 @@ func aptlyRepoEdit(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to edit: %s", err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to edit: %s", err)
 	}
@@ -53,7 +53,7 @@ func aptlyRepoEdit(cmd *commander.Command, args []string) error {
 		}
 	}
 
-	err = collectionFactory.LocalRepoCollection().Update(repo)
+	err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to edit: %s", err)
 	}

cmd/repo_include.go

@@ -67,7 +67,7 @@ func aptlyRepoInclude(cmd *commander.Command, args []string) error {
 	_, failedFiles2, err = deb.ImportChangesFiles(
 		changesFiles, reporter, acceptUnsigned, ignoreSignatures, forceReplace, noRemoveFiles, verifier, repoTemplate,
 		context.Progress(), collectionFactory.LocalRepoCollection(), collectionFactory.PackageCollection(),
-		context.PackagePool(), collectionFactory.ChecksumCollection,
+		collectionFactory.RefListCollection(), context.PackagePool(), collectionFactory.ChecksumCollection,
 		uploaders, query.Parse)
 	failedFiles = append(failedFiles, failedFiles2...)
 

cmd/repo_list.go

@@ -32,11 +32,11 @@ func aptlyRepoListTxt(cmd *commander.Command, _ []string) error {
 	collectionFactory := context.NewCollectionFactory()
 	repos := make([]string, collectionFactory.LocalRepoCollection().Len())
 	i := 0
-	_ = collectionFactory.LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
+	collectionFactory.LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
 		if raw {
 			repos[i] = repo.Name
 		} else {
-			e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
+			e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 			if e != nil {
 				return e
 			}
@@ -47,7 +47,7 @@ func aptlyRepoListTxt(cmd *commander.Command, _ []string) error {
 		return nil
 	})
 
-	_ = context.CloseDatabase()
+	context.CloseDatabase()
 
 	sort.Strings(repos)
 
@@ -76,8 +76,9 @@ func aptlyRepoListJSON(_ *commander.Command, _ []string) error {
 
 	repos := make([]*deb.LocalRepo, context.NewCollectionFactory().LocalRepoCollection().Len())
 	i := 0
-	_ = context.NewCollectionFactory().LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
-		e := context.NewCollectionFactory().LocalRepoCollection().LoadComplete(repo)
+	context.NewCollectionFactory().LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
+		collectionFactory := context.NewCollectionFactory()
+		e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -87,7 +88,7 @@ func aptlyRepoListJSON(_ *commander.Command, _ []string) error {
 		return nil
 	})
 
-	_ = context.CloseDatabase()
+	context.CloseDatabase()
 
 	sort.Slice(repos, func(i, j int) bool {
 		return repos[i].Name < repos[j].Name

cmd/repo_move.go

@@ -25,13 +25,13 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to %s: %s", command, err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(dstRepo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(dstRepo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to %s: %s", command, err)
 	}
 
 	var (
-		srcRefList *deb.PackageRefList
+		srcRefList *deb.SplitRefList
 		srcRepo    *deb.LocalRepo
 	)
 
@@ -45,7 +45,7 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to %s: source and destination are the same", command)
 		}
 
-		err = collectionFactory.LocalRepoCollection().LoadComplete(srcRepo)
+		err = collectionFactory.LocalRepoCollection().LoadComplete(srcRepo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to %s: %s", command, err)
 		}
@@ -59,12 +59,12 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to %s: %s", command, err)
 		}
 
-		err = collectionFactory.RemoteRepoCollection().LoadComplete(srcRemoteRepo)
+		err = collectionFactory.RemoteRepoCollection().LoadComplete(srcRemoteRepo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to %s: %s", command, err)
 		}
 
-		if srcRemoteRepo.RefList() == nil {
+		if srcRemoteRepo.RefList().Len() == 0 {
 			return fmt.Errorf("unable to %s: mirror not updated", command)
 		}
 
@@ -161,17 +161,17 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
 	if context.Flags().Lookup("dry-run").Value.Get().(bool) {
 		context.Progress().Printf("\nChanges not saved, as dry run has been requested.\n")
 	} else {
-		dstRepo.UpdateRefList(deb.NewPackageRefListFromPackageList(dstList))
+		dstRepo.UpdateRefList(deb.NewSplitRefListFromPackageList(dstList))
 
-		err = collectionFactory.LocalRepoCollection().Update(dstRepo)
+		err = collectionFactory.LocalRepoCollection().Update(dstRepo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to save: %s", err)
 		}
 
 		if command == "move" { // nolint: goconst
-			srcRepo.UpdateRefList(deb.NewPackageRefListFromPackageList(srcList))
+			srcRepo.UpdateRefList(deb.NewSplitRefListFromPackageList(srcList))
 
-			err = collectionFactory.LocalRepoCollection().Update(srcRepo)
+			err = collectionFactory.LocalRepoCollection().Update(srcRepo, collectionFactory.RefListCollection())
 			if err != nil {
 				return fmt.Errorf("unable to save: %s", err)
 			}

cmd/repo_remove.go

@@ -24,7 +24,7 @@ func aptlyRepoRemove(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to remove: %s", err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to remove: %s", err)
 	}
@@ -54,7 +54,7 @@ func aptlyRepoRemove(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to remove: %s", err)
 	}
 
-	_ = toRemove.ForEach(func(p *deb.Package) error {
+	toRemove.ForEach(func(p *deb.Package) error {
 		list.Remove(p)
 		context.Progress().ColoredPrintf("@r[-]@| %s removed", p)
 		return nil
@@ -63,9 +63,9 @@ func aptlyRepoRemove(cmd *commander.Command, args []string) error {
 	if context.Flags().Lookup("dry-run").Value.Get().(bool) {
 		context.Progress().Printf("\nChanges not saved, as dry run has been requested.\n")
 	} else {
-		repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
+		repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
 
-		err = collectionFactory.LocalRepoCollection().Update(repo)
+		err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to save: %s", err)
 		}

cmd/repo_rename.go

@@ -32,7 +32,7 @@ func aptlyRepoRename(cmd *commander.Command, args []string) error {
 	}
 
 	repo.Name = newName
-	err = collectionFactory.LocalRepoCollection().Update(repo)
+	err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to rename: %s", err)
 	}

cmd/repo_show.go

@@ -36,7 +36,7 @@ func aptlyRepoShowTxt(_ *commander.Command, args []string) error {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -52,7 +52,7 @@ func aptlyRepoShowTxt(_ *commander.Command, args []string) error {
 
 	withPackages := context.Flags().Lookup("with-packages").Value.Get().(bool)
 	if withPackages {
-		_ = ListPackagesRefList(repo.RefList(), collectionFactory)
+		ListPackagesRefList(repo.RefList(), collectionFactory)
 	}
 
 	return err
@@ -63,12 +63,13 @@ func aptlyRepoShowJSON(_ *commander.Command, args []string) error {
 
 	name := args[0]
 
-	repo, err := context.NewCollectionFactory().LocalRepoCollection().ByName(name)
+	collectionFactory := context.NewCollectionFactory()
+	repo, err := collectionFactory.LocalRepoCollection().ByName(name)
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = context.NewCollectionFactory().LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -79,7 +80,7 @@ func aptlyRepoShowJSON(_ *commander.Command, args []string) error {
 	if withPackages {
 		if repo.RefList() != nil {
 			var list *deb.PackageList
-			list, err = deb.NewPackageListFromRefList(repo.RefList(), context.NewCollectionFactory().PackageCollection(), context.Progress())
+			list, err = deb.NewPackageListFromRefList(repo.RefList(), collectionFactory.PackageCollection(), context.Progress())
 			if err == nil {
 				packageList = list.FullNames()
 			}

cmd/snapshot_create.go

@@ -30,7 +30,7 @@ func aptlySnapshotCreate(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}
 
-		err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}
@@ -50,7 +50,7 @@ func aptlySnapshotCreate(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}
 
-		err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}
@@ -71,7 +71,7 @@ func aptlySnapshotCreate(cmd *commander.Command, args []string) error {
 		return commander.ErrCommandError
 	}
 
-	err = collectionFactory.SnapshotCollection().Add(snapshot)
+	err = collectionFactory.SnapshotCollection().Add(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to add snapshot: %s", err)
 	}

cmd/snapshot_diff.go

@@ -23,7 +23,7 @@ func aptlySnapshotDiff(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to load snapshot A: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(snapshotA)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshotA, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to load snapshot A: %s", err)
 	}
@@ -34,13 +34,13 @@ func aptlySnapshotDiff(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to load snapshot B: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(snapshotB)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshotB, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to load snapshot B: %s", err)
 	}
 
 	// Calculate diff
-	diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection())
+	diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection(), nil)
 	if err != nil {
 		return fmt.Errorf("unable to calculate diff: %s", err)
 	}

cmd/snapshot_filter.go

@@ -27,7 +27,7 @@ func aptlySnapshotFilter(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to filter: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(source)
+	err = collectionFactory.SnapshotCollection().LoadComplete(source, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to filter: %s", err)
 	}
@@ -87,7 +87,7 @@ func aptlySnapshotFilter(cmd *commander.Command, args []string) error {
 	destination := deb.NewSnapshotFromPackageList(args[1], []*deb.Snapshot{source}, result,
 		fmt.Sprintf("Filtered '%s', query was: '%s'", source.Name, strings.Join(args[2:], " ")))
 
-	err = collectionFactory.SnapshotCollection().Add(destination)
+	err = collectionFactory.SnapshotCollection().Add(destination, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to create snapshot: %s", err)
 	}

cmd/snapshot_list.go

@@ -33,7 +33,7 @@ func aptlySnapshotListTxt(cmd *commander.Command, _ []string) error {
 	collection := collectionFactory.SnapshotCollection()
 
 	if raw {
-		_ = collection.ForEachSorted(sortMethodString, func(snapshot *deb.Snapshot) error {
+		collection.ForEachSorted(sortMethodString, func(snapshot *deb.Snapshot) error {
 			fmt.Printf("%s\n", snapshot.Name)
 			return nil
 		})
@@ -68,7 +68,7 @@ func aptlySnapshotListJSON(cmd *commander.Command, _ []string) error {
 
 	jsonSnapshots := make([]*deb.Snapshot, collection.Len())
 	i := 0
-	_ = collection.ForEachSorted(sortMethodString, func(snapshot *deb.Snapshot) error {
+	collection.ForEachSorted(sortMethodString, func(snapshot *deb.Snapshot) error {
 		jsonSnapshots[i] = snapshot
 		i++
 		return nil

cmd/snapshot_merge.go

@@ -24,7 +24,7 @@ func aptlySnapshotMerge(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to load snapshot: %s", err)
 		}
 
-		err = collectionFactory.SnapshotCollection().LoadComplete(sources[i])
+		err = collectionFactory.SnapshotCollection().LoadComplete(sources[i], collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to load snapshot: %s", err)
 		}
@@ -57,7 +57,7 @@ func aptlySnapshotMerge(cmd *commander.Command, args []string) error {
 	destination := deb.NewSnapshotFromRefList(args[0], sources, result,
 		fmt.Sprintf("Merged from sources: %s", strings.Join(sourceDescription, ", ")))
 
-	err = collectionFactory.SnapshotCollection().Add(destination)
+	err = collectionFactory.SnapshotCollection().Add(destination, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to create snapshot: %s", err)
 	}

cmd/snapshot_pull.go

@@ -29,7 +29,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to pull: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to pull: %s", err)
 	}
@@ -40,7 +40,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to pull: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(source)
+	err = collectionFactory.SnapshotCollection().LoadComplete(source, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to pull: %s", err)
 	}
@@ -116,7 +116,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
 
 	alreadySeen := map[string]bool{}
 
-	_ = result.ForEachIndexed(func(pkg *deb.Package) error {
+	result.ForEachIndexed(func(pkg *deb.Package) error {
 		key := pkg.Architecture + "_" + pkg.Name
 		_, seen := alreadySeen[key]
 
@@ -132,7 +132,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
 
 		// If !allMatches, add only first matching name-arch package
 		if !seen || allMatches {
-			_ = packageList.Add(pkg)
+			packageList.Add(pkg)
 			context.Progress().ColoredPrintf("@g[+]@| %s added", pkg)
 		}
 
@@ -149,7 +149,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
 		destination := deb.NewSnapshotFromPackageList(args[2], []*deb.Snapshot{snapshot, source}, packageList,
 			fmt.Sprintf("Pulled into '%s' with '%s' as source, pull request was: '%s'", snapshot.Name, source.Name, strings.Join(args[3:], " ")))
 
-		err = collectionFactory.SnapshotCollection().Add(destination)
+		err = collectionFactory.SnapshotCollection().Add(destination, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}

cmd/snapshot_rename.go

@@ -32,7 +32,7 @@ func aptlySnapshotRename(cmd *commander.Command, args []string) error {
 	}
 
 	snapshot.Name = newName
-	err = collectionFactory.SnapshotCollection().Update(snapshot)
+	err = collectionFactory.SnapshotCollection().Update(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to rename: %s", err)
 	}

cmd/snapshot_search.go

@@ -25,7 +25,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 	command := cmd.Parent.Name()
 	collectionFactory := context.NewCollectionFactory()
 
-	var reflist *deb.PackageRefList
+	var reflist *deb.SplitRefList
 
 	if command == "snapshot" { // nolint: goconst
 		var snapshot *deb.Snapshot
@@ -34,7 +34,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 			return fmt.Errorf("unable to search: %s", err)
 		}
 
-		err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to search: %s", err)
 		}
@@ -47,7 +47,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 			return fmt.Errorf("unable to search: %s", err)
 		}
 
-		err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to search: %s", err)
 		}
@@ -60,7 +60,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 			return fmt.Errorf("unable to search: %s", err)
 		}
 
-		err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to search: %s", err)
 		}
@@ -123,7 +123,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 	}
 
 	format := context.Flags().Lookup("format").Value.String()
-	_ = PrintPackageList(result, format, "")
+	PrintPackageList(result, format, "")
 
 	return err
 }

cmd/snapshot_show.go

@@ -35,7 +35,7 @@ func aptlySnapshotShowTxt(_ *commander.Command, args []string) error {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -79,23 +79,24 @@ func aptlySnapshotShowTxt(_ *commander.Command, args []string) error {
 
 	withPackages := context.Flags().Lookup("with-packages").Value.Get().(bool)
 	if withPackages {
-		_ = ListPackagesRefList(snapshot.RefList(), collectionFactory)
+		ListPackagesRefList(snapshot.RefList(), collectionFactory)
 	}
 
 	return err
 }
 
 func aptlySnapshotShowJSON(_ *commander.Command, args []string) error {
+	collectionFactory := context.NewCollectionFactory()
 	var err error
 
 	name := args[0]
 
-	snapshot, err := context.NewCollectionFactory().SnapshotCollection().ByName(name)
+	snapshot, err := collectionFactory.SnapshotCollection().ByName(name)
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = context.NewCollectionFactory().SnapshotCollection().LoadComplete(snapshot)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -105,14 +106,14 @@ func aptlySnapshotShowJSON(_ *commander.Command, args []string) error {
 		for _, sourceID := range snapshot.SourceIDs {
 			if snapshot.SourceKind == deb.SourceSnapshot {
 				var source *deb.Snapshot
-				source, err = context.NewCollectionFactory().SnapshotCollection().ByUUID(sourceID)
+				source, err = collectionFactory.SnapshotCollection().ByUUID(sourceID)
 				if err != nil {
 					continue
 				}
 				snapshot.Snapshots = append(snapshot.Snapshots, source)
 			} else if snapshot.SourceKind == deb.SourceLocalRepo {
 				var source *deb.LocalRepo
-				source, err = context.NewCollectionFactory().LocalRepoCollection().ByUUID(sourceID)
+				source, err = collectionFactory.LocalRepoCollection().ByUUID(sourceID)
 				if err != nil {
 					continue
 				}
@@ -133,13 +134,13 @@ func aptlySnapshotShowJSON(_ *commander.Command, args []string) error {
 	if withPackages {
 		if snapshot.RefList() != nil {
 			var list *deb.PackageList
-			list, err = deb.NewPackageListFromRefList(snapshot.RefList(), context.NewCollectionFactory().PackageCollection(), context.Progress())
+			list, err = deb.NewPackageListFromRefList(snapshot.RefList(), collectionFactory.PackageCollection(), context.Progress())
 			if err != nil {
 				return fmt.Errorf("unable to get package list: %s", err)
 			}
 
 			list.PrepareIndex()
-			_ = list.ForEachIndexed(func(p *deb.Package) error {
+			list.ForEachIndexed(func(p *deb.Package) error {
 				snapshot.Packages = append(snapshot.Packages, p.GetFullName())
 				return nil
 			})

cmd/snapshot_verify.go

@@ -23,7 +23,7 @@ func aptlySnapshotVerify(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to verify: %s", err)
 		}
 
-		err = collectionFactory.SnapshotCollection().LoadComplete(snapshots[i])
+		err = collectionFactory.SnapshotCollection().LoadComplete(snapshots[i], collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to verify: %s", err)
 		}

cmd/task_run.go

@@ -6,7 +6,7 @@ import (
 	"os"
 	"strings"
 
-	shellwords "github.com/mattn/go-shellwords"
+	"github.com/mattn/go-shellwords"
 	"github.com/smira/commander"
 )
 
@@ -31,7 +31,7 @@ func aptlyTaskRun(cmd *commander.Command, args []string) error {
 		if err != nil {
 			return err
 		}
-		defer func() { _ = file.Close() }()
+		defer file.Close()
 
 		scanner := bufio.NewScanner(file)
 

completion.d/aptly

@@ -22,36 +22,34 @@
 
 __aptly_mirror_list()
 {
-  aptly ${aptly_global_opts[@]} mirror list -raw
+  aptly mirror list -raw
 }
 
 __aptly_repo_list()
 {
-  aptly ${aptly_global_opts[@]} repo list -raw
+  aptly repo list -raw
 }
 
 __aptly_snapshot_list()
 {
-  aptly ${aptly_global_opts[@]} snapshot list -raw
+  aptly snapshot list -raw
 }
 
 __aptly_published_distributions()
 {
-  aptly ${aptly_global_opts[@]} publish list -raw | cut -d ' ' -f 2 | sort | uniq
+  aptly publish list -raw | cut -d ' ' -f 2 | sort | uniq
 }
 
 __aptly_published_prefixes()
 {
-  aptly ${aptly_global_opts[@]} publish list -raw | cut -d ' ' -f 1 | sort | uniq
+  aptly publish list -raw | cut -d ' ' -f 1 | sort | uniq
 }
 
 __aptly_prefixes_for_distribution()
 {
-  aptly ${aptly_global_opts[@]} publish list -raw | awk -v dist="$1" '{ if (dist == $2) print $1 }' | sort | uniq
+  aptly publish list -raw | awk -v dist="$1" '{ if (dist == $2) print $1 }' | sort | uniq
 }
 
-
-
 _aptly()
 {
     cur="${COMP_WORDS[COMP_CWORD]}"
@@ -59,12 +57,7 @@ _aptly()
     prevprev="${COMP_WORDS[COMP_CWORD-2]}"
 
     commands="api config db graph mirror package publish repo serve snapshot task version"
-
-    options="-architectures -config -db-open-attempts -dep-follow-all-variants -dep-follow-recommends -dep-follow-source -dep-follow-suggests -dep-verbose-resolve -gpg-provider"
-    options_without_arg="-dep-follow-all-variants -dep-follow-recommends -dep-follow-source -dep-follow-suggests -dep-verbose-resolve"
-    options_with_arg="-architectures -db-open-attempts -gpg-provider"
-    options_with_path_arg="-config"
-
+    options="-architectures= -config= -db-open-attempts= -dep-follow-all-variants -dep-follow-recommends -dep-follow-source -dep-follow-suggests -dep-verbose-resolve -gpg-provider="
     db_subcommands="cleanup recover"
     mirror_subcommands="create drop edit show list rename search update"
     publish_subcommands="drop list repo snapshot switch update source"
@@ -76,41 +69,12 @@ _aptly()
     config_subcommands="show"
     api_subcommands="serve"
 
-    local cmd subcmd numargs numoptions i aptly_global_opts
+    local cmd subcmd numargs numoptions i
 
     numargs=0
     numoptions=0
 
-    for opt in "${options_with_path_arg[@]}"; do
-        [[ "$prev" == "$opt" ]] || continue
-        compopt -o filenames 2>/dev/null
-        _filedir
-        return 0
-    done
-
     for (( i=1; i < $COMP_CWORD; i++ )); do
-      word=${COMP_WORDS[i]}
-      if [[ "$word" == -*=* ]]; then
-        for o in "${options[@]}"; do
-          [[ ${word%%=*} == "$o" ]] && aptly_global_opts+=("$word")
-        done
-      else
-        for o in "${options_with_arg[@]}" ""${options_with_path_arg[@]}"" ; do
-          if [[ "$word" == "$o" ]]; then
-            if (( i + 1 < COMP_CWORD )); then
-                aptly_global_opts+=("$word" "${COMP_WORDS[i+1]}")
-            else
-                aptly_global_opts+=("$word")
-            fi
-            (( i++ ))
-            continue 2
-          fi
-        done
-      fi
-      for o in ${options_without_arg[@]}; do
-          [[ "$word" == "$o" ]] && aptly_global_opts+=("$word")
-      done
-
       if [[ -n "$cmd" ]]; then
         if [[ ! -n "$subcmd" ]]; then
           subcmd=${COMP_WORDS[i]}
@@ -375,7 +339,7 @@ _aptly()
                 if [[ "$cur" == -* ]]; then
                   COMPREPLY=($(compgen -W "-accept-unsigned -force-replace -ignore-signatures -keyring= -no-remove-files -repo= -uploaders-file=" -- ${cur}))
                 else
-                  compopt -o filenames 2>/dev/null
+                  comptopt -o filenames 2>/dev/null
                   COMPREPLY=($(compgen -f -- ${cur}))
                   return 0
                 fi

console/progress.go

@@ -6,7 +6,6 @@ import (
 	"strings"
 
 	"github.com/aptly-dev/aptly/aptly"
-	"github.com/aptly-dev/aptly/utils"
 	"github.com/cheggaaa/pb"
 	"github.com/rs/zerolog/log"
 	"github.com/wsxiaoys/terminal/color"
@@ -79,7 +78,7 @@ func (p *Progress) InitBar(count int64, isBytes bool, _ aptly.BarType) {
 	if p.bar != nil {
 		panic("bar already initialized")
 	}
-	if utils.RunningOnTerminal() {
+	if RunningOnTerminal() {
 		p.bar = pb.New(0)
 		p.bar.Total = count
 		p.bar.NotPrint = true
@@ -142,7 +141,7 @@ func (p *Progress) PrintfStdErr(msg string, a ...interface{}) {
 
 // ColoredPrintf does printf in colored way + newline
 func (p *Progress) ColoredPrintf(msg string, a ...interface{}) {
-	if utils.RunningOnTerminal() {
+	if RunningOnTerminal() {
 		p.queue <- printTask{code: codePrint, message: color.Sprintf(msg, a...) + "\n"}
 	} else {
 		// stip color marks

console/terminal.go

@@ -0,0 +1,12 @@
+package console
+
+import (
+	"syscall"
+
+	"golang.org/x/term"
+)
+
+// RunningOnTerminal checks whether stdout is terminal
+func RunningOnTerminal() bool {
+	return term.IsTerminal(syscall.Stdout)
+}

context/context.go

@@ -115,7 +115,7 @@ func (context *AptlyContext) config() *utils.ConfigStructure {
 			if err != nil {
 				fmt.Fprintf(os.Stderr, "Config file not found, creating default config at %s\n\n", homeLocation)
 
-				_ = utils.SaveConfigRaw(homeLocation, aptly.AptlyConf)
+				utils.SaveConfigRaw(homeLocation, aptly.AptlyConf)
 				err = utils.LoadConfig(homeLocation, &utils.Config)
 				if err != nil {
 					Fatal(fmt.Errorf("error loading config file %s: %s", homeLocation, err))
@@ -123,14 +123,6 @@ func (context *AptlyContext) config() *utils.ConfigStructure {
 			}
 		}
 
-		if utils.Config.LogFormat == "json" {
-			context.StructuredLogging(true)
-			utils.SetupJSONLogger(utils.Config.LogLevel, os.Stdout)
-		} else {
-			context.StructuredLogging(false)
-			utils.SetupDefaultLogger(utils.Config.LogLevel)
-		}
-
 		context.configLoaded = true
 
 	}
@@ -241,7 +233,7 @@ func (context *AptlyContext) newDownloader(progress aptly.Progress) aptly.Downlo
 		// If flag is defined prefer it to global setting
 		maxTries = maxTriesFlag.Value.Get().(int)
 	}
-	var downloader = context.config().Downloader
+	var downloader string = context.config().Downloader
 	downloaderFlag := context.flags.Lookup("downloader")
 	if downloaderFlag != nil {
 		downloader = downloaderFlag.Value.String()
@@ -303,8 +295,8 @@ func (context *AptlyContext) _database() (database.Storage, error) {
 		switch context.config().DatabaseBackend.Type {
 		case "leveldb":
 			dbPath := filepath.Join(context.config().GetRootDir(), "db")
-			if len(context.config().DatabaseBackend.DBPath) != 0 {
-				dbPath = context.config().DatabaseBackend.DBPath
+			if len(context.config().DatabaseBackend.DbPath) != 0 {
+				dbPath = context.config().DatabaseBackend.DbPath
 			}
 			context.database, err = goleveldb.NewDB(dbPath)
 		case "etcd":
@@ -452,7 +444,7 @@ func (context *AptlyContext) GetPublishedStorage(name string) aptly.PublishedSto
 		} else if strings.HasPrefix(name, "azure:") {
 			params, ok := context.config().AzurePublishRoots[name[6:]]
 			if !ok {
-				Fatal(fmt.Errorf("published Azure storage %v not configured", name[6:]))
+				Fatal(fmt.Errorf("Published Azure storage %v not configured", name[6:]))
 			}
 
 			var err error
@@ -597,17 +589,17 @@ func (context *AptlyContext) Shutdown() {
 
 	if aptly.EnableDebug {
 		if context.fileMemProfile != nil {
-			_ = pprof.WriteHeapProfile(context.fileMemProfile)
-			_ = context.fileMemProfile.Close()
+			pprof.WriteHeapProfile(context.fileMemProfile)
+			context.fileMemProfile.Close()
 			context.fileMemProfile = nil
 		}
 		if context.fileCPUProfile != nil {
 			pprof.StopCPUProfile()
-			_ = context.fileCPUProfile.Close()
+			context.fileCPUProfile.Close()
 			context.fileCPUProfile = nil
 		}
 		if context.fileMemProfile != nil {
-			_ = context.fileMemProfile.Close()
+			context.fileMemProfile.Close()
 			context.fileMemProfile = nil
 		}
 	}
@@ -615,7 +607,7 @@ func (context *AptlyContext) Shutdown() {
 		context.taskList.Stop()
 	}
 	if context.database != nil {
-		_ = context.database.Close()
+		context.database.Close()
 		context.database = nil
 	}
 	if context.downloader != nil {
@@ -660,7 +652,7 @@ func NewContext(flags *flag.FlagSet) (*AptlyContext, error) {
 			if err != nil {
 				return nil, err
 			}
-			_ = pprof.StartCPUProfile(context.fileCPUProfile)
+			pprof.StartCPUProfile(context.fileCPUProfile)
 		}
 
 		memprofile := flags.Lookup("memprofile").Value.String()
@@ -680,7 +672,7 @@ func NewContext(flags *flag.FlagSet) (*AptlyContext, error) {
 				return nil, err
 			}
 
-			_, _ = context.fileMemStats.WriteString("# Time\tHeapSys\tHeapAlloc\tHeapIdle\tHeapReleased\n")
+			context.fileMemStats.WriteString("# Time\tHeapSys\tHeapAlloc\tHeapIdle\tHeapReleased\n")
 
 			go func() {
 				var stats runtime.MemStats
@@ -690,7 +682,7 @@ func NewContext(flags *flag.FlagSet) (*AptlyContext, error) {
 				for {
 					runtime.ReadMemStats(&stats)
 					if context.fileMemStats != nil {
-						_, _ = context.fileMemStats.WriteString(fmt.Sprintf("%d\t%d\t%d\t%d\t%d\n",
+						context.fileMemStats.WriteString(fmt.Sprintf("%d\t%d\t%d\t%d\t%d\n",
 							(time.Now().UnixNano()-start)/1000000, stats.HeapSys, stats.HeapAlloc, stats.HeapIdle, stats.HeapReleased))
 						time.Sleep(interval)
 					} else {

database/database.go

@@ -48,6 +48,8 @@ type Storage interface {
 
 	CreateTemporary() (Storage, error)
 
+	GetRecommendedMaxKVSize() int
+
 	Open() error
 	Close() error
 	CompactDB() error

database/etcddb/database_test.go

@@ -14,6 +14,7 @@ func Test(t *testing.T) {
 }
 
 type EtcDDBSuite struct {
+	url string
 	db  database.Storage
 }
 
@@ -66,17 +67,17 @@ func (s *EtcDDBSuite) TestDelete(c *C) {
 func (s *EtcDDBSuite) TestByPrefix(c *C) {
 	//c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{})
 
-	_ = s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
-	_ = s.db.Put([]byte{0x80, 0x03}, []byte{0x03})
-	_ = s.db.Put([]byte{0x80, 0x02}, []byte{0x02})
+	s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
+	s.db.Put([]byte{0x80, 0x03}, []byte{0x03})
+	s.db.Put([]byte{0x80, 0x02}, []byte{0x02})
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x01}, {0x02}, {0x03}})
 	c.Check(s.db.KeysByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x80, 0x01}, {0x80, 0x02}, {0x80, 0x03}})
 
-	_ = s.db.Put([]byte{0x90, 0x01}, []byte{0x04})
+	s.db.Put([]byte{0x90, 0x01}, []byte{0x04})
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x01}, {0x02}, {0x03}})
 	c.Check(s.db.KeysByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x80, 0x01}, {0x80, 0x02}, {0x80, 0x03}})
 
-	_ = s.db.Put([]byte{0x00, 0x01}, []byte{0x05})
+	s.db.Put([]byte{0x00, 0x01}, []byte{0x05})
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x01}, {0x02}, {0x03}})
 	c.Check(s.db.KeysByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x80, 0x01}, {0x80, 0x02}, {0x80, 0x03}})
 
@@ -108,7 +109,7 @@ func (s *EtcDDBSuite) TestHasPrefix(c *C) {
 	//c.Check(s.db.HasPrefix([]byte(nil)), Equals, false)
 	//c.Check(s.db.HasPrefix([]byte{0x80}), Equals, false)
 
-	_ = s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
+	s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
 
 	c.Check(s.db.HasPrefix([]byte(nil)), Equals, true)
 	c.Check(s.db.HasPrefix([]byte{0x80}), Equals, true)
@@ -123,15 +124,13 @@ func (s *EtcDDBSuite) TestTransactionCommit(c *C) {
 		value2 = []byte("value2")
 	)
 	transaction, err := s.db.OpenTransaction()
-	c.Assert(err, IsNil)
 
 	err = s.db.Put(key, value)
 	c.Assert(err, IsNil)
 
 	c.Assert(err, IsNil)
-	_ = transaction.Put(key2, value2)
+	transaction.Put(key2, value2)
 	v, err := s.db.Get(key)
-	c.Assert(err, IsNil)
 	c.Check(v, DeepEquals, value)
         err = transaction.Delete(key)
 	c.Assert(err, IsNil)

database/etcddb/storage.go

@@ -145,7 +145,7 @@ func (s *EtcDStorage) Close() error {
 	return err
 }
 
-// Open returns the database
+// Reopen tries to open (re-open) the database
 func (s *EtcDStorage) Open() error {
 	if s.db != nil {
 		return nil

database/etcddb/transaction.go

@@ -67,7 +67,8 @@ func (t *transaction) Commit() (err error) {
 // Discard is safe to call after Commit(), it would be no-op
 func (t *transaction) Discard() {
 	t.ops = []clientv3.Op{}
-	_ = t.tmpdb.Drop()
+	t.tmpdb.Drop()
+	return
 }
 
 // transaction should implement database.Transaction

database/goleveldb/database.go

@@ -9,10 +9,13 @@ import (
 	"github.com/aptly-dev/aptly/database"
 )
 
+const blockSize = 4 * 1024
+
 func internalOpen(path string, throttleCompaction bool) (*leveldb.DB, error) {
 	o := &opt.Options{
 		Filter:                 filter.NewBloomFilter(10),
 		OpenFilesCacheCapacity: 256,
+		BlockSize:              blockSize,
 	}
 
 	if throttleCompaction {
@@ -51,8 +54,8 @@ func RecoverDB(path string) error {
 		return err
 	}
 
-	_ = db.Close()
-	_ = stor.Close()
+	db.Close()
+	stor.Close()
 
 	return nil
 }

database/goleveldb/database_test.go

@@ -119,17 +119,17 @@ func (s *LevelDBSuite) TestDelete(c *C) {
 func (s *LevelDBSuite) TestByPrefix(c *C) {
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{})
 
-	_ = s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
-	_ = s.db.Put([]byte{0x80, 0x03}, []byte{0x03})
-	_ = s.db.Put([]byte{0x80, 0x02}, []byte{0x02})
+	s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
+	s.db.Put([]byte{0x80, 0x03}, []byte{0x03})
+	s.db.Put([]byte{0x80, 0x02}, []byte{0x02})
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x01}, {0x02}, {0x03}})
 	c.Check(s.db.KeysByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x80, 0x01}, {0x80, 0x02}, {0x80, 0x03}})
 
-	_ = s.db.Put([]byte{0x90, 0x01}, []byte{0x04})
+	s.db.Put([]byte{0x90, 0x01}, []byte{0x04})
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x01}, {0x02}, {0x03}})
 	c.Check(s.db.KeysByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x80, 0x01}, {0x80, 0x02}, {0x80, 0x03}})
 
-	_ = s.db.Put([]byte{0x00, 0x01}, []byte{0x05})
+	s.db.Put([]byte{0x00, 0x01}, []byte{0x05})
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x01}, {0x02}, {0x03}})
 	c.Check(s.db.KeysByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x80, 0x01}, {0x80, 0x02}, {0x80, 0x03}})
 
@@ -161,7 +161,7 @@ func (s *LevelDBSuite) TestHasPrefix(c *C) {
 	c.Check(s.db.HasPrefix([]byte(nil)), Equals, false)
 	c.Check(s.db.HasPrefix([]byte{0x80}), Equals, false)
 
-	_ = s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
+	s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
 
 	c.Check(s.db.HasPrefix([]byte(nil)), Equals, true)
 	c.Check(s.db.HasPrefix([]byte{0x80}), Equals, true)
@@ -180,8 +180,8 @@ func (s *LevelDBSuite) TestBatch(c *C) {
 	c.Assert(err, IsNil)
 
 	batch := s.db.CreateBatch()
-	_ = batch.Put(key2, value2)
-	_ = batch.Delete(key)
+	batch.Put(key2, value2)
+	batch.Delete(key)
 
 	v, err := s.db.Get(key)
 	c.Check(err, IsNil)
@@ -202,9 +202,9 @@ func (s *LevelDBSuite) TestBatch(c *C) {
 }
 
 func (s *LevelDBSuite) TestCompactDB(c *C) {
-	_ = s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
-	_ = s.db.Put([]byte{0x80, 0x03}, []byte{0x03})
-	_ = s.db.Put([]byte{0x80, 0x02}, []byte{0x02})
+	s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
+	s.db.Put([]byte{0x80, 0x03}, []byte{0x03})
+	s.db.Put([]byte{0x80, 0x02}, []byte{0x02})
 
 	c.Check(s.db.CompactDB(), IsNil)
 }

database/goleveldb/storage.go

@@ -16,6 +16,17 @@ type storage struct {
 	db   *leveldb.DB
 }
 
+func (s *storage) GetRecommendedMaxKVSize() int {
+	// The block size configured is not actually a *set* block size, but rather a
+	// *minimum*. LevelDB only checks if a block is full after a new key/value pair is
+	// written, meaning that blocks will tend to overflow a bit.
+	// Therefore, using the default block size as the max value size will ensure
+	// that a new block will only contain a single value and that the size will
+	// only ever be as large as around double the block size (if the block was
+	// nearly full before the new items were added).
+	return blockSize
+}
+
 // CreateTemporary creates new DB of the same type in temp dir
 func (s *storage) CreateTemporary() (database.Storage, error) {
 	tempdir, err := os.MkdirTemp("", "aptly")

deb/changes.go

@@ -60,14 +60,14 @@ func (c *Changes) VerifyAndParse(acceptUnsigned, ignoreSignature bool, verifier
 	if err != nil {
 		return err
 	}
-	defer func() { _ = input.Close() }()
+	defer input.Close()
 
 	isClearSigned, err := verifier.IsClearSigned(input)
 	if err != nil {
 		return err
 	}
 
-	_, _ = input.Seek(0, 0)
+	input.Seek(0, 0)
 
 	if !isClearSigned && !acceptUnsigned {
 		return fmt.Errorf(".changes file is not signed and unsigned processing hasn't been enabled")
@@ -79,7 +79,7 @@ func (c *Changes) VerifyAndParse(acceptUnsigned, ignoreSignature bool, verifier
 		if err != nil {
 			return err
 		}
-		_, _ = input.Seek(0, 0)
+		input.Seek(0, 0)
 
 		c.SignatureKeys = keyInfo.GoodKeys
 	}
@@ -91,7 +91,7 @@ func (c *Changes) VerifyAndParse(acceptUnsigned, ignoreSignature bool, verifier
 		if err != nil {
 			return err
 		}
-		defer func() { _ = text.Close() }()
+		defer text.Close()
 	} else {
 		text = input
 	}
@@ -291,7 +291,8 @@ func CollectChangesFiles(locations []string, reporter aptly.ResultReporter) (cha
 // ImportChangesFiles imports referenced files in changes files into local repository
 func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, acceptUnsigned, ignoreSignatures, forceReplace, noRemoveFiles bool,
 	verifier pgp.Verifier, repoTemplate *template.Template, progress aptly.Progress, localRepoCollection *LocalRepoCollection, packageCollection *PackageCollection,
-	pool aptly.PackagePool, checksumStorageProvider aptly.ChecksumStorageProvider, uploaders *Uploaders, parseQuery parseQuery) (processedFiles []string, failedFiles []string, err error) {
+	reflistCollection *RefListCollection, pool aptly.PackagePool, checksumStorageProvider aptly.ChecksumStorageProvider, uploaders *Uploaders,
+	parseQuery parseQuery) (processedFiles []string, failedFiles []string, err error) {
 
 	for _, path := range changesFiles {
 		var changes *Changes
@@ -307,7 +308,7 @@ func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, ac
 		if err != nil {
 			failedFiles = append(failedFiles, path)
 			reporter.Warning("unable to process file %s: %s", changes.ChangesName, err)
-			_ = changes.Cleanup()
+			changes.Cleanup()
 			continue
 		}
 
@@ -315,7 +316,7 @@ func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, ac
 		if err != nil {
 			failedFiles = append(failedFiles, path)
 			reporter.Warning("unable to process file %s: %s", changes.ChangesName, err)
-			_ = changes.Cleanup()
+			changes.Cleanup()
 			continue
 		}
 
@@ -334,7 +335,7 @@ func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, ac
 		if err != nil {
 			failedFiles = append(failedFiles, path)
 			reporter.Warning("unable to process file %s: %s", changes.ChangesName, err)
-			_ = changes.Cleanup()
+			changes.Cleanup()
 			continue
 		}
 
@@ -354,12 +355,12 @@ func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, ac
 				failedFiles = append(failedFiles, path)
 				reporter.Warning("changes file skipped due to uploaders config: %s, keys %#v: %s",
 					changes.ChangesName, changes.SignatureKeys, err)
-				_ = changes.Cleanup()
+				changes.Cleanup()
 				continue
 			}
 		}
 
-		err = localRepoCollection.LoadComplete(repo)
+		err = localRepoCollection.LoadComplete(repo, reflistCollection)
 		if err != nil {
 			return nil, nil, fmt.Errorf("unable to load repo: %s", err)
 		}
@@ -382,9 +383,9 @@ func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, ac
 			return nil, nil, fmt.Errorf("unable to import package files: %s", err)
 		}
 
-		repo.UpdateRefList(NewPackageRefListFromPackageList(list))
+		repo.UpdateRefList(NewSplitRefListFromPackageList(list))
 
-		err = localRepoCollection.Update(repo)
+		err = localRepoCollection.Update(repo, reflistCollection)
 		if err != nil {
 			return nil, nil, fmt.Errorf("unable to save: %s", err)
 		}

deb/changes_test.go

@@ -21,6 +21,7 @@ type ChangesSuite struct {
 	db                  database.Storage
 	localRepoCollection *LocalRepoCollection
 	packageCollection   *PackageCollection
+	reflistCollection   *RefListCollection
 	packagePool         aptly.PackagePool
 	checksumStorage     aptly.ChecksumStorage
 	progress            aptly.Progress
@@ -42,6 +43,7 @@ func (s *ChangesSuite) SetUpTest(c *C) {
 	s.db, _ = goleveldb.NewOpenDB(c.MkDir())
 	s.localRepoCollection = NewLocalRepoCollection(s.db)
 	s.packageCollection = NewPackageCollection(s.db)
+	s.reflistCollection = NewRefListCollection(s.db)
 
 	s.checksumStorage = files.NewMockChecksumStorage()
 	s.packagePool = files.NewPackagePool(s.Dir, false)
@@ -51,7 +53,7 @@ func (s *ChangesSuite) SetUpTest(c *C) {
 
 func (s *ChangesSuite) TearDownTest(c *C) {
 	s.progress.Shutdown()
-	_ = s.db.Close()
+	s.db.Close()
 }
 
 func (s *ChangesSuite) TestParseAndVerify(c *C) {
@@ -88,7 +90,7 @@ func (s *ChangesSuite) TestCollectChangesFiles(c *C) {
 
 func (s *ChangesSuite) TestImportChangesFiles(c *C) {
 	repo := NewLocalRepo("test", "Test Comment")
-	c.Assert(s.localRepoCollection.Add(repo), IsNil)
+	c.Assert(s.localRepoCollection.Add(repo, s.reflistCollection), IsNil)
 
 	origFailedFiles := []string{
 		"testdata/changes/calamares.changes",
@@ -108,13 +110,13 @@ func (s *ChangesSuite) TestImportChangesFiles(c *C) {
 
 	for _, path := range origFailedFiles {
 		filename := filepath.Join(s.Dir, filepath.Base(path))
-		_ = utils.CopyFile(path, filename)
+		utils.CopyFile(path, filename)
 		expectedFailedFiles = append(expectedFailedFiles, filename)
 	}
 
 	for _, path := range origProcessedFiles {
 		filename := filepath.Join(s.Dir, filepath.Base(path))
-		_ = utils.CopyFile(path, filename)
+		utils.CopyFile(path, filename)
 		expectedProcessedFiles = append(expectedProcessedFiles, filename)
 	}
 
@@ -124,7 +126,8 @@ func (s *ChangesSuite) TestImportChangesFiles(c *C) {
 	processedFiles, failedFiles, err := ImportChangesFiles(
 		append(changesFiles, "testdata/changes/notexistent.changes"),
 		s.Reporter, true, true, false, false, &NullVerifier{},
-		template.Must(template.New("test").Parse("test")), s.progress, s.localRepoCollection, s.packageCollection, s.packagePool, func(database.ReaderWriter) aptly.ChecksumStorage { return s.checksumStorage },
+		template.Must(template.New("test").Parse("test")), s.progress, s.localRepoCollection, s.packageCollection, s.reflistCollection, s.packagePool,
+		func(database.ReaderWriter) aptly.ChecksumStorage { return s.checksumStorage },
 		nil, nil)
 	c.Assert(err, IsNil)
 	c.Check(failedFiles, DeepEquals, append(expectedFailedFiles, "testdata/changes/notexistent.changes"))
@@ -133,7 +136,7 @@ func (s *ChangesSuite) TestImportChangesFiles(c *C) {
 
 func (s *ChangesSuite) TestImportDbgsymWithVersionedSourceField(c *C) {
 	repo := NewLocalRepo("test", "Test Comment")
-	c.Assert(s.localRepoCollection.Add(repo), IsNil)
+	c.Assert(s.localRepoCollection.Add(repo, s.reflistCollection), IsNil)
 
 	changesFiles, failedFiles := CollectChangesFiles(
 		[]string{"testdata/dbgsym-with-source-version"}, s.Reporter)
@@ -142,7 +145,8 @@ func (s *ChangesSuite) TestImportDbgsymWithVersionedSourceField(c *C) {
 
 	_, failedFiles, err := ImportChangesFiles(
 		changesFiles, s.Reporter, true, true, false, true, &NullVerifier{},
-		template.Must(template.New("test").Parse("test")), s.progress, s.localRepoCollection, s.packageCollection, s.packagePool, func(database.ReaderWriter) aptly.ChecksumStorage { return s.checksumStorage },
+		template.Must(template.New("test").Parse("test")), s.progress, s.localRepoCollection, s.packageCollection, s.reflistCollection, s.packagePool,
+		func(database.ReaderWriter) aptly.ChecksumStorage { return s.checksumStorage },
 		nil, nil)
 	c.Assert(err, IsNil)
 	c.Check(failedFiles, IsNil)

deb/checksum_collection_test.go

@@ -28,7 +28,7 @@ func (s *ChecksumCollectionSuite) SetUpTest(c *C) {
 }
 
 func (s *ChecksumCollectionSuite) TearDownTest(c *C) {
-	_ = s.db.Close()
+	s.db.Close()
 }
 
 func (s *ChecksumCollectionSuite) TestFlow(c *C) {

deb/collections.go

@@ -16,6 +16,7 @@ type CollectionFactory struct {
 	snapshots      *SnapshotCollection
 	localRepos     *LocalRepoCollection
 	publishedRepos *PublishedRepoCollection
+	reflists       *RefListCollection
 	checksums      *ChecksumCollection
 }
 
@@ -91,6 +92,17 @@ func (factory *CollectionFactory) PublishedRepoCollection() *PublishedRepoCollec
 	return factory.publishedRepos
 }
 
+func (factory *CollectionFactory) RefListCollection() *RefListCollection {
+	factory.Lock()
+	defer factory.Unlock()
+
+	if factory.reflists == nil {
+		factory.reflists = NewRefListCollection(factory.db)
+	}
+
+	return factory.reflists
+}
+
 // ChecksumCollection returns (or creates) new ChecksumCollection
 func (factory *CollectionFactory) ChecksumCollection(db database.ReaderWriter) aptly.ChecksumStorage {
 	factory.Lock()

deb/deb.go

@@ -17,7 +17,7 @@ import (
 	"github.com/aptly-dev/aptly/pgp"
 	"github.com/kjk/lzma"
 	"github.com/klauspost/compress/zstd"
-	xz "github.com/smira/go-xz"
+	"github.com/smira/go-xz"
 )
 
 // Source kinds
@@ -35,7 +35,7 @@ func GetControlFileFromDeb(packageFile string) (Stanza, error) {
 	if err != nil {
 		return nil, err
 	}
-	defer func() { _ = file.Close() }()
+	defer file.Close()
 
 	library := ar.NewReader(file)
 	for {
@@ -66,14 +66,14 @@ func GetControlFileFromDeb(packageFile string) (Stanza, error) {
 				if err != nil {
 					return nil, errors.Wrapf(err, "unable to ungzip %s from %s", header.Name, packageFile)
 				}
-				defer func() { _ = ungzip.Close() }()
+				defer ungzip.Close()
 				tarInput = ungzip
 			case "control.tar.xz":
 				unxz, err := xz.NewReader(bufReader)
 				if err != nil {
 					return nil, errors.Wrapf(err, "unable to unxz %s from %s", header.Name, packageFile)
 				}
-				defer func() { _ = unxz.Close() }()
+				defer unxz.Close()
 				tarInput = unxz
 			case "control.tar.zst":
 				unzstd, err := zstd.NewReader(bufReader)
@@ -116,10 +116,10 @@ func GetControlFileFromDsc(dscFile string, verifier pgp.Verifier) (Stanza, error
 	if err != nil {
 		return nil, err
 	}
-	defer func() { _ = file.Close() }()
+	defer file.Close()
 
 	isClearSigned, err := verifier.IsClearSigned(file)
-	_, _ = file.Seek(0, 0)
+	file.Seek(0, 0)
 
 	if err != nil {
 		return nil, err
@@ -132,7 +132,7 @@ func GetControlFileFromDsc(dscFile string, verifier pgp.Verifier) (Stanza, error
 		if err != nil {
 			return nil, err
 		}
-		defer func() { _ = text.Close() }()
+		defer text.Close()
 	} else {
 		text = file
 	}
@@ -181,7 +181,7 @@ func GetContentsFromDeb(file io.Reader, packageFile string) ([]string, error) {
 					if err != nil {
 						return nil, errors.Wrapf(err, "unable to ungzip data.tar.gz from %s", packageFile)
 					}
-					defer func() { _ = ungzip.Close() }()
+					defer ungzip.Close()
 					tarInput = ungzip
 				}
 			case "data.tar.bz2":
@@ -191,11 +191,11 @@ func GetContentsFromDeb(file io.Reader, packageFile string) ([]string, error) {
 				if err != nil {
 					return nil, errors.Wrapf(err, "unable to unxz data.tar.xz from %s", packageFile)
 				}
-				defer func() { _ = unxz.Close() }()
+				defer unxz.Close()
 				tarInput = unxz
 			case "data.tar.lzma":
 				unlzma := lzma.NewReader(bufReader)
-				defer func() { _ = unlzma.Close() }()
+				defer unlzma.Close()
 				tarInput = unlzma
 			case "data.tar.zst":
 				unzstd, err := zstd.NewReader(bufReader)

deb/find_dangling.go

@@ -1,45 +0,0 @@
-package deb
-
-import (
-	"errors"
-	"fmt"
-
-	"github.com/aptly-dev/aptly/database"
-)
-
-// FindDanglingReferences finds references that exist in the given PackageRefList, but not in the given PackageCollection.
-// It returns all such references, so they can be removed from the database.
-func FindDanglingReferences(reflist *PackageRefList, packages *PackageCollection) (dangling *PackageRefList, err error) {
-	dangling = &PackageRefList{}
-
-	err = reflist.ForEach(func(key []byte) error {
-		ok, err := isDangling(packages, key)
-		if err != nil {
-			return err
-		}
-
-		if ok {
-			dangling.Refs = append(dangling.Refs, key)
-		}
-
-		return nil
-	})
-	if err != nil {
-		return nil, err
-	}
-
-	return dangling, nil
-}
-
-func isDangling(packages *PackageCollection, key []byte) (bool, error) {
-	_, err := packages.ByKey(key)
-	if errors.Is(err, database.ErrNotFound) {
-		return true, nil
-	}
-
-	if err != nil {
-		return false, fmt.Errorf("get reference %q: %w", key, err)
-	}
-
-	return false, nil
-}

deb/find_dangling_test.go

@@ -1,46 +0,0 @@
-package deb_test
-
-import (
-	"bytes"
-	"testing"
-
-	"github.com/aptly-dev/aptly/database/goleveldb"
-	"github.com/aptly-dev/aptly/deb"
-)
-
-func TestFindDanglingReferences(t *testing.T) {
-	reflist := deb.NewPackageRefList()
-	reflist.Refs = [][]byte{[]byte("P existing 1.2.3"), []byte("P dangling 1.2.3")}
-
-	db, _ := goleveldb.NewOpenDB(t.TempDir())
-	packages := deb.NewPackageCollection(db)
-
-	if err := packages.Update(&deb.Package{Name: "existing", Version: "1.2.3"}); err != nil {
-		t.Fatal(err)
-	}
-
-	dangling, err := deb.FindDanglingReferences(reflist, packages)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	exp := &deb.PackageRefList{
-		Refs: [][]byte{[]byte("P dangling 1.2.3")},
-	}
-
-	compareRefs(t, exp, dangling)
-}
-
-func compareRefs(t *testing.T, exp, got *deb.PackageRefList) {
-	t.Helper()
-
-	if len(exp.Refs) != len(got.Refs) {
-		t.Fatalf("refs length mismatch: exp %d, got %d", len(exp.Refs), len(got.Refs))
-	}
-
-	for i := range exp.Refs {
-		if !bytes.Equal(exp.Refs[i], got.Refs[i]) {
-			t.Fatalf("refs do not match: exp %q, got %q", exp.Refs[i], got.Refs[i])
-		}
-	}
-}

deb/format_test.go

@@ -163,7 +163,7 @@ func (s *ControlFileSuite) TestCanonicalCase(c *C) {
 func (s *ControlFileSuite) TestLongFields(c *C) {
 	f, err := os.Open("long.stanza")
 	c.Assert(err, IsNil)
-	defer func() { _ = f.Close() }()
+	defer f.Close()
 
 	r := NewControlFileReader(f, false, false)
 	stanza, e := r.ReadStanza()

deb/graph.go

@@ -12,15 +12,15 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 	var err error
 
 	graph := gographviz.NewEscape()
-	_ = graph.SetDir(true)
-	_ = graph.SetName("aptly")
+	graph.SetDir(true)
+	graph.SetName("aptly")
 
 	var labelStart string
 	var labelEnd string
 
 	switch layout {
 	case "vertical":
-		_ = graph.AddAttr("aptly", "rankdir", "LR")
+		graph.AddAttr("aptly", "rankdir", "LR")
 		labelStart = ""
 		labelEnd = ""
 	case "horizontal":
@@ -33,12 +33,12 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 	existingNodes := map[string]bool{}
 
 	err = collectionFactory.RemoteRepoCollection().ForEach(func(repo *RemoteRepo) error {
-		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
 
-		_ = graph.AddNode("aptly", repo.UUID, map[string]string{
+		graph.AddNode("aptly", repo.UUID, map[string]string{
 			"shape":     "Mrecord",
 			"style":     "filled",
 			"fillcolor": "darkgoldenrod1",
@@ -55,12 +55,12 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 	}
 
 	err = collectionFactory.LocalRepoCollection().ForEach(func(repo *LocalRepo) error {
-		e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
 
-		_ = graph.AddNode("aptly", repo.UUID, map[string]string{
+		graph.AddNode("aptly", repo.UUID, map[string]string{
 			"shape":     "Mrecord",
 			"style":     "filled",
 			"fillcolor": "mediumseagreen",
@@ -75,13 +75,13 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 		return nil, err
 	}
 
-	_ = collectionFactory.SnapshotCollection().ForEach(func(snapshot *Snapshot) error {
+	collectionFactory.SnapshotCollection().ForEach(func(snapshot *Snapshot) error {
 		existingNodes[snapshot.UUID] = true
 		return nil
 	})
 
 	err = collectionFactory.SnapshotCollection().ForEach(func(snapshot *Snapshot) error {
-		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -91,7 +91,7 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 			description = "Snapshot from repo"
 		}
 
-		_ = graph.AddNode("aptly", snapshot.UUID, map[string]string{
+		graph.AddNode("aptly", snapshot.UUID, map[string]string{
 			"shape":     "Mrecord",
 			"style":     "filled",
 			"fillcolor": "cadetblue1",
@@ -103,7 +103,7 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 			for _, uuid := range snapshot.SourceIDs {
 				_, exists := existingNodes[uuid]
 				if exists {
-					_ = graph.AddEdge(uuid, snapshot.UUID, true, nil)
+					graph.AddEdge(uuid, snapshot.UUID, true, nil)
 				}
 			}
 		}
@@ -114,8 +114,8 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 		return nil, err
 	}
 
-	_ = collectionFactory.PublishedRepoCollection().ForEach(func(repo *PublishedRepo) error {
-		_ = graph.AddNode("aptly", repo.UUID, map[string]string{
+	collectionFactory.PublishedRepoCollection().ForEach(func(repo *PublishedRepo) error {
+		graph.AddNode("aptly", repo.UUID, map[string]string{
 			"shape":     "Mrecord",
 			"style":     "filled",
 			"fillcolor": "darkolivegreen1",
@@ -127,7 +127,7 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 		for _, uuid := range repo.Sources {
 			_, exists := existingNodes[uuid]
 			if exists {
-				_ = graph.AddEdge(uuid, repo.UUID, true, nil)
+				graph.AddEdge(uuid, repo.UUID, true, nil)
 			}
 		}
 

deb/index_files.go

@@ -59,24 +59,24 @@ func (file *indexFile) Finalize(signer pgp.Signer) error {
 		if file.discardable {
 			return nil
 		}
-		_, _ = file.BufWriter()
+		file.BufWriter()
 	}
 
 	err := file.w.Flush()
 	if err != nil {
-		_ = file.tempFile.Close()
+		file.tempFile.Close()
 		return fmt.Errorf("unable to write to index file: %s", err)
 	}
 
 	if file.compressable {
 		err = utils.CompressFile(file.tempFile, file.onlyGzip || file.parent.skipBz2)
 		if err != nil {
-			_ = file.tempFile.Close()
+			file.tempFile.Close()
 			return fmt.Errorf("unable to compress index file: %s", err)
 		}
 	}
 
-	_ = file.tempFile.Close()
+	file.tempFile.Close()
 
 	exts := []string{""}
 	cksumExts := exts
@@ -220,11 +220,11 @@ func packageIndexByHash(file *indexFile, ext string, hash string, sum string) er
 				// If we managed to resolve the link target: delete it. This is the
 				// oldest physical index file we no longer need. Once we drop our
 				// old symlink we'll essentially forget about it existing at all.
-				_ = file.parent.publishedStorage.Remove(linkTarget)
+				file.parent.publishedStorage.Remove(linkTarget)
 			}
-			_ = file.parent.publishedStorage.Remove(oldIndexPath)
+			file.parent.publishedStorage.Remove(oldIndexPath)
 		}
-		_ = file.parent.publishedStorage.RenameFile(indexPath, oldIndexPath)
+		file.parent.publishedStorage.RenameFile(indexPath, oldIndexPath)
 	}
 
 	// create symlink

deb/list.go

@@ -91,7 +91,7 @@ func NewPackageListWithDuplicates(duplicates bool, capacity int) *PackageList {
 }
 
 // NewPackageListFromRefList loads packages list from PackageRefList
-func NewPackageListFromRefList(reflist *PackageRefList, collection *PackageCollection, progress aptly.Progress) (*PackageList, error) {
+func NewPackageListFromRefList(reflist AnyRefList, collection *PackageCollection, progress aptly.Progress) (*PackageList, error) {
 	// empty reflist
 	if reflist == nil {
 		return NewPackageList(), nil
@@ -438,7 +438,7 @@ func (l *PackageList) Scan(q PackageQuery) (result *PackageList) {
 	result = NewPackageListWithDuplicates(l.duplicatesAllowed, 0)
 	for _, pkg := range l.packages {
 		if q.Matches(pkg) {
-			_ = result.Add(pkg)
+			result.Add(pkg)
 		}
 	}
 
@@ -456,7 +456,7 @@ func (l *PackageList) SearchByKey(arch, name, version string) (result *PackageLi
 
 	pkg := l.packages["P"+arch+" "+name+" "+version]
 	if pkg != nil {
-		_ = result.Add(pkg)
+		result.Add(pkg)
 	}
 
 	return

deb/list_test.go

@@ -96,7 +96,7 @@ func (s *PackageListSuite) SetUpTest(c *C) {
 		{Name: "dpkg", Version: "1.7", Architecture: "source", SourceArchitecture: "any", IsSource: true, deps: &PackageDependencies{}},
 	}
 	for _, p := range s.packages {
-		_ = s.il.Add(p)
+		s.il.Add(p)
 	}
 	s.il.PrepareIndex()
 
@@ -110,7 +110,7 @@ func (s *PackageListSuite) SetUpTest(c *C) {
 		{Name: "app", Version: "3.0", Architecture: "amd64", deps: &PackageDependencies{PreDepends: []string{"dpkg >= 1.6)"}, Depends: []string{"lib (>> 0.9)", "data (>= 1.0)"}}},
 	}
 	for _, p := range s.packages2 {
-		_ = s.il2.Add(p)
+		s.il2.Add(p)
 	}
 	s.il2.PrepareIndex()
 
@@ -202,8 +202,8 @@ func (s *PackageListSuite) TestRemoveWhenIndexed(c *C) {
 }
 
 func (s *PackageListSuite) TestForeach(c *C) {
-	_ = s.list.Add(s.p1)
-	_ = s.list.Add(s.p3)
+	s.list.Add(s.p1)
+	s.list.Add(s.p3)
 
 	Len := 0
 	err := s.list.ForEach(func(*Package) error {
@@ -232,21 +232,21 @@ func (s *PackageListSuite) TestIndex(c *C) {
 }
 
 func (s *PackageListSuite) TestAppend(c *C) {
-	_ = s.list.Add(s.p1)
-	_ = s.list.Add(s.p3)
+	s.list.Add(s.p1)
+	s.list.Add(s.p3)
 
 	err := s.list.Append(s.il)
 	c.Check(err, IsNil)
 	c.Check(s.list.Len(), Equals, 16)
 
 	list := NewPackageList()
-	_ = list.Add(s.p4)
+	list.Add(s.p4)
 
 	err = s.list.Append(list)
 	c.Check(err, ErrorMatches, "package already exists and is different: .*")
 
 	s.list.PrepareIndex()
-	c.Check(func() { _ = s.list.Append(s.il) }, Panics, "Append not supported when indexed")
+	c.Check(func() { s.list.Append(s.il) }, Panics, "Append not supported when indexed")
 }
 
 func (s *PackageListSuite) TestSearch(c *C) {
@@ -312,7 +312,7 @@ func (s *PackageListSuite) TestSearch(c *C) {
 
 func (s *PackageListSuite) TestFilter(c *C) {
 	c.Check(func() {
-		_, _ = s.list.Filter(FilterOptions{
+		s.list.Filter(FilterOptions{
 			Queries: []PackageQuery{&PkgQuery{"abcd", "0.3", "i386"}},
 		})
 	}, Panics, "list not indexed, can't filter")
@@ -479,7 +479,7 @@ func (s *PackageListSuite) TestVerifyDependencies(c *C) {
 		{Pkg: "mail-agent", Relation: VersionDontCare, Version: "", Architecture: "arm"}})
 
 	for _, p := range s.sourcePackages {
-		_ = s.il.Add(p)
+		s.il.Add(p)
 	}
 
 	missing, err = s.il.VerifyDependencies(DepFollowSource, []string{"i386", "amd64"}, s.il, nil)

deb/local.go

@@ -26,7 +26,7 @@ type LocalRepo struct {
 	// Uploaders configuration
 	Uploaders *Uploaders `codec:"Uploaders,omitempty" json:"-"`
 	// "Snapshot" of current list of packages
-	packageRefs *PackageRefList
+	packageRefs *SplitRefList
 }
 
 // NewLocalRepo creates new instance of Debian local repository
@@ -55,13 +55,13 @@ func (repo *LocalRepo) NumPackages() int {
 }
 
 // RefList returns package list for repo
-func (repo *LocalRepo) RefList() *PackageRefList {
+func (repo *LocalRepo) RefList() *SplitRefList {
 	return repo.packageRefs
 }
 
 // UpdateRefList changes package list for local repo
-func (repo *LocalRepo) UpdateRefList(reflist *PackageRefList) {
-	repo.packageRefs = reflist
+func (repo *LocalRepo) UpdateRefList(sl *SplitRefList) {
+	repo.packageRefs = sl
 }
 
 // Encode does msgpack encoding of LocalRepo
@@ -69,7 +69,7 @@ func (repo *LocalRepo) Encode() []byte {
 	var buf bytes.Buffer
 
 	encoder := codec.NewEncoder(&buf, &codec.MsgpackHandle{})
-	_ = encoder.Encode(repo)
+	encoder.Encode(repo)
 
 	return buf.Bytes()
 }
@@ -116,7 +116,7 @@ func (collection *LocalRepoCollection) search(filter func(*LocalRepo) bool, uniq
 		return result
 	}
 
-	_ = collection.db.ProcessByPrefix([]byte("L"), func(_, blob []byte) error {
+	collection.db.ProcessByPrefix([]byte("L"), func(_, blob []byte) error {
 		r := &LocalRepo{}
 		if err := r.Decode(blob); err != nil {
 			log.Printf("Error decoding local repo: %s\n", err)
@@ -140,14 +140,14 @@ func (collection *LocalRepoCollection) search(filter func(*LocalRepo) bool, uniq
 }
 
 // Add appends new repo to collection and saves it
-func (collection *LocalRepoCollection) Add(repo *LocalRepo) error {
+func (collection *LocalRepoCollection) Add(repo *LocalRepo, reflistCollection *RefListCollection) error {
 	_, err := collection.ByName(repo.Name)
 
 	if err == nil {
 		return fmt.Errorf("local repo with name %s already exists", repo.Name)
 	}
 
-	err = collection.Update(repo)
+	err = collection.Update(repo, reflistCollection)
 	if err != nil {
 		return err
 	}
@@ -157,27 +157,25 @@ func (collection *LocalRepoCollection) Add(repo *LocalRepo) error {
 }
 
 // Update stores updated information about repo in DB
-func (collection *LocalRepoCollection) Update(repo *LocalRepo) error {
+func (collection *LocalRepoCollection) Update(repo *LocalRepo, reflistCollection *RefListCollection) error {
 	batch := collection.db.CreateBatch()
-	_ = batch.Put(repo.Key(), repo.Encode())
+	batch.Put(repo.Key(), repo.Encode())
 	if repo.packageRefs != nil {
-		_ = batch.Put(repo.RefKey(), repo.packageRefs.Encode())
+		bc := reflistCollection.NewBatch(batch)
+		reflistCollection.UpdateInBatch(repo.packageRefs, repo.RefKey(), bc)
 	}
 	return batch.Write()
 }
 
 // LoadComplete loads additional information for local repo
-func (collection *LocalRepoCollection) LoadComplete(repo *LocalRepo) error {
-	encoded, err := collection.db.Get(repo.RefKey())
+func (collection *LocalRepoCollection) LoadComplete(repo *LocalRepo, reflistCollection *RefListCollection) error {
+	repo.packageRefs = NewSplitRefList()
+	err := reflistCollection.LoadComplete(repo.packageRefs, repo.RefKey())
 	if err == database.ErrNotFound {
 		return nil
 	}
-	if err != nil {
-		return err
-	}
 
-	repo.packageRefs = &PackageRefList{}
-	return repo.packageRefs.Decode(encoded)
+	return err
 }
 
 // ByName looks up repository by name
@@ -247,7 +245,7 @@ func (collection *LocalRepoCollection) Drop(repo *LocalRepo) error {
 	delete(collection.cache, repo.UUID)
 
 	batch := collection.db.CreateBatch()
-	_ = batch.Delete(repo.Key())
-	_ = batch.Delete(repo.RefKey())
+	batch.Delete(repo.Key())
+	batch.Delete(repo.RefKey())
 	return batch.Write()
 }

deb/local_test.go

@@ -12,7 +12,7 @@ import (
 type LocalRepoSuite struct {
 	db      database.Storage
 	list    *PackageList
-	reflist *PackageRefList
+	reflist *SplitRefList
 	repo    *LocalRepo
 }
 
@@ -21,17 +21,17 @@ var _ = Suite(&LocalRepoSuite{})
 func (s *LocalRepoSuite) SetUpTest(c *C) {
 	s.db, _ = goleveldb.NewOpenDB(c.MkDir())
 	s.list = NewPackageList()
-	_ = s.list.Add(&Package{Name: "lib", Version: "1.7", Architecture: "i386"})
-	_ = s.list.Add(&Package{Name: "app", Version: "1.9", Architecture: "amd64"})
+	s.list.Add(&Package{Name: "lib", Version: "1.7", Architecture: "i386"})
+	s.list.Add(&Package{Name: "app", Version: "1.9", Architecture: "amd64"})
 
-	s.reflist = NewPackageRefListFromPackageList(s.list)
+	s.reflist = NewSplitRefListFromPackageList(s.list)
 
 	s.repo = NewLocalRepo("lrepo", "Super repo")
 	s.repo.packageRefs = s.reflist
 }
 
 func (s *LocalRepoSuite) TearDownTest(c *C) {
-	_ = s.db.Close()
+	s.db.Close()
 }
 
 func (s *LocalRepoSuite) TestString(c *C) {
@@ -77,8 +77,9 @@ func (s *LocalRepoSuite) TestRefKey(c *C) {
 type LocalRepoCollectionSuite struct {
 	db                database.Storage
 	collection        *LocalRepoCollection
+	reflistCollection *RefListCollection
 	list              *PackageList
-	reflist    *PackageRefList
+	reflist           *SplitRefList
 }
 
 var _ = Suite(&LocalRepoCollectionSuite{})
@@ -86,16 +87,17 @@ var _ = Suite(&LocalRepoCollectionSuite{})
 func (s *LocalRepoCollectionSuite) SetUpTest(c *C) {
 	s.db, _ = goleveldb.NewOpenDB(c.MkDir())
 	s.collection = NewLocalRepoCollection(s.db)
+	s.reflistCollection = NewRefListCollection(s.db)
 
 	s.list = NewPackageList()
-	_ = s.list.Add(&Package{Name: "lib", Version: "1.7", Architecture: "i386"})
-	_ = s.list.Add(&Package{Name: "app", Version: "1.9", Architecture: "amd64"})
+	s.list.Add(&Package{Name: "lib", Version: "1.7", Architecture: "i386"})
+	s.list.Add(&Package{Name: "app", Version: "1.9", Architecture: "amd64"})
 
-	s.reflist = NewPackageRefListFromPackageList(s.list)
+	s.reflist = NewSplitRefListFromRefList(NewPackageRefListFromPackageList(s.list))
 }
 
 func (s *LocalRepoCollectionSuite) TearDownTest(c *C) {
-	_ = s.db.Close()
+	s.db.Close()
 }
 
 func (s *LocalRepoCollectionSuite) TestAddByName(c *C) {
@@ -103,8 +105,8 @@ func (s *LocalRepoCollectionSuite) TestAddByName(c *C) {
 	c.Assert(err, ErrorMatches, "*.not found")
 
 	repo := NewLocalRepo("local1", "Comment 1")
-	c.Assert(s.collection.Add(repo), IsNil)
-	c.Assert(s.collection.Add(repo), ErrorMatches, ".*already exists")
+	c.Assert(s.collection.Add(repo, s.reflistCollection), IsNil)
+	c.Assert(s.collection.Add(repo, s.reflistCollection), ErrorMatches, ".*already exists")
 
 	r, err := s.collection.ByName("local1")
 	c.Assert(err, IsNil)
@@ -121,7 +123,7 @@ func (s *LocalRepoCollectionSuite) TestByUUID(c *C) {
 	c.Assert(err, ErrorMatches, "*.not found")
 
 	repo := NewLocalRepo("local1", "Comment 1")
-	c.Assert(s.collection.Add(repo), IsNil)
+	c.Assert(s.collection.Add(repo, s.reflistCollection), IsNil)
 
 	r, err := s.collection.ByUUID(repo.UUID)
 	c.Assert(err, IsNil)
@@ -135,7 +137,7 @@ func (s *LocalRepoCollectionSuite) TestByUUID(c *C) {
 
 func (s *LocalRepoCollectionSuite) TestUpdateLoadComplete(c *C) {
 	repo := NewLocalRepo("local1", "Comment 1")
-	c.Assert(s.collection.Update(repo), IsNil)
+	c.Assert(s.collection.Update(repo, s.reflistCollection), IsNil)
 
 	collection := NewLocalRepoCollection(s.db)
 	r, err := collection.ByName("local1")
@@ -143,20 +145,20 @@ func (s *LocalRepoCollectionSuite) TestUpdateLoadComplete(c *C) {
 	c.Assert(r.packageRefs, IsNil)
 
 	repo.packageRefs = s.reflist
-	c.Assert(s.collection.Update(repo), IsNil)
+	c.Assert(s.collection.Update(repo, s.reflistCollection), IsNil)
 
 	collection = NewLocalRepoCollection(s.db)
 	r, err = collection.ByName("local1")
 	c.Assert(err, IsNil)
 	c.Assert(r.packageRefs, IsNil)
 	c.Assert(r.NumPackages(), Equals, 0)
-	c.Assert(s.collection.LoadComplete(r), IsNil)
+	c.Assert(s.collection.LoadComplete(r, s.reflistCollection), IsNil)
 	c.Assert(r.NumPackages(), Equals, 2)
 }
 
 func (s *LocalRepoCollectionSuite) TestForEachAndLen(c *C) {
 	repo := NewLocalRepo("local1", "Comment 1")
-	_ = s.collection.Add(repo)
+	s.collection.Add(repo, s.reflistCollection)
 
 	count := 0
 	err := s.collection.ForEach(func(*LocalRepo) error {
@@ -178,10 +180,10 @@ func (s *LocalRepoCollectionSuite) TestForEachAndLen(c *C) {
 
 func (s *LocalRepoCollectionSuite) TestDrop(c *C) {
 	repo1 := NewLocalRepo("local1", "Comment 1")
-	_ = s.collection.Add(repo1)
+	s.collection.Add(repo1, s.reflistCollection)
 
 	repo2 := NewLocalRepo("local2", "Comment 2")
-	_ = s.collection.Add(repo2)
+	s.collection.Add(repo2, s.reflistCollection)
 
 	r1, _ := s.collection.ByUUID(repo1.UUID)
 	c.Check(r1, Equals, repo1)
@@ -208,6 +210,6 @@ func (s *LocalRepoCollectionSuite) TestDropNonExisting(c *C) {
 	_, err := s.collection.ByUUID(repo.UUID)
 	c.Check(err, ErrorMatches, "local repo .* not found")
 
-	_ = s.collection.Drop(repo)
+	err = s.collection.Drop(repo)
 	c.Check(s.collection.Drop(repo), ErrorMatches, "local repo not found")
 }

deb/package.go

@@ -565,7 +565,7 @@ func (p *Package) CalculateContents(packagePool aptly.PackagePool, progress aptl
 		}
 		return nil, err
 	}
-	defer func() { _ = reader.Close() }()
+	defer reader.Close()
 
 	contents, err := GetContentsFromDeb(reader, file.Filename)
 	if err != nil {

deb/package_collection.go

@@ -309,7 +309,7 @@ func (collection *PackageCollection) Scan(q PackageQuery) (result *PackageList)
 		}
 
 		if q.Matches(pkg) {
-			_ = result.Add(pkg)
+			result.Add(pkg)
 		}
 	}
 
@@ -337,7 +337,7 @@ func (collection *PackageCollection) SearchByKey(arch, name, version string) (re
 		}
 
 		if pkg.Architecture == arch && pkg.Name == name && pkg.Version == version {
-			_ = result.Add(pkg)
+			result.Add(pkg)
 		}
 	}
 

deb/package_collection_test.go

@@ -23,7 +23,7 @@ func (s *PackageCollectionSuite) SetUpTest(c *C) {
 }
 
 func (s *PackageCollectionSuite) TearDownTest(c *C) {
-	_ = s.db.Close()
+	s.db.Close()
 }
 
 func (s *PackageCollectionSuite) TestUpdate(c *C) {
@@ -67,7 +67,7 @@ func (s *PackageCollectionSuite) TestByKey(c *C) {
 
 func (s *PackageCollectionSuite) TestByKeyOld0_3(c *C) {
 	key := []byte("Pi386 vmware-view-open-client 4.5.0-297975+dfsg-4+b1")
-	_ = s.db.Put(key, old0_3Package)
+	s.db.Put(key, old0_3Package)
 
 	p, err := s.collection.ByKey(key)
 	c.Check(err, IsNil)

deb/package_files.go

@@ -64,7 +64,7 @@ func (files PackageFiles) Hash() uint64 {
 
 	for _, f := range files {
 		h.Write([]byte(f.Filename))
-		_ = binary.Write(h, binary.BigEndian, f.Checksums.Size)
+		binary.Write(h, binary.BigEndian, f.Checksums.Size)
 		h.Write([]byte(f.Checksums.MD5))
 		h.Write([]byte(f.Checksums.SHA1))
 		h.Write([]byte(f.Checksums.SHA256))

deb/package_files_test.go

@@ -1,7 +1,7 @@
 package deb
 
 import (
-	"os"
+	"io/ioutil"
 	"path/filepath"
 
 	"github.com/aptly-dev/aptly/aptly"
@@ -39,7 +39,7 @@ func (s *PackageFilesSuite) TestVerify(c *C) {
 	c.Check(result, Equals, false)
 
 	tmpFilepath := filepath.Join(c.MkDir(), "file")
-	c.Assert(os.WriteFile(tmpFilepath, []byte("abcde"), 0777), IsNil)
+	c.Assert(ioutil.WriteFile(tmpFilepath, []byte("abcde"), 0777), IsNil)
 
 	s.files[0].PoolPath, _ = packagePool.Import(tmpFilepath, s.files[0].Filename, &s.files[0].Checksums, false, s.cs)
 

deb/package_test.go

@@ -2,7 +2,7 @@ package deb
 
 import (
 	"bytes"
-	"os"
+	"io/ioutil"
 	"path/filepath"
 	"regexp"
 
@@ -395,7 +395,7 @@ func (s *PackageSuite) TestLinkFromPool(c *C) {
 	p := NewPackageFromControlFile(s.stanza)
 
 	tmpFilepath := filepath.Join(c.MkDir(), "file")
-	c.Assert(os.WriteFile(tmpFilepath, nil, 0777), IsNil)
+	c.Assert(ioutil.WriteFile(tmpFilepath, nil, 0777), IsNil)
 
 	p.Files()[0].PoolPath, _ = packagePool.Import(tmpFilepath, p.Files()[0].Filename, &p.Files()[0].Checksums, false, cs)
 
@@ -434,7 +434,7 @@ func (s *PackageSuite) TestDownloadList(c *C) {
 	})
 
 	tmpFilepath := filepath.Join(c.MkDir(), "file")
-	c.Assert(os.WriteFile(tmpFilepath, []byte("abcde"), 0777), IsNil)
+	c.Assert(ioutil.WriteFile(tmpFilepath, []byte("abcde"), 0777), IsNil)
 	p.Files()[0].PoolPath, _ = packagePool.Import(tmpFilepath, p.Files()[0].Filename, &p.Files()[0].Checksums, false, cs)
 
 	list, err = p.DownloadList(packagePool, cs)
@@ -449,7 +449,7 @@ func (s *PackageSuite) TestVerifyFiles(c *C) {
 	cs := files.NewMockChecksumStorage()
 
 	tmpFilepath := filepath.Join(c.MkDir(), "file")
-	c.Assert(os.WriteFile(tmpFilepath, []byte("abcde"), 0777), IsNil)
+	c.Assert(ioutil.WriteFile(tmpFilepath, []byte("abcde"), 0777), IsNil)
 
 	p.Files()[0].PoolPath, _ = packagePool.Import(tmpFilepath, p.Files()[0].Filename, &p.Files()[0].Checksums, false, cs)
 

deb/publish.go

@@ -37,7 +37,7 @@ type repoSourceItem struct {
 	// Pointer to local repo if SourceKind == "local"
 	localRepo *LocalRepo
 	// Package references is SourceKind == "local"
-	packageRefs *PackageRefList
+	packageRefs *SplitRefList
 }
 
 // PublishedRepo is a published for http/ftp representation of snapshot as Debian repository
@@ -609,7 +609,7 @@ func (p *PublishedRepo) RefKey(component string) []byte {
 }
 
 // RefList returns list of package refs in local repo
-func (p *PublishedRepo) RefList(component string) *PackageRefList {
+func (p *PublishedRepo) RefList(component string) *SplitRefList {
 	item := p.sourceItems[component]
 	if p.SourceKind == SourceLocalRepo {
 		return item.packageRefs
@@ -631,7 +631,7 @@ func (p *PublishedRepo) Components() []string {
 	return result
 }
 
-// SourceNames returns sorted list of published repo source names
+// Components returns sorted list of published repo source names
 func (p *PublishedRepo) SourceNames() []string {
 	var sources = []string{}
 
@@ -702,7 +702,7 @@ func (p *PublishedRepo) Encode() []byte {
 	var buf bytes.Buffer
 
 	encoder := codec.NewEncoder(&buf, &codec.MsgpackHandle{})
-	_ = encoder.Encode(p)
+	encoder.Encode(p)
 
 	return buf.Bytes()
 }
@@ -884,7 +884,7 @@ func (p *PublishedRepo) Publish(packagePool aptly.PackagePool, publishedStorageP
 	if err != nil {
 		return err
 	}
-	defer func() { _ = os.RemoveAll(tempDir) }()
+	defer os.RemoveAll(tempDir)
 
 	indexes := newIndexFiles(publishedStorage, basePath, tempDir, suffix, p.AcquireByHash, p.SkipBz2)
 
@@ -970,7 +970,7 @@ func (p *PublishedRepo) Publish(packagePool aptly.PackagePool, publishedStorageP
 								contentIndexesMap[key] = contentIndex
 							}
 
-							_ = contentIndex.Push(qualifiedName, contents, batch)
+							contentIndex.Push(qualifiedName, contents, batch)
 						}
 					}
 
@@ -1243,14 +1243,14 @@ func (collection *PublishedRepoCollection) loadList() {
 }
 
 // Add appends new repo to collection and saves it
-func (collection *PublishedRepoCollection) Add(repo *PublishedRepo) error {
+func (collection *PublishedRepoCollection) Add(repo *PublishedRepo, reflistCollection *RefListCollection) error {
 	collection.loadList()
 
 	if collection.CheckDuplicate(repo) != nil {
 		return fmt.Errorf("published repo with storage/prefix/distribution %s/%s/%s already exists", repo.Storage, repo.Prefix, repo.Distribution)
 	}
 
-	err := collection.Update(repo)
+	err := collection.Update(repo, reflistCollection)
 	if err != nil {
 		return err
 	}
@@ -1273,13 +1273,14 @@ func (collection *PublishedRepoCollection) CheckDuplicate(repo *PublishedRepo) *
 }
 
 // Update stores updated information about repo in DB
-func (collection *PublishedRepoCollection) Update(repo *PublishedRepo) error {
+func (collection *PublishedRepoCollection) Update(repo *PublishedRepo, reflistCollection *RefListCollection) error {
 	batch := collection.db.CreateBatch()
-	_ = batch.Put(repo.Key(), repo.Encode())
+	batch.Put(repo.Key(), repo.Encode())
 
 	if repo.SourceKind == SourceLocalRepo {
+		rb := reflistCollection.NewBatch(batch)
 		for component, item := range repo.sourceItems {
-			_ = batch.Put(repo.RefKey(component), item.packageRefs.Encode())
+			reflistCollection.UpdateInBatch(item.packageRefs, repo.RefKey(component), rb)
 		}
 	}
 	return batch.Write()
@@ -1312,7 +1313,7 @@ func (collection *PublishedRepoCollection) LoadShallow(repo *PublishedRepo, coll
 				return
 			}
 
-			item.packageRefs = &PackageRefList{}
+			item.packageRefs = NewSplitRefList()
 			repo.sourceItems[component] = item
 		}
 	} else {
@@ -1324,39 +1325,33 @@ func (collection *PublishedRepoCollection) LoadShallow(repo *PublishedRepo, coll
 
 // LoadComplete loads complete information on the sources of the repo *and* their packages
 func (collection *PublishedRepoCollection) LoadComplete(repo *PublishedRepo, collectionFactory *CollectionFactory) (err error) {
-	_ = collection.LoadShallow(repo, collectionFactory)
+	collection.LoadShallow(repo, collectionFactory)
 
 	if repo.SourceKind == SourceSnapshot {
 		for _, item := range repo.sourceItems {
-			err = collectionFactory.SnapshotCollection().LoadComplete(item.snapshot)
+			err = collectionFactory.SnapshotCollection().LoadComplete(item.snapshot, collectionFactory.RefListCollection())
 			if err != nil {
 				return
 			}
 		}
 	} else if repo.SourceKind == SourceLocalRepo {
 		for component, item := range repo.sourceItems {
-			err = collectionFactory.LocalRepoCollection().LoadComplete(item.localRepo)
+			err = collectionFactory.LocalRepoCollection().LoadComplete(item.localRepo, collectionFactory.RefListCollection())
 			if err != nil {
 				return
 			}
 
-			var encoded []byte
-			encoded, err = collection.db.Get(repo.RefKey(component))
+			err = collectionFactory.RefListCollection().LoadComplete(item.packageRefs, repo.RefKey(component))
 			if err != nil {
 				// < 0.6 saving w/o component name
 				if err == database.ErrNotFound && len(repo.Sources) == 1 {
-					encoded, err = collection.db.Get(repo.RefKey(""))
+					err = collectionFactory.RefListCollection().LoadComplete(item.packageRefs, repo.RefKey(""))
 				}
 
 				if err != nil {
 					return
 				}
 			}
-
-			err = item.packageRefs.Decode(encoded)
-			if err != nil {
-				return
-			}
 		}
 	} else {
 		panic("unknown SourceKind")
@@ -1461,6 +1456,11 @@ func (collection *PublishedRepoCollection) listReferencedFilesByComponent(prefix
 	referencedFiles := map[string][]string{}
 	processedComponentRefs := map[string]*PackageRefList{}
 
+	processedComponentBuckets := map[string]*RefListDigestSet{}
+	for _, component := range components {
+		processedComponentBuckets[component] = NewRefListDigestSet()
+	}
+
 	for _, r := range collection.list {
 		if r.Prefix == prefix && !r.MultiDist {
 			matches := false
@@ -1484,7 +1484,15 @@ func (collection *PublishedRepoCollection) listReferencedFilesByComponent(prefix
 
 			for _, component := range components {
 				if utils.StrSliceHasItem(repoComponents, component) {
-					unseenRefs := r.RefList(component)
+					processedBuckets := processedComponentBuckets[component]
+
+					err := r.RefList(component).ForEachBucket(func(digest []byte, bucket *PackageRefList) error {
+						if processedBuckets.Has(digest) {
+							return nil
+						}
+						processedBuckets.Add(digest)
+
+						unseenRefs := bucket
 						processedRefs := processedComponentRefs[component]
 						if processedRefs != nil {
 							unseenRefs = unseenRefs.Subtract(processedRefs)
@@ -1493,16 +1501,16 @@ func (collection *PublishedRepoCollection) listReferencedFilesByComponent(prefix
 						}
 
 						if unseenRefs.Len() == 0 {
-						continue
+							return nil
 						}
 						processedComponentRefs[component] = processedRefs.Merge(unseenRefs, false, true)
 
 						packageList, err := NewPackageListFromRefList(unseenRefs, collectionFactory.PackageCollection(), progress)
 						if err != nil {
-						return nil, err
+							return err
 						}
 
-					_ = packageList.ForEach(func(p *Package) error {
+						packageList.ForEach(func(p *Package) error {
 							poolDir, err := p.PoolDirectory()
 							if err != nil {
 								return err
@@ -1514,6 +1522,13 @@ func (collection *PublishedRepoCollection) listReferencedFilesByComponent(prefix
 
 							return nil
 						})
+
+						return nil
+					})
+
+					if err != nil {
+						return nil, err
+					}
 				}
 			}
 		}
@@ -1575,7 +1590,7 @@ func (collection *PublishedRepoCollection) CleanupPrefixComponentFiles(published
 				return err
 			}
 
-			_ = packageList.ForEach(func(p *Package) error {
+			packageList.ForEach(func(p *Package) error {
 				poolDir, err := p.PoolDirectory()
 				if err != nil {
 					return err
@@ -1709,10 +1724,10 @@ func (collection *PublishedRepoCollection) Remove(publishedStorageProvider aptly
 	}
 
 	batch := collection.db.CreateBatch()
-	_ = batch.Delete(repo.Key())
+	batch.Delete(repo.Key())
 
 	for _, component := range repo.Components() {
-		_ = batch.Delete(repo.RefKey(component))
+		batch.Delete(repo.RefKey(component))
 	}
 
 	return batch.Write()

deb/publish_bench_test.go

@@ -19,18 +19,19 @@ func BenchmarkListReferencedFiles(b *testing.B) {
 	if err != nil {
 		b.Fatal(err)
 	}
-	defer func() { _ = os.RemoveAll(tmpDir) }()
+	defer os.RemoveAll(tmpDir)
 
 	db, err := goleveldb.NewOpenDB(tmpDir)
 	if err != nil {
 		b.Fatal(err)
 	}
-	defer func() { _ = db.Close() }()
+	defer db.Close()
 
 	factory := NewCollectionFactory(db)
 	packageCollection := factory.PackageCollection()
 	repoCollection := factory.LocalRepoCollection()
 	publishCollection := factory.PublishedRepoCollection()
+	reflistCollection := factory.RefListCollection()
 
 	sharedRefs := NewPackageRefList()
 	{
@@ -49,7 +50,7 @@ func BenchmarkListReferencedFiles(b *testing.B) {
 				Filename: fmt.Sprintf("pkg-shared_%d.deb", pkgIndex),
 			}})
 
-			_ = packageCollection.UpdateInTransaction(p, transaction)
+			packageCollection.UpdateInTransaction(p, transaction)
 			sharedRefs.Refs = append(sharedRefs.Refs, p.Key(""))
 		}
 
@@ -78,7 +79,7 @@ func BenchmarkListReferencedFiles(b *testing.B) {
 				Filename: fmt.Sprintf("pkg%d_%d.deb", repoIndex, pkgIndex),
 			}})
 
-			_ = packageCollection.UpdateInTransaction(p, transaction)
+			packageCollection.UpdateInTransaction(p, transaction)
 			refs.Refs = append(refs.Refs, p.Key(""))
 		}
 
@@ -91,17 +92,17 @@ func BenchmarkListReferencedFiles(b *testing.B) {
 		repo := NewLocalRepo(fmt.Sprintf("repo%d", repoIndex), "comment")
 		repo.DefaultDistribution = fmt.Sprintf("dist%d", repoIndex)
 		repo.DefaultComponent = defaultComponent
-		repo.UpdateRefList(refs.Merge(sharedRefs, false, true))
-		_ = repoCollection.Add(repo)
+		repo.UpdateRefList(NewSplitRefListFromRefList(refs.Merge(sharedRefs, false, true)))
+		repoCollection.Add(repo, reflistCollection)
 
 		publish, err := NewPublishedRepo("", "test", "", nil, []string{defaultComponent}, []interface{}{repo}, factory, false)
 		if err != nil {
 			b.Fatal(err)
 		}
-		_ = publishCollection.Add(publish)
+		publishCollection.Add(publish, reflistCollection)
 	}
 
-	_ = db.CompactDB()
+	db.CompactDB()
 
 	b.ResetTimer()
 	for i := 0; i < b.N; i++ {

deb/publish_test.go

@@ -5,6 +5,7 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
+	"io/ioutil"
 	"os"
 	"path/filepath"
 	"sort"
@@ -50,11 +51,11 @@ func (n *NullSigner) SetPassphrase(passphrase, passphraseFile string) {
 }
 
 func (n *NullSigner) DetachedSign(source string, destination string) error {
-	return os.WriteFile(destination, []byte{}, 0644)
+	return ioutil.WriteFile(destination, []byte{}, 0644)
 }
 
 func (n *NullSigner) ClearSign(source string, destination string) error {
-	return os.WriteFile(destination, []byte{}, 0644)
+	return ioutil.WriteFile(destination, []byte{}, 0644)
 }
 
 type FakeStorageProvider struct {
@@ -82,6 +83,7 @@ type PublishedRepoSuite struct {
 	db                                  database.Storage
 	factory                             *CollectionFactory
 	packageCollection                   *PackageCollection
+	reflistCollection                   *RefListCollection
 }
 
 var _ = Suite(&PublishedRepoSuite{})
@@ -103,7 +105,7 @@ func (s *PublishedRepoSuite) SetUpTest(c *C) {
 	s.cs = files.NewMockChecksumStorage()
 
 	tmpFilepath := filepath.Join(c.MkDir(), "file")
-	c.Assert(os.WriteFile(tmpFilepath, nil, 0777), IsNil)
+	c.Assert(ioutil.WriteFile(tmpFilepath, nil, 0777), IsNil)
 
 	var err error
 	s.p1.Files()[0].PoolPath, err = s.packagePool.Import(tmpFilepath, s.p1.Files()[0].Filename, &s.p1.Files()[0].Checksums, false, s.cs)
@@ -113,26 +115,27 @@ func (s *PublishedRepoSuite) SetUpTest(c *C) {
 	s.p2.UpdateFiles(s.p1.Files())
 	s.p3.UpdateFiles(s.p1.Files())
 
-	s.reflist = NewPackageRefListFromPackageList(s.list)
+	s.reflist = NewSplitRefListFromPackageList(s.list)
+	s.reflistCollection = s.factory.RefListCollection()
 
 	repo, _ := NewRemoteRepo("yandex", "http://mirror.yandex.ru/debian/", "squeeze", []string{"main"}, []string{}, false, false, false)
 	repo.packageRefs = s.reflist
-	_ = s.factory.RemoteRepoCollection().Add(repo)
+	s.factory.RemoteRepoCollection().Add(repo, s.reflistCollection)
 
 	s.localRepo = NewLocalRepo("local1", "comment1")
 	s.localRepo.packageRefs = s.reflist
-	_ = s.factory.LocalRepoCollection().Add(s.localRepo)
+	s.factory.LocalRepoCollection().Add(s.localRepo, s.reflistCollection)
 
 	s.snapshot, _ = NewSnapshotFromRepository("snap", repo)
-	_ = s.factory.SnapshotCollection().Add(s.snapshot)
+	s.factory.SnapshotCollection().Add(s.snapshot, s.reflistCollection)
 
 	s.snapshot2, _ = NewSnapshotFromRepository("snap", repo)
-	_ = s.factory.SnapshotCollection().Add(s.snapshot2)
+	s.factory.SnapshotCollection().Add(s.snapshot2, s.reflistCollection)
 
 	s.packageCollection = s.factory.PackageCollection()
-	_ = s.packageCollection.Update(s.p1)
-	_ = s.packageCollection.Update(s.p2)
-	_ = s.packageCollection.Update(s.p3)
+	s.packageCollection.Update(s.p1)
+	s.packageCollection.Update(s.p2)
+	s.packageCollection.Update(s.p3)
 
 	s.repo, _ = NewPublishedRepo("", "ppa", "squeeze", nil, []string{"main"}, []interface{}{s.snapshot}, s.factory, false)
 	s.repo.SkipContents = true
@@ -151,7 +154,7 @@ func (s *PublishedRepoSuite) SetUpTest(c *C) {
 }
 
 func (s *PublishedRepoSuite) TearDownTest(c *C) {
-	_ = s.db.Close()
+	s.db.Close()
 }
 
 func (s *PublishedRepoSuite) TestNewPublishedRepo(c *C) {
@@ -178,12 +181,12 @@ func (s *PublishedRepoSuite) TestNewPublishedRepo(c *C) {
 	c.Check(s.repo3.RefList("main").Len(), Equals, 3)
 	c.Check(s.repo3.RefList("contrib").Len(), Equals, 3)
 
-	c.Check(func() { _, _ = NewPublishedRepo("", ".", "a", nil, nil, nil, s.factory, false) }, PanicMatches, "publish with empty sources")
+	c.Check(func() { NewPublishedRepo("", ".", "a", nil, nil, nil, s.factory, false) }, PanicMatches, "publish with empty sources")
 	c.Check(func() {
-		_, _ = NewPublishedRepo("", ".", "a", nil, []string{"main"}, []interface{}{s.snapshot, s.snapshot2}, s.factory, false)
+		NewPublishedRepo("", ".", "a", nil, []string{"main"}, []interface{}{s.snapshot, s.snapshot2}, s.factory, false)
 	}, PanicMatches, "sources and components should be equal in size")
 	c.Check(func() {
-		_, _ = NewPublishedRepo("", ".", "a", nil, []string{"main", "contrib"}, []interface{}{s.localRepo, s.snapshot2}, s.factory, false)
+		NewPublishedRepo("", ".", "a", nil, []string{"main", "contrib"}, []interface{}{s.localRepo, s.snapshot2}, s.factory, false)
 	}, PanicMatches, "interface conversion:.*")
 
 	_, err := NewPublishedRepo("", ".", "a", nil, []string{"main", "main"}, []interface{}{s.snapshot, s.snapshot2}, s.factory, false)
@@ -336,7 +339,7 @@ func (s *PublishedRepoSuite) TestDistributionComponentGuessing(c *C) {
 
 	s.localRepo.DefaultDistribution = "precise"
 	s.localRepo.DefaultComponent = "contrib"
-	_ = s.factory.LocalRepoCollection().Update(s.localRepo)
+	s.factory.LocalRepoCollection().Update(s.localRepo, s.reflistCollection)
 
 	repo, err = NewPublishedRepo("", "ppa", "", nil, []string{""}, []interface{}{s.localRepo}, s.factory, false)
 	c.Check(err, IsNil)
@@ -525,8 +528,8 @@ func (s *PublishedRepoSuite) TestPublishedRepoRevision(c *C) {
 	bytes, err := json.Marshal(revision)
 	c.Assert(err, IsNil)
 
-	jsonExpected := `{"Sources":[{"Component":"main","Name":"local1"},{"Component":"test1","Name":"snap1"},{"Component":"test2","Name":"snap2"}]}`
-	c.Assert(string(bytes), Equals, jsonExpected)
+	json_expected := `{"Sources":[{"Component":"main","Name":"local1"},{"Component":"test1","Name":"snap1"},{"Component":"test2","Name":"snap2"}]}`
+	c.Assert(string(bytes), Equals, json_expected)
 
 	c.Assert(s.repo2.DropRevision(), DeepEquals, revision)
 	c.Assert(s.repo2.Revision, IsNil)
@@ -537,6 +540,7 @@ type PublishedRepoCollectionSuite struct {
 	db                                database.Storage
 	factory                           *CollectionFactory
 	snapshotCollection                *SnapshotCollection
+	reflistCollection                 *RefListCollection
 	collection                        *PublishedRepoCollection
 	snap1, snap2                      *Snapshot
 	localRepo                         *LocalRepo
@@ -552,22 +556,23 @@ func (s *PublishedRepoCollectionSuite) SetUpTest(c *C) {
 	s.factory = NewCollectionFactory(s.db)
 
 	s.snapshotCollection = s.factory.SnapshotCollection()
+	s.reflistCollection = s.factory.RefListCollection()
 
 	snap1Refs := NewPackageRefList()
 	snap1Refs.Refs = [][]byte{s.p1.Key(""), s.p2.Key("")}
 	sort.Sort(snap1Refs)
-	s.snap1 = NewSnapshotFromRefList("snap1", []*Snapshot{}, snap1Refs, "desc1")
+	s.snap1 = NewSnapshotFromRefList("snap1", []*Snapshot{}, NewSplitRefListFromRefList(snap1Refs), "desc1")
 
 	snap2Refs := NewPackageRefList()
 	snap2Refs.Refs = [][]byte{s.p3.Key("")}
 	sort.Sort(snap2Refs)
-	s.snap2 = NewSnapshotFromRefList("snap2", []*Snapshot{}, snap2Refs, "desc2")
+	s.snap2 = NewSnapshotFromRefList("snap2", []*Snapshot{}, NewSplitRefListFromRefList(snap2Refs), "desc2")
 
-	_ = s.snapshotCollection.Add(s.snap1)
-	_ = s.snapshotCollection.Add(s.snap2)
+	s.snapshotCollection.Add(s.snap1, s.reflistCollection)
+	s.snapshotCollection.Add(s.snap2, s.reflistCollection)
 
 	s.localRepo = NewLocalRepo("local1", "comment1")
-	_ = s.factory.LocalRepoCollection().Add(s.localRepo)
+	s.factory.LocalRepoCollection().Add(s.localRepo, s.reflistCollection)
 
 	s.repo1, _ = NewPublishedRepo("", "ppa", "anaconda", []string{}, []string{"main"}, []interface{}{s.snap1}, s.factory, false)
 	s.repo2, _ = NewPublishedRepo("", "", "anaconda", []string{}, []string{"main", "contrib"}, []interface{}{s.snap2, s.snap1}, s.factory, false)
@@ -579,21 +584,21 @@ func (s *PublishedRepoCollectionSuite) SetUpTest(c *C) {
 }
 
 func (s *PublishedRepoCollectionSuite) TearDownTest(c *C) {
-	_ = s.db.Close()
+	s.db.Close()
 }
 
 func (s *PublishedRepoCollectionSuite) TestAddByStoragePrefixDistribution(c *C) {
 	_, err := s.collection.ByStoragePrefixDistribution("", "ppa", "anaconda")
 	c.Assert(err, ErrorMatches, "*.not found")
 
-	c.Assert(s.collection.Add(s.repo1), IsNil)
-	c.Assert(s.collection.Add(s.repo1), ErrorMatches, ".*already exists")
+	c.Assert(s.collection.Add(s.repo1, s.reflistCollection), IsNil)
+	c.Assert(s.collection.Add(s.repo1, s.reflistCollection), ErrorMatches, ".*already exists")
 	c.Assert(s.collection.CheckDuplicate(s.repo2), IsNil)
-	c.Assert(s.collection.Add(s.repo2), IsNil)
-	c.Assert(s.collection.Add(s.repo3), ErrorMatches, ".*already exists")
+	c.Assert(s.collection.Add(s.repo2, s.reflistCollection), IsNil)
+	c.Assert(s.collection.Add(s.repo3, s.reflistCollection), ErrorMatches, ".*already exists")
 	c.Assert(s.collection.CheckDuplicate(s.repo3), Equals, s.repo1)
-	c.Assert(s.collection.Add(s.repo4), IsNil)
-	c.Assert(s.collection.Add(s.repo5), IsNil)
+	c.Assert(s.collection.Add(s.repo4, s.reflistCollection), IsNil)
+	c.Assert(s.collection.Add(s.repo5, s.reflistCollection), IsNil)
 
 	r, err := s.collection.ByStoragePrefixDistribution("", "ppa", "anaconda")
 	c.Assert(err, IsNil)
@@ -619,7 +624,7 @@ func (s *PublishedRepoCollectionSuite) TestByUUID(c *C) {
 	_, err := s.collection.ByUUID(s.repo1.UUID)
 	c.Assert(err, ErrorMatches, "*.not found")
 
-	c.Assert(s.collection.Add(s.repo1), IsNil)
+	c.Assert(s.collection.Add(s.repo1, s.reflistCollection), IsNil)
 
 	r, err := s.collection.ByUUID(s.repo1.UUID)
 	c.Assert(err, IsNil)
@@ -630,8 +635,8 @@ func (s *PublishedRepoCollectionSuite) TestByUUID(c *C) {
 }
 
 func (s *PublishedRepoCollectionSuite) TestUpdateLoadComplete(c *C) {
-	c.Assert(s.collection.Update(s.repo1), IsNil)
-	c.Assert(s.collection.Update(s.repo4), IsNil)
+	c.Assert(s.collection.Update(s.repo1, s.reflistCollection), IsNil)
+	c.Assert(s.collection.Update(s.repo4, s.reflistCollection), IsNil)
 
 	collection := NewPublishedRepoCollection(s.db)
 	r, err := collection.ByStoragePrefixDistribution("", "ppa", "anaconda")
@@ -676,10 +681,10 @@ func (s *PublishedRepoCollectionSuite) TestLoadPre0_6(c *C) {
 	var buf bytes.Buffer
 
 	encoder := codec.NewEncoder(&buf, &codec.MsgpackHandle{})
-	_ = encoder.Encode(&old)
+	encoder.Encode(&old)
 
 	c.Assert(s.db.Put(s.repo1.Key(), buf.Bytes()), IsNil)
-	c.Assert(s.db.Put(s.repo1.RefKey(""), s.localRepo.RefList().Encode()), IsNil)
+	c.Assert(s.db.Put(s.repo1.RefKey(""), NewPackageRefList().Encode()), IsNil)
 
 	collection := NewPublishedRepoCollection(s.db)
 	repo, err := collection.ByStoragePrefixDistribution("", "ppa", "anaconda")
@@ -694,7 +699,7 @@ func (s *PublishedRepoCollectionSuite) TestLoadPre0_6(c *C) {
 }
 
 func (s *PublishedRepoCollectionSuite) TestForEachAndLen(c *C) {
-	_ = s.collection.Add(s.repo1)
+	s.collection.Add(s.repo1, s.reflistCollection)
 
 	count := 0
 	err := s.collection.ForEach(func(*PublishedRepo) error {
@@ -715,17 +720,17 @@ func (s *PublishedRepoCollectionSuite) TestForEachAndLen(c *C) {
 }
 
 func (s *PublishedRepoCollectionSuite) TestBySnapshot(c *C) {
-	c.Check(s.collection.Add(s.repo1), IsNil)
-	c.Check(s.collection.Add(s.repo2), IsNil)
+	c.Check(s.collection.Add(s.repo1, s.reflistCollection), IsNil)
+	c.Check(s.collection.Add(s.repo2, s.reflistCollection), IsNil)
 
 	c.Check(s.collection.BySnapshot(s.snap1), DeepEquals, []*PublishedRepo{s.repo1, s.repo2})
 	c.Check(s.collection.BySnapshot(s.snap2), DeepEquals, []*PublishedRepo{s.repo2})
 }
 
 func (s *PublishedRepoCollectionSuite) TestByLocalRepo(c *C) {
-	c.Check(s.collection.Add(s.repo1), IsNil)
-	c.Check(s.collection.Add(s.repo4), IsNil)
-	c.Check(s.collection.Add(s.repo5), IsNil)
+	c.Check(s.collection.Add(s.repo1, s.reflistCollection), IsNil)
+	c.Check(s.collection.Add(s.repo4, s.reflistCollection), IsNil)
+	c.Check(s.collection.Add(s.repo5, s.reflistCollection), IsNil)
 
 	c.Check(s.collection.ByLocalRepo(s.localRepo), DeepEquals, []*PublishedRepo{s.repo4, s.repo5})
 }
@@ -735,10 +740,10 @@ func (s *PublishedRepoCollectionSuite) TestListReferencedFiles(c *C) {
 	c.Check(s.factory.PackageCollection().Update(s.p2), IsNil)
 	c.Check(s.factory.PackageCollection().Update(s.p3), IsNil)
 
-	c.Check(s.collection.Add(s.repo1), IsNil)
-	c.Check(s.collection.Add(s.repo2), IsNil)
-	c.Check(s.collection.Add(s.repo4), IsNil)
-	c.Check(s.collection.Add(s.repo5), IsNil)
+	c.Check(s.collection.Add(s.repo1, s.reflistCollection), IsNil)
+	c.Check(s.collection.Add(s.repo2, s.reflistCollection), IsNil)
+	c.Check(s.collection.Add(s.repo4, s.reflistCollection), IsNil)
+	c.Check(s.collection.Add(s.repo5, s.reflistCollection), IsNil)
 
 	files, err := s.collection.listReferencedFilesByComponent(".", []string{"main", "contrib"}, s.factory, nil)
 	c.Assert(err, IsNil)
@@ -754,12 +759,12 @@ func (s *PublishedRepoCollectionSuite) TestListReferencedFiles(c *C) {
 	})
 
 	snap3 := NewSnapshotFromRefList("snap3", []*Snapshot{}, s.snap2.RefList(), "desc3")
-	_ = s.snapshotCollection.Add(snap3)
+	s.snapshotCollection.Add(snap3, s.reflistCollection)
 
 	// Ensure that adding a second publish point with matching files doesn't give duplicate results.
 	repo3, err := NewPublishedRepo("", "", "anaconda-2", []string{}, []string{"main"}, []interface{}{snap3}, s.factory, false)
 	c.Check(err, IsNil)
-	c.Check(s.collection.Add(repo3), IsNil)
+	c.Check(s.collection.Add(repo3, s.reflistCollection), IsNil)
 
 	files, err = s.collection.listReferencedFilesByComponent(".", []string{"main", "contrib"}, s.factory, nil)
 	c.Assert(err, IsNil)
@@ -780,6 +785,7 @@ type PublishedRepoRemoveSuite struct {
 	db                                  database.Storage
 	factory                             *CollectionFactory
 	snapshotCollection                  *SnapshotCollection
+	reflistCollection                   *RefListCollection
 	collection                          *PublishedRepoCollection
 	root, root2                         string
 	provider                            *FakeStorageProvider
@@ -795,10 +801,11 @@ func (s *PublishedRepoRemoveSuite) SetUpTest(c *C) {
 	s.factory = NewCollectionFactory(s.db)
 
 	s.snapshotCollection = s.factory.SnapshotCollection()
+	s.reflistCollection = s.factory.RefListCollection()
 
 	s.snap1 = NewSnapshotFromPackageList("snap1", []*Snapshot{}, NewPackageList(), "desc1")
 
-	_ = s.snapshotCollection.Add(s.snap1)
+	s.snapshotCollection.Add(s.snap1, s.reflistCollection)
 
 	s.repo1, _ = NewPublishedRepo("", "ppa", "anaconda", []string{}, []string{"main"}, []interface{}{s.snap1}, s.factory, false)
 	s.repo2, _ = NewPublishedRepo("", "", "anaconda", []string{}, []string{"main"}, []interface{}{s.snap1}, s.factory, false)
@@ -807,26 +814,26 @@ func (s *PublishedRepoRemoveSuite) SetUpTest(c *C) {
 	s.repo5, _ = NewPublishedRepo("files:other", "ppa", "osminog", []string{}, []string{"contrib"}, []interface{}{s.snap1}, s.factory, false)
 
 	s.collection = s.factory.PublishedRepoCollection()
-	_ = s.collection.Add(s.repo1)
-	_ = s.collection.Add(s.repo2)
-	_ = s.collection.Add(s.repo3)
-	_ = s.collection.Add(s.repo4)
-	_ = s.collection.Add(s.repo5)
+	s.collection.Add(s.repo1, s.reflistCollection)
+	s.collection.Add(s.repo2, s.reflistCollection)
+	s.collection.Add(s.repo3, s.reflistCollection)
+	s.collection.Add(s.repo4, s.reflistCollection)
+	s.collection.Add(s.repo5, s.reflistCollection)
 
 	s.root = c.MkDir()
 	s.publishedStorage = files.NewPublishedStorage(s.root, "", "")
-	_ = s.publishedStorage.MkDir("ppa/dists/anaconda")
-	_ = s.publishedStorage.MkDir("ppa/dists/meduza")
-	_ = s.publishedStorage.MkDir("ppa/dists/osminog")
-	_ = s.publishedStorage.MkDir("ppa/pool/main")
-	_ = s.publishedStorage.MkDir("ppa/pool/contrib")
-	_ = s.publishedStorage.MkDir("dists/anaconda")
-	_ = s.publishedStorage.MkDir("pool/main")
+	s.publishedStorage.MkDir("ppa/dists/anaconda")
+	s.publishedStorage.MkDir("ppa/dists/meduza")
+	s.publishedStorage.MkDir("ppa/dists/osminog")
+	s.publishedStorage.MkDir("ppa/pool/main")
+	s.publishedStorage.MkDir("ppa/pool/contrib")
+	s.publishedStorage.MkDir("dists/anaconda")
+	s.publishedStorage.MkDir("pool/main")
 
 	s.root2 = c.MkDir()
 	s.publishedStorage2 = files.NewPublishedStorage(s.root2, "", "")
-	_ = s.publishedStorage2.MkDir("ppa/dists/osminog")
-	_ = s.publishedStorage2.MkDir("ppa/pool/contrib")
+	s.publishedStorage2.MkDir("ppa/dists/osminog")
+	s.publishedStorage2.MkDir("ppa/pool/contrib")
 
 	s.provider = &FakeStorageProvider{map[string]aptly.PublishedStorage{
 		"":            s.publishedStorage,
@@ -834,11 +841,11 @@ func (s *PublishedRepoRemoveSuite) SetUpTest(c *C) {
 }
 
 func (s *PublishedRepoRemoveSuite) TearDownTest(c *C) {
-	_ = s.db.Close()
+	s.db.Close()
 }
 
 func (s *PublishedRepoRemoveSuite) TestRemoveFilesOnlyDist(c *C) {
-	_ = s.repo1.RemoveFiles(s.provider, false, []string{}, nil)
+	s.repo1.RemoveFiles(s.provider, false, []string{}, nil)
 
 	c.Check(filepath.Join(s.publishedStorage.PublicPath(), "ppa/dists/anaconda"), Not(PathExists))
 	c.Check(filepath.Join(s.publishedStorage.PublicPath(), "ppa/dists/meduza"), PathExists)
@@ -852,7 +859,7 @@ func (s *PublishedRepoRemoveSuite) TestRemoveFilesOnlyDist(c *C) {
 }
 
 func (s *PublishedRepoRemoveSuite) TestRemoveFilesWithPool(c *C) {
-	_ = s.repo1.RemoveFiles(s.provider, false, []string{"main"}, nil)
+	s.repo1.RemoveFiles(s.provider, false, []string{"main"}, nil)
 
 	c.Check(filepath.Join(s.publishedStorage.PublicPath(), "ppa/dists/anaconda"), Not(PathExists))
 	c.Check(filepath.Join(s.publishedStorage.PublicPath(), "ppa/dists/meduza"), PathExists)
@@ -866,7 +873,7 @@ func (s *PublishedRepoRemoveSuite) TestRemoveFilesWithPool(c *C) {
 }
 
 func (s *PublishedRepoRemoveSuite) TestRemoveFilesWithTwoPools(c *C) {
-	_ = s.repo1.RemoveFiles(s.provider, false, []string{"main", "contrib"}, nil)
+	s.repo1.RemoveFiles(s.provider, false, []string{"main", "contrib"}, nil)
 
 	c.Check(filepath.Join(s.publishedStorage.PublicPath(), "ppa/dists/anaconda"), Not(PathExists))
 	c.Check(filepath.Join(s.publishedStorage.PublicPath(), "ppa/dists/meduza"), PathExists)
@@ -880,7 +887,7 @@ func (s *PublishedRepoRemoveSuite) TestRemoveFilesWithTwoPools(c *C) {
 }
 
 func (s *PublishedRepoRemoveSuite) TestRemoveFilesWithPrefix(c *C) {
-	_ = s.repo1.RemoveFiles(s.provider, true, []string{"main"}, nil)
+	s.repo1.RemoveFiles(s.provider, true, []string{"main"}, nil)
 
 	c.Check(filepath.Join(s.publishedStorage.PublicPath(), "ppa/dists/anaconda"), Not(PathExists))
 	c.Check(filepath.Join(s.publishedStorage.PublicPath(), "ppa/dists/meduza"), Not(PathExists))
@@ -894,7 +901,7 @@ func (s *PublishedRepoRemoveSuite) TestRemoveFilesWithPrefix(c *C) {
 }
 
 func (s *PublishedRepoRemoveSuite) TestRemoveFilesWithPrefixRoot(c *C) {
-	_ = s.repo2.RemoveFiles(s.provider, true, []string{"main"}, nil)
+	s.repo2.RemoveFiles(s.provider, true, []string{"main"}, nil)
 
 	c.Check(filepath.Join(s.publishedStorage.PublicPath(), "ppa/dists/anaconda"), PathExists)
 	c.Check(filepath.Join(s.publishedStorage.PublicPath(), "ppa/dists/meduza"), PathExists)

deb/query.go

@@ -89,7 +89,7 @@ func (q *OrQuery) Fast(list PackageCatalog) bool {
 func (q *OrQuery) Query(list PackageCatalog) (result *PackageList) {
 	if q.Fast(list) {
 		result = q.L.Query(list)
-		_ = result.Append(q.R.Query(list))
+		result.Append(q.R.Query(list))
 	} else {
 		result = list.Scan(q)
 	}
@@ -245,7 +245,7 @@ func (q *DependencyQuery) Query(list PackageCatalog) (result *PackageList) {
 	if q.Fast(list) {
 		result = NewPackageList()
 		for _, pkg := range list.Search(q.Dep, true, true) {
-			_ = result.Add(pkg)
+			result.Add(pkg)
 		}
 	} else {
 		result = list.Scan(q)