handle packageRefs == nil

In current aptly, each repository and snapshot has its own reflist in
the database. This brings a few problems with it:

- Given a sufficiently large repositories and snapshots, these lists can
  get enormous, reaching >1MB. This is a problem for LevelDB's overall
  performance, as it tends to prefer values around the confiruged block
  size (defaults to just 4KiB).
- When you take these large repositories and snapshot them, you have a
  full, new copy of the reflist, even if only a few packages changed.
  This means that having a lot of snapshots with a few changes causes
  the database to basically be full of largely duplicate reflists.
- All the duplication also means that many of the same refs are being
  loaded repeatedly, which can cause some slowdown but, more notably,
  eats up huge amounts of memory.
- Adding on more and more new repositories and snapshots will cause the
  time and memory spent on things like cleanup and publishing to grow
  roughly linearly.

At the core, there are two problems here:

- Reflists get very big because there are just a lot of packages.
- Different reflists can tend to duplicate much of the same contents.

*Split reflists* aim at solving this by separating reflists into 64
*buckets*. Package refs are sorted into individual buckets according to
the following system:

- Take the first 3 letters of the package name, after dropping a `lib`
  prefix. (Using only the first 3 letters will cause packages with
  similar prefixes to end up in the same bucket, under the assumption
  that packages with similar names tend to be updated together.)
- Take the 64-bit xxhash of these letters. (xxhash was chosen because it
  relatively good distribution across the individual bits, which is
  important for the next step.)
- Use the first 6 bits of the hash (range [0:63]) as an index into the
  buckets.

Once refs are placed in buckets, a sha256 digest of all the refs in the
bucket is taken. These buckets are then stored in the database, split
into roughly block-sized segments, and all the repositories and
snapshots simply store an array of bucket digests.

This approach means that *repositories and snapshots can share their
reflist buckets*. If a snapshot is taken of a repository, it will have
the same contents, so its split reflist will point to the same buckets
as the base repository, and only one copy of each bucket is stored in
the database. When some packages in the repository change, only the
buckets containing those packages will be modified; all the other
buckets will remain unchanged, and thus their contents will still be
shared. Later on, when these reflists are loaded, each bucket is only
loaded once, short-cutting loaded many megabytes of data. In effect,
split reflists are essentially copy-on-write, with only the changed
buckets stored individually.

Changing the disk format means that a migration needs to take place, so
that task is moved into the database cleanup step, which will migrate
reflists over to split reflists, as well as delete any unused reflist
buckets.

All the reflist tests are also changed to additionally test out split
reflists; although the internal logic is all shared (since buckets are,
themselves, just normal reflists), some special additions are needed to
have native versions of the various reflist helper methods.

In our tests, we've observed the following improvements:

- Memory usage during publish and database cleanup, with
  `GOMEMLIMIT=2GiB`, goes down from ~3.2GiB (larger than the memory
  limit!) to ~0.7GiB, a decrease of ~4.5x.
- Database size decreases from 1.3GB to 367MB.

*In my local tests*, publish times had also decreased down to mere
seconds but the same effect wasn't observed on the server, with the
times staying around the same. My suspicions are that this is due to I/O
performance: my local system is an M1 MBP, which almost certainly has
much faster disk speeds than our DigitalOcean block volumes. Split
reflists include a side effect of requiring more random accesses from
reading all the buckets by their keys, so if your random I/O
performance is slower, it might cancel out the benefits. That being
said, even in that case, the memory usage and database size advantages
still persist.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>

.github/workflows/ci.yml

@@ -17,29 +17,6 @@ env:
   DEBIAN_FRONTEND: noninteractive
 
 jobs:
-  unit-test:
-    name: "Unit Tests (Debian 13)"
-    runs-on: ubuntu-22.04
-    continue-on-error: false
-    timeout-minutes: 30
-    steps:
-      - name: "Checkout Repository"
-        uses: actions/checkout@v4
-        with:
-          # fetch the whole repo for `git describe` to work
-          fetch-depth: 0
-      - name: "Docker Image"
-        run: |
-          make docker-image
-      - name: "Unit Test"
-        run: |
-          make docker-unit-test
-      - name: "Upload Code Coverage"
-        uses: codecov/codecov-action@v2
-        with:
-          token: ${{ secrets.CODECOV_TOKEN }}
-          files: unit.out
-
   test:
     name: "Test (Ubuntu 22.04)"
     runs-on: ubuntu-22.04
@@ -53,10 +30,10 @@ jobs:
       GOPROXY: "https://proxy.golang.org"
 
     steps:
-      - name: "Install Test Packages"
+      - name: "Install Packages"
         run: |
           sudo apt-get update
-          sudo apt-get install -y --no-install-recommends graphviz gnupg2 gpgv2 git gcc make devscripts python3 python3-requests-unixsocket python3-termcolor python3-swiftclient python3-boto python3-azure-storage python3-etcd3 python3-plyvel flake8 faketime
+          sudo apt-get install -y --no-install-recommends graphviz gnupg2 gpgv2 git gcc make devscripts python3 python3-requests-unixsocket python3-termcolor python3-swiftclient python3-boto python3-azure-storage python3-etcd3 python3-plyvel flake8
 
       - name: "Checkout Repository"
         uses: actions/checkout@v4
@@ -86,6 +63,18 @@ jobs:
         with:
           directory: ${{ runner.temp }}
 
+      - name: "Run Unit Tests"
+        env:
+          RUN_LONG_TESTS: 'yes'
+          AZURE_STORAGE_ENDPOINT: "http://127.0.0.1:10000/devstoreaccount1"
+          AZURE_STORAGE_ACCOUNT: "devstoreaccount1"
+          AZURE_STORAGE_ACCESS_KEY: "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        run: |
+          sudo mkdir -p /srv ; sudo chown runner /srv
+          COVERAGE_DIR=${{ runner.temp }} make test
+
       - name: "Run Benchmark"
         run: |
           COVERAGE_DIR=${{ runner.temp }} make bench
@@ -104,9 +93,8 @@ jobs:
 
       - name: "Merge Code Coverage"
         run: |
-          # go install github.com/wadey/gocovmerge@v0.0.0-20160331181800-b5bfa59ec0ad
+          go install github.com/wadey/gocovmerge@latest
-          # ~/go/bin/gocovmerge coverage/*.out > coverage.txt
+          ~/go/bin/gocovmerge unit.out ${{ runner.temp }}/*.out > coverage.txt
-          awk 'FNR==1 && NR!=1 {next} {print}' coverage/*.out > coverage.txt
 
       - name: "Upload Code Coverage"
         uses: codecov/codecov-action@v2
@@ -116,8 +104,7 @@ jobs:
 
   ci-debian-build:
     name: "Build"
-    needs:
+    needs: test
-      - test
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
@@ -152,7 +139,7 @@ jobs:
         APT_LISTCHANGES_FRONTEND: none
         DEBIAN_FRONTEND: noninteractive
     steps:
-      - name: "Install Build Packages"
+      - name: "Install packages"
         run: |
           apt-get update
           apt-get install -y --no-install-recommends make ca-certificates git curl build-essential devscripts dh-golang jq bash-completion lintian \
@@ -220,28 +207,9 @@ jobs:
         run: |
           .github/workflows/scripts/upload-artifacts.sh release ${{ matrix.suite }}
 
-      - name: "Get aptly version"
-        env:
-          FORCE_CI: ${{ steps.force_ci.outputs.FORCE_CI }}
-        run: |
-           aptlyver=$(make -s version)
-           echo "Aptly Version: $aptlyver"
-           echo "VERSION=$aptlyver" >> $GITHUB_OUTPUT
-        id: releaseversion
-
-      - name: "Upload CI Artifacts"
-        if: github.ref != 'refs/heads/master' && !startsWith(github.event.ref, 'refs/tags')
-        uses: actions/upload-artifact@v4
-        with:
-          name: aptly_${{ steps.releaseversion.outputs.VERSION }}_${{ matrix.suite }}_${{ matrix.arch }}
-          path: build/
-          retention-days: 7
-
   ci-binary-build:
     name: "Build"
-    needs:
+    needs: test
-      - unit-test
-      - test
     runs-on: ubuntu-latest
     strategy:
       matrix:
@@ -299,15 +267,6 @@ jobs:
           path: build/aptly_${{ steps.releaseversion.outputs.VERSION }}_${{ matrix.goos }}_${{ matrix.goarch }}.zip
           compression-level: 0 # no compression
 
-      - name: "Upload CI Artifacts"
-        uses: actions/upload-artifact@v4
-        if: "!startsWith(github.event.ref, 'refs/tags')"
-        with:
-          name: aptly_${{ steps.releaseversion.outputs.VERSION }}_${{ matrix.goos }}_${{ matrix.goarch }}
-          path: build/aptly_${{ steps.releaseversion.outputs.VERSION }}_${{ matrix.goos }}_${{ matrix.goarch }}.zip
-          compression-level: 0 # no compression
-          retention-days: 7
-
   gh-release:
     name: "Github Release"
     runs-on: ubuntu-latest

.github/workflows/golangci-lint.yml

@@ -44,7 +44,7 @@ jobs:
           # Require: The version of golangci-lint to use.
           # When `install-mode` is `binary` (default) the value can be v1.2 or v1.2.3 or `latest` to use the latest version.
           # When `install-mode` is `goinstall` the value can be v1.2.3, `latest`, or the hash of a commit.
-          version: v1.64.5
+          version: v1.54.1
 
           # Optional: working directory, useful for monorepos
           # working-directory: somedir

.golangci.yml

@@ -1,11 +1,16 @@
-version: "2"
+run:
+  tests: false
+
+
 linters:
-  settings:
+  disable-all: true
-    staticcheck:
+  enable:
-      checks:
+   - goconst
-      - "all"
+   - gofmt
-      - "-QF1004"  # could use strings.ReplaceAll instead
+   - goimports
-      - "-QF1012"  # Use fmt.Fprintf(...) instead of WriteString(fmt.Sprintf(...))
+   - govet
-      - "-QF1003"  # could use tagged switch
+   - ineffassign
-      - "-ST1000"  # at least one file in a package should have a package comment
+   - misspell
-      - "-QF1001"  # could apply De Morgan's law
+   - revive
+   - staticcheck
+   - vetshadow

AUTHORS

@@ -68,11 +68,3 @@ List of contributors, in chronological order:
 * Blake Kostner (https://github.com/btkostner)
 * Leigh London (https://github.com/leighlondon)
 * Gordian Schoenherr (https://github.com/schoenherrg)
-* Silke Hofstra (https://github.com/silkeh)
-* Itay Porezky (https://github.com/itayporezky)
-* JupiterRider (https://github.com/JupiterRider)
-* Tobias Assarsson (https://github.com/daedaluz)
-* Yaksh Bariya (https://github.com/thunder-coding)
-* Brian Witt (https://github.com/bwitt)
-* Ales Bregar (https://github.com/abregar)
-* Tim Foerster (https://github.com/tonobo)

CONTRIBUTING.md

@@ -16,7 +16,7 @@ Please report unacceptable behavior on [https://github.com/aptly-dev/aptly/discu
 ### List of Repositories
 
 * [aptly-dev/aptly](https://github.com/aptly-dev/aptly) - aptly source code, functional tests, man page
-* [aptly-dev/aptly-dev.github.io](https://github.com/aptly-dev/aptly-dev.github.io) - aptly website (https://www.aptly.info/)
+* [apty-dev/aptly-dev.github.io](https://github.com/aptly-dev/aptly-dev.github.io) - aptly website (https://www.aptly.info/)
 * [aptly-dev/aptly-fixture-db](https://github.com/aptly-dev/aptly-fixture-db) & [aptly-dev/aptly-fixture-pool](https://github.com/aptly-dev/aptly-fixture-pool) provide
   fixtures for aptly functional tests
 

Makefile

@@ -2,32 +2,13 @@ GOPATH=$(shell go env GOPATH)
 VERSION=$(shell make -s version)
 PYTHON?=python3
 BINPATH?=$(GOPATH)/bin
-GOLANGCI_LINT_VERSION=v2.0.2  # version supporting go 1.24
+GOLANGCI_LINT_VERSION=v1.54.1  # version supporting go 1.19
 COVERAGE_DIR?=$(shell mktemp -d)
 GOOS=$(shell go env GOHOSTOS)
 GOARCH=$(shell go env GOHOSTARCH)
 
-export PODMAN_USERNS = keep-id
+# Uncomment to update system test gold files
-DOCKER_RUN = docker run --security-opt label=disable --user 0:0 --rm -v ${PWD}:/work/src
+# CAPTURE := "--capture"
-
-# Setting TZ for certificates
-export TZ=UTC
-# Unit Tests and some sysmte tests rely on expired certificates, turn back the time
-export TEST_FAKETIME := 2025-01-02 03:04:05
-
-# run with 'COVERAGE_SKIP=1' to skip coverage checks during system tests
-ifeq ($(COVERAGE_SKIP),1)
-COVERAGE_ARG_BUILD :=
-COVERAGE_ARG_TEST := --coverage-skip
-else
-COVERAGE_ARG_BUILD := -coverpkg="./..."
-COVERAGE_ARG_TEST := --coverage-dir $(COVERAGE_DIR)
-endif
-
-# export CAPUTRE=1 for regenrating test gold files
-ifeq ($(CAPTURE),1)
-CAPTURE_ARG := --capture
-endif
 
 help:  ## Print this help
 	@grep -E '^[a-zA-Z][a-zA-Z0-9_-]*:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
@@ -69,15 +50,15 @@ swagger-install:
 	echo "// @version $(VERSION)" >> docs/swagger.conf
 
 azurite-start:
-	azurite -l /tmp/aptly-azurite > ~/.azurite.log 2>&1 & \
+	azurite -l /tmp/aptly-azurite & \
 	echo $$! > ~/.azurite.pid
 
 azurite-stop:
 	@kill `cat ~/.azurite.pid`
 
-swagger: #swagger-install
+swagger: swagger-install
 	# Generate swagger docs
-	#@PATH=$(BINPATH)/:$(PATH) swag init --parseDependency --parseInternal --markdownFiles docs --generalInfo docs/swagger.conf
+	@PATH=$(BINPATH)/:$(PATH) swag init --parseDependency --parseInternal --markdownFiles docs --generalInfo docs/swagger.conf
 
 etcd-install:
 	# Install etcd
@@ -88,9 +69,9 @@ flake8:  ## run flake8 on system test python files
 
 lint: prepare
 	# Install golangci-lint
-	@test -f $(BINPATH)/golangci-lint || go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@$(GOLANGCI_LINT_VERSION)
+	@test -f $(BINPATH)/golangci-lint || go install github.com/golangci/golangci-lint/cmd/golangci-lint@$(GOLANGCI_LINT_VERSION)
 	# Running lint
-	@NO_COLOR=true PATH=$(BINPATH)/:$(PATH) golangci-lint run --max-issues-per-linter=0 --max-same-issues=0
+	@PATH=$(BINPATH)/:$(PATH) golangci-lint run
 
 
 build: prepare swagger  ## Build aptly
@@ -103,11 +84,11 @@ install:
 	# go install -v
 	@out=`mktemp`; if ! go install -v > $$out 2>&1; then cat $$out; rm -f $$out; echo "\nBuild failed\n"; exit 1; else rm -f $$out; fi
 
-test: prepare swagger etcd-install  ## Run unit tests (add TEST=regex to specify which tests to run)
+test: prepare swagger etcd-install  ## Run unit tests
 	@echo "\e[33m\e[1mStarting etcd ...\e[0m"
 	@mkdir -p /tmp/aptly-etcd-data; system/t13_etcd/start-etcd.sh > /tmp/aptly-etcd-data/etcd.log 2>&1 &
 	@echo "\e[33m\e[1mRunning go test ...\e[0m"
-	faketime "$(TEST_FAKETIME)" go test -v ./... -gocheck.v=true -check.f "$(TEST)" -coverprofile=unit.out; echo $$? > .unit-test.ret
+	go test -v ./... -gocheck.v=true -coverprofile=unit.out; echo $$? > .unit-test.ret
 	@echo "\e[33m\e[1mStopping etcd ...\e[0m"
 	@pid=`cat /tmp/etcd.pid`; kill $$pid
 	@rm -f /tmp/aptly-etcd-data/etcd.log
@@ -115,13 +96,13 @@ test: prepare swagger etcd-install  ## Run unit tests (add TEST=regex to specify
 
 system-test: prepare swagger etcd-install  ## Run system tests
 	# build coverage binary
-	go test -v $(COVERAGE_ARG_BUILD) -c -tags testruncli
+	go test -v -coverpkg="./..." -c -tags testruncli
 	# Download fixture-db, fixture-pool, etcd.db
 	if [ ! -e ~/aptly-fixture-db ]; then git clone https://github.com/aptly-dev/aptly-fixture-db.git ~/aptly-fixture-db/; fi
 	if [ ! -e ~/aptly-fixture-pool ]; then git clone https://github.com/aptly-dev/aptly-fixture-pool.git ~/aptly-fixture-pool/; fi
 	test -f ~/etcd.db || (curl -o ~/etcd.db.xz http://repo.aptly.info/system-tests/etcd.db.xz && xz -d ~/etcd.db.xz)
 	# Run system tests
-	PATH=$(BINPATH)/:$(PATH) FORCE_COLOR=1 $(PYTHON) system/run.py --long $(COVERAGE_ARG_TEST) $(CAPTURE_ARG) $(TEST)
+	PATH=$(BINPATH)/:$(PATH) && FORCE_COLOR=1 $(PYTHON) system/run.py --long --coverage-dir $(COVERAGE_DIR) $(CAPTURE) $(TEST)
 
 bench:
 	@echo "\e[33m\e[1mRunning benchmark ...\e[0m"
@@ -131,7 +112,6 @@ serve: prepare swagger-install  ## Run development server (auto recompiling)
 	test -f $(BINPATH)/air || go install github.com/air-verse/air@v1.52.3
 	cp debian/aptly.conf ~/.aptly.conf
 	sed -i /enable_swagger_endpoint/s/false/true/ ~/.aptly.conf
-	sed -i /enable_metrics_endpoint/s/false/true/ ~/.aptly.conf
 	PATH=$(BINPATH):$$PATH air -build.pre_cmd 'swag init -q --markdownFiles docs --generalInfo docs/swagger.conf' -build.exclude_dir docs,system,debian,pgp/keyrings,pgp/test-bins,completion.d,man,deb/testdata,console,_man,systemd,obj-x86_64-linux-gnu -- api serve -listen 0.0.0.0:3142
 
 dpkg: prepare swagger  ## Build debian packages
@@ -182,49 +162,44 @@ binaries: prepare swagger  ## Build binary releases (FreeBSD, macOS, Linux gener
 docker-image:  ## Build aptly-dev docker image
 	@docker build -f system/Dockerfile . -t aptly-dev
 
-docker-image-no-cache:  ## Build aptly-dev docker image (no cache)
-	@docker build --no-cache -f system/Dockerfile . -t aptly-dev
-
 docker-build:  ## Build aptly in docker container
-	@$(DOCKER_RUN) -t aptly-dev /work/src/system/docker-wrapper build
+	@docker run -it --rm -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper build
 
 docker-shell:  ## Run aptly and other commands in docker container
-	@$(DOCKER_RUN) -it -p 3142:3142 aptly-dev /work/src/system/docker-wrapper || true
+	@docker run -it --rm -p 3142:3142 -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper || true
 
 docker-deb:  ## Build debian packages in docker container
-	@$(DOCKER_RUN) -t aptly-dev /work/src/system/docker-wrapper dpkg DEBARCH=amd64
+	@docker run -it --rm -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper dpkg DEBARCH=amd64
 
-docker-unit-test:  ## Run unit tests in docker container (add TEST=regex to specify which tests to run)
+docker-unit-test:  ## Run unit tests in docker container
-	$(DOCKER_RUN) -t --tmpfs /smallfs:rw,size=1m aptly-dev /work/src/system/docker-wrapper \
+	@docker run -it --rm -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper \
 		azurite-start \
 		AZURE_STORAGE_ENDPOINT=http://127.0.0.1:10000/devstoreaccount1 \
 		AZURE_STORAGE_ACCOUNT=devstoreaccount1 \
 		AZURE_STORAGE_ACCESS_KEY="Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==" \
-		test TEST=$(TEST) \
+		test \
 		azurite-stop
 
 docker-system-test:  ## Run system tests in docker container (add TEST=t04_mirror or TEST=UpdateMirror26Test to run only specific tests)
-	@$(DOCKER_RUN) -t aptly-dev /work/src/system/docker-wrapper \
+	@docker run -it --rm -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper \
 		azurite-start \
 		AZURE_STORAGE_ENDPOINT=http://127.0.0.1:10000/devstoreaccount1 \
 		AZURE_STORAGE_ACCOUNT=devstoreaccount1 \
 		AZURE_STORAGE_ACCESS_KEY="Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==" \
-		AWS_ACCESS_KEY_ID=$(AWS_ACCESS_KEY_ID) \
+		system-test TEST=$(TEST) \
-		AWS_SECRET_ACCESS_KEY=$(AWS_SECRET_ACCESS_KEY) \
-		system-test TEST=$(TEST) CAPTURE=$(CAPTURE) COVERAGE_SKIP=$(COVERAGE_SKIP) \
 		azurite-stop
 
 docker-serve:  ## Run development server (auto recompiling) on http://localhost:3142
-	@$(DOCKER_RUN) -it -p 3142:3142 -v /tmp/cache-go-aptly:/var/lib/aptly/.cache/go-build aptly-dev /work/src/system/docker-wrapper serve || true
+	@docker run -it --rm -p 3142:3142 -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper serve || true
 
 docker-lint:  ## Run golangci-lint in docker container
-	@$(DOCKER_RUN) -t aptly-dev /work/src/system/docker-wrapper lint
+	@docker run -it --rm -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper lint
 
 docker-binaries:  ## Build binary releases (FreeBSD, macOS, Linux generic) in docker container
-	@$(DOCKER_RUN) -t aptly-dev /work/src/system/docker-wrapper binaries
+	@docker run -it --rm -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper binaries
 
 docker-man:  ## Create man page in docker container
-	@$(DOCKER_RUN) -t aptly-dev /work/src/system/docker-wrapper man
+	@docker run -it --rm -v ${PWD}:/work/src aptly-dev /work/src/system/docker-wrapper man
 
 mem.png: mem.dat mem.gp
 	gnuplot mem.gp
@@ -241,4 +216,4 @@ clean:  ## remove local build and module cache
 	rm -f unit.out aptly.test VERSION docs/docs.go docs/swagger.json docs/swagger.yaml docs/swagger.conf
 	find system/ -type d -name __pycache__ -exec rm -rf {} \; 2>/dev/null || true
 
-.PHONY: help man prepare swagger version binaries build docker-release docker-system-tests docker-unit-test docker-lint docker-build docker-image docker-man docker-shell docker-serve clean releasetype dpkg serve flake8
+.PHONY: help man prepare swagger version binaries build docker-release docker-system-test docker-unit-test docker-lint docker-build docker-image docker-man docker-shell docker-serve clean releasetype dpkg serve flake8

Releasing.md

@@ -12,6 +12,5 @@ git push origin v$version master
 ```
 - run swagger locally (`make docker-serve`)
 - copy generated docs/swagger.json to https://github.com/aptly-dev/www.aptly.info/tree/master/static/swagger/aptly_1.x.y.json
-- add new version to select tag in content/doc/api/swagger.md line 48
+- releae www.aptly.info
-- push commit to master
 - create release announcement on https://github.com/aptly-dev/aptly/discussions

api/api.go

@@ -41,10 +41,7 @@ type aptlyVersion struct {
 // @Success 200 {object} aptlyVersion
 // @Router /api/version [get]
 func apiVersion(c *gin.Context) {
-	version := aptlyVersion{
+	c.JSON(200, gin.H{"Version": aptly.Version})
-		Version: aptly.Version,
-	}
-	c.JSON(200, version)
 }
 
 type aptlyStatus struct {
@@ -70,8 +67,7 @@ func apiReady(isReady *atomic.Value) func(*gin.Context) {
 			return
 		}
 
-		status := aptlyStatus{Status: "Aptly is ready"}
+		c.JSON(200, gin.H{"Status": "Aptly is ready"})
-		c.JSON(200, status)
 	}
 }
 
@@ -169,7 +165,7 @@ func runTaskInBackground(name string, resources []string, proc task.Process) (ta
 			return nil, err
 		}
 
-		defer func() { _ = releaseDatabaseConnection() }()
+		defer releaseDatabaseConnection()
 		return proc(out, detail)
 	})
 }
@@ -178,18 +174,18 @@ func truthy(value interface{}) bool {
 	if value == nil {
 		return false
 	}
-	switch v := value.(type) {
+	switch value.(type) {
 	case string:
-		switch strings.ToLower(v) {
+		switch strings.ToLower(value.(string)) {
 		case "n", "no", "f", "false", "0", "off":
 			return false
 		default:
 			return true
 		}
 	case int:
-		return v != 0
+		return !(value.(int) == 0)
 	case bool:
-		return v
+		return value.(bool)
 	}
 	return true
 }
@@ -214,11 +210,11 @@ func maybeRunTaskInBackground(c *gin.Context, name string, resources []string, p
 		}
 
 		// wait for task to finish
-		_, _ = context.TaskList().WaitForTaskByID(task.ID)
+		context.TaskList().WaitForTaskByID(task.ID)
 
 		retValue, _ := context.TaskList().GetTaskReturnValueByID(task.ID)
 		err, _ := context.TaskList().GetTaskErrorByID(task.ID)
-		_, _ = context.TaskList().DeleteTaskByID(task.ID)
+		context.TaskList().DeleteTaskByID(task.ID)
 		if err != nil {
 			AbortWithJSONError(c, retValue.Code, err)
 			return
@@ -233,7 +229,7 @@ func maybeRunTaskInBackground(c *gin.Context, name string, resources []string, p
 
 // Common piece of code to show list of packages,
 // with searching & details if requested
-func showPackages(c *gin.Context, reflist *deb.PackageRefList, collectionFactory *deb.CollectionFactory) {
+func showPackages(c *gin.Context, reflist deb.AnyRefList, collectionFactory *deb.CollectionFactory) {
 	result := []*deb.Package{}
 
 	list, err := deb.NewPackageListFromRefList(reflist, collectionFactory.PackageCollection(), nil)
@@ -286,11 +282,11 @@ func showPackages(c *gin.Context, reflist *deb.PackageRefList, collectionFactory
 	// filter packages by version
 	if c.Request.URL.Query().Get("maximumVersion") == "1" {
 		list.PrepareIndex()
-		_ = list.ForEach(func(p *deb.Package) error {
+		list.ForEach(func(p *deb.Package) error {
 			versionQ, err := query.Parse(fmt.Sprintf("Name (%s), $Version (<= %s)", p.Name, p.Version))
 			if err != nil {
 				fmt.Println("filter packages by version, query string parse err: ", err)
-				_ = c.AbortWithError(500, fmt.Errorf("unable to parse %s maximum version query string: %s", p.Name, err))
+				c.AbortWithError(500, fmt.Errorf("unable to parse %s maximum version query string: %s", p.Name, err))
 			} else {
 				tmpList, err := list.Filter(deb.FilterOptions{
 					Queries: []deb.PackageQuery{versionQ},
@@ -298,15 +294,15 @@ func showPackages(c *gin.Context, reflist *deb.PackageRefList, collectionFactory
 
 				if err == nil {
 					if tmpList.Len() > 0 {
-						_ = tmpList.ForEach(func(tp *deb.Package) error {
+						tmpList.ForEach(func(tp *deb.Package) error {
 							list.Remove(tp)
 							return nil
 						})
-						_ = list.Add(p)
+						list.Add(p)
 					}
 				} else {
 					fmt.Println("filter packages by version, filter err: ", err)
-					_ = c.AbortWithError(500, fmt.Errorf("unable to get %s maximum version: %s", p.Name, err))
+					c.AbortWithError(500, fmt.Errorf("unable to get %s maximum version: %s", p.Name, err))
 				}
 			}
 
@@ -315,7 +311,7 @@ func showPackages(c *gin.Context, reflist *deb.PackageRefList, collectionFactory
 	}
 
 	if c.Request.URL.Query().Get("format") == "details" {
-		_ = list.ForEach(func(p *deb.Package) error {
+		list.ForEach(func(p *deb.Package) error {
 			result = append(result, p)
 			return nil
 		})
@@ -326,7 +322,7 @@ func showPackages(c *gin.Context, reflist *deb.PackageRefList, collectionFactory
 	}
 }
 
-func AbortWithJSONError(c *gin.Context, code int, err error) {
+func AbortWithJSONError(c *gin.Context, code int, err error) *gin.Error {
 	c.Writer.Header().Set("Content-Type", "application/json; charset=utf-8")
-	_ = c.AbortWithError(code, err)
+	return c.AbortWithError(code, err)
 }

api/api_test.go

@@ -24,14 +24,14 @@ func Test(t *testing.T) {
 	TestingT(t)
 }
 
-type APISuite struct {
+type ApiSuite struct {
 	context    *ctx.AptlyContext
 	flags      *flag.FlagSet
 	configFile *os.File
 	router     http.Handler
 }
 
-var _ = Suite(&APISuite{})
+var _ = Suite(&ApiSuite{})
 
 func createTestConfig() *os.File {
 	file, err := os.CreateTemp("", "aptly")
@@ -45,11 +45,11 @@ func createTestConfig() *os.File {
 	if err != nil {
 		return nil
 	}
-	_, _ = file.Write(jsonString)
+	file.Write(jsonString)
 	return file
 }
 
-func (s *APISuite) setupContext() error {
+func (s *ApiSuite) setupContext() error {
 	aptly.Version = "testVersion"
 	file := createTestConfig()
 	if nil == file {
@@ -75,23 +75,23 @@ func (s *APISuite) setupContext() error {
 	return nil
 }
 
-func (s *APISuite) SetUpSuite(c *C) {
+func (s *ApiSuite) SetUpSuite(c *C) {
 	err := s.setupContext()
 	c.Assert(err, IsNil)
 }
 
-func (s *APISuite) TearDownSuite(c *C) {
+func (s *ApiSuite) TearDownSuite(c *C) {
-	_ = os.Remove(s.configFile.Name())
+	os.Remove(s.configFile.Name())
 	s.context.Shutdown()
 }
 
-func (s *APISuite) SetUpTest(c *C) {
+func (s *ApiSuite) SetUpTest(c *C) {
 }
 
-func (s *APISuite) TearDownTest(c *C) {
+func (s *ApiSuite) TearDownTest(c *C) {
 }
 
-func (s *APISuite) HTTPRequest(method string, url string, body io.Reader) (*httptest.ResponseRecorder, error) {
+func (s *ApiSuite) HTTPRequest(method string, url string, body io.Reader) (*httptest.ResponseRecorder, error) {
 	w := httptest.NewRecorder()
 	req, err := http.NewRequest(method, url, body)
 	if err != nil {
@@ -102,32 +102,32 @@ func (s *APISuite) HTTPRequest(method string, url string, body io.Reader) (*http
 	return w, nil
 }
 
-func (s *APISuite) TestGinRunsInReleaseMode(c *C) {
+func (s *ApiSuite) TestGinRunsInReleaseMode(c *C) {
 	c.Check(gin.Mode(), Equals, gin.ReleaseMode)
 }
 
-func (s *APISuite) TestGetVersion(c *C) {
+func (s *ApiSuite) TestGetVersion(c *C) {
 	response, err := s.HTTPRequest("GET", "/api/version", nil)
 	c.Assert(err, IsNil)
 	c.Check(response.Code, Equals, 200)
 	c.Check(response.Body.String(), Matches, "{\"Version\":\""+aptly.Version+"\"}")
 }
 
-func (s *APISuite) TestGetReadiness(c *C) {
+func (s *ApiSuite) TestGetReadiness(c *C) {
 	response, err := s.HTTPRequest("GET", "/api/ready", nil)
 	c.Assert(err, IsNil)
 	c.Check(response.Code, Equals, 200)
 	c.Check(response.Body.String(), Matches, "{\"Status\":\"Aptly is ready\"}")
 }
 
-func (s *APISuite) TestGetHealthiness(c *C) {
+func (s *ApiSuite) TestGetHealthiness(c *C) {
 	response, err := s.HTTPRequest("GET", "/api/healthy", nil)
 	c.Assert(err, IsNil)
 	c.Check(response.Code, Equals, 200)
 	c.Check(response.Body.String(), Matches, "{\"Status\":\"Aptly is healthy\"}")
 }
 
-func (s *APISuite) TestGetMetrics(c *C) {
+func (s *ApiSuite) TestGetMetrics(c *C) {
 	response, err := s.HTTPRequest("GET", "/api/metrics", nil)
 	c.Assert(err, IsNil)
 	c.Check(response.Code, Equals, 200)
@@ -141,7 +141,7 @@ func (s *APISuite) TestGetMetrics(c *C) {
 	c.Check(b, Matches, ".*aptly_build_info.*version=\"testVersion\".*")
 }
 
-func (s *APISuite) TestRepoCreate(c *C) {
+func (s *ApiSuite) TestRepoCreate(c *C) {
 	body, err := json.Marshal(gin.H{
 		"Name": "dummy",
 	})
@@ -150,7 +150,7 @@ func (s *APISuite) TestRepoCreate(c *C) {
 	c.Assert(err, IsNil)
 }
 
-func (s *APISuite) TestTruthy(c *C) {
+func (s *ApiSuite) TestTruthy(c *C) {
 	c.Check(truthy("no"), Equals, false)
 	c.Check(truthy("n"), Equals, false)
 	c.Check(truthy("off"), Equals, false)

api/db.go

@@ -5,6 +5,7 @@ import (
 	"sort"
 
 	"github.com/aptly-dev/aptly/aptly"
+	"github.com/aptly-dev/aptly/database"
 	"github.com/aptly-dev/aptly/deb"
 	"github.com/aptly-dev/aptly/task"
 	"github.com/aptly-dev/aptly/utils"
@@ -21,25 +22,29 @@ import (
 // @Success 200 {object} string "Output"
 // @Failure 404 {object} Error "Not Found"
 // @Router /api/db/cleanup [post]
-func apiDBCleanup(c *gin.Context) {
+func apiDbCleanup(c *gin.Context) {
 	resources := []string{string(task.AllResourcesKey)}
 	maybeRunTaskInBackground(c, "Clean up db", resources, func(out aptly.Progress, detail *task.Detail) (*task.ProcessReturnValue, error) {
 		var err error
 
 		collectionFactory := context.NewCollectionFactory()
 
-		// collect information about referenced packages...
+		// collect information about referenced packages and their reflist buckets...
-		existingPackageRefs := deb.NewPackageRefList()
+		existingPackageRefs := deb.NewSplitRefList()
+		existingBuckets := deb.NewRefListDigestSet()
+
+		reflistMigration := collectionFactory.RefListCollection().NewMigration()
 
 		out.Printf("Loading mirrors, local repos, snapshots and published repos...")
 		err = collectionFactory.RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
-			e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+			sl := deb.NewSplitRefList()
-			if e != nil {
+			e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
+			if e != nil && e != database.ErrNotFound {
 				return e
 			}
-			if repo.RefList() != nil {
+
-				existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
+			existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
-			}
+			existingBuckets.AddAllInRefList(sl)
 
 			return nil
 		})
@@ -48,14 +53,14 @@ func apiDBCleanup(c *gin.Context) {
 		}
 
 		err = collectionFactory.LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
-			e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
+			sl := deb.NewSplitRefList()
-			if e != nil {
+			e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
+			if e != nil && e != database.ErrNotFound {
 				return e
 			}
 
-			if repo.RefList() != nil {
+			existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
-				existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
+			existingBuckets.AddAllInRefList(sl)
-			}
 
 			return nil
 		})
@@ -64,12 +69,14 @@ func apiDBCleanup(c *gin.Context) {
 		}
 
 		err = collectionFactory.SnapshotCollection().ForEach(func(snapshot *deb.Snapshot) error {
-			e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+			sl := deb.NewSplitRefList()
+			e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, snapshot.RefKey(), reflistMigration)
 			if e != nil {
 				return e
 			}
 
-			existingPackageRefs = existingPackageRefs.Merge(snapshot.RefList(), false, true)
+			existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+			existingBuckets.AddAllInRefList(sl)
 
 			return nil
 		})
@@ -81,13 +88,16 @@ func apiDBCleanup(c *gin.Context) {
 			if published.SourceKind != deb.SourceLocalRepo {
 				return nil
 			}
-			e := collectionFactory.PublishedRepoCollection().LoadComplete(published, collectionFactory)
-			if e != nil {
-				return e
-			}
 
 			for _, component := range published.Components() {
-				existingPackageRefs = existingPackageRefs.Merge(published.RefList(component), false, true)
+				sl := deb.NewSplitRefList()
+				e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, published.RefKey(component), reflistMigration)
+				if e != nil {
+					return e
+				}
+
+				existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+				existingBuckets.AddAllInRefList(sl)
 			}
 			return nil
 		})
@@ -95,11 +105,20 @@ func apiDBCleanup(c *gin.Context) {
 			return nil, err
 		}
 
+		err = reflistMigration.Flush()
+		if err != nil {
+			return nil, err
+		}
+		if stats := reflistMigration.Stats(); stats.Reflists > 0 {
+			out.Printf("Split %d reflist(s) into %d bucket(s) (%d segment(s))",
+				stats.Reflists, stats.Buckets, stats.Segments)
+		}
+
 		// ... and compare it to the list of all packages
 		out.Printf("Loading list of all packages...")
 		allPackageRefs := collectionFactory.PackageCollection().AllPackageRefs()
 
-		toDelete := allPackageRefs.Subtract(existingPackageRefs)
+		toDelete := allPackageRefs.Subtract(existingPackageRefs.Flatten())
 
 		// delete packages that are no longer referenced
 		out.Printf("Deleting unreferenced packages (%d)...", toDelete.Len())
@@ -109,8 +128,8 @@ func apiDBCleanup(c *gin.Context) {
 
 		if toDelete.Len() > 0 {
 			batch := db.CreateBatch()
-			_ = toDelete.ForEach(func(ref []byte) error {
+			toDelete.ForEach(func(ref []byte) error {
-				_ = collectionFactory.PackageCollection().DeleteByKey(ref, batch)
+				collectionFactory.PackageCollection().DeleteByKey(ref, batch)
 				return nil
 			})
 
@@ -120,6 +139,28 @@ func apiDBCleanup(c *gin.Context) {
 			}
 		}
 
+		bucketsToDelete, err := collectionFactory.RefListCollection().AllBucketDigests()
+		if err != nil {
+			return nil, err
+		}
+
+		bucketsToDelete.RemoveAll(existingBuckets)
+
+		out.Printf("Deleting unreferenced reflist buckets (%d)...", bucketsToDelete.Len())
+		if bucketsToDelete.Len() > 0 {
+			batch := db.CreateBatch()
+			err := bucketsToDelete.ForEach(func(digest []byte) error {
+				return collectionFactory.RefListCollection().UnsafeDropBucket(digest, batch)
+			})
+			if err != nil {
+				return nil, err
+			}
+
+			if err := batch.Write(); err != nil {
+				return nil, err
+			}
+		}
+
 		// now, build a list of files that should be present in Repository (package pool)
 		out.Printf("Building list of files referenced by packages...")
 		referencedFiles := make([]string, 0, existingPackageRefs.Len())

api/files.go

@@ -13,10 +13,6 @@ import (
 	"github.com/saracen/walker"
 )
 
-// syncFile is a seam to allow tests to force fsync failures (e.g. ENOSPC).
-// In production it calls (*os.File).Sync().
-var syncFile = func(f *os.File) error { return f.Sync() }
-
 func verifyPath(path string) bool {
 	path = filepath.Clean(path)
 	for _, part := range strings.Split(path, string(filepath.Separator)) {
@@ -118,69 +114,34 @@ func apiFilesUpload(c *gin.Context) {
 	}
 
 	stored := []string{}
-	openFiles := []*os.File{}
 
-	// Write all files first
 	for _, files := range c.Request.MultipartForm.File {
 		for _, file := range files {
 			src, err := file.Open()
 			if err != nil {
-				// Close any files we've opened
-				for _, f := range openFiles {
-					_ = f.Close()
-				}
 				AbortWithJSONError(c, 500, err)
 				return
 			}
+			defer src.Close()
 
 			destPath := filepath.Join(path, filepath.Base(file.Filename))
 			dst, err := os.Create(destPath)
 			if err != nil {
-				_ = src.Close()
-				// Close any files we've opened
-				for _, f := range openFiles {
-					_ = f.Close()
-				}
 				AbortWithJSONError(c, 500, err)
 				return
 			}
+			defer dst.Close()
 
 			_, err = io.Copy(dst, src)
-			_ = src.Close()
 			if err != nil {
-				_ = dst.Close()
-				// Close any files we've opened
-				for _, f := range openFiles {
-					_ = f.Close()
-				}
 				AbortWithJSONError(c, 500, err)
 				return
 			}
 
-			// Keep file open for batch sync
-			openFiles = append(openFiles, dst)
 			stored = append(stored, filepath.Join(c.Params.ByName("dir"), filepath.Base(file.Filename)))
 		}
 	}
 
-	// Sync all files at once to catch ENOSPC errors
-	for i, dst := range openFiles {
-		err := syncFile(dst)
-		if err != nil {
-			// Close all files
-			for _, f := range openFiles {
-				_ = f.Close()
-			}
-			AbortWithJSONError(c, 500, fmt.Errorf("error syncing file %s: %s", stored[i], err))
-			return
-		}
-	}
-
-	// Close all files
-	for _, dst := range openFiles {
-		_ = dst.Close()
-	}
-
 	apiFilesUploadedCounter.WithLabelValues(c.Params.ByName("dir")).Inc()
 	c.JSON(200, stored)
 }
@@ -209,11 +170,7 @@ func apiFilesListFiles(c *gin.Context) {
 	listLock := &sync.Mutex{}
 	root := filepath.Join(context.UploadPath(), utils.SanitizePath(c.Params.ByName("dir")))
 
-	err := filepath.Walk(root, func(path string, _ os.FileInfo, err error) error {
+	err := walker.Walk(root, func(path string, _ os.FileInfo) error {
-		if err != nil {
-			return err
-		}
-
 		if path == root {
 			return nil
 		}

api/files_test.go

@@ -1,476 +0,0 @@
-package api
-
-import (
-	"bytes"
-	"encoding/json"
-	"io"
-	"mime/multipart"
-	"net/http"
-	"net/http/httptest"
-	"os"
-	"path/filepath"
-	"strings"
-	"syscall"
-
-	"github.com/aptly-dev/aptly/aptly"
-	ctx "github.com/aptly-dev/aptly/context"
-	"github.com/gin-gonic/gin"
-	"github.com/smira/flag"
-
-	. "gopkg.in/check.v1"
-)
-
-type FilesUploadDiskFullSuite struct {
-	aptlyContext *ctx.AptlyContext
-	flags        *flag.FlagSet
-	configFile   *os.File
-	router       http.Handler
-}
-
-var _ = Suite(&FilesUploadDiskFullSuite{})
-
-func (s *FilesUploadDiskFullSuite) SetUpTest(c *C) {
-	aptly.Version = "testVersion"
-
-	file, err := os.CreateTemp("", "aptly")
-	c.Assert(err, IsNil)
-	s.configFile = file
-
-	jsonString, err := json.Marshal(gin.H{
-		"architectures": []string{},
-		"rootDir":       c.MkDir(),
-	})
-	c.Assert(err, IsNil)
-	_, err = file.Write(jsonString)
-	c.Assert(err, IsNil)
-	_ = file.Close()
-
-	flags := flag.NewFlagSet("fakeFlags", flag.ContinueOnError)
-	flags.Bool("no-lock", false, "dummy")
-	flags.Int("db-open-attempts", 3, "dummy")
-	flags.String("config", s.configFile.Name(), "dummy")
-	flags.String("architectures", "", "dummy")
-	s.flags = flags
-
-	aptlyContext, err := ctx.NewContext(s.flags)
-	c.Assert(err, IsNil)
-
-	s.aptlyContext = aptlyContext
-	s.router = Router(aptlyContext)
-	context = aptlyContext
-}
-
-func (s *FilesUploadDiskFullSuite) TearDownTest(c *C) {
-	if s.configFile != nil {
-		_ = os.Remove(s.configFile.Name())
-	}
-	if s.aptlyContext != nil {
-		s.aptlyContext.Shutdown()
-	}
-}
-
-func (s *FilesUploadDiskFullSuite) TestUploadSuccessWithSync(c *C) {
-	testContent := []byte("test file content for upload")
-
-	body := &bytes.Buffer{}
-	writer := multipart.NewWriter(body)
-
-	part, err := writer.CreateFormFile("file", "testfile.txt")
-	c.Assert(err, IsNil)
-
-	_, err = part.Write(testContent)
-	c.Assert(err, IsNil)
-
-	err = writer.Close()
-	c.Assert(err, IsNil)
-
-	req, err := http.NewRequest("POST", "/api/files/testdir", body)
-	c.Assert(err, IsNil)
-	req.Header.Set("Content-Type", writer.FormDataContentType())
-
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-	c.Assert(w.Code, Equals, 200)
-
-	uploadedFile := filepath.Join(s.aptlyContext.Config().GetRootDir(), "upload", "testdir", "testfile.txt")
-	content, err := os.ReadFile(uploadedFile)
-	c.Assert(err, IsNil)
-	c.Check(content, DeepEquals, testContent)
-}
-
-func (s *FilesUploadDiskFullSuite) TestUploadVerifiesFileIntegrity(c *C) {
-	testContent := bytes.Repeat([]byte("A"), 10000)
-
-	body := &bytes.Buffer{}
-	writer := multipart.NewWriter(body)
-
-	part, err := writer.CreateFormFile("file", "largefile.bin")
-	c.Assert(err, IsNil)
-
-	_, err = io.Copy(part, bytes.NewReader(testContent))
-	c.Assert(err, IsNil)
-
-	err = writer.Close()
-	c.Assert(err, IsNil)
-
-	req, err := http.NewRequest("POST", "/api/files/testdir2", body)
-	c.Assert(err, IsNil)
-	req.Header.Set("Content-Type", writer.FormDataContentType())
-
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-
-	c.Assert(w.Code, Equals, 200)
-
-	uploadedFile := filepath.Join(s.aptlyContext.Config().GetRootDir(), "upload", "testdir2", "largefile.bin")
-	content, err := os.ReadFile(uploadedFile)
-	c.Assert(err, IsNil)
-	c.Check(len(content), Equals, len(testContent))
-	c.Check(content, DeepEquals, testContent)
-}
-
-func (s *FilesUploadDiskFullSuite) TestUploadMultipleFilesWithBatchSync(c *C) {
-	testFiles := map[string][]byte{
-		"file1.txt": []byte("content of file 1"),
-		"file2.txt": bytes.Repeat([]byte("B"), 5000),
-		"file3.deb": []byte("debian package content"),
-	}
-
-	body := &bytes.Buffer{}
-	writer := multipart.NewWriter(body)
-
-	for filename, content := range testFiles {
-		part, err := writer.CreateFormFile("file", filename)
-		c.Assert(err, IsNil)
-		_, err = part.Write(content)
-		c.Assert(err, IsNil)
-	}
-
-	err := writer.Close()
-	c.Assert(err, IsNil)
-
-	req, err := http.NewRequest("POST", "/api/files/multitest", body)
-	c.Assert(err, IsNil)
-	req.Header.Set("Content-Type", writer.FormDataContentType())
-
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-
-	c.Assert(w.Code, Equals, 200)
-
-	uploadDir := filepath.Join(s.aptlyContext.Config().GetRootDir(), "upload", "multitest")
-	for filename, expectedContent := range testFiles {
-		uploadedFile := filepath.Join(uploadDir, filename)
-		content, err := os.ReadFile(uploadedFile)
-		c.Assert(err, IsNil, Commentf("Failed to read %s", filename))
-		c.Check(content, DeepEquals, expectedContent, Commentf("Content mismatch for %s", filename))
-	}
-}
-
-func (s *FilesUploadDiskFullSuite) TestUploadReturnsErrorOnSyncFailure(c *C) {
-	oldSyncFile := syncFile
-	syncFile = func(f *os.File) error {
-		if filepath.Base(f.Name()) == "syncfail.txt" {
-			return syscall.ENOSPC
-		}
-		return nil
-	}
-	defer func() { syncFile = oldSyncFile }()
-
-	body := &bytes.Buffer{}
-	writer := multipart.NewWriter(body)
-
-	part1, err := writer.CreateFormFile("file", "ok.txt")
-	c.Assert(err, IsNil)
-	_, err = part1.Write([]byte("ok"))
-	c.Assert(err, IsNil)
-
-	part2, err := writer.CreateFormFile("file", "syncfail.txt")
-	c.Assert(err, IsNil)
-	_, err = part2.Write([]byte("will fail on sync"))
-	c.Assert(err, IsNil)
-
-	err = writer.Close()
-	c.Assert(err, IsNil)
-
-	req, err := http.NewRequest("POST", "/api/files/syncfaildir", body)
-	c.Assert(err, IsNil)
-	req.Header.Set("Content-Type", writer.FormDataContentType())
-
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-
-	c.Assert(w.Code, Equals, 500)
-	c.Check(bytes.Contains(w.Body.Bytes(), []byte("error syncing file")), Equals, true)
-}
-
-func (s *FilesUploadDiskFullSuite) TestVerifyPath(c *C) {
-	c.Check(verifyPath("a/b/c"), Equals, true)
-	c.Check(verifyPath("../x"), Equals, false)
-	c.Check(verifyPath("./x"), Equals, true)
-	c.Check(verifyPath(".."), Equals, false)
-	c.Check(verifyPath("."), Equals, false)
-}
-
-func (s *FilesUploadDiskFullSuite) TestListDirsEmptyWhenUploadMissing(c *C) {
-	_ = os.RemoveAll(s.aptlyContext.UploadPath())
-
-	req, err := http.NewRequest("GET", "/api/files", nil)
-	c.Assert(err, IsNil)
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-
-	c.Assert(w.Code, Equals, 200)
-	c.Check(strings.TrimSpace(w.Body.String()), Equals, "[]")
-}
-
-func (s *FilesUploadDiskFullSuite) TestListDirsReturnsDirectories(c *C) {
-	uploadRoot := s.aptlyContext.UploadPath()
-	c.Assert(os.MkdirAll(filepath.Join(uploadRoot, "d1"), 0777), IsNil)
-	c.Assert(os.MkdirAll(filepath.Join(uploadRoot, "d2"), 0777), IsNil)
-	c.Assert(os.WriteFile(filepath.Join(uploadRoot, "rootfile"), []byte("x"), 0644), IsNil)
-
-	req, err := http.NewRequest("GET", "/api/files", nil)
-	c.Assert(err, IsNil)
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-
-	c.Assert(w.Code, Equals, 200)
-	body := w.Body.String()
-	c.Check(strings.Contains(body, "d1"), Equals, true)
-	c.Check(strings.Contains(body, "d2"), Equals, true)
-}
-
-func (s *FilesUploadDiskFullSuite) TestListFilesNotFound(c *C) {
-	req, err := http.NewRequest("GET", "/api/files/does-not-exist", nil)
-	c.Assert(err, IsNil)
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-	c.Assert(w.Code, Equals, 404)
-}
-
-func (s *FilesUploadDiskFullSuite) TestListFilesReturnsFiles(c *C) {
-	base := filepath.Join(s.aptlyContext.UploadPath(), "dir")
-	c.Assert(os.MkdirAll(base, 0777), IsNil)
-	c.Assert(os.WriteFile(filepath.Join(base, "a.txt"), []byte("a"), 0644), IsNil)
-	c.Assert(os.WriteFile(filepath.Join(base, "b.txt"), []byte("b"), 0644), IsNil)
-
-	req, err := http.NewRequest("GET", "/api/files/dir", nil)
-	c.Assert(err, IsNil)
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-
-	c.Assert(w.Code, Equals, 200)
-	body := w.Body.String()
-	c.Check(strings.Contains(body, "a.txt"), Equals, true)
-	c.Check(strings.Contains(body, "b.txt"), Equals, true)
-}
-
-func (s *FilesUploadDiskFullSuite) TestDeleteDirRemovesDirectory(c *C) {
-	base := filepath.Join(s.aptlyContext.UploadPath(), "todel")
-	c.Assert(os.MkdirAll(base, 0777), IsNil)
-	c.Assert(os.WriteFile(filepath.Join(base, "a.txt"), []byte("a"), 0644), IsNil)
-
-	req, err := http.NewRequest("DELETE", "/api/files/todel", nil)
-	c.Assert(err, IsNil)
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-	c.Assert(w.Code, Equals, 200)
-
-	_, statErr := os.Stat(base)
-	c.Check(os.IsNotExist(statErr), Equals, true)
-}
-
-func (s *FilesUploadDiskFullSuite) TestDeleteFileRemovesFile(c *C) {
-	base := filepath.Join(s.aptlyContext.UploadPath(), "todel2")
-	c.Assert(os.MkdirAll(base, 0777), IsNil)
-	c.Assert(os.WriteFile(filepath.Join(base, "a.txt"), []byte("a"), 0644), IsNil)
-
-	req, err := http.NewRequest("DELETE", "/api/files/todel2/a.txt", nil)
-	c.Assert(err, IsNil)
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-	c.Assert(w.Code, Equals, 200)
-
-	_, statErr := os.Stat(filepath.Join(base, "a.txt"))
-	c.Check(os.IsNotExist(statErr), Equals, true)
-}
-
-func (s *FilesUploadDiskFullSuite) TestDeleteFileNotFoundStillOk(c *C) {
-	base := filepath.Join(s.aptlyContext.UploadPath(), "todel3")
-	c.Assert(os.MkdirAll(base, 0777), IsNil)
-
-	req, err := http.NewRequest("DELETE", "/api/files/todel3/nope.txt", nil)
-	c.Assert(err, IsNil)
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-	c.Assert(w.Code, Equals, 200)
-}
-
-func (s *FilesUploadDiskFullSuite) TestRejectsInvalidDir(c *C) {
-	req, err := http.NewRequest("DELETE", "/api/files/..", nil)
-	c.Assert(err, IsNil)
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-	c.Assert(w.Code, Equals, 400)
-}
-
-func (s *FilesUploadDiskFullSuite) TestRejectsInvalidFileName(c *C) {
-	base := filepath.Join(s.aptlyContext.UploadPath(), "dirx")
-	c.Assert(os.MkdirAll(base, 0777), IsNil)
-
-	req, err := http.NewRequest("DELETE", "/api/files/dirx/..", nil)
-	c.Assert(err, IsNil)
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-	c.Assert(w.Code, Equals, 400)
-}
-
-func (s *FilesUploadDiskFullSuite) TestListDirsEmptyIfUploadPathIsNotDir(c *C) {
-	_ = os.RemoveAll(s.aptlyContext.UploadPath())
-	c.Assert(os.WriteFile(s.aptlyContext.UploadPath(), []byte("not a dir"), 0644), IsNil)
-
-	req, err := http.NewRequest("GET", "/api/files", nil)
-	c.Assert(err, IsNil)
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-
-	c.Assert(w.Code, Equals, 200)
-	c.Check(strings.TrimSpace(w.Body.String()), Equals, "[]")
-}
-
-func (s *FilesUploadDiskFullSuite) TestListFilesReturns500OnPermissionError(c *C) {
-	base := filepath.Join(s.aptlyContext.UploadPath(), "noperms")
-	c.Assert(os.MkdirAll(base, 0777), IsNil)
-	c.Assert(os.WriteFile(filepath.Join(base, "a.txt"), []byte("a"), 0644), IsNil)
-	c.Assert(os.Chmod(base, 0), IsNil)
-	defer func() { _ = os.Chmod(base, 0777) }()
-
-	req, err := http.NewRequest("GET", "/api/files/noperms", nil)
-	c.Assert(err, IsNil)
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-
-	c.Assert(w.Code, Equals, 500)
-}
-
-func (s *FilesUploadDiskFullSuite) TestDeleteFileReturns500OnNonNotExistError(c *C) {
-	base := filepath.Join(s.aptlyContext.UploadPath(), "dirisfile")
-	c.Assert(os.MkdirAll(base, 0777), IsNil)
-	subdir := filepath.Join(base, "subdir")
-	c.Assert(os.MkdirAll(subdir, 0777), IsNil)
-	c.Assert(os.WriteFile(filepath.Join(subdir, "x"), []byte("x"), 0644), IsNil)
-
-	req, err := http.NewRequest("DELETE", "/api/files/dirisfile/subdir", nil)
-	c.Assert(err, IsNil)
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-
-	c.Assert(w.Code, Equals, 500)
-}
-
-func (s *FilesUploadDiskFullSuite) TestUploadBadMultipartReturns400(c *C) {
-	req, err := http.NewRequest("POST", "/api/files/badmultipart", bytes.NewBufferString("not multipart"))
-	c.Assert(err, IsNil)
-	req.Header.Set("Content-Type", "multipart/form-data; boundary=missing")
-
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-
-	c.Assert(w.Code, Equals, 400)
-}
-
-func (s *FilesUploadDiskFullSuite) TestUploadRejectsInvalidDir(c *C) {
-	body := &bytes.Buffer{}
-	writer := multipart.NewWriter(body)
-	part, err := writer.CreateFormFile("file", "a.txt")
-	c.Assert(err, IsNil)
-	_, err = part.Write([]byte("x"))
-	c.Assert(err, IsNil)
-	c.Assert(writer.Close(), IsNil)
-
-	req, err := http.NewRequest("POST", "/api/files/..", body)
-	c.Assert(err, IsNil)
-	req.Header.Set("Content-Type", writer.FormDataContentType())
-
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-	c.Assert(w.Code, Equals, 400)
-}
-
-func (s *FilesUploadDiskFullSuite) TestUploadReturns500IfUploadRootIsNotDir(c *C) {
-	_ = os.RemoveAll(s.aptlyContext.UploadPath())
-	c.Assert(os.WriteFile(s.aptlyContext.UploadPath(), []byte("not a dir"), 0644), IsNil)
-
-	body := &bytes.Buffer{}
-	writer := multipart.NewWriter(body)
-	part, err := writer.CreateFormFile("file", "a.txt")
-	c.Assert(err, IsNil)
-	_, err = part.Write([]byte("x"))
-	c.Assert(err, IsNil)
-	c.Assert(writer.Close(), IsNil)
-
-	req, err := http.NewRequest("POST", "/api/files/testdir", body)
-	c.Assert(err, IsNil)
-	req.Header.Set("Content-Type", writer.FormDataContentType())
-
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-	c.Assert(w.Code, Equals, 500)
-}
-
-func (s *FilesUploadDiskFullSuite) TestUploadReturns500OnFileOpenFailure(c *C) {
-	// Pre-populate MultipartForm to inject a FileHeader that fails on Open().
-	form := &multipart.Form{
-		File: map[string][]*multipart.FileHeader{
-			"file": {{Filename: "broken.bin"}},
-		},
-	}
-
-	req, err := http.NewRequest("POST", "/api/files/openfaildir", nil)
-	c.Assert(err, IsNil)
-	req.MultipartForm = form
-
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-	c.Assert(w.Code, Equals, 500)
-}
-
-func (s *FilesUploadDiskFullSuite) TestUploadReturns500OnCreateFailure(c *C) {
-	base := filepath.Join(s.aptlyContext.UploadPath(), "readonly")
-	c.Assert(os.MkdirAll(base, 0777), IsNil)
-	c.Assert(os.Chmod(base, 0555), IsNil)
-	defer func() { _ = os.Chmod(base, 0777) }()
-
-	body := &bytes.Buffer{}
-	writer := multipart.NewWriter(body)
-	part, err := writer.CreateFormFile("file", "a.txt")
-	c.Assert(err, IsNil)
-	_, err = part.Write([]byte("x"))
-	c.Assert(err, IsNil)
-	c.Assert(writer.Close(), IsNil)
-
-	req, err := http.NewRequest("POST", "/api/files/readonly", body)
-	c.Assert(err, IsNil)
-	req.Header.Set("Content-Type", writer.FormDataContentType())
-
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-	c.Assert(w.Code, Equals, 500)
-}
-
-func (s *FilesUploadDiskFullSuite) TestDeleteDirReturns500OnRemoveFailure(c *C) {
-	parent := s.aptlyContext.UploadPath()
-	base := filepath.Join(parent, "cantremove")
-	c.Assert(os.MkdirAll(base, 0777), IsNil)
-	c.Assert(os.WriteFile(filepath.Join(base, "a.txt"), []byte("a"), 0644), IsNil)
-
-	c.Assert(os.Chmod(parent, 0555), IsNil)
-	defer func() { _ = os.Chmod(parent, 0777) }()
-
-	req, err := http.NewRequest("DELETE", "/api/files/cantremove", nil)
-	c.Assert(err, IsNil)
-	w := httptest.NewRecorder()
-	s.router.ServeHTTP(w, req)
-	c.Assert(w.Code, Equals, 500)
-}

api/gpg.go

@@ -13,34 +13,26 @@ import (
 )
 
 type gpgAddKeyParams struct {
+	// Keyserver, when downloading GpgKeyIDs
+	Keyserver string `json:"Keyserver"       example:"hkp://keyserver.ubuntu.com:80"`
+	// GpgKeyIDs to download from Keyserver, comma separated list
+	GpgKeyID string `json:"GpgKeyID"        example:"EF0F382A1A7B6500,8B48AD6246925553"`
+	// Armored gpg public ket, instead of downloading from keyserver
+	GpgKeyArmor string `json:"GpgKeyArmor"     example:""`
 	// Keyring for adding the keys (default: trustedkeys.gpg)
 	Keyring string `json:"Keyring"         example:"trustedkeys.gpg"`
-
-	// Add ASCII armored gpg public key, do not download from keyserver
-	GpgKeyArmor string `json:"GpgKeyArmor"     example:""`
-
-	// Keyserver to download keys provided in `GpgKeyID`
-	Keyserver string `json:"Keyserver"       example:"hkp://keyserver.ubuntu.com:80"`
-	// Keys do download from `Keyserver`, separated by space
-	GpgKeyID string `json:"GpgKeyID"        example:"EF0F382A1A7B6500 8B48AD6246925553"`
 }
 
 // @Summary Add GPG Keys
 // @Description **Adds GPG keys to aptly keyring**
 // @Description
-// @Description Add GPG public keys for verifying remote repositories for mirroring.
+// @Description Add GPG public keys for veryfing remote repositories for mirroring.
-// @Description
-// @Description Keys can be added in two ways:
-// @Description * By providing the ASCII armord key in `GpgKeyArmor` (leave Keyserver and GpgKeyID empty)
-// @Description * By providing a `Keyserver` and one or more key IDs in `GpgKeyID`, separated by space (leave GpgKeyArmor empty)
-// @Description
 // @Tags Mirrors
-// @Consume  json
-// @Param request body gpgAddKeyParams true "Parameters"
 // @Produce json
 // @Success 200 {object} string "OK"
 // @Failure 400 {object} Error "Bad Request"
-// @Router /api/gpg/key [post]
+// @Failure 404 {object} Error "Not Found"
+// @Router /api/gpg [post]
 func apiGPGAddKey(c *gin.Context) {
 	b := gpgAddKeyParams{}
 	if c.Bind(&b) != nil {
@@ -68,7 +60,7 @@ func apiGPGAddKey(c *gin.Context) {
 			AbortWithJSONError(c, 400, err)
 			return
 		}
-		defer func() { _ = os.RemoveAll(tempdir) }()
+		defer os.RemoveAll(tempdir)
 
 		keypath := filepath.Join(tempdir, "key")
 		keyfile, e := os.Create(keypath)

api/metrics.go

@@ -102,7 +102,7 @@ func countPackagesByRepos() {
 
 		components := repo.Components()
 		for _, c := range components {
-			count := float64(len(repo.RefList(c).Refs))
+			count := float64(repo.RefList(c).Len())
 			apiReposPackageCountGauge.WithLabelValues(fmt.Sprintf("%s", (repo.SourceNames())), repo.Distribution, c).Set(count)
 		}
 

api/middleware_test.go

@@ -67,17 +67,17 @@ func (s *MiddlewareSuite) TestJSONMiddleware4xx(c *C) {
 	outC := make(chan string)
 	go func() {
 		var buf bytes.Buffer
-		_, _ = io.Copy(&buf, s.logReader)
+		io.Copy(&buf, s.logReader)
 		fmt.Println(buf.String())
 		outC <- buf.String()
 	}()
 
 	s.HTTPRequest(http.MethodGet, "/", nil)
-	_ = s.logWriter.Close()
+	s.logWriter.Close()
 	capturedOutput := <-outC
 
 	var jsonMap map[string]interface{}
-	_ = json.Unmarshal([]byte(capturedOutput), &jsonMap)
+	json.Unmarshal([]byte(capturedOutput), &jsonMap)
 
 	if val, ok := jsonMap["level"]; ok {
 		c.Check(val, Equals, "warn")
@@ -130,17 +130,17 @@ func (s *MiddlewareSuite) TestJSONMiddleware2xx(c *C) {
 	outC := make(chan string)
 	go func() {
 		var buf bytes.Buffer
-		_, _ = io.Copy(&buf, s.logReader)
+		io.Copy(&buf, s.logReader)
 		fmt.Println(buf.String())
 		outC <- buf.String()
 	}()
 
 	s.HTTPRequest(http.MethodGet, "/api/healthy", nil)
-	_ = s.logWriter.Close()
+	s.logWriter.Close()
 	capturedOutput := <-outC
 
 	var jsonMap map[string]interface{}
-	_ = json.Unmarshal([]byte(capturedOutput), &jsonMap)
+	json.Unmarshal([]byte(capturedOutput), &jsonMap)
 
 	if val, ok := jsonMap["level"]; ok {
 		c.Check(val, Equals, "info")
@@ -153,17 +153,17 @@ func (s *MiddlewareSuite) TestJSONMiddleware5xx(c *C) {
 	outC := make(chan string)
 	go func() {
 		var buf bytes.Buffer
-		_, _ = io.Copy(&buf, s.logReader)
+		io.Copy(&buf, s.logReader)
 		fmt.Println(buf.String())
 		outC <- buf.String()
 	}()
 
 	s.HTTPRequest(http.MethodGet, "/api/ready", nil)
-	_ = s.logWriter.Close()
+	s.logWriter.Close()
 	capturedOutput := <-outC
 
 	var jsonMap map[string]interface{}
-	_ = json.Unmarshal([]byte(capturedOutput), &jsonMap)
+	json.Unmarshal([]byte(capturedOutput), &jsonMap)
 
 	if val, ok := jsonMap["level"]; ok {
 		c.Check(val, Equals, "error")
@@ -176,17 +176,17 @@ func (s *MiddlewareSuite) TestJSONMiddlewareRaw(c *C) {
 	outC := make(chan string)
 	go func() {
 		var buf bytes.Buffer
-		_, _ = io.Copy(&buf, s.logReader)
+		io.Copy(&buf, s.logReader)
 		fmt.Println(buf.String())
 		outC <- buf.String()
 	}()
 
 	s.HTTPRequest(http.MethodGet, "/api/healthy?test=raw", nil)
-	_ = s.logWriter.Close()
+	s.logWriter.Close()
 	capturedOutput := <-outC
 
 	var jsonMap map[string]interface{}
-	_ = json.Unmarshal([]byte(capturedOutput), &jsonMap)
+	json.Unmarshal([]byte(capturedOutput), &jsonMap)
 
 	fmt.Println(capturedOutput)
 

api/mirror.go

@@ -43,7 +43,7 @@ func apiMirrorsList(c *gin.Context) {
 	collection := collectionFactory.RemoteRepoCollection()
 
 	result := []*deb.RemoteRepo{}
-	_ = collection.ForEach(func(repo *deb.RemoteRepo) error {
+	collection.ForEach(func(repo *deb.RemoteRepo) error {
 		result = append(result, repo)
 		return nil
 	})
@@ -150,7 +150,7 @@ func apiMirrorsCreate(c *gin.Context) {
 		return
 	}
 
-	err = collection.Add(repo)
+	err = collection.Add(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, fmt.Errorf("unable to add mirror: %s", err))
 		return
@@ -175,9 +175,9 @@ func apiMirrorsDrop(c *gin.Context) {
 	name := c.Params.ByName("name")
 	force := c.Request.URL.Query().Get("force") == "1"
 
-	// Phase 1: Pre-task validation (shallow load for 404 check only)
 	collectionFactory := context.NewCollectionFactory()
 	mirrorCollection := collectionFactory.RemoteRepoCollection()
+	snapshotCollection := collectionFactory.SnapshotCollection()
 
 	repo, err := mirrorCollection.ByName(name)
 	if err != nil {
@@ -187,34 +187,21 @@ func apiMirrorsDrop(c *gin.Context) {
 
 	resources := []string{string(repo.Key())}
 	taskName := fmt.Sprintf("Delete mirror %s", name)
-
 	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		// Phase 2: Inside task lock - create fresh collections
+		err := repo.CheckLock()
-		taskCollectionFactory := context.NewCollectionFactory()
-		taskMirrorCollection := taskCollectionFactory.RemoteRepoCollection()
-		taskSnapshotCollection := taskCollectionFactory.SnapshotCollection()
-
-		// Fresh load after lock acquired
-		repo, err := taskMirrorCollection.ByName(name)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to drop: %v", err)
-		}
-
-		err = repo.CheckLock()
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to drop: %v", err)
 		}
 
 		if !force {
-			// Fresh checks with current collections
+			snapshots := snapshotCollection.ByRemoteRepoSource(repo)
-			snapshots := taskSnapshotCollection.ByRemoteRepoSource(repo)
 
 			if len(snapshots) > 0 {
 				return &task.ProcessReturnValue{Code: http.StatusForbidden, Value: nil}, fmt.Errorf("won't delete mirror with snapshots, use 'force=1' to override")
 			}
 		}
 
-		err = taskMirrorCollection.Drop(repo)
+		err = mirrorCollection.Drop(repo)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to drop: %v", err)
 		}
@@ -242,10 +229,9 @@ func apiMirrorsShow(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(repo)
+	err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, fmt.Errorf("unable to show: %s", err))
-		return
 	}
 
 	c.JSON(200, repo)
@@ -274,7 +260,7 @@ func apiMirrorsPackages(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(repo)
+	err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, fmt.Errorf("unable to show: %s", err))
 	}
@@ -333,7 +319,7 @@ func apiMirrorsPackages(c *gin.Context) {
 	}
 
 	if c.Request.URL.Query().Get("format") == "details" {
-		_ = list.ForEach(func(p *deb.Package) error {
+		list.ForEach(func(p *deb.Package) error {
 			result = append(result, p)
 			return nil
 		})
@@ -347,8 +333,26 @@ func apiMirrorsPackages(c *gin.Context) {
 type mirrorUpdateParams struct {
 	// Change mirror name to `Name`
 	Name string `                 json:"Name"                   example:"mirror1"`
-	// Gpg keyring(s) for verifying Release file
+	// Url of the archive to mirror
+	ArchiveURL string `           json:"ArchiveURL"             example:"http://deb.debian.org/debian"`
+	// Package query that is applied to mirror packages
+	Filter string `               json:"Filter"                 example:"xserver-xorg"`
+	// Limit mirror to those architectures, if not specified aptly would fetch all architectures
+	Architectures []string `      json:"Architectures"          example:"amd64"`
+	// Components to mirror, if not specified aptly would fetch all components
+	Components []string `         json:"Components"             example:"main"`
+	// Gpg keyring(s) for verifing Release file
 	Keyrings []string `           json:"Keyrings"               example:"trustedkeys.gpg"`
+	// Set "true" to include dependencies of matching packages when filtering
+	FilterWithDeps bool `         json:"FilterWithDeps"`
+	// Set "true" to mirror source packages
+	DownloadSources bool `        json:"DownloadSources"`
+	// Set "true" to mirror udeb files
+	DownloadUdebs bool `          json:"DownloadUdebs"`
+	// Set "true" to skip checking if the given components are in the Release file
+	SkipComponentCheck bool `     json:"SkipComponentCheck"`
+	// Set "true" to skip checking if the given architectures are in the Release file
+	SkipArchitectureCheck bool `  json:"SkipArchitectureCheck"`
 	// Set "true" to ignore checksum errors
 	IgnoreChecksums bool `        json:"IgnoreChecksums"`
 	// Set "true" to skip the verification of Release file signatures
@@ -383,14 +387,21 @@ func apiMirrorsUpdate(c *gin.Context) {
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.RemoteRepoCollection()
 
-	name := c.Params.ByName("name")
+	remote, err = collection.ByName(c.Params.ByName("name"))
-	remote, err = collection.ByName(name)
 	if err != nil {
 		AbortWithJSONError(c, 404, err)
 		return
 	}
 
 	b.Name = remote.Name
+	b.DownloadUdebs = remote.DownloadUdebs
+	b.DownloadSources = remote.DownloadSources
+	b.SkipComponentCheck = remote.SkipComponentCheck
+	b.SkipArchitectureCheck = remote.SkipArchitectureCheck
+	b.FilterWithDeps = remote.FilterWithDeps
+	b.Filter = remote.Filter
+	b.Architectures = remote.Architectures
+	b.Components = remote.Components
 	b.IgnoreSignatures = context.Config().GpgDisableVerify
 
 	log.Info().Msgf("%s: Starting mirror update", b.Name)
@@ -399,7 +410,6 @@ func apiMirrorsUpdate(c *gin.Context) {
 		return
 	}
 
-	// Pre-task validation of new name if provided
 	if b.Name != remote.Name {
 		_, err = collection.ByName(b.Name)
 		if err == nil {
@@ -408,6 +418,27 @@ func apiMirrorsUpdate(c *gin.Context) {
 		}
 	}
 
+	if b.DownloadUdebs != remote.DownloadUdebs {
+		if remote.IsFlat() && b.DownloadUdebs {
+			AbortWithJSONError(c, 400, fmt.Errorf("unable to update: flat mirrors don't support udebs"))
+			return
+		}
+	}
+
+	if b.ArchiveURL != "" {
+		remote.SetArchiveRoot(b.ArchiveURL)
+	}
+
+	remote.Name = b.Name
+	remote.DownloadUdebs = b.DownloadUdebs
+	remote.DownloadSources = b.DownloadSources
+	remote.SkipComponentCheck = b.SkipComponentCheck
+	remote.SkipArchitectureCheck = b.SkipArchitectureCheck
+	remote.FilterWithDeps = b.FilterWithDeps
+	remote.Filter = b.Filter
+	remote.Architectures = b.Architectures
+	remote.Components = b.Components
+
 	verifier, err := getVerifier(b.Keyrings)
 	if err != nil {
 		AbortWithJSONError(c, 400, fmt.Errorf("unable to initialize GPG verifier: %s", err))
@@ -416,26 +447,9 @@ func apiMirrorsUpdate(c *gin.Context) {
 
 	resources := []string{string(remote.Key())}
 	maybeRunTaskInBackground(c, "Update mirror "+b.Name, resources, func(out aptly.Progress, detail *task.Detail) (*task.ProcessReturnValue, error) {
-		// Phase 2: Inside task lock - create fresh factory
-		taskCollectionFactory := context.NewCollectionFactory()
-		taskCollection := taskCollectionFactory.RemoteRepoCollection()
-
-		// Fresh load after lock acquired (use captured `name` variable, not gin context)
-		remote, err := taskCollection.ByName(name)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
-		}
-
-		// Fresh rename check inside lock (if renaming)
-		if b.Name != remote.Name {
-			_, err := taskCollection.ByName(b.Name)
-			if err == nil {
-				return &task.ProcessReturnValue{Code: http.StatusConflict, Value: nil}, fmt.Errorf("unable to rename: mirror %s already exists", b.Name)
-			}
-		}
 
 		downloader := context.NewDownloader(out)
-		err = remote.Fetch(downloader, verifier, b.IgnoreSignatures)
+		err := remote.Fetch(downloader, verifier, b.IgnoreSignatures)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
 		}
@@ -447,7 +461,7 @@ func apiMirrorsUpdate(c *gin.Context) {
 			}
 		}
 
-		err = remote.DownloadPackageIndexes(out, downloader, verifier, taskCollectionFactory, b.IgnoreSignatures, remote.SkipComponentCheck)
+		err = remote.DownloadPackageIndexes(out, downloader, verifier, collectionFactory, b.IgnoreSignatures, b.SkipComponentCheck)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
 		}
@@ -466,8 +480,8 @@ func apiMirrorsUpdate(c *gin.Context) {
 			}
 		}
 
-		queue, downloadSize, err := remote.BuildDownloadQueue(context.PackagePool(), taskCollectionFactory.PackageCollection(),
+		queue, downloadSize, err := remote.BuildDownloadQueue(context.PackagePool(), collectionFactory.PackageCollection(),
-			taskCollectionFactory.ChecksumCollection(nil), b.SkipExistingPackages)
+			collectionFactory.ChecksumCollection(nil), b.SkipExistingPackages)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
 		}
@@ -477,12 +491,12 @@ func apiMirrorsUpdate(c *gin.Context) {
 			e := context.ReOpenDatabase()
 			if e == nil {
 				remote.MarkAsIdle()
-				_ = taskCollection.Update(remote)
+				collection.Update(remote, collectionFactory.RefListCollection())
 			}
 		}()
 
 		remote.MarkAsUpdating()
-		err = taskCollection.Update(remote)
+		err = collection.Update(remote, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
 		}
@@ -565,7 +579,7 @@ func apiMirrorsUpdate(c *gin.Context) {
 							file, e = os.CreateTemp("", task.File.Filename)
 							if e == nil {
 								task.TempDownPath = file.Name()
-								_ = file.Close()
+								file.Close()
 							}
 						}
 						if e != nil {
@@ -586,7 +600,7 @@ func apiMirrorsUpdate(c *gin.Context) {
 						}
 
 						// and import it back to the pool
-						task.File.PoolPath, err = context.PackagePool().Import(task.TempDownPath, task.File.Filename, &task.File.Checksums, true, taskCollectionFactory.ChecksumCollection(nil))
+						task.File.PoolPath, err = context.PackagePool().Import(task.TempDownPath, task.File.Filename, &task.File.Checksums, true, collectionFactory.ChecksumCollection(nil))
 						if err != nil {
 							//return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to import file: %s", err)
 							pushError(err)
@@ -639,8 +653,8 @@ func apiMirrorsUpdate(c *gin.Context) {
 		}
 
 		log.Info().Msgf("%s: Finalizing download...", b.Name)
-		_ = remote.FinalizeDownload(taskCollectionFactory, out)
+		remote.FinalizeDownload(collectionFactory, out)
-		err = taskCollection.Update(remote)
+		err = collectionFactory.RemoteRepoCollection().Update(remote, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
 		}

api/mirror_test.go

@@ -9,7 +9,7 @@ import (
 )
 
 type MirrorSuite struct {
-	APISuite
+	ApiSuite
 }
 
 var _ = Suite(&MirrorSuite{})

api/packages_test.go

@@ -5,7 +5,7 @@ import (
 )
 
 type PackagesSuite struct {
-	APISuite
+	ApiSuite
 }
 
 var _ = Suite(&PackagesSuite{})

api/publish.go

@@ -16,8 +16,8 @@ import (
 type signingParams struct {
 	// Don't sign published repository
 	Skip bool `                json:"Skip"           example:"false"`
-	// GPG key ID(s) to use when signing the release, separated by comma, and if not specified, default configured key(s) are used
+	// GPG key ID to use when signing the release, if not specified default key is used
-	GpgKey string `            json:"GpgKey"         example:"KEY_ID_a, KEY_ID_b"`
+	GpgKey string `            json:"GpgKey"         example:"A0546A43624A8331"`
 	// GPG keyring to use (instead of default)
 	Keyring string `           json:"Keyring"        example:"trustedkeys.gpg"`
 	// GPG secret keyring to use (instead of default) Note: depreciated with gpg2
@@ -41,21 +41,7 @@ func getSigner(options *signingParams) (pgp.Signer, error) {
 	}
 
 	signer := context.GetSigner()
-
+	signer.SetKey(options.GpgKey)
-	var multiGpgKeys []string
-	// REST params have priority over config
-	if options.GpgKey != "" {
-		for _, p := range strings.Split(options.GpgKey, ",") {
-			if t := strings.TrimSpace(p); t != "" {
-				multiGpgKeys = append(multiGpgKeys, t)
-			}
-		}
-	} else if len(context.Config().GpgKeys) > 0 {
-		multiGpgKeys = context.Config().GpgKeys
-	}
-	for _, gpgKey := range multiGpgKeys {
-		signer.SetKey(gpgKey)
-	}
 	signer.SetKeyRing(options.Keyring, options.SecretKeyring)
 	signer.SetPassphrase(options.Passphrase, options.PassphraseFile)
 
@@ -124,7 +110,7 @@ func apiPublishList(c *gin.Context) {
 // @Description See also: `aptly publish show`
 // @Tags Publish
 // @Produce json
-// @Param prefix path string true "publishing prefix, use `:.` instead of `.` because it is ambiguous in URLs"
+// @Param prefix path string true "publishing prefix, use `:.` instead of `.` because it is ambigious in URLs"
 // @Param distribution path string true "distribution name"
 // @Success 200 {object} deb.PublishedRepo
 // @Failure 404 {object} Error "Published repository not found"
@@ -160,6 +146,10 @@ type publishedRepoCreateParams struct {
 	Sources []sourceParams `binding:"required"    json:"Sources"`
 	// Distribution name, if missing Aptly would try to guess from sources
 	Distribution string `                         json:"Distribution"          example:"bookworm"`
+	// Value of Label: field in published repository stanza
+	Label string `                                json:"Label"                 example:""`
+	// Value of Origin: field in published repository stanza
+	Origin string `                               json:"Origin"                example:""`
 	// when publishing, overwrite files in pool/ directory without notice
 	ForceOverwrite bool `                         json:"ForceOverwrite"        example:"false"`
 	// Override list of published architectures
@@ -192,7 +182,7 @@ type publishedRepoCreateParams struct {
 // @Description **Example:**
 // @Description ```
 // @Description $ curl -X POST -H 'Content-Type: application/json' --data '{"Distribution": "wheezy", "Sources": [{"Name": "aptly-repo"}]}' http://localhost:8080/api/publish//repos
-// @Description {"Architectures":["i386"],"Distribution":"wheezy","Prefix":".","SourceKind":"local","Sources":[{"Component":"main","Name":"aptly-repo"}],"Storage":""}
+// @Description {"Architectures":["i386"],"Distribution":"wheezy","Label":"","Origin":"","Prefix":".","SourceKind":"local","Sources":[{"Component":"main","Name":"aptly-repo"}],"Storage":""}
 // @Description ```
 // @Description
 // @Description See also: `aptly publish create`
@@ -259,7 +249,7 @@ func apiPublishRepoOrSnapshot(c *gin.Context) {
 				return
 			}
 
-			resources = append(resources, string(snapshot.Key()))
+			resources = append(resources, string(snapshot.ResourceKey()))
 			sources = append(sources, snapshot)
 		}
 	} else if b.SourceKind == deb.SourceLocalRepo {
@@ -290,24 +280,11 @@ func apiPublishRepoOrSnapshot(c *gin.Context) {
 		multiDist = *b.MultiDist
 	}
 
-	// Non-MultiDist publishes share a single pool/ directory under the
+	collection := collectionFactory.PublishedRepoCollection()
-	// prefix.  Lock at the prefix level so that concurrent publish/drop
-	// operations on sibling distributions cannot race during cleanup.
-	if !multiDist {
-		storagePrefix := prefix
-		if storage != "" {
-			storagePrefix = storage + ":" + prefix
-		}
-
-		resources = append(resources, deb.PrefixPoolLockKey(storagePrefix))
-	}
 
 	taskName := fmt.Sprintf("Publish %s repository %s/%s with components \"%s\" and sources \"%s\"",
 		b.SourceKind, param, b.Distribution, strings.Join(components, `", "`), strings.Join(names, `", "`))
 	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, detail *task.Detail) (*task.ProcessReturnValue, error) {
-		taskCollectionFactory := context.NewCollectionFactory()
-		taskCollection := taskCollectionFactory.PublishedRepoCollection()
-
 		taskDetail := task.PublishDetail{
 			Detail: detail,
 		}
@@ -319,11 +296,11 @@ func apiPublishRepoOrSnapshot(c *gin.Context) {
 		for _, source := range sources {
 			switch s := source.(type) {
 			case *deb.Snapshot:
-				snapshotCollection := taskCollectionFactory.SnapshotCollection()
+				snapshotCollection := collectionFactory.SnapshotCollection()
-				err = snapshotCollection.LoadComplete(s)
+				err = snapshotCollection.LoadComplete(s, collectionFactory.RefListCollection())
 			case *deb.LocalRepo:
-				localCollection := taskCollectionFactory.LocalRepoCollection()
+				localCollection := collectionFactory.LocalRepoCollection()
-				err = localCollection.LoadComplete(s)
+				err = localCollection.LoadComplete(s, collectionFactory.RefListCollection())
 			default:
 				err = fmt.Errorf("unexpected type for source: %T", source)
 			}
@@ -332,17 +309,23 @@ func apiPublishRepoOrSnapshot(c *gin.Context) {
 			}
 		}
 
-		published, err := deb.NewPublishedRepo(storage, prefix, b.Distribution, b.Architectures, components, sources, taskCollectionFactory, multiDist)
+		published, err := deb.NewPublishedRepo(storage, prefix, b.Distribution, b.Architectures, components, sources, collectionFactory, multiDist)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to publish: %s", err)
 		}
 
+		resources = append(resources, string(published.Key()))
+
+		if b.Origin != "" {
+			published.Origin = b.Origin
+		}
 		if b.NotAutomatic != "" {
 			published.NotAutomatic = b.NotAutomatic
 		}
 		if b.ButAutomaticUpgrades != "" {
 			published.ButAutomaticUpgrades = b.ButAutomaticUpgrades
 		}
+		published.Label = b.Label
 
 		published.SkipContents = context.Config().SkipContentsPublishing
 		if b.SkipContents != nil {
@@ -358,18 +341,18 @@ func apiPublishRepoOrSnapshot(c *gin.Context) {
 			published.AcquireByHash = *b.AcquireByHash
 		}
 
-		duplicate := taskCollection.CheckDuplicate(published)
+		duplicate := collection.CheckDuplicate(published)
 		if duplicate != nil {
-			_ = taskCollectionFactory.PublishedRepoCollection().LoadComplete(duplicate, taskCollectionFactory)
+			collectionFactory.PublishedRepoCollection().LoadComplete(duplicate, collectionFactory)
 			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, fmt.Errorf("prefix/distribution already used by another published repo: %s", duplicate)
 		}
 
-		err = published.Publish(context.PackagePool(), context, taskCollectionFactory, signer, publishOutput, b.ForceOverwrite, context.SkelPath())
+		err = published.Publish(context.PackagePool(), context, collectionFactory, signer, publishOutput, b.ForceOverwrite, context.SkelPath())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to publish: %s", err)
 		}
 
-		err = taskCollection.Add(published)
+		err = collection.Add(published, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
 		}
@@ -410,6 +393,7 @@ type publishedRepoUpdateSwitchParams struct {
 // @Description
 // @Description See also: `aptly publish update` / `aptly publish switch`
 // @Tags Publish
+// @Produce json
 // @Param prefix path string true "publishing prefix"
 // @Param distribution path string true "distribution name"
 // @Param _async query bool false "Run in background and return task object"
@@ -441,7 +425,6 @@ func apiPublishUpdateSwitch(c *gin.Context) {
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.PublishedRepoCollection()
 	snapshotCollection := collectionFactory.SnapshotCollection()
-	localRepoCollection := collectionFactory.LocalRepoCollection()
 
 	published, err := collection.ByStoragePrefixDistribution(storage, prefix, distribution)
 	if err != nil {
@@ -449,74 +432,46 @@ func apiPublishUpdateSwitch(c *gin.Context) {
 		return
 	}
 
-	resources := []string{string(published.Key())}
-
 	if published.SourceKind == deb.SourceLocalRepo {
 		if len(b.Snapshots) > 0 {
 			AbortWithJSONError(c, http.StatusBadRequest, fmt.Errorf("snapshots shouldn't be given when updating local repo"))
 			return
 		}
-		for _, uuid := range published.Sources {
-			repo, err2 := localRepoCollection.ByUUID(uuid)
-			if err2 != nil {
-				AbortWithJSONError(c, http.StatusNotFound, err2)
-				return
-			}
-			resources = append(resources, string(repo.Key()))
-		}
 	} else if published.SourceKind == deb.SourceSnapshot {
 		for _, snapshotInfo := range b.Snapshots {
-			snapshot, err2 := snapshotCollection.ByName(snapshotInfo.Name)
+			_, err2 := snapshotCollection.ByName(snapshotInfo.Name)
 			if err2 != nil {
 				AbortWithJSONError(c, http.StatusNotFound, err2)
 				return
 			}
-			resources = append(resources, string(snapshot.Key()))
 		}
 	} else {
 		AbortWithJSONError(c, http.StatusInternalServerError, fmt.Errorf("unknown published repository type"))
 		return
 	}
 
-	// Non-MultiDist distributions share a single pool/ directory under the
+	if b.SkipContents != nil {
-	// prefix.  Acquire the prefix-level pool lock so that concurrent updates
+		published.SkipContents = *b.SkipContents
-	// on sibling distributions are serialised and cannot race during cleanup.
-	if !published.MultiDist {
-		resources = append(resources, deb.PrefixPoolLockKey(published.StoragePrefix()))
 	}
 
-	// Field mutations and fresh DB load are deferred to inside the task so
+	if b.SkipBz2 != nil {
-	// they always operate on a consistent state after the lock is held.
+		published.SkipBz2 = *b.SkipBz2
+	}
+
+	if b.AcquireByHash != nil {
+		published.AcquireByHash = *b.AcquireByHash
+	}
+
+	if b.MultiDist != nil {
+		published.MultiDist = *b.MultiDist
+	}
+
+	resources := []string{string(published.Key())}
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
 	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		taskCollectionFactory := context.NewCollectionFactory()
+		err = collection.LoadComplete(published, collectionFactory, collectionFactory.RefListCollection())
-		taskCollection := taskCollectionFactory.PublishedRepoCollection()
-
-		published, err := taskCollection.ByStoragePrefixDistribution(storage, prefix, distribution)
 		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
+			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("Unable to update: %s", err)
-		}
-
-		err = taskCollection.LoadComplete(published, taskCollectionFactory)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
-		}
-
-		// Capture MultiDist before mutations to detect a false→true transition.
-		prevMultiDist := published.MultiDist
-
-		// Apply field mutations on the freshly loaded object.
-		if b.SkipContents != nil {
-			published.SkipContents = *b.SkipContents
-		}
-		if b.SkipBz2 != nil {
-			published.SkipBz2 = *b.SkipBz2
-		}
-		if b.AcquireByHash != nil {
-			published.AcquireByHash = *b.AcquireByHash
-		}
-		if b.MultiDist != nil {
-			published.MultiDist = *b.MultiDist
 		}
 
 		revision := published.ObtainRevision()
@@ -530,17 +485,17 @@ func apiPublishUpdateSwitch(c *gin.Context) {
 			}
 		}
 
-		result, err := published.Update(taskCollectionFactory, out)
+		result, err := published.Update(collectionFactory, out)
 		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
+			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("Unable to update: %s", err)
 		}
 
-		err = published.Publish(context.PackagePool(), context, taskCollectionFactory, signer, out, b.ForceOverwrite, context.SkelPath())
+		err = published.Publish(context.PackagePool(), context, collectionFactory, signer, out, b.ForceOverwrite, context.SkelPath())
 		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
+			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("Unable to update: %s", err)
 		}
 
-		err = taskCollection.Update(published)
+		err = collection.Update(published, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
 		}
@@ -548,19 +503,10 @@ func apiPublishUpdateSwitch(c *gin.Context) {
 		if b.SkipCleanup == nil || !*b.SkipCleanup {
 			cleanComponents := make([]string, 0, len(result.UpdatedSources)+len(result.RemovedSources))
 			cleanComponents = append(append(cleanComponents, result.UpdatedComponents()...), result.RemovedComponents()...)
-			err = taskCollection.CleanupPrefixComponentFiles(context, published, cleanComponents, taskCollectionFactory, out)
+			err = collection.CleanupPrefixComponentFiles(context, published, cleanComponents, collectionFactory, out)
 			if err != nil {
 				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
 			}
-			// When MultiDist is toggled, the old pool layout still has files that
-			// CleanupPrefixComponentFiles won't touch (it only scans the new layout).
-			// Run a second pass over the previous layout to remove stale files.
-			if prevMultiDist != published.MultiDist {
-				err = taskCollection.CleanupAfterMultiDistToggle(context, published, prevMultiDist, cleanComponents, taskCollectionFactory, out)
-				if err != nil {
-					return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to clean legacy pool: %s", err)
-				}
-			}
 		}
 
 		return &task.ProcessReturnValue{Code: http.StatusOK, Value: published}, nil
@@ -605,19 +551,10 @@ func apiPublishDrop(c *gin.Context) {
 	}
 
 	resources := []string{string(published.Key())}
-	// Non-MultiDist distributions share a single pool/ directory under the
-	// prefix.  Acquire the prefix-level pool lock so that a drop cannot race
-	// with a concurrent update or drop of a sibling distribution during cleanup.
-	if !published.MultiDist {
-		resources = append(resources, deb.PrefixPoolLockKey(published.StoragePrefix()))
-	}
 	taskName := fmt.Sprintf("Delete published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
 	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		taskCollectionFactory := context.NewCollectionFactory()
+		err := collection.Remove(context, storage, prefix, distribution,
-		taskCollection := taskCollectionFactory.PublishedRepoCollection()
+			collectionFactory, out, force, skipCleanup)
-
-		err := taskCollection.Remove(context, storage, prefix, distribution,
-			taskCollectionFactory, out, force, skipCleanup)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to drop: %s", err)
 		}
@@ -653,52 +590,43 @@ func apiPublishAddSource(c *gin.Context) {
 	storage, prefix := deb.ParsePrefix(param)
 	distribution := slashEscape(c.Params.ByName("distribution"))
 
-	if c.Bind(&b) != nil {
-		return
-	}
-
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.PublishedRepoCollection()
 
-	// Load shallowly (no LoadComplete) to verify existence and obtain the
-	// resource key and task name.  The actual mutation is performed inside
-	// the task on a freshly loaded copy to prevent lost-update races.
 	published, err := collection.ByStoragePrefixDistribution(storage, prefix, distribution)
 	if err != nil {
 		AbortWithJSONError(c, http.StatusNotFound, fmt.Errorf("unable to create: %s", err))
 		return
 	}
 
+	err = collection.LoadComplete(published, collectionFactory)
+	if err != nil {
+		AbortWithJSONError(c, http.StatusInternalServerError, fmt.Errorf("unable to create: %s", err))
+		return
+	}
+
+	if c.Bind(&b) != nil {
+		return
+	}
+
+	revision := published.ObtainRevision()
+	sources := revision.Sources
+
+	component := b.Component
+	name := b.Name
+
+	_, exists := sources[component]
+	if exists {
+		AbortWithJSONError(c, http.StatusBadRequest, fmt.Errorf("unable to create: Component '%s' already exists", component))
+		return
+	}
+
+	sources[component] = name
+
 	resources := []string{string(published.Key())}
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
-	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
+	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		taskCollectionFactory := context.NewCollectionFactory()
+		err = collection.Update(published)
-		taskCollection := taskCollectionFactory.PublishedRepoCollection()
-
-		published, err := taskCollection.ByStoragePrefixDistribution(storage, prefix, distribution)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusNotFound, Value: nil}, fmt.Errorf("unable to create: %s", err)
-		}
-
-		err = taskCollection.LoadComplete(published, taskCollectionFactory)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to create: %s", err)
-		}
-
-		revision := published.ObtainRevision()
-		sources := revision.Sources
-
-		component := b.Component
-		name := b.Name
-
-		_, exists := sources[component]
-		if exists {
-			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, fmt.Errorf("unable to create: Component '%s' already exists", component)
-		}
-
-		sources[component] = name
-
-		err = taskCollection.Update(published)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
 		}
@@ -780,48 +708,39 @@ func apiPublishSetSources(c *gin.Context) {
 	storage, prefix := deb.ParsePrefix(param)
 	distribution := slashEscape(c.Params.ByName("distribution"))
 
-	if c.Bind(&b) != nil {
-		return
-	}
-
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.PublishedRepoCollection()
 
-	// Load shallowly for 404 check, resource key, and task name.
-	// Full load and mutation happen inside the task.
 	published, err := collection.ByStoragePrefixDistribution(storage, prefix, distribution)
 	if err != nil {
 		AbortWithJSONError(c, http.StatusNotFound, fmt.Errorf("unable to update: %s", err))
 		return
 	}
 
+	err = collection.LoadComplete(published, collectionFactory)
+	if err != nil {
+		AbortWithJSONError(c, http.StatusInternalServerError, fmt.Errorf("unable to update: %s", err))
+		return
+	}
+
+	if c.Bind(&b) != nil {
+		return
+	}
+
+	revision := published.ObtainRevision()
+	sources := make(map[string]string, len(b))
+	revision.Sources = sources
+
+	for _, source := range b {
+		component := source.Component
+		name := source.Name
+		sources[component] = name
+	}
+
 	resources := []string{string(published.Key())}
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
-	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
+	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		taskCollectionFactory := context.NewCollectionFactory()
+		err = collection.Update(published)
-		taskCollection := taskCollectionFactory.PublishedRepoCollection()
-
-		published, err := taskCollection.ByStoragePrefixDistribution(storage, prefix, distribution)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusNotFound, Value: nil}, fmt.Errorf("unable to update: %s", err)
-		}
-
-		err = taskCollection.LoadComplete(published, taskCollectionFactory)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
-		}
-
-		revision := published.ObtainRevision()
-		sources := make(map[string]string, len(b))
-		revision.Sources = sources
-
-		for _, source := range b {
-			component := source.Component
-			name := source.Name
-			sources[component] = name
-		}
-
-		err = taskCollection.Update(published)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
 		}
@@ -854,33 +773,24 @@ func apiPublishDropChanges(c *gin.Context) {
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.PublishedRepoCollection()
 
-	// Load shallowly for 404 check, resource key, and task name.
-	// Full load and DropRevision happen inside the task.
 	published, err := collection.ByStoragePrefixDistribution(storage, prefix, distribution)
 	if err != nil {
 		AbortWithJSONError(c, http.StatusNotFound, fmt.Errorf("unable to delete: %s", err))
 		return
 	}
 
+	err = collection.LoadComplete(published, collectionFactory)
+	if err != nil {
+		AbortWithJSONError(c, http.StatusInternalServerError, fmt.Errorf("unable to delete: %s", err))
+		return
+	}
+
+	published.DropRevision()
+
 	resources := []string{string(published.Key())}
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
-	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
+	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		taskCollectionFactory := context.NewCollectionFactory()
+		err = collection.Update(published)
-		taskCollection := taskCollectionFactory.PublishedRepoCollection()
-
-		published, err := taskCollection.ByStoragePrefixDistribution(storage, prefix, distribution)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusNotFound, Value: nil}, fmt.Errorf("unable to delete: %s", err)
-		}
-
-		err = taskCollection.LoadComplete(published, taskCollectionFactory)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to delete: %s", err)
-		}
-
-		published.DropRevision()
-
-		err = taskCollection.Update(published)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
 		}
@@ -916,58 +826,51 @@ func apiPublishUpdateSource(c *gin.Context) {
 	param := slashEscape(c.Params.ByName("prefix"))
 	storage, prefix := deb.ParsePrefix(param)
 	distribution := slashEscape(c.Params.ByName("distribution"))
-	urlComponent := slashEscape(c.Params.ByName("component"))
+	component := slashEscape(c.Params.ByName("component"))
-
-	// Default component to the URL path segment; the body may rename it.
-	b.Component = urlComponent
-	if c.Bind(&b) != nil {
-		return
-	}
 
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.PublishedRepoCollection()
 
-	// Load shallowly for 404 check, resource key, and task name.
-	// Full load and mutation happen inside the task.
 	published, err := collection.ByStoragePrefixDistribution(storage, prefix, distribution)
 	if err != nil {
 		AbortWithJSONError(c, http.StatusNotFound, fmt.Errorf("unable to update: %s", err))
 		return
 	}
 
+	err = collection.LoadComplete(published, collectionFactory)
+	if err != nil {
+		AbortWithJSONError(c, http.StatusInternalServerError, fmt.Errorf("unable to update: %s", err))
+		return
+	}
+
+	revision := published.ObtainRevision()
+	sources := revision.Sources
+
+	_, exists := sources[component]
+	if !exists {
+		AbortWithJSONError(c, http.StatusNotFound, fmt.Errorf("unable to update: Component '%s' does not exist", component))
+		return
+	}
+
+	b.Component = component
+	b.Name = revision.Sources[component]
+
+	if c.Bind(&b) != nil {
+		return
+	}
+
+	if b.Component != component {
+		delete(sources, component)
+	}
+
+	component = b.Component
+	name := b.Name
+	sources[component] = name
+
 	resources := []string{string(published.Key())}
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
-	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
+	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		taskCollectionFactory := context.NewCollectionFactory()
+		err = collection.Update(published)
-		taskCollection := taskCollectionFactory.PublishedRepoCollection()
-
-		published, err := taskCollection.ByStoragePrefixDistribution(storage, prefix, distribution)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusNotFound, Value: nil}, fmt.Errorf("unable to update: %s", err)
-		}
-
-		err = taskCollection.LoadComplete(published, taskCollectionFactory)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
-		}
-
-		revision := published.ObtainRevision()
-		sources := revision.Sources
-
-		_, exists := sources[urlComponent]
-		if !exists {
-			return &task.ProcessReturnValue{Code: http.StatusNotFound, Value: nil}, fmt.Errorf("unable to update: Component '%s' does not exist", urlComponent)
-		}
-
-		if b.Component != urlComponent {
-			delete(sources, urlComponent)
-		}
-
-		newComponent := b.Component
-		name := b.Name
-		sources[newComponent] = name
-
-		err = taskCollection.Update(published)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
 		}
@@ -1004,41 +907,33 @@ func apiPublishRemoveSource(c *gin.Context) {
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.PublishedRepoCollection()
 
-	// Load shallowly for 404 check, resource key, and task name.
-	// Full load and mutation happen inside the task.
 	published, err := collection.ByStoragePrefixDistribution(storage, prefix, distribution)
 	if err != nil {
 		AbortWithJSONError(c, http.StatusNotFound, fmt.Errorf("unable to delete: %s", err))
 		return
 	}
 
+	err = collection.LoadComplete(published, collectionFactory)
+	if err != nil {
+		AbortWithJSONError(c, http.StatusInternalServerError, fmt.Errorf("unable to delete: %s", err))
+		return
+	}
+
+	revision := published.ObtainRevision()
+	sources := revision.Sources
+
+	_, exists := sources[component]
+	if !exists {
+		AbortWithJSONError(c, http.StatusNotFound, fmt.Errorf("unable to delete: Component '%s' does not exist", component))
+		return
+	}
+
+	delete(sources, component)
+
 	resources := []string{string(published.Key())}
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
-	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
+	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		taskCollectionFactory := context.NewCollectionFactory()
+		err = collection.Update(published)
-		taskCollection := taskCollectionFactory.PublishedRepoCollection()
-
-		published, err := taskCollection.ByStoragePrefixDistribution(storage, prefix, distribution)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusNotFound, Value: nil}, fmt.Errorf("unable to delete: %s", err)
-		}
-
-		err = taskCollection.LoadComplete(published, taskCollectionFactory)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to delete: %s", err)
-		}
-
-		revision := published.ObtainRevision()
-		sources := revision.Sources
-
-		_, exists := sources[component]
-		if !exists {
-			return &task.ProcessReturnValue{Code: http.StatusNotFound, Value: nil}, fmt.Errorf("unable to delete: Component '%s' does not exist", component)
-		}
-
-		delete(sources, component)
-
-		err = taskCollection.Update(published)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
 		}
@@ -1102,92 +997,48 @@ func apiPublishUpdate(c *gin.Context) {
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.PublishedRepoCollection()
 
-	// Load shallowly for 404 check, resource key, and task name.
-	// Full load and field mutations happen inside the task.
 	published, err := collection.ByStoragePrefixDistribution(storage, prefix, distribution)
 	if err != nil {
 		AbortWithJSONError(c, http.StatusNotFound, fmt.Errorf("unable to update: %s", err))
 		return
 	}
 
+	err = collection.LoadComplete(published, collectionFactory)
+	if err != nil {
+		AbortWithJSONError(c, http.StatusInternalServerError, fmt.Errorf("unable to update: %s", err))
+		return
+	}
+
+	if b.SkipContents != nil {
+		published.SkipContents = *b.SkipContents
+	}
+
+	if b.SkipBz2 != nil {
+		published.SkipBz2 = *b.SkipBz2
+	}
+
+	if b.AcquireByHash != nil {
+		published.AcquireByHash = *b.AcquireByHash
+	}
+
+	if b.MultiDist != nil {
+		published.MultiDist = *b.MultiDist
+	}
+
 	resources := []string{string(published.Key())}
-
-	// Non-MultiDist distributions share a single pool/ directory under the
-	// prefix.  Acquire the prefix-level pool lock so that concurrent updates
-	// on sibling distributions are serialised and cannot race during cleanup.
-	if !published.MultiDist {
-		resources = append(resources, deb.PrefixPoolLockKey(published.StoragePrefix()))
-	}
-
-	// Lock source repos / snapshots the same way apiPublishUpdateSwitch does,
-	// because published.Update() reads from them and concurrent modification
-	// would produce an inconsistent view.
-	snapshotCollection := collectionFactory.SnapshotCollection()
-	localRepoCollection := collectionFactory.LocalRepoCollection()
-
-	if published.SourceKind == deb.SourceLocalRepo {
-		for _, uuid := range published.Sources {
-			repo, err2 := localRepoCollection.ByUUID(uuid)
-			if err2 != nil {
-				AbortWithJSONError(c, http.StatusNotFound, err2)
-				return
-			}
-			resources = append(resources, string(repo.Key()))
-		}
-	} else if published.SourceKind == deb.SourceSnapshot {
-		for _, uuid := range published.Sources {
-			snapshot, err2 := snapshotCollection.ByUUID(uuid)
-			if err2 != nil {
-				AbortWithJSONError(c, http.StatusNotFound, err2)
-				return
-			}
-			resources = append(resources, string(snapshot.Key()))
-		}
-	}
-
 	taskName := fmt.Sprintf("Update published %s repository %s/%s", published.SourceKind, published.StoragePrefix(), published.Distribution)
 	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		taskCollectionFactory := context.NewCollectionFactory()
+		result, err := published.Update(collectionFactory, out)
-		taskCollection := taskCollectionFactory.PublishedRepoCollection()
-
-		published, err := taskCollection.ByStoragePrefixDistribution(storage, prefix, distribution)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
 		}
 
-		err = taskCollection.LoadComplete(published, taskCollectionFactory)
+		err = published.Publish(context.PackagePool(), context, collectionFactory, signer, out, b.ForceOverwrite, context.SkelPath())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
 		}
 
-		// Capture MultiDist before mutations to detect a false→true transition.
+		err = collection.Update(published)
-		prevMultiDist := published.MultiDist
-
-		// Apply field mutations on the freshly loaded object.
-		if b.SkipContents != nil {
-			published.SkipContents = *b.SkipContents
-		}
-		if b.SkipBz2 != nil {
-			published.SkipBz2 = *b.SkipBz2
-		}
-		if b.AcquireByHash != nil {
-			published.AcquireByHash = *b.AcquireByHash
-		}
-		if b.MultiDist != nil {
-			published.MultiDist = *b.MultiDist
-		}
-
-		result, err := published.Update(taskCollectionFactory, out)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
-		}
-
-		err = published.Publish(context.PackagePool(), context, taskCollectionFactory, signer, out, b.ForceOverwrite, context.SkelPath())
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
-		}
-
-		err = taskCollection.Update(published)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save to DB: %s", err)
 		}
@@ -1195,19 +1046,10 @@ func apiPublishUpdate(c *gin.Context) {
 		if b.SkipCleanup == nil || !*b.SkipCleanup {
 			cleanComponents := make([]string, 0, len(result.UpdatedSources)+len(result.RemovedSources))
 			cleanComponents = append(append(cleanComponents, result.UpdatedComponents()...), result.RemovedComponents()...)
-			err = taskCollection.CleanupPrefixComponentFiles(context, published, cleanComponents, taskCollectionFactory, out)
+			err = collection.CleanupPrefixComponentFiles(context, published, cleanComponents, collectionFactory, out)
 			if err != nil {
 				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to update: %s", err)
 			}
-			// When MultiDist is toggled, the old pool layout still has files that
-			// CleanupPrefixComponentFiles won't touch (it only scans the new layout).
-			// Run a second pass over the previous layout to remove stale files.
-			if prevMultiDist != published.MultiDist {
-				err = taskCollection.CleanupAfterMultiDistToggle(context, published, prevMultiDist, cleanComponents, taskCollectionFactory, out)
-				if err != nil {
-					return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to clean legacy pool: %s", err)
-				}
-			}
 		}
 
 		return &task.ProcessReturnValue{Code: http.StatusOK, Value: published}, nil

api/published_file_missing_test.go

@@ -1,737 +0,0 @@
-package api
-
-import (
-	"bytes"
-	"encoding/json"
-	"fmt"
-	"net/http"
-	"net/http/httptest"
-	"os"
-	"os/exec"
-	"path/filepath"
-	"sync"
-	"time"
-
-	"github.com/aptly-dev/aptly/aptly"
-	ctx "github.com/aptly-dev/aptly/context"
-	"github.com/aptly-dev/aptly/deb"
-	"github.com/gin-gonic/gin"
-	"github.com/smira/flag"
-
-	. "gopkg.in/check.v1"
-)
-
-// PublishedFileMissingSuite reproduces the exact bug where:
-// - Package import succeeds
-// - Metadata is updated (Packages.gz shows the package)
-// - Publish reports success
-// - BUT the .deb file is missing from the published pool directory
-// - Result: apt-get returns 404 when trying to download the package
-type PublishedFileMissingSuite struct {
-	context    *ctx.AptlyContext
-	flags      *flag.FlagSet
-	configFile *os.File
-	router     http.Handler
-	tempDir    string
-	poolPath   string
-	publicPath string
-}
-
-var _ = Suite(&PublishedFileMissingSuite{})
-
-func (s *PublishedFileMissingSuite) SetUpSuite(c *C) {
-	aptly.Version = "publishedFileMissingTest"
-
-	tempDir, err := os.MkdirTemp("", "aptly-published-missing-test")
-	c.Assert(err, IsNil)
-	s.tempDir = tempDir
-	s.poolPath = filepath.Join(tempDir, "pool")
-	s.publicPath = filepath.Join(tempDir, "public")
-
-	file, err := os.CreateTemp("", "aptly-published-missing-config")
-	c.Assert(err, IsNil)
-	s.configFile = file
-
-	config := gin.H{
-		"rootDir":                    tempDir,
-		"downloadDir":                filepath.Join(tempDir, "download"),
-		"architectures":              []string{"amd64"},
-		"dependencyFollowSuggests":   false,
-		"dependencyFollowRecommends": false,
-		"gpgDisableSign":             true,
-		"gpgDisableVerify":           true,
-		"gpgProvider":                "internal",
-		"skipLegacyPool":             true,
-		"enableMetricsEndpoint":      false,
-	}
-
-	jsonString, err := json.Marshal(config)
-	c.Assert(err, IsNil)
-	_, err = file.Write(jsonString)
-	c.Assert(err, IsNil)
-
-	flags := flag.NewFlagSet("publishedFileMissingTestFlags", flag.ContinueOnError)
-	flags.Bool("no-lock", true, "disable database locking for test")
-	flags.Int("db-open-attempts", 3, "dummy")
-	flags.String("config", s.configFile.Name(), "config file")
-	flags.String("architectures", "", "dummy")
-	s.flags = flags
-
-	context, err := ctx.NewContext(s.flags)
-	c.Assert(err, IsNil)
-
-	s.context = context
-	s.router = Router(context)
-}
-
-func (s *PublishedFileMissingSuite) TearDownSuite(c *C) {
-	if s.configFile != nil {
-		_ = os.Remove(s.configFile.Name())
-	}
-	if s.context != nil {
-		s.context.Shutdown()
-	}
-	if s.tempDir != "" {
-		_ = os.RemoveAll(s.tempDir)
-	}
-}
-
-func (s *PublishedFileMissingSuite) SetUpTest(c *C) {
-	collectionFactory := s.context.NewCollectionFactory()
-
-	localRepoCollection := collectionFactory.LocalRepoCollection()
-	_ = localRepoCollection.ForEach(func(repo *deb.LocalRepo) error {
-		_ = localRepoCollection.Drop(repo)
-		return nil
-	})
-
-	publishedCollection := collectionFactory.PublishedRepoCollection()
-	_ = publishedCollection.ForEach(func(published *deb.PublishedRepo) error {
-		_ = publishedCollection.Remove(s.context, published.Storage, published.Prefix,
-			published.Distribution, collectionFactory, nil, true, true)
-		return nil
-	})
-}
-
-func (s *PublishedFileMissingSuite) TearDownTest(c *C) {
-	s.SetUpTest(c)
-}
-
-func (s *PublishedFileMissingSuite) httpRequest(c *C, method string, url string, body []byte) *httptest.ResponseRecorder {
-	w := httptest.NewRecorder()
-	var req *http.Request
-	var err error
-
-	if body != nil {
-		req, err = http.NewRequest(method, url, bytes.NewReader(body))
-	} else {
-		req, err = http.NewRequest(method, url, nil)
-	}
-	c.Assert(err, IsNil)
-	req.Header.Add("Content-Type", "application/json")
-	s.router.ServeHTTP(w, req)
-	return w
-}
-
-func (s *PublishedFileMissingSuite) createDebPackage(c *C, uploadID, packageName, version string) {
-	uploadPath := s.context.UploadPath()
-	uploadDir := filepath.Join(uploadPath, uploadID)
-	err := os.MkdirAll(uploadDir, 0755)
-	c.Assert(err, IsNil)
-
-	tempDir, err := os.MkdirTemp("", "deb-build")
-	c.Assert(err, IsNil)
-	defer func() { _ = os.RemoveAll(tempDir) }()
-
-	debianDir := filepath.Join(tempDir, "DEBIAN")
-	err = os.MkdirAll(debianDir, 0755)
-	c.Assert(err, IsNil)
-
-	controlContent := fmt.Sprintf(`Package: %s
-Version: %s
-Section: libs
-Priority: optional
-Architecture: amd64
-Maintainer: Test <test@example.com>
-Description: Test package
- Test package for published file missing bug.
-`, packageName, version)
-
-	err = os.WriteFile(filepath.Join(debianDir, "control"), []byte(controlContent), 0644)
-	c.Assert(err, IsNil)
-
-	usrDir := filepath.Join(tempDir, "usr", "lib")
-	err = os.MkdirAll(usrDir, 0755)
-	c.Assert(err, IsNil)
-	err = os.WriteFile(filepath.Join(usrDir, "lib.so"), []byte("library"), 0644)
-	c.Assert(err, IsNil)
-
-	debFile := filepath.Join(uploadDir, fmt.Sprintf("%s_%s_amd64.deb", packageName, version))
-	cmd := exec.Command("dpkg-deb", "--build", tempDir, debFile)
-	err = cmd.Run()
-	c.Assert(err, IsNil)
-}
-
-// TestPublishedFileGoMissing reproduces the exact production bug
-func (s *PublishedFileMissingSuite) TestPublishedFileGoMissing(c *C) {
-	c.Log("=== Reproducing: Package in metadata but 404 on download ===")
-
-	// Create and publish a repository
-	repoName := "test-repo"
-	distribution := "bullseye"
-
-	createBody, _ := json.Marshal(gin.H{
-		"Name":                repoName,
-		"DefaultDistribution": distribution,
-		"DefaultComponent":    "main",
-	})
-	resp := s.httpRequest(c, "POST", "/api/repos", createBody)
-	c.Assert(resp.Code, Equals, 201, Commentf("Failed to create repo: %s", resp.Body.String()))
-
-	publishBody, _ := json.Marshal(gin.H{
-		"SourceKind":    "local",
-		"Distribution":  distribution,
-		"Architectures": []string{"amd64"},
-		"Sources": []gin.H{
-			{"Component": "main", "Name": repoName},
-		},
-		"Signing": gin.H{"Skip": true},
-	})
-	resp = s.httpRequest(c, "POST", "/api/publish/hrt", publishBody)
-	c.Assert(resp.Code, Equals, 201, Commentf("Failed to publish: %s", resp.Body.String()))
-
-	// Create package
-	packageName := "hrt-libblobbyclient1"
-	version := "20250926.152427+hrtdeb11"
-	uploadID := "test-upload-1"
-
-	s.createDebPackage(c, uploadID, packageName, version)
-
-	// Add package
-	resp = s.httpRequest(c, "POST", fmt.Sprintf("/api/repos/%s/file/%s?noRemove=0", repoName, uploadID), nil)
-	c.Assert(resp.Code, Equals, 200, Commentf("Failed to add package: %s", resp.Body.String()))
-
-	// Update publish
-	updateBody, _ := json.Marshal(gin.H{
-		"Signing":        gin.H{"Skip": true},
-		"ForceOverwrite": true,
-	})
-	resp = s.httpRequest(c, "PUT", fmt.Sprintf("/api/publish/hrt/%s", distribution), updateBody)
-	c.Assert(resp.Code, Equals, 200, Commentf("Failed to update publish: %s", resp.Body.String()))
-
-	// Now check if the file is actually accessible in the published location
-	publishedStorage, err := s.context.GetPublishedStorage("")
-	c.Assert(err, IsNil)
-	publicPath := publishedStorage.(aptly.FileSystemPublishedStorage).PublicPath()
-
-	// Expected file path: hrt/pool/main/h/hrt-libblobbyclient1/hrt-libblobbyclient1_20250926.152427+hrtdeb11_amd64.deb
-	expectedPath := filepath.Join(publicPath, "hrt", "pool", "main", "h", packageName,
-		fmt.Sprintf("%s_%s_amd64.deb", packageName, version))
-
-	c.Logf("Checking for published file at: %s", expectedPath)
-
-	fileInfo, err := os.Stat(expectedPath)
-	fileExists := err == nil
-
-	c.Logf("File exists: %v", fileExists)
-	if fileExists {
-		c.Logf("File size: %d bytes", fileInfo.Size())
-	}
-
-	// Check metadata
-	resp = s.httpRequest(c, "GET", fmt.Sprintf("/api/repos/%s/packages", repoName), nil)
-	var packages []string
-	err = json.Unmarshal(resp.Body.Bytes(), &packages)
-	c.Assert(err, IsNil)
-	c.Logf("Packages in metadata: %d", len(packages))
-
-	// THE BUG: Metadata says package exists, but file is missing from published location
-	if len(packages) > 0 && !fileExists {
-		c.Logf("★★★ BUG REPRODUCED! ★★★")
-		c.Logf("Metadata shows %d package(s) but file is missing at: %s", len(packages), expectedPath)
-		c.Logf("This is exactly what causes: 404 Not Found [IP: 10.20.72.62 3142]")
-
-		c.Fatal("BUG CONFIRMED: Package in metadata but missing from published directory!")
-	}
-
-	c.Assert(fileExists, Equals, true, Commentf(
-		"Published file should exist at %s when package is in metadata", expectedPath))
-}
-
-// TestConcurrentPublishRace tries to trigger the race with concurrent publishes
-func (s *PublishedFileMissingSuite) TestConcurrentPublishRace(c *C) {
-	c.Log("=== Testing concurrent publish race condition ===")
-
-	const numIterations = 4
-
-	for iteration := 0; iteration < numIterations; iteration++ {
-		c.Logf("--- Iteration %d/%d ---", iteration+1, numIterations)
-
-		// Create repo
-		repoName := fmt.Sprintf("race-repo-%d", iteration)
-		distribution := fmt.Sprintf("dist-%d", iteration)
-
-		createBody, _ := json.Marshal(gin.H{
-			"Name":                repoName,
-			"DefaultDistribution": distribution,
-			"DefaultComponent":    "main",
-		})
-		resp := s.httpRequest(c, "POST", "/api/repos", createBody)
-		c.Assert(resp.Code, Equals, 201)
-
-		publishBody, _ := json.Marshal(gin.H{
-			"SourceKind":    "local",
-			"Distribution":  distribution,
-			"Architectures": []string{"amd64"},
-			"Sources": []gin.H{
-				{"Component": "main", "Name": repoName},
-			},
-			"Signing": gin.H{"Skip": true},
-		})
-		resp = s.httpRequest(c, "POST", "/api/publish/concurrent", publishBody)
-		c.Assert(resp.Code, Equals, 201)
-
-		// Create multiple packages
-		var wg sync.WaitGroup
-		numPackages := 5
-
-		for i := 0; i < numPackages; i++ {
-			wg.Add(1)
-			go func(idx int) {
-				defer wg.Done()
-
-				packageName := fmt.Sprintf("pkg-%d-%d", iteration, idx)
-				version := "1.0.0"
-				uploadID := fmt.Sprintf("upload-%d-%d", iteration, idx)
-
-				s.createDebPackage(c, uploadID, packageName, version)
-
-				// Add package
-				resp := s.httpRequest(c, "POST", fmt.Sprintf("/api/repos/%s/file/%s?noRemove=0", repoName, uploadID), nil)
-				c.Logf("Package %d add: %d", idx, resp.Code)
-
-				// Small delay
-				time.Sleep(time.Duration(5+idx*2) * time.Millisecond)
-
-				// Publish
-				updateBody, _ := json.Marshal(gin.H{
-					"Signing":        gin.H{"Skip": true},
-					"ForceOverwrite": true,
-				})
-				resp = s.httpRequest(c, "PUT", fmt.Sprintf("/api/publish/concurrent/%s", distribution), updateBody)
-				c.Logf("Publish %d: %d", idx, resp.Code)
-			}(i)
-		}
-
-		wg.Wait()
-		time.Sleep(100 * time.Millisecond)
-
-		// Check all packages
-		resp = s.httpRequest(c, "GET", fmt.Sprintf("/api/repos/%s/packages", repoName), nil)
-		var packages []string
-		err := json.Unmarshal(resp.Body.Bytes(), &packages)
-		c.Assert(err, IsNil)
-
-		// Check published files
-		publishedStorage, err := s.context.GetPublishedStorage("")
-		c.Assert(err, IsNil)
-		publicPath := publishedStorage.(aptly.FileSystemPublishedStorage).PublicPath()
-
-		missingFiles := []string{}
-		for i := 0; i < numPackages; i++ {
-			packageName := fmt.Sprintf("pkg-%d-%d", iteration, i)
-			version := "1.0.0"
-
-			// Calculate pool path
-			poolSubdir := string(packageName[0])
-			expectedPath := filepath.Join(publicPath, "concurrent", "pool", "main", poolSubdir, packageName,
-				fmt.Sprintf("%s_%s_amd64.deb", packageName, version))
-
-			if _, err := os.Stat(expectedPath); os.IsNotExist(err) {
-				missingFiles = append(missingFiles, expectedPath)
-			}
-		}
-
-		if len(missingFiles) > 0 {
-			c.Logf("★★★ BUG DETECTED in iteration %d/%d! ★★★", iteration+1, numIterations)
-			c.Logf("Metadata shows %d packages, but %d files are MISSING:", len(packages), len(missingFiles))
-			for i, f := range missingFiles {
-				c.Logf("  [iter %d] File MISSING %d/%d: %s", iteration+1, i+1, len(missingFiles), f)
-			}
-
-			c.Fatalf("BUG REPRODUCED in iteration %d/%d: %d published files missing", iteration+1, numIterations, len(missingFiles))
-		} else {
-			c.Logf("[iter %d/%d] All %d files present - OK", iteration+1, numIterations, numPackages)
-		}
-	}
-
-	c.Logf("All %d iterations passed - bug not reproduced with current timing", numIterations)
-}
-
-// TestIdenticalPackageRace tests the specific case of identical SHA256 packages
-func (s *PublishedFileMissingSuite) TestIdenticalPackageRace(c *C) {
-	c.Log("=== AGGRESSIVE test: identical package (same SHA256) race ===")
-
-	const numIterations = 4
-	packageName := "shared-package"
-
-	for iter := 0; iter < numIterations; iter++ {
-		c.Logf("Iteration %d/%d", iter+1, numIterations)
-
-		// Create two repos that will get the SAME package (unique per iteration)
-		repos := []string{fmt.Sprintf("identical-a-%d", iter), fmt.Sprintf("identical-b-%d", iter)}
-		dists := []string{fmt.Sprintf("dist-a-%d", iter), fmt.Sprintf("dist-b-%d", iter)}
-
-		for i := range repos {
-			createBody, _ := json.Marshal(gin.H{
-				"Name":                repos[i],
-				"DefaultDistribution": dists[i],
-				"DefaultComponent":    "main",
-			})
-			resp := s.httpRequest(c, "POST", "/api/repos", createBody)
-			c.Assert(resp.Code, Equals, 201)
-
-			publishBody, _ := json.Marshal(gin.H{
-				"SourceKind":    "local",
-				"Distribution":  dists[i],
-				"Architectures": []string{"amd64"},
-				"Sources": []gin.H{
-					{"Component": "main", "Name": repos[i]},
-				},
-				"Signing": gin.H{"Skip": true},
-				"SkipBz2": true,
-			})
-			resp = s.httpRequest(c, "POST", "/api/publish/identical", publishBody)
-			c.Assert(resp.Code, Equals, 201)
-		}
-
-		// Create IDENTICAL package file with UNIQUE VERSION per iteration
-		version := fmt.Sprintf("1.0.%d", iter)
-		uploadID1 := fmt.Sprintf("identical-upload-1-%d", iter)
-		uploadID2 := fmt.Sprintf("identical-upload-2-%d", iter)
-
-		s.createDebPackage(c, uploadID1, packageName, version)
-
-		// Copy to second upload (same SHA256)
-		uploadPath := s.context.UploadPath()
-		src := filepath.Join(uploadPath, uploadID1, fmt.Sprintf("%s_%s_amd64.deb", packageName, version))
-		destDir := filepath.Join(uploadPath, uploadID2)
-		err := os.MkdirAll(destDir, 0755)
-		c.Assert(err, IsNil)
-		dest := filepath.Join(destDir, fmt.Sprintf("%s_%s_amd64.deb", packageName, version))
-
-		srcData, readErr := os.ReadFile(src)
-		c.Assert(readErr, IsNil)
-		err = os.WriteFile(dest, srcData, 0644)
-		c.Assert(err, IsNil)
-
-		// Race: add and publish both simultaneously
-		var wg sync.WaitGroup
-		wg.Add(2)
-
-		go func() {
-			defer wg.Done()
-			s.httpRequest(c, "POST", fmt.Sprintf("/api/repos/%s/file/%s?noRemove=0", repos[0], uploadID1), nil)
-			updateBody, _ := json.Marshal(gin.H{"Signing": gin.H{"Skip": true}, "ForceOverwrite": true, "SkipBz2": true})
-			s.httpRequest(c, "PUT", fmt.Sprintf("/api/publish/identical/%s", dists[0]), updateBody)
-		}()
-
-		go func() {
-			defer wg.Done()
-			s.httpRequest(c, "POST", fmt.Sprintf("/api/repos/%s/file/%s?noRemove=0", repos[1], uploadID2), nil)
-			updateBody, _ := json.Marshal(gin.H{"Signing": gin.H{"Skip": true}, "ForceOverwrite": true, "SkipBz2": true})
-			s.httpRequest(c, "PUT", fmt.Sprintf("/api/publish/identical/%s", dists[1]), updateBody)
-		}()
-
-		wg.Wait()
-		time.Sleep(200 * time.Millisecond)
-		c.Logf("[iter %d] All operations complete", iter)
-
-		// Check the shared pool location
-		publishedStorage, err := s.context.GetPublishedStorage("")
-		c.Assert(err, IsNil)
-		publicPath := publishedStorage.(aptly.FileSystemPublishedStorage).PublicPath()
-
-		poolSubdir := string(packageName[0])
-		sharedPoolPath := filepath.Join(publicPath, "identical", "pool", "main", poolSubdir, packageName,
-			fmt.Sprintf("%s_%s_amd64.deb", packageName, version))
-
-		fileInfo, err := os.Stat(sharedPoolPath)
-		fileExists := err == nil
-
-		if fileExists {
-			c.Logf("[iter %d] File EXISTS at %s (size: %d)", iter, sharedPoolPath, fileInfo.Size())
-		} else {
-			c.Logf("[iter %d] File MISSING at %s (error: %v)", iter, sharedPoolPath, err)
-		}
-
-		// Check metadata
-		var packagesA, packagesB []string
-		resp := s.httpRequest(c, "GET", fmt.Sprintf("/api/repos/%s/packages", repos[0]), nil)
-		err = json.Unmarshal(resp.Body.Bytes(), &packagesA)
-		c.Assert(err, IsNil)
-		resp = s.httpRequest(c, "GET", fmt.Sprintf("/api/repos/%s/packages", repos[1]), nil)
-		err = json.Unmarshal(resp.Body.Bytes(), &packagesB)
-		c.Assert(err, IsNil)
-
-		c.Logf("[iter %d] Packages in metadata: A=%d, B=%d", iter, len(packagesA), len(packagesB))
-
-		// THE BUG: Both repos show packages in metadata, but the shared pool file is missing
-		if (len(packagesA) > 0 || len(packagesB) > 0) && !fileExists {
-			c.Logf("★★★ BUG REPRODUCED in iteration %d! ★★★", iter+1)
-			c.Logf("Packages in metadata A: %d, B: %d", len(packagesA), len(packagesB))
-			c.Logf("Shared pool file exists: %v", fileExists)
-			c.Logf("Pool path: %s", sharedPoolPath)
-
-			// List what files ARE in the pool directory
-			poolDir := filepath.Dir(sharedPoolPath)
-			if entries, err := os.ReadDir(poolDir); err == nil {
-				c.Logf("Files in pool directory %s:", poolDir)
-				for _, entry := range entries {
-					c.Logf("  - %s", entry.Name())
-				}
-			}
-
-			c.Fatalf("Metadata shows packages but shared pool file is missing (iteration %d)", iter+1)
-		}
-	}
-
-	c.Logf("All %d iterations passed - bug not reproduced", numIterations)
-}
-
-// TestConcurrentSnapshotPublishToSamePrefix reproduces the EXACT production bug:
-// Multiple snapshots are published concurrently to the SAME prefix but different distributions.
-// Example from production logs:
-//   - trixie-pgdg published to "external/postgres-auto/trixie"
-//   - bullseye-pgdg published to "external/postgres-auto/bullseye"
-// Both share the same pool directory, causing cleanup race conditions.
-func (s *PublishedFileMissingSuite) TestConcurrentSnapshotPublishToSamePrefix(c *C) {
-	const numIterations = 4
-
-	for iter := 0; iter < numIterations; iter++ {
-		c.Logf("--- Iteration %d/%d ---", iter+1, numIterations)
-
-		// Create two repos with different packages (simulating trixie-pgdg and bullseye-pgdg)
-		repoTrixie := fmt.Sprintf("trixie-pgdg-%d", iter)
-		repoBullseye := fmt.Sprintf("bullseye-pgdg-%d", iter)
-
-		// Create trixie repo
-		createBody, _ := json.Marshal(gin.H{
-			"Name":                repoTrixie,
-			"DefaultDistribution": "trixie",
-			"DefaultComponent":    "main",
-		})
-		resp := s.httpRequest(c, "POST", "/api/repos", createBody)
-		c.Assert(resp.Code, Equals, 201, Commentf("Failed to create trixie repo"))
-
-		// Create bullseye repo
-		createBody, _ = json.Marshal(gin.H{
-			"Name":                repoBullseye,
-			"DefaultDistribution": "bullseye",
-			"DefaultComponent":    "main",
-		})
-		resp = s.httpRequest(c, "POST", "/api/repos", createBody)
-		c.Assert(resp.Code, Equals, 201, Commentf("Failed to create bullseye repo"))
-
-		// Add packages to both repos
-		numPackages := 3
-
-		// Add packages to trixie repo
-		for i := 0; i < numPackages; i++ {
-			packageName := fmt.Sprintf("postgresql-17-trixie-pkg%d", i)
-			version := fmt.Sprintf("17.0.%d", iter)
-			uploadID := fmt.Sprintf("trixie-upload-%d-%d", iter, i)
-
-			s.createDebPackage(c, uploadID, packageName, version)
-			resp = s.httpRequest(c, "POST", fmt.Sprintf("/api/repos/%s/file/%s?noRemove=0", repoTrixie, uploadID), nil)
-			c.Assert(resp.Code, Equals, 200, Commentf("Failed to add package to trixie"))
-		}
-
-		// Add packages to bullseye repo
-		for i := 0; i < numPackages; i++ {
-			packageName := fmt.Sprintf("postgresql-17-bullseye-pkg%d", i)
-			version := fmt.Sprintf("17.0.%d", iter)
-			uploadID := fmt.Sprintf("bullseye-upload-%d-%d", iter, i)
-
-			s.createDebPackage(c, uploadID, packageName, version)
-			resp = s.httpRequest(c, "POST", fmt.Sprintf("/api/repos/%s/file/%s?noRemove=0", repoBullseye, uploadID), nil)
-			c.Assert(resp.Code, Equals, 200, Commentf("Failed to add package to bullseye"))
-		}
-
-		// Create snapshots from both repos
-		snapshotTrixie := fmt.Sprintf("%s-snap", repoTrixie)
-		snapshotBullseye := fmt.Sprintf("%s-snap", repoBullseye)
-
-		createSnapshotBody, _ := json.Marshal(gin.H{"Name": snapshotTrixie})
-		resp = s.httpRequest(c, "POST", fmt.Sprintf("/api/repos/%s/snapshots", repoTrixie), createSnapshotBody)
-		c.Assert(resp.Code, Equals, 201, Commentf("Failed to create trixie snapshot"))
-
-		createSnapshotBody, _ = json.Marshal(gin.H{"Name": snapshotBullseye})
-		resp = s.httpRequest(c, "POST", fmt.Sprintf("/api/repos/%s/snapshots", repoBullseye), createSnapshotBody)
-		c.Assert(resp.Code, Equals, 201, Commentf("Failed to create bullseye snapshot"))
-
-		// Publish both snapshots CONCURRENTLY to the SAME prefix
-		// This mimics production where both are published to "external/postgres-auto"
-		// Use the SAME prefix across all iterations to trigger the race more aggressively
-		sharedPrefix := "postgres-auto"
-
-		var wg sync.WaitGroup
-		var trixiePublishCode, bullseyePublishCode int
-
-		wg.Add(2)
-
-		// Publish or update trixie snapshot
-		go func() {
-			defer wg.Done()
-
-			var resp *httptest.ResponseRecorder
-			if iter == 0 {
-				// First iteration: CREATE
-				publishBody, _ := json.Marshal(gin.H{
-					"SourceKind":    "snapshot",
-					"Distribution":  "trixie",
-					"Architectures": []string{"amd64"},
-					"Sources": []gin.H{
-						{"Name": snapshotTrixie},
-					},
-					"Signing":        gin.H{"Skip": true},
-					"SkipBz2":        true,
-					"ForceOverwrite": true,
-					"SkipCleanup":    false, // Force cleanup to run
-				})
-				resp = s.httpRequest(c, "POST", fmt.Sprintf("/api/publish/%s", sharedPrefix), publishBody)
-			} else {
-				// Subsequent iterations: UPDATE (this is what happens in production)
-				updateBody, _ := json.Marshal(gin.H{
-					"Snapshots": []gin.H{
-						{"Component": "main", "Name": snapshotTrixie},
-					},
-					"Signing":        gin.H{"Skip": true},
-					"SkipBz2":        true,
-					"ForceOverwrite": true,
-					"SkipCleanup":    false,
-				})
-				resp = s.httpRequest(c, "PUT", fmt.Sprintf("/api/publish/%s/trixie", sharedPrefix), updateBody)
-			}
-			trixiePublishCode = resp.Code
-			c.Logf("[iter %d] Trixie publish/update completed: %d", iter, resp.Code)
-		}()
-
-		// Publish or update bullseye snapshot
-		go func() {
-			defer wg.Done()
-
-			var resp *httptest.ResponseRecorder
-			if iter == 0 {
-				// First iteration: CREATE
-				publishBody, _ := json.Marshal(gin.H{
-					"SourceKind":    "snapshot",
-					"Distribution":  "bullseye",
-					"Architectures": []string{"amd64"},
-					"Sources": []gin.H{
-						{"Name": snapshotBullseye},
-					},
-					"Signing":        gin.H{"Skip": true},
-					"SkipBz2":        true,
-					"ForceOverwrite": true,
-					"SkipCleanup":    false,
-				})
-				resp = s.httpRequest(c, "POST", fmt.Sprintf("/api/publish/%s", sharedPrefix), publishBody)
-			} else {
-				// Subsequent iterations: UPDATE
-				updateBody, _ := json.Marshal(gin.H{
-					"Snapshots": []gin.H{
-						{"Component": "main", "Name": snapshotBullseye},
-					},
-					"Signing":        gin.H{"Skip": true},
-					"SkipBz2":        true,
-					"ForceOverwrite": true,
-					"SkipCleanup":    false,
-				})
-				resp = s.httpRequest(c, "PUT", fmt.Sprintf("/api/publish/%s/bullseye", sharedPrefix), updateBody)
-			}
-			bullseyePublishCode = resp.Code
-			c.Logf("[iter %d] Bullseye publish/update completed: %d", iter, resp.Code)
-		}()
-
-		wg.Wait()
-		time.Sleep(50 * time.Millisecond)
-
-		// Verify publishes succeeded (201 for create, 200 for update)
-		expectedCode := 201
-		if iter > 0 {
-			expectedCode = 200
-		}
-		c.Assert(trixiePublishCode, Equals, expectedCode, Commentf("Trixie publish/update should succeed"))
-		c.Assert(bullseyePublishCode, Equals, expectedCode, Commentf("Bullseye publish/update should succeed"))
-
-		// Verify ALL package files exist in the published pool
-		publishedStorage, err := s.context.GetPublishedStorage("")
-		c.Assert(err, IsNil)
-		publicPath := publishedStorage.(aptly.FileSystemPublishedStorage).PublicPath()
-
-		missingFiles := []string{}
-		expectedFiles := []string{}
-
-		// Check trixie packages
-		for i := 0; i < numPackages; i++ {
-			packageName := fmt.Sprintf("postgresql-17-trixie-pkg%d", i)
-			version := fmt.Sprintf("17.0.%d", iter)
-
-			poolSubdir := string(packageName[0])
-			expectedPath := filepath.Join(publicPath, sharedPrefix, "pool", "main", poolSubdir, packageName,
-				fmt.Sprintf("%s_%s_amd64.deb", packageName, version))
-
-			expectedFiles = append(expectedFiles, expectedPath)
-			if _, err := os.Stat(expectedPath); os.IsNotExist(err) {
-				missingFiles = append(missingFiles, fmt.Sprintf("TRIXIE: %s", filepath.Base(expectedPath)))
-			}
-		}
-
-		// Check bullseye packages
-		for i := 0; i < numPackages; i++ {
-			packageName := fmt.Sprintf("postgresql-17-bullseye-pkg%d", i)
-			version := fmt.Sprintf("17.0.%d", iter)
-
-			poolSubdir := string(packageName[0])
-			expectedPath := filepath.Join(publicPath, sharedPrefix, "pool", "main", poolSubdir, packageName,
-				fmt.Sprintf("%s_%s_amd64.deb", packageName, version))
-
-			expectedFiles = append(expectedFiles, expectedPath)
-			if _, err := os.Stat(expectedPath); os.IsNotExist(err) {
-				missingFiles = append(missingFiles, fmt.Sprintf("BULLSEYE: %s", filepath.Base(expectedPath)))
-			}
-		}
-
-		// BUG: Files from one distribution are deleted by the other's cleanup
-		if len(missingFiles) > 0 {
-			c.Logf("★★★ BUG REPRODUCED in iteration %d/%d! ★★★", iter+1, numIterations)
-			c.Logf("Both publishes to prefix '%s' succeeded, but %d files are MISSING:", sharedPrefix, len(missingFiles))
-			for i, f := range missingFiles {
-				c.Logf("  Missing file %d/%d: %s", i+1, len(missingFiles), f)
-			}
-
-			c.Logf("\nThis reproduces the exact production bug where:")
-			c.Logf("  1. Mirror updates complete successfully")
-			c.Logf("  2. Snapshots are created")
-			c.Logf("  3. Both snapshots publish to same prefix (different distributions)")
-			c.Logf("  4. Cleanup from one publish DELETES files from the other")
-			c.Logf("  5. Result: apt-get returns 404 when downloading packages")
-
-			// List what's actually in the pool
-			poolDir := filepath.Join(publicPath, sharedPrefix, "pool", "main")
-			if entries, err := os.ReadDir(poolDir); err == nil {
-				c.Logf("\nActual pool directory contents (%s):", poolDir)
-				for _, entry := range entries {
-					c.Logf("  - %s/", entry.Name())
-				}
-			}
-
-			c.Fatalf("BUG CONFIRMED (iteration %d/%d): %d files missing from shared pool",
-				iter+1, numIterations, len(missingFiles))
-		} else {
-			c.Logf("[iter %d/%d] All %d files present - OK", iter+1, numIterations, len(expectedFiles))
-		}
-	}
-	c.Logf("✓ All %d iterations passed - no files missing", numIterations)
-}

api/repos.go

@@ -24,19 +24,19 @@ import (
 // @Tags Repos
 // @Produce html
 // @Success 200 {object} string "HTML"
-// @Router /repos [get]
+// @Router /api/repos [get]
 func reposListInAPIMode(localRepos map[string]utils.FileSystemPublishRoot) gin.HandlerFunc {
 	return func(c *gin.Context) {
 		c.Writer.Header().Set("Content-Type", "text/html; charset=utf-8")
 		c.Writer.Flush()
-		_, _ = c.Writer.WriteString("<pre>\n")
+		c.Writer.WriteString("<pre>\n")
 		if len(localRepos) == 0 {
-			_, _ = c.Writer.WriteString("<a href=\"-/\">default</a>\n")
+			c.Writer.WriteString("<a href=\"-/\">default</a>\n")
 		}
 		for publishPrefix := range localRepos {
-			_, _ = c.Writer.WriteString(fmt.Sprintf("<a href=\"%[1]s/\">%[1]s</a>\n", publishPrefix))
+			c.Writer.WriteString(fmt.Sprintf("<a href=\"%[1]s/\">%[1]s</a>\n", publishPrefix))
 		}
-		_, _ = c.Writer.WriteString("</pre>")
+		c.Writer.WriteString("</pre>")
 		c.Writer.Flush()
 	}
 }
@@ -49,7 +49,7 @@ func reposListInAPIMode(localRepos map[string]utils.FileSystemPublishRoot) gin.H
 // @Param pkgPath path string true "Package Path" allowReserved=true
 // @Produce json
 // @Success 200 ""
-// @Router /repos/{storage}/{pkgPath} [get]
+// @Router /api/{storage}/{pkgPath} [get]
 func reposServeInAPIMode(c *gin.Context) {
 	pkgpath := c.Param("pkgPath")
 
@@ -60,12 +60,7 @@ func reposServeInAPIMode(c *gin.Context) {
 		storage = "filesystem:" + storage
 	}
 
-	ps, err := context.GetPublishedStorage(storage)
+	publicPath := context.GetPublishedStorage(storage).(aptly.FileSystemPublishedStorage).PublicPath()
-	if err != nil {
-		AbortWithJSONError(c, http.StatusNotFound, err)
-		return
-	}
-	publicPath := ps.(aptly.FileSystemPublishedStorage).PublicPath()
 	c.FileFromFS(pkgpath, http.Dir(publicPath))
 }
 
@@ -81,7 +76,7 @@ func apiReposList(c *gin.Context) {
 
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.LocalRepoCollection()
-	_ = collection.ForEach(func(r *deb.LocalRepo) error {
+	collection.ForEach(func(r *deb.LocalRepo) error {
 		result = append(result, r)
 		return nil
 	})
@@ -98,7 +93,7 @@ type repoCreateParams struct {
 	DefaultDistribution string `     json:"DefaultDistribution"  example:"stable"`
 	// Default component when publishing from this local repo
 	DefaultComponent string `        json:"DefaultComponent"     example:"main"`
-	// Snapshot name to create repository from (optional)
+	// Snapshot name to create repoitory from (optional)
 	FromSnapshot string `            json:"FromSnapshot"         example:""`
 }
 
@@ -112,9 +107,9 @@ type repoCreateParams struct {
 // @Description {"Name":"aptly-repo","Comment":"","DefaultDistribution":"","DefaultComponent":""}
 // @Description ```
 // @Tags Repos
+// @Produce  json
 // @Consume  json
 // @Param request body repoCreateParams true "Parameters"
-// @Produce  json
 // @Success 201 {object} deb.LocalRepo
 // @Failure 404 {object} Error "Source snapshot not found"
 // @Failure 409 {object} Error "Local repo already exists"
@@ -127,82 +122,71 @@ func apiReposCreate(c *gin.Context) {
 		return
 	}
 
-	// Handler: Pre-task validations (shallow)
+	repo := deb.NewLocalRepo(b.Name, b.Comment)
+	repo.DefaultComponent = b.DefaultComponent
+	repo.DefaultDistribution = b.DefaultDistribution
+
 	collectionFactory := context.NewCollectionFactory()
 
-	var resources []string
 	if b.FromSnapshot != "" {
-		snapshot, err := collectionFactory.SnapshotCollection().ByName(b.FromSnapshot)
+		var snapshot *deb.Snapshot
+
+		snapshotCollection := collectionFactory.SnapshotCollection()
+
+		snapshot, err := snapshotCollection.ByName(b.FromSnapshot)
 		if err != nil {
 			AbortWithJSONError(c, http.StatusNotFound, fmt.Errorf("source snapshot not found: %s", err))
 			return
 		}
-		resources = append(resources, string(snapshot.Key()))
+
+		err = snapshotCollection.LoadComplete(snapshot)
+		if err != nil {
+			AbortWithJSONError(c, http.StatusInternalServerError, fmt.Errorf("unable to load source snapshot: %s", err))
+			return
+		}
+
+		repo.UpdateRefList(snapshot.RefList())
 	}
 
-	taskName := fmt.Sprintf("Create repository %s", b.Name)
+	localRepoCollection := collectionFactory.LocalRepoCollection()
 
-	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
+	if _, err := localRepoCollection.ByName(b.Name); err == nil {
-		// Task: Create fresh collection and check/create ATOMIC inside task
+		AbortWithJSONError(c, http.StatusConflict, fmt.Errorf("local repo with name %s already exists", b.Name))
-		taskCollectionFactory := context.NewCollectionFactory()
+		return
-		taskCollection := taskCollectionFactory.LocalRepoCollection()
+	}
 
-		// Check duplicate inside lock
+	collection := collectionFactory.LocalRepoCollection()
-		if _, err := taskCollection.ByName(b.Name); err == nil {
+	err := collection.Add(repo, collectionFactory.RefListCollection())
-			return &task.ProcessReturnValue{Code: http.StatusConflict, Value: nil}, 
+	if err != nil {
-				fmt.Errorf("local repo with name %s already exists", b.Name)
+		AbortWithJSONError(c, 400, err)
-		}
+		return
+	}
 
-		// Create repo
+	err := localRepoCollection.Add(repo)
-		repo := deb.NewLocalRepo(b.Name, b.Comment)
+	if err != nil {
-		repo.DefaultComponent = b.DefaultComponent
+		AbortWithJSONError(c, http.StatusInternalServerError, err)
-		repo.DefaultDistribution = b.DefaultDistribution
+		return
+	}
 
-		if b.FromSnapshot != "" {
+	c.JSON(http.StatusCreated, repo)
-			snapshotCollection := taskCollectionFactory.SnapshotCollection()
-
-			snapshot, err := snapshotCollection.ByName(b.FromSnapshot)
-			if err != nil {
-				return &task.ProcessReturnValue{Code: http.StatusNotFound, Value: nil}, 
-					fmt.Errorf("source snapshot not found: %s", err)
-			}
-
-			err = snapshotCollection.LoadComplete(snapshot)
-			if err != nil {
-				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, 
-					fmt.Errorf("unable to load source snapshot: %s", err)
-			}
-
-			repo.UpdateRefList(snapshot.RefList())
-		}
-
-		err := taskCollection.Add(repo)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
-		}
-
-		return &task.ProcessReturnValue{Code: http.StatusCreated, Value: repo}, nil
-	})
 }
 
 type reposEditParams struct {
 	// Name of repository to modify
-	Name *string `                       json:"Name"                 example:"new-repo-name"`
+	Name *string `binding:"required"     json:"Name"                 example:"repo1"`
 	// Change Comment of repository
 	Comment *string `                    json:"Comment"              example:"example repo"`
 	// Change Default Distribution for publishing
 	DefaultDistribution *string `        json:"DefaultDistribution"  example:""`
-	// Change Default Component for publishing
+	// Change Devault Component for publishing
 	DefaultComponent *string `        json:"DefaultComponent"     example:""`
 }
 
 // @Summary Update Repository
 // @Description **Update local repository meta information**
 // @Tags Repos
-// @Param name path string true "Repository name to modify"
-// @Consume  json
-// @Param request body reposEditParams true "Parameters"
 // @Produce json
+// @Param request body reposEditParams true "Parameters"
 // @Success 200 {object} deb.LocalRepo "msg"
 // @Failure 404 {object} Error "Not Found"
 // @Failure 500 {object} Error "Internal Server Error"
@@ -212,74 +196,50 @@ func apiReposEdit(c *gin.Context) {
 	if c.Bind(&b) != nil {
 		return
 	}
-	// Load shallowly for 404 check and resource key.
+
-	// Mutation and duplicate check happen inside the task for atomicity.
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.LocalRepoCollection()
 
-	name := c.Params.ByName("name")
+	repo, err := collection.ByName(c.Params.ByName("name"))
-	repo, err := collection.ByName(name)
 	if err != nil {
 		AbortWithJSONError(c, 404, err)
 		return
 	}
 
-	if b.Name != nil && *b.Name != name {
+	if b.Name != nil {
-		if _, err = collection.ByName(*b.Name); err == nil {
+		_, err := collection.ByName(*b.Name)
-			AbortWithJSONError(c, 409, fmt.Errorf("unable to rename: local repo %q already exists", *b.Name))
+		if err == nil {
+			// already exists
+			AbortWithJSONError(c, 404, err)
 			return
 		}
+		repo.Name = *b.Name
+	}
+	if b.Comment != nil {
+		repo.Comment = *b.Comment
+	}
+	if b.DefaultDistribution != nil {
+		repo.DefaultDistribution = *b.DefaultDistribution
+	}
+	if b.DefaultComponent != nil {
+		repo.DefaultComponent = *b.DefaultComponent
 	}
 
-	resources := []string{string(repo.Key())}
+	err = collection.Update(repo, collectionFactory.RefListCollection())
-	taskName := fmt.Sprintf("Edit repository %s", name)
+	if err != nil {
+		AbortWithJSONError(c, 500, err)
+		return
+	}
 
-	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
+	c.JSON(200, repo)
-		// Task: Create fresh collection inside task after lock
-		taskCollectionFactory := context.NewCollectionFactory()
-		taskCollection := taskCollectionFactory.LocalRepoCollection()
-
-		// Fresh load after lock acquired
-		repo, err := taskCollection.ByName(name)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusNotFound, Value: nil}, err
-		}
-
-		// Check and update ATOMIC (inside lock)
-		if b.Name != nil && *b.Name != name {
-			_, err := taskCollection.ByName(*b.Name)
-			if err == nil {
-				// already exists
-				return &task.ProcessReturnValue{Code: http.StatusConflict, Value: nil}, 
-					fmt.Errorf("local repo with name %q already exists", *b.Name)
-			}
-			repo.Name = *b.Name
-		}
-		if b.Comment != nil {
-			repo.Comment = *b.Comment
-		}
-		if b.DefaultDistribution != nil {
-			repo.DefaultDistribution = *b.DefaultDistribution
-		}
-		if b.DefaultComponent != nil {
-			repo.DefaultComponent = *b.DefaultComponent
-		}
-
-		err = taskCollection.Update(repo)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
-		}
-
-		return &task.ProcessReturnValue{Code: http.StatusOK, Value: repo}, nil
-	})
 }
 
 // GET /api/repos/:name
 // @Summary Get Repository Info
 // @Description Returns basic information about local repository.
 // @Tags Repos
-// @Param name path string true "Repository name"
 // @Produce  json
+// @Param name path string true "Repository name"
 // @Success 200 {object} deb.LocalRepo
 // @Failure 404 {object} Error "Repository not found"
 // @Router /api/repos/{name} [get]
@@ -301,10 +261,9 @@ func apiReposShow(c *gin.Context) {
 // @Description Cannot drop repos that are published.
 // @Description Needs force=1 to drop repos used as source by other repos.
 // @Tags Repos
-// @Param name path string true "Repository name"
+// @Produce json
 // @Param _async query bool false "Run in background and return task object"
 // @Param force query int false "force: 1 to enable"
-// @Produce json
 // @Success 200 {object} task.ProcessReturnValue "Repo object"
 // @Failure 404 {object} Error "Not Found"
 // @Failure 404 {object} Error "Repo Conflict"
@@ -313,10 +272,10 @@ func apiReposDrop(c *gin.Context) {
 	force := c.Request.URL.Query().Get("force") == "1"
 	name := c.Params.ByName("name")
 
-	// Load shallowly for 404 check, resource key, and task name.
-	// Full checks (published/snapshots) happen inside the task.
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.LocalRepoCollection()
+	snapshotCollection := collectionFactory.SnapshotCollection()
+	publishedCollection := collectionFactory.PublishedRepoCollection()
 
 	repo, err := collection.ByName(name)
 	if err != nil {
@@ -327,32 +286,19 @@ func apiReposDrop(c *gin.Context) {
 	resources := []string{string(repo.Key())}
 	taskName := fmt.Sprintf("Delete repo %s", name)
 	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		// Task: Create fresh collections inside task after lock acquired
+		published := publishedCollection.ByLocalRepo(repo)
-		taskCollectionFactory := context.NewCollectionFactory()
-		taskCollection := taskCollectionFactory.LocalRepoCollection()
-		taskSnapshotCollection := taskCollectionFactory.SnapshotCollection()
-		taskPublishedCollection := taskCollectionFactory.PublishedRepoCollection()
-
-		// Re-read repo with fresh collection after lock
-		repo, err := taskCollection.ByName(name)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusConflict, Value: nil}, fmt.Errorf("unable to drop: %s", err)
-		}
-
-		// Check with fresh collections
-		published := taskPublishedCollection.ByLocalRepo(repo)
 		if len(published) > 0 {
 			return &task.ProcessReturnValue{Code: http.StatusConflict, Value: nil}, fmt.Errorf("unable to drop, local repo is published")
 		}
 
 		if !force {
-			snapshots := taskSnapshotCollection.ByLocalRepoSource(repo)
+			snapshots := snapshotCollection.ByLocalRepoSource(repo)
 			if len(snapshots) > 0 {
 				return &task.ProcessReturnValue{Code: http.StatusConflict, Value: nil}, fmt.Errorf("unable to drop, local repo has snapshots, use ?force=1 to override")
 			}
 		}
 
-		return &task.ProcessReturnValue{Code: http.StatusOK, Value: gin.H{}}, taskCollection.Drop(repo)
+		return &task.ProcessReturnValue{Code: http.StatusOK, Value: gin.H{}}, collection.Drop(repo)
 	})
 }
 
@@ -367,12 +313,12 @@ func apiReposDrop(c *gin.Context) {
 // @Description ["Pi386 aptly 0.8 966561016b44ed80"]
 // @Description ```
 // @Tags Repos
-// @Param name path string true "Repository name"
+// @Produce json
+// @Param name path string true "Snapshot to search"
 // @Param q query string true "Package query (e.g Name%20(~%20matlab))"
 // @Param withDeps query string true "Set to 1 to include dependencies when evaluating package query"
 // @Param format query string true "Set to 'details' to return extra info about each package"
 // @Param maximumVersion query string true "Set to 1 to only return the highest version for each package name"
-// @Produce json
 // @Success 200 {object} string "msg"
 // @Failure 404 {object} Error "Not Found"
 // @Failure 404 {object} Error "Internal Server Error"
@@ -387,7 +333,7 @@ func apiReposPackagesShow(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(repo)
+	err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -409,13 +355,10 @@ func apiReposPackagesAddDelete(c *gin.Context, taskNamePrefix string, cb func(li
 		return
 	}
 
-	// Load shallowly for 404 check and resource key.
-	// Full load and mutations happen inside the task.
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.LocalRepoCollection()
 
-	name := c.Params.ByName("name")
+	repo, err := collection.ByName(c.Params.ByName("name"))
-	repo, err := collection.ByName(name)
 	if err != nil {
 		AbortWithJSONError(c, 404, err)
 		return
@@ -424,23 +367,13 @@ func apiReposPackagesAddDelete(c *gin.Context, taskNamePrefix string, cb func(li
 	resources := []string{string(repo.Key())}
 
 	maybeRunTaskInBackground(c, taskNamePrefix+repo.Name, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		// Task: Create fresh factory and collection inside task after lock
+		err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
-		taskCollectionFactory := context.NewCollectionFactory()
-		taskCollection := taskCollectionFactory.LocalRepoCollection()
-
-		// Fresh load after lock acquired (use captured `name` variable, not gin context)
-		repo, err := taskCollection.ByName(name)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusNotFound, Value: nil}, err
-		}
-
-		err = taskCollection.LoadComplete(repo)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
 
 		out.Printf("Loading packages...\n")
-		list, err := deb.NewPackageListFromRefList(repo.RefList(), taskCollectionFactory.PackageCollection(), nil)
+		list, err := deb.NewPackageListFromRefList(repo.RefList(), collectionFactory.PackageCollection(), nil)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -449,7 +382,7 @@ func apiReposPackagesAddDelete(c *gin.Context, taskNamePrefix string, cb func(li
 		for _, ref := range b.PackageRefs {
 			var p *deb.Package
 
-			p, err = taskCollectionFactory.PackageCollection().ByKey([]byte(ref))
+			p, err = collectionFactory.PackageCollection().ByKey([]byte(ref))
 			if err != nil {
 				if err == database.ErrNotFound {
 					return &task.ProcessReturnValue{Code: http.StatusNotFound, Value: nil}, fmt.Errorf("packages %s: %s", ref, err)
@@ -463,9 +396,9 @@ func apiReposPackagesAddDelete(c *gin.Context, taskNamePrefix string, cb func(li
 			}
 		}
 
-		repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
+		repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
 
-		err = taskCollection.Update(repo)
+		err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save: %s", err)
 		}
@@ -480,11 +413,9 @@ func apiReposPackagesAddDelete(c *gin.Context, taskNamePrefix string, cb func(li
 // @Description
 // @Description API verifies that packages actually exist in aptly database and checks constraint that conflicting packages can’t be part of the same local repository.
 // @Tags Repos
-// @Param name path string true "Repository name"
+// @Produce json
-// @Consume  json
 // @Param request body reposPackagesAddDeleteParams true "Parameters"
 // @Param _async query bool false "Run in background and return task object"
-// @Produce json
 // @Success 200 {object} string "msg"
 // @Failure 400 {object} Error "Bad Request"
 // @Failure 404 {object} Error "Not Found"
@@ -502,11 +433,9 @@ func apiReposPackagesAdd(c *gin.Context) {
 // @Description
 // @Description Any package(s) can be removed from a local repository. Package references from a local repository can be retrieved with GET /api/repos/:name/packages.
 // @Tags Repos
-// @Param name path string true "Repository name"
-// @Param _async query bool false "Run in background and return task object"
-// @Consume  json
-// @Param request body reposPackagesAddDeleteParams true "Parameters"
 // @Produce json
+// @Param request body reposPackagesAddDeleteParams true "Parameters"
+// @Param _async query bool false "Run in background and return task object"
 // @Success 200 {object} string "msg"
 // @Failure 400 {object} Error "Bad Request"
 // @Failure 404 {object} Error "Not Found"
@@ -527,7 +456,7 @@ func apiReposPackagesDelete(c *gin.Context) {
 // @Tags Repos
 // @Param name path string true "Repository name"
 // @Param dir path string true "Directory of packages"
-// @Param file path string true "Filename"
+// @Param file path string false "Filename (optional)"
 // @Param _async query bool false "Run in background and return task object"
 // @Produce json
 // @Success 200 {string} string "OK"
@@ -572,8 +501,6 @@ func apiReposPackageFromDir(c *gin.Context) {
 		return
 	}
 
-	// Load shallowly for 404 check and resource key.
-	// Full load and mutations happen inside the task.
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.LocalRepoCollection()
 
@@ -597,17 +524,7 @@ func apiReposPackageFromDir(c *gin.Context) {
 	resources := []string{string(repo.Key())}
 	resources = append(resources, sources...)
 	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		// Task: Create fresh factory and collection inside task after lock
+		err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
-		taskCollectionFactory := context.NewCollectionFactory()
-		taskCollection := taskCollectionFactory.LocalRepoCollection()
-
-		// Fresh load after lock acquired
-		repo, err := taskCollection.ByName(name)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
-		}
-
-		err = taskCollection.LoadComplete(repo)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -628,13 +545,13 @@ func apiReposPackageFromDir(c *gin.Context) {
 
 		packageFiles, otherFiles, failedFiles = deb.CollectPackageFiles(sources, reporter)
 
-		list, err = deb.NewPackageListFromRefList(repo.RefList(), taskCollectionFactory.PackageCollection(), nil)
+		list, err := deb.NewPackageListFromRefList(repo.RefList(), collectionFactory.PackageCollection(), nil)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to load packages: %s", err)
 		}
 
 		processedFiles, failedFiles2, err = deb.ImportPackageFiles(list, packageFiles, forceReplace, verifier, context.PackagePool(),
-			taskCollectionFactory.PackageCollection(), reporter, nil, taskCollectionFactory.ChecksumCollection)
+			collectionFactory.PackageCollection(), reporter, nil, collectionFactory.ChecksumCollection)
 		failedFiles = append(failedFiles, failedFiles2...)
 		processedFiles = append(processedFiles, otherFiles...)
 
@@ -642,9 +559,9 @@ func apiReposPackageFromDir(c *gin.Context) {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to import package files: %s", err)
 		}
 
-		repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
+		repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
 
-		err = taskCollection.Update(repo)
+		err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save: %s", err)
 		}
@@ -660,7 +577,7 @@ func apiReposPackageFromDir(c *gin.Context) {
 			}
 
 			// atempt to remove dir, if it fails, that's fine: probably it's not empty
-			_ = os.Remove(filepath.Join(context.UploadPath(), dirParam))
+			os.Remove(filepath.Join(context.UploadPath(), dirParam))
 		}
 
 		if failedFiles == nil {
@@ -697,11 +614,11 @@ type reposCopyPackageParams struct {
 // @Summary Copy Package
 // @Description Copies a package from a source to destination repository
 // @Tags Repos
-// @Param name path string true "Destination repo"
+// @Produce json
-// @Param src path string true "Source repo"
+// @Param name path string true "Source repo"
+// @Param src path string true "Destination repo"
 // @Param file path string true "File/packages to copy"
 // @Param _async query bool false "Run in background and return task object"
-// @Produce json
 // @Success 200 {object} task.ProcessReturnValue "msg"
 // @Failure 400 {object} Error "Bad Request"
 // @Failure 404 {object} Error "Not Found"
@@ -723,8 +640,6 @@ func apiReposCopyPackage(c *gin.Context) {
 		return
 	}
 
-	// Load shallowly for 404 check and resource keys.
-	// Full load and mutations happen inside the task.
 	collectionFactory := context.NewCollectionFactory()
 	dstRepo, err := collectionFactory.LocalRepoCollection().ByName(dstRepoName)
 	if err != nil {
@@ -748,26 +663,12 @@ func apiReposCopyPackage(c *gin.Context) {
 	resources := []string{string(dstRepo.Key()), string(srcRepo.Key())}
 
 	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		// Task: Create fresh factory and collections inside task after lock
+		err = collectionFactory.LocalRepoCollection().LoadComplete(dstRepo)
-		taskCollectionFactory := context.NewCollectionFactory()
-
-		// Fresh load of both repos after lock acquired
-		dstRepo, err := taskCollectionFactory.LocalRepoCollection().ByName(dstRepoName)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, fmt.Errorf("dest repo error: %s", err)
 		}
 
-		srcRepo, err := taskCollectionFactory.LocalRepoCollection().ByName(srcRepoName)
+		err = collectionFactory.LocalRepoCollection().LoadComplete(srcRepo)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, fmt.Errorf("src repo error: %s", err)
-		}
-
-		err = taskCollectionFactory.LocalRepoCollection().LoadComplete(dstRepo)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, fmt.Errorf("dest repo error: %s", err)
-		}
-
-		err = taskCollectionFactory.LocalRepoCollection().LoadComplete(srcRepo)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, fmt.Errorf("src repo error: %s", err)
 		}
@@ -780,12 +681,12 @@ func apiReposCopyPackage(c *gin.Context) {
 			RemovedLines: []string{},
 		}
 
-		dstList, err := deb.NewPackageListFromRefList(dstRepo.RefList(), taskCollectionFactory.PackageCollection(), context.Progress())
+		dstList, err := deb.NewPackageListFromRefList(dstRepo.RefList(), collectionFactory.PackageCollection(), context.Progress())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to load packages in dest: %s", err)
 		}
 
-		srcList, err := deb.NewPackageListFromRefList(srcRefList, taskCollectionFactory.PackageCollection(), context.Progress())
+		srcList, err := deb.NewPackageListFromRefList(srcRefList, collectionFactory.PackageCollection(), context.Progress())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to load packages in src: %s", err)
 		}
@@ -853,7 +754,7 @@ func apiReposCopyPackage(c *gin.Context) {
 		} else {
 			dstRepo.UpdateRefList(deb.NewPackageRefListFromPackageList(dstList))
 
-			err = taskCollectionFactory.LocalRepoCollection().Update(dstRepo)
+			err = collectionFactory.LocalRepoCollection().Update(dstRepo)
 			if err != nil {
 				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to save: %s", err)
 			}
@@ -868,15 +769,12 @@ func apiReposCopyPackage(c *gin.Context) {
 // @Summary Include File from Directory
 // @Description Allows automatic processing of .changes file controlling package upload (uploaded using File Upload API) to the local repository. i.e. Exposes repo include command in api.
 // @Tags Repos
-// @Param name path string true "Repository name"
+// @Produce json
-// @Param dir path string true "Directory of packages"
-// @Param file path string true "File/packages to include"
 // @Param forceReplace query int false "when value is set to 1, when adding package that conflicts with existing package, remove existing package"
 // @Param noRemoveFiles query int false "when value is set to 1, don’t remove files that have been imported successfully into repository"
 // @Param acceptUnsigned query int false "when value is set to 1, accept unsigned .changes files"
 // @Param ignoreSignature query int false "when value is set to 1 disable verification of .changes file signature"
 // @Param _async query bool false "Run in background and return task object"
-// @Produce json
 // @Success 200 {object} string "msg"
 // @Failure 404 {object} Error "Not Found"
 // @Router /api/repos/{name}/include/{dir}/{file} [post]
@@ -885,22 +783,26 @@ func apiReposIncludePackageFromFile(c *gin.Context) {
 	apiReposIncludePackageFromDir(c)
 }
 
+type reposIncludePackageFromDirReport struct {
+	Warnings []string
+	Added    []string
+	Deleted  []string
+}
+
 type reposIncludePackageFromDirResponse struct {
-	Report      *aptly.RecordingResultReporter
+	Report      reposIncludePackageFromDirReport
 	FailedFiles []string
 }
 
 // @Summary Include Directory
 // @Description Allows automatic processing of .changes file controlling package upload (uploaded using File Upload API) to the local repository. i.e. Exposes repo include command in api.
 // @Tags Repos
-// @Param name path string true "Repository name"
+// @Produce json
-// @Param dir path string true "Directory of packages"
 // @Param forceReplace query int false "when value is set to 1, when adding package that conflicts with existing package, remove existing package"
 // @Param noRemoveFiles query int false "when value is set to 1, don’t remove files that have been imported successfully into repository"
 // @Param acceptUnsigned query int false "when value is set to 1, accept unsigned .changes files"
 // @Param ignoreSignature query int false "when value is set to 1 disable verification of .changes file signature"
 // @Param _async query bool false "Run in background and return task object"
-// @Produce json
 // @Success 200 {object} reposIncludePackageFromDirResponse "Response"
 // @Failure 404 {object} Error "Not Found"
 // @Router /api/repos/{name}/include/{dir} [post]
@@ -941,7 +843,7 @@ func apiReposIncludePackageFromDir(c *gin.Context) {
 	}
 
 	var resources []string
-	if len(repoTemplate.Root.Nodes) > 1 {
+	if len(repoTemplate.Tree.Root.Nodes) > 1 {
 		resources = append(resources, task.AllLocalReposResourcesKey)
 	} else {
 		// repo template string is simple text so only use resource key of specific repository
@@ -956,9 +858,6 @@ func apiReposIncludePackageFromDir(c *gin.Context) {
 	resources = append(resources, sources...)
 
 	maybeRunTaskInBackground(c, taskName, resources, func(out aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		// Task: Create fresh factory and collection inside task after lock
-		taskCollectionFactory := context.NewCollectionFactory()
-
 		var (
 			err                       error
 			verifier                  = context.GetVerifier()
@@ -974,8 +873,8 @@ func apiReposIncludePackageFromDir(c *gin.Context) {
 		changesFiles, failedFiles = deb.CollectChangesFiles(sources, reporter)
 		_, failedFiles2, err = deb.ImportChangesFiles(
 			changesFiles, reporter, acceptUnsigned, ignoreSignature, forceReplace, noRemoveFiles, verifier,
-			repoTemplate, context.Progress(), taskCollectionFactory.LocalRepoCollection(), taskCollectionFactory.PackageCollection(),
+			repoTemplate, context.Progress(), collectionFactory.LocalRepoCollection(), collectionFactory.PackageCollection(),
-			context.PackagePool(), taskCollectionFactory.ChecksumCollection, nil, query.Parse)
+			collectionFactory.RefListCollection(), context.PackagePool(), collectionFactory.ChecksumCollection, nil, query.Parse)
 		failedFiles = append(failedFiles, failedFiles2...)
 
 		if err != nil {
@@ -984,7 +883,7 @@ func apiReposIncludePackageFromDir(c *gin.Context) {
 
 		if !noRemoveFiles {
 			// atempt to remove dir, if it fails, that's fine: probably it's not empty
-			_ = os.Remove(filepath.Join(context.UploadPath(), dirParam))
+			os.Remove(filepath.Join(context.UploadPath(), dirParam))
 		}
 
 		if failedFiles == nil {
@@ -1004,10 +903,9 @@ func apiReposIncludePackageFromDir(c *gin.Context) {
 			out.Printf("Failed files: %s\n", strings.Join(failedFiles, ", "))
 		}
 
-		ret := reposIncludePackageFromDirResponse{
+		return &task.ProcessReturnValue{Code: http.StatusOK, Value: gin.H{
-			Report:      reporter,
+			"Report":      reporter,
-			FailedFiles: failedFiles,
+			"FailedFiles": failedFiles,
-		}
+		}}, nil
-		return &task.ProcessReturnValue{Code: http.StatusOK, Value: ret}, nil
 	})
 }

api/router.go

@@ -2,6 +2,7 @@ package api
 
 import (
 	"net/http"
+	"os"
 	"sync/atomic"
 
 	"github.com/aptly-dev/aptly/aptly"
@@ -11,19 +12,13 @@ import (
 	"github.com/prometheus/client_golang/prometheus/promhttp"
 	"github.com/rs/zerolog/log"
 
-	// _ "github.com/aptly-dev/aptly/docs" // import docs
+	"github.com/aptly-dev/aptly/docs"
-	// swaggerFiles "github.com/swaggo/files"
+	swaggerFiles "github.com/swaggo/files"
-	// ginSwagger "github.com/swaggo/gin-swagger"
+	ginSwagger "github.com/swaggo/gin-swagger"
 )
 
 var context *ctx.AptlyContext
 
-// @Summary Get Metrics
-// @Description **Get Prometheus Metrics**
-// @Tags Status
-// @Produce text/plain
-// @Success 200 {string} string Metrics
-// @Router /api/metrics [get]
 func apiMetricsGet() gin.HandlerFunc {
 	return func(c *gin.Context) {
 		countPackagesByRepos()
@@ -31,21 +26,21 @@ func apiMetricsGet() gin.HandlerFunc {
 	}
 }
 
-// func redirectSwagger(c *gin.Context) {
+func redirectSwagger(c *gin.Context) {
-// 	if c.Request.URL.Path == "/docs/index.html" {
+	if c.Request.URL.Path == "/docs/index.html" {
-// 		c.Redirect(http.StatusMovedPermanently, "/docs.html")
+		c.Redirect(http.StatusMovedPermanently, "/docs.html")
-// 		return
+		return
-// 	}
+	}
-// 	if c.Request.URL.Path == "/docs/" {
+	if c.Request.URL.Path == "/docs/" {
-// 		c.Redirect(http.StatusMovedPermanently, "/docs.html")
+		c.Redirect(http.StatusMovedPermanently, "/docs.html")
-// 		return
+		return
-// 	}
+	}
-// 	if c.Request.URL.Path == "/docs" {
+	if c.Request.URL.Path == "/docs" {
-// 		c.Redirect(http.StatusMovedPermanently, "/docs.html")
+		c.Redirect(http.StatusMovedPermanently, "/docs.html")
-// 		return
+		return
-// 	}
+	}
-// 	c.Next()
+	c.Next()
-// }
+}
 
 // Router returns prebuilt with routes http.Handler
 func Router(c *ctx.AptlyContext) http.Handler {
@@ -61,22 +56,26 @@ func Router(c *ctx.AptlyContext) http.Handler {
 	router.UseRawPath = true
 
 	if c.Config().LogFormat == "json" {
+		c.StructuredLogging(true)
+		utils.SetupJSONLogger(c.Config().LogLevel, os.Stdout)
 		gin.DefaultWriter = utils.LogWriter{Logger: log.Logger}
 		router.Use(JSONLogger())
 	} else {
+		c.StructuredLogging(false)
+		utils.SetupDefaultLogger(c.Config().LogLevel)
 		router.Use(gin.Logger())
 	}
 
 	router.Use(gin.Recovery(), gin.ErrorLogger())
 
-	// if c.Config().EnableSwaggerEndpoint {
+	if c.Config().EnableSwaggerEndpoint {
-	// 	router.GET("docs.html", func(c *gin.Context) {
+		router.GET("docs.html", func(c *gin.Context) {
-	// 		c.Data(http.StatusOK, "text/html; charset=utf-8", docs.DocsHTML)
+			c.Data(http.StatusOK, "text/html; charset=utf-8", docs.DocsHTML)
-	// 	})
+		})
-	// 	router.Use(redirectSwagger)
+		router.Use(redirectSwagger)
-	// 	url := ginSwagger.URL("/docs/doc.json")
+		url := ginSwagger.URL("/docs/doc.json")
-	// 	router.GET("/docs/*any", ginSwagger.WrapHandler(swaggerFiles.Handler, url))
+		router.GET("/docs/*any", ginSwagger.WrapHandler(swaggerFiles.Handler, url))
-	// }
+	}
 
 	if c.Config().EnableMetricsEndpoint {
 		MetricsCollectorRegistrar.Register(router)
@@ -221,7 +220,7 @@ func Router(c *ctx.AptlyContext) http.Handler {
 		api.GET("/graph.:ext", apiGraph)
 	}
 	{
-		api.POST("/db/cleanup", apiDBCleanup)
+		api.POST("/db/cleanup", apiDbCleanup)
 	}
 	{
 		api.GET("/tasks", apiTasksList)

api/s3.go

@@ -14,8 +14,7 @@ import (
 // @Router /api/s3 [get]
 func apiS3List(c *gin.Context) {
 	keys := []string{}
-	s3Roots := context.Config().S3PublishRoots
+	for k := range context.Config().S3PublishRoots {
-	for k := range s3Roots {
 		keys = append(keys, k)
 	}
 	c.JSON(200, keys)

api/snapshot.go

@@ -33,7 +33,7 @@ func apiSnapshotsList(c *gin.Context) {
 	}
 
 	result := []*deb.Snapshot{}
-	_ = collection.ForEachSorted(SortMethodString, func(snapshot *deb.Snapshot) error {
+	collection.ForEachSorted(SortMethodString, func(snapshot *deb.Snapshot) error {
 		result = append(result, snapshot)
 		return nil
 	})
@@ -74,33 +74,26 @@ func apiSnapshotsCreateFromMirror(c *gin.Context) {
 	}
 
 	collectionFactory := context.NewCollectionFactory()
+	collection := collectionFactory.RemoteRepoCollection()
+	snapshotCollection := collectionFactory.SnapshotCollection()
 	name := c.Params.ByName("name")
 
-	repo, err = collectionFactory.RemoteRepoCollection().ByName(name)
+	repo, err = collection.ByName(name)
 	if err != nil {
 		AbortWithJSONError(c, 404, err)
 		return
 	}
 
 	// including snapshot resource key
-	resources := []string{string(repo.Key())}
+	resources := []string{string(repo.Key()), "S" + b.Name}
 	taskName := fmt.Sprintf("Create snapshot of mirror %s", name)
 	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		taskCollectionFactory := context.NewCollectionFactory()
+		err := repo.CheckLock()
-		taskMirrorCollection := taskCollectionFactory.RemoteRepoCollection()
-		taskSnapshotCollection := taskCollectionFactory.SnapshotCollection()
-
-		repo, err := taskMirrorCollection.ByName(name)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
-		}
-
-		err = repo.CheckLock()
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusConflict, Value: nil}, err
 		}
 
-		err = taskMirrorCollection.LoadComplete(repo)
+		err = collection.LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -114,7 +107,7 @@ func apiSnapshotsCreateFromMirror(c *gin.Context) {
 			snapshot.Description = b.Description
 		}
 
-		err = taskSnapshotCollection.Add(snapshot)
+		err = snapshotCollection.Add(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, err
 		}
@@ -163,7 +156,6 @@ func apiSnapshotsCreate(c *gin.Context) {
 		}
 	}
 
-	// Phase 1: Pre-task validation (shallow load for 404 checks only)
 	collectionFactory := context.NewCollectionFactory()
 	snapshotCollection := collectionFactory.SnapshotCollection()
 	var resources []string
@@ -177,62 +169,37 @@ func apiSnapshotsCreate(c *gin.Context) {
 			return
 		}
 
-		resources = append(resources, string(sources[i].Key()))
+		resources = append(resources, string(sources[i].ResourceKey()))
 	}
 
 	maybeRunTaskInBackground(c, "Create snapshot "+b.Name, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		// Phase 2: Inside task lock - create fresh factory
+		for i := range sources {
-		taskCollectionFactory := context.NewCollectionFactory()
+			err = snapshotCollection.LoadComplete(sources[i], collectionFactory.RefListCollection())
-		taskSnapshotCollection := taskCollectionFactory.SnapshotCollection()
-		taskPackageCollection := taskCollectionFactory.PackageCollection()
-
-		// Fresh load of all sources after lock acquired
-		freshSources := make([]*deb.Snapshot, len(b.SourceSnapshots))
-		for i := range b.SourceSnapshots {
-			freshSources[i], err = taskSnapshotCollection.ByName(b.SourceSnapshots[i])
-			if err != nil {
-				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
-			}
-			// LoadComplete on fresh copy
-			err = taskSnapshotCollection.LoadComplete(freshSources[i])
 			if err != nil {
 				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 			}
 		}
 
-		// Merge packages from all source snapshots
+		list := deb.NewPackageList()
-		var refList *deb.PackageRefList
+
-		if len(freshSources) > 0 {
+		// verify package refs and build package list
-			refList = freshSources[0].RefList()
+		for _, ref := range b.PackageRefs {
-			for i := 1; i < len(freshSources); i++ {
+			p, err := collectionFactory.PackageCollection().ByKey([]byte(ref))
-				refList = refList.Merge(freshSources[i].RefList(), true, false)
+			if err != nil {
+				if err == database.ErrNotFound {
+					return &task.ProcessReturnValue{Code: http.StatusNotFound, Value: nil}, fmt.Errorf("package %s: %s", ref, err)
+				}
+				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
+			}
+			err = list.Add(p)
+			if err != nil {
+				return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, err
 			}
-		} else {
-			refList = deb.NewPackageRefList()
 		}
 
-		// Add any explicitly specified package refs on top
+		snapshot = deb.NewSnapshotFromRefList(b.Name, sources, deb.NewSplitRefListFromPackageList(list), b.Description)
-		if len(b.PackageRefs) > 0 {
-			list := deb.NewPackageList()
-			for _, ref := range b.PackageRefs {
-				p, err := taskPackageCollection.ByKey([]byte(ref))
-				if err != nil {
-					if err == database.ErrNotFound {
-						return &task.ProcessReturnValue{Code: http.StatusNotFound, Value: nil}, fmt.Errorf("package %s: %s", ref, err)
-					}
-					return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
-				}
-				err = list.Add(p)
-				if err != nil {
-					return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, err
-				}
-			}
-			refList = refList.Merge(deb.NewPackageRefListFromPackageList(list), true, false)
-		}
 
-		snapshot = deb.NewSnapshotFromRefList(b.Name, freshSources, refList, b.Description)
+		err = snapshotCollection.Add(snapshot, collectionFactory.RefListCollection())
-
-		err = taskSnapshotCollection.Add(snapshot)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, err
 		}
@@ -250,9 +217,10 @@ type snapshotsCreateFromRepositoryParams struct {
 // @Summary Snapshot Repository
 // @Description **Create a snapshot of a repository by name**
 // @Tags Snapshots
+// @Param name path string true "Repository name"
 // @Consume json
 // @Param request body snapshotsCreateFromRepositoryParams true "Parameters"
-// @Param name path string true "Repository name"
+// @Param name path string true "Name of the snapshot"
 // @Param _async query bool false "Run in background and return task object"
 // @Produce json
 // @Success 201 {object} deb.Snapshot "Created snapshot object"
@@ -273,28 +241,21 @@ func apiSnapshotsCreateFromRepository(c *gin.Context) {
 	}
 
 	collectionFactory := context.NewCollectionFactory()
+	collection := collectionFactory.LocalRepoCollection()
+	snapshotCollection := collectionFactory.SnapshotCollection()
 	name := c.Params.ByName("name")
 
-	repo, err = collectionFactory.LocalRepoCollection().ByName(name)
+	repo, err = collection.ByName(name)
 	if err != nil {
 		AbortWithJSONError(c, 404, err)
 		return
 	}
 
 	// including snapshot resource key
-	resources := []string{string(repo.Key())}
+	resources := []string{string(repo.Key()), "S" + b.Name}
 	taskName := fmt.Sprintf("Create snapshot of repo %s", name)
 	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		taskCollectionFactory := context.NewCollectionFactory()
+		err := collection.LoadComplete(repo, collectionFactory.RefListCollection())
-		taskRepoCollection := taskCollectionFactory.LocalRepoCollection()
-		taskSnapshotCollection := taskCollectionFactory.SnapshotCollection()
-
-		repo, err := taskRepoCollection.ByName(name)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
-		}
-
-		err = taskRepoCollection.LoadComplete(repo)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -308,7 +269,7 @@ func apiSnapshotsCreateFromRepository(c *gin.Context) {
 			snapshot.Description = b.Description
 		}
 
-		err = taskSnapshotCollection.Add(snapshot)
+		err = snapshotCollection.Add(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusBadRequest, Value: nil}, err
 		}
@@ -346,7 +307,6 @@ func apiSnapshotsUpdate(c *gin.Context) {
 		return
 	}
 
-	// Phase 1: Pre-task validation (shallow load for 404 check only)
 	collectionFactory := context.NewCollectionFactory()
 	collection := collectionFactory.SnapshotCollection()
 	name := c.Params.ByName("name")
@@ -357,38 +317,14 @@ func apiSnapshotsUpdate(c *gin.Context) {
 		return
 	}
 
-	// Pre-task validation of new name if provided (skip if renaming to same name)
+	resources := []string{string(snapshot.ResourceKey()), "S" + b.Name}
-	if b.Name != "" && b.Name != name {
-		_, err = collection.ByName(b.Name)
-		if err == nil {
-			AbortWithJSONError(c, 409, fmt.Errorf("unable to rename: snapshot %s already exists", b.Name))
-			return
-		}
-	}
-
-	resources := []string{string(snapshot.Key())}
 	taskName := fmt.Sprintf("Update snapshot %s", name)
-
 	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		// Phase 2: Inside task lock - create fresh factory
+		_, err := collection.ByName(b.Name)
-		taskCollectionFactory := context.NewCollectionFactory()
+		if err == nil {
-		taskCollection := taskCollectionFactory.SnapshotCollection()
+			return &task.ProcessReturnValue{Code: http.StatusConflict, Value: nil}, fmt.Errorf("unable to rename: snapshot %s already exists", b.Name)
-
-		// Fresh load after lock acquired
-		snapshot, err = taskCollection.ByName(name)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
 
-		// Fresh duplicate check inside lock
-		if b.Name != "" {
-			_, err := taskCollection.ByName(b.Name)
-			if err == nil {
-				return &task.ProcessReturnValue{Code: http.StatusConflict, Value: nil}, fmt.Errorf("unable to rename: snapshot %s already exists", b.Name)
-			}
-		}
-
-		// Update fresh copy
 		if b.Name != "" {
 			snapshot.Name = b.Name
 		}
@@ -397,7 +333,7 @@ func apiSnapshotsUpdate(c *gin.Context) {
 			snapshot.Description = b.Description
 		}
 
-		err = taskCollection.Update(snapshot)
+		err = collectionFactory.SnapshotCollection().Update(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -424,7 +360,7 @@ func apiSnapshotsShow(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(snapshot)
+	err = collection.LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -451,9 +387,9 @@ func apiSnapshotsDrop(c *gin.Context) {
 	name := c.Params.ByName("name")
 	force := c.Request.URL.Query().Get("force") == "1"
 
-	// Phase 1: Pre-task validation (shallow load for 404 check only)
 	collectionFactory := context.NewCollectionFactory()
 	snapshotCollection := collectionFactory.SnapshotCollection()
+	publishedCollection := collectionFactory.PublishedRepoCollection()
 
 	snapshot, err := snapshotCollection.ByName(name)
 	if err != nil {
@@ -461,37 +397,23 @@ func apiSnapshotsDrop(c *gin.Context) {
 		return
 	}
 
-	resources := []string{string(snapshot.Key())}
+	resources := []string{string(snapshot.ResourceKey())}
 	taskName := fmt.Sprintf("Delete snapshot %s", name)
-
 	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		// Phase 2: Inside task lock - create fresh collections
+		published := publishedCollection.BySnapshot(snapshot)
-		taskCollectionFactory := context.NewCollectionFactory()
-		taskSnapshotCollection := taskCollectionFactory.SnapshotCollection()
-		taskPublishedCollection := taskCollectionFactory.PublishedRepoCollection()
-
-		// Fresh load after lock acquired
-		snapshot, err := taskSnapshotCollection.ByName(name)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
-		}
-
-		// Fresh checks with current collections
-		published := taskPublishedCollection.BySnapshot(snapshot)
 
 		if len(published) > 0 {
 			return &task.ProcessReturnValue{Code: http.StatusConflict, Value: nil}, fmt.Errorf("unable to drop: snapshot is published")
 		}
 
 		if !force {
-			// Using fresh collection for dependency check
+			snapshots := snapshotCollection.BySnapshotSource(snapshot)
-			snapshots := taskSnapshotCollection.BySnapshotSource(snapshot)
 			if len(snapshots) > 0 {
 				return &task.ProcessReturnValue{Code: http.StatusConflict, Value: nil}, fmt.Errorf("won't delete snapshot that was used as source for other snapshots, use ?force=1 to override")
 			}
 		}
 
-		err = taskSnapshotCollection.Drop(snapshot)
+		err = snapshotCollection.Drop(snapshot)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -530,20 +452,20 @@ func apiSnapshotsDiff(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(snapshotA)
+	err = collection.LoadComplete(snapshotA, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
 	}
 
-	err = collection.LoadComplete(snapshotB)
+	err = collection.LoadComplete(snapshotB, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
 	}
 
 	// Calculate diff
-	diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection())
+	diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection(), nil)
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -586,7 +508,7 @@ func apiSnapshotsSearchPackages(c *gin.Context) {
 		return
 	}
 
-	err = collection.LoadComplete(snapshot)
+	err = collection.LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		AbortWithJSONError(c, 500, err)
 		return
@@ -633,7 +555,7 @@ func apiSnapshotsMerge(c *gin.Context) {
 	}
 
 	if len(body.Sources) < 1 {
-		AbortWithJSONError(c, http.StatusBadRequest, fmt.Errorf("minimum one source snapshot is required"))
+		AbortWithJSONError(c, http.StatusBadRequest, fmt.Errorf("At least one source snapshot is required"))
 		return
 	}
 
@@ -646,7 +568,6 @@ func apiSnapshotsMerge(c *gin.Context) {
 		return
 	}
 
-	// Phase 1: Pre-task validation (shallow load for 404 checks only)
 	collectionFactory := context.NewCollectionFactory()
 	snapshotCollection := collectionFactory.SnapshotCollection()
 
@@ -659,47 +580,36 @@ func apiSnapshotsMerge(c *gin.Context) {
 			return
 		}
 
-		resources[i] = string(sources[i].Key())
+		resources[i] = string(sources[i].ResourceKey())
 	}
 
 	maybeRunTaskInBackground(c, "Merge snapshot "+name, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		// Phase 2: Inside task lock - create fresh factory
+		err = snapshotCollection.LoadComplete(sources[0], collectionFactory.RefListCollection())
-		taskCollectionFactory := context.NewCollectionFactory()
+		if err != nil {
-		taskSnapshotCollection := taskCollectionFactory.SnapshotCollection()
+			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
-
-		// Fresh load of all sources inside task
-		freshSources := make([]*deb.Snapshot, len(body.Sources))
-		for i := range body.Sources {
-			freshSources[i], err = taskSnapshotCollection.ByName(body.Sources[i])
-			if err != nil {
-				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
-			}
-			// LoadComplete on fresh copy
-			err = taskSnapshotCollection.LoadComplete(freshSources[i])
-			if err != nil {
-				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
-			}
 		}
-
+		result := sources[0].RefList()
-		// Merge using fresh sources
+		for i := 1; i < len(sources); i++ {
-		result := freshSources[0].RefList()
+			err = snapshotCollection.LoadComplete(sources[i], collectionFactory.RefListCollection())
-		for i := 1; i < len(freshSources); i++ {
+			if err != nil {
-			result = result.Merge(freshSources[i].RefList(), overrideMatching, false)
+				return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
+			}
+			result = result.Merge(sources[i].RefList(), overrideMatching, false)
 		}
 
 		if latest {
 			result.FilterLatestRefs()
 		}
 
-		sourceDescription := make([]string, len(freshSources))
+		sourceDescription := make([]string, len(sources))
-		for i, s := range freshSources {
+		for i, s := range sources {
 			sourceDescription[i] = fmt.Sprintf("'%s'", s.Name)
 		}
 
-		snapshot = deb.NewSnapshotFromRefList(name, freshSources, result,
+		snapshot = deb.NewSnapshotFromRefList(name, sources, result,
 			fmt.Sprintf("Merged from sources: %s", strings.Join(sourceDescription, ", ")))
 
-		err = taskCollectionFactory.SnapshotCollection().Add(snapshot)
+		err = collectionFactory.SnapshotCollection().Add(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, fmt.Errorf("unable to create snapshot: %s", err)
 		}
@@ -780,32 +690,24 @@ func apiSnapshotsPull(c *gin.Context) {
 		return
 	}
 
-	resources := []string{string(sourceSnapshot.Key()), string(toSnapshot.Key())}
+	resources := []string{string(sourceSnapshot.ResourceKey()), string(toSnapshot.ResourceKey())}
 	taskName := fmt.Sprintf("Pull snapshot %s into %s and save as %s", body.Source, name, body.Destination)
 	maybeRunTaskInBackground(c, taskName, resources, func(_ aptly.Progress, _ *task.Detail) (*task.ProcessReturnValue, error) {
-		// Phase 2: Inside task lock - create fresh factory
+		err = collectionFactory.SnapshotCollection().LoadComplete(toSnapshot)
-		taskCollectionFactory := context.NewCollectionFactory()
-
-		// Fresh load of snapshots after lock acquired
-		freshToSnapshot, err := taskCollectionFactory.SnapshotCollection().ByName(name)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
-		freshSourceSnapshot, err := taskCollectionFactory.SnapshotCollection().ByName(body.Source)
+		err = collectionFactory.SnapshotCollection().LoadComplete(sourceSnapshot)
-		if err != nil {
-			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
-		}
-		err = taskCollectionFactory.SnapshotCollection().LoadComplete(freshSourceSnapshot)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
 
 		// convert snapshots to package list
-		toPackageList, err := deb.NewPackageListFromRefList(freshToSnapshot.RefList(), taskCollectionFactory.PackageCollection(), context.Progress())
+		toPackageList, err := deb.NewPackageListFromRefList(toSnapshot.RefList(), collectionFactory.PackageCollection(), context.Progress())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
-		sourcePackageList, err := deb.NewPackageListFromRefList(freshSourceSnapshot.RefList(), taskCollectionFactory.PackageCollection(), context.Progress())
+		sourcePackageList, err := deb.NewPackageListFromRefList(sourceSnapshot.RefList(), collectionFactory.PackageCollection(), context.Progress())
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}
@@ -863,7 +765,7 @@ func apiSnapshotsPull(c *gin.Context) {
 		addedPackages := []string{}
 		alreadySeen := map[string]bool{}
 
-		_ = destinationPackageList.ForEachIndexed(func(pkg *deb.Package) error {
+		destinationPackageList.ForEachIndexed(func(pkg *deb.Package) error {
 			key := pkg.Architecture + "_" + pkg.Name
 			_, seen := alreadySeen[key]
 
@@ -879,7 +781,7 @@ func apiSnapshotsPull(c *gin.Context) {
 
 			// If !allMatches, add only first matching name-arch package
 			if !seen || allMatches {
-				_ = toPackageList.Add(pkg)
+				toPackageList.Add(pkg)
 				addedPackages = append(addedPackages, pkg.String())
 			}
 
@@ -902,10 +804,10 @@ func apiSnapshotsPull(c *gin.Context) {
 		}
 
 		// Create <destination> snapshot
-		destinationSnapshot = deb.NewSnapshotFromPackageList(body.Destination, []*deb.Snapshot{freshToSnapshot, freshSourceSnapshot}, toPackageList,
+		destinationSnapshot = deb.NewSnapshotFromPackageList(body.Destination, []*deb.Snapshot{toSnapshot, sourceSnapshot}, toPackageList,
-			fmt.Sprintf("Pulled into '%s' with '%s' as source, pull request was: '%s'", freshToSnapshot.Name, freshSourceSnapshot.Name, strings.Join(body.Queries, ", ")))
+			fmt.Sprintf("Pulled into '%s' with '%s' as source, pull request was: '%s'", toSnapshot.Name, sourceSnapshot.Name, strings.Join(body.Queries, ", ")))
 
-		err = taskCollectionFactory.SnapshotCollection().Add(destinationSnapshot)
+		err = collectionFactory.SnapshotCollection().Add(destinationSnapshot)
 		if err != nil {
 			return &task.ProcessReturnValue{Code: http.StatusInternalServerError, Value: nil}, err
 		}

aptly/conf.go

@@ -1,4 +1,4 @@
 package aptly
 
-// AptlyConf holds the default aptly.conf (filled in at link time)
+// Default aptly.conf (filled in at link time)
 var AptlyConf []byte

aptly/interfaces.go

@@ -95,8 +95,8 @@ type FileSystemPublishedStorage interface {
 
 // PublishedStorageProvider is a thing that returns PublishedStorage by name
 type PublishedStorageProvider interface {
-	// GetPublishedStorage returns PublishedStorage by name, or an error if the storage is not configured
+	// GetPublishedStorage returns PublishedStorage by name
-	GetPublishedStorage(name string) (PublishedStorage, error)
+	GetPublishedStorage(name string) PublishedStorage
 }
 
 // BarType used to differentiate between different progress bars

azure/azure.go

@@ -5,28 +5,35 @@ package azure
 import (
 	"context"
 	"encoding/hex"
+	"errors"
 	"fmt"
 	"io"
-	"net/url"
+	"os"
 	"path/filepath"
 	"time"
 
-	"github.com/Azure/azure-storage-blob-go/azblob"
+	"github.com/Azure/azure-sdk-for-go/sdk/azcore"
+	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
+	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
 	"github.com/aptly-dev/aptly/aptly"
 )
 
 func isBlobNotFound(err error) bool {
-	storageError, ok := err.(azblob.StorageError)
+	var respErr *azcore.ResponseError
-	return ok && storageError.ServiceCode() == azblob.ServiceCodeBlobNotFound
+	if errors.As(err, &respErr) {
+		return respErr.StatusCode == 404 // BlobNotFound
+	}
+	return false
 }
 
 type azContext struct {
-	container azblob.ContainerURL
+	client    *azblob.Client
+	container string
 	prefix    string
 }
 
 func newAzContext(accountName, accountKey, container, prefix, endpoint string) (*azContext, error) {
-	credential, err := azblob.NewSharedKeyCredential(accountName, accountKey)
+	cred, err := azblob.NewSharedKeyCredential(accountName, accountKey)
 	if err != nil {
 		return nil, err
 	}
@@ -35,15 +42,14 @@ func newAzContext(accountName, accountKey, container, prefix, endpoint string) (
 		endpoint = fmt.Sprintf("https://%s.blob.core.windows.net", accountName)
 	}
 
-	url, err := url.Parse(fmt.Sprintf("%s/%s", endpoint, container))
+	serviceClient, err := azblob.NewClientWithSharedKeyCredential(endpoint, cred, nil)
 	if err != nil {
 		return nil, err
 	}
 
-	containerURL := azblob.NewContainerURL(*url, azblob.NewPipeline(credential, azblob.PipelineOptions{}))
-
 	result := &azContext{
-		container: containerURL,
+		client:    serviceClient,
+		container: container,
 		prefix:    prefix,
 	}
 
@@ -54,10 +60,6 @@ func (az *azContext) blobPath(path string) string {
 	return filepath.Join(az.prefix, path)
 }
 
-func (az *azContext) blobURL(path string) azblob.BlobURL {
-	return az.container.NewBlobURL(az.blobPath(path))
-}
-
 func (az *azContext) internalFilelist(prefix string, progress aptly.Progress) (paths []string, md5s []string, err error) {
 	const delimiter = "/"
 	paths = make([]string, 0, 1024)
@@ -67,27 +69,33 @@ func (az *azContext) internalFilelist(prefix string, progress aptly.Progress) (p
 		prefix += delimiter
 	}
 
-	for marker := (azblob.Marker{}); marker.NotDone(); {
+	ctx := context.Background()
-		listBlob, err := az.container.ListBlobsFlatSegment(
+	maxResults := int32(1)
-			context.Background(), marker, azblob.ListBlobsSegmentOptions{
+	pager := az.client.NewListBlobsFlatPager(az.container, &azblob.ListBlobsFlatOptions{
-				Prefix:     prefix,
+		Prefix:     &prefix,
-				MaxResults: 1,
+		MaxResults: &maxResults,
-				Details:    azblob.BlobListingDetails{Metadata: true}})
+		Include:    azblob.ListBlobsInclude{Metadata: true},
+	})
+
+	// Iterate over each page
+	for pager.More() {
+		page, err := pager.NextPage(ctx)
 		if err != nil {
 			return nil, nil, fmt.Errorf("error listing under prefix %s in %s: %s", prefix, az, err)
 		}
 
-		marker = listBlob.NextMarker
+		for _, blob := range page.Segment.BlobItems {
-
-		for _, blob := range listBlob.Segment.BlobItems {
 			if prefix == "" {
-				paths = append(paths, blob.Name)
+				paths = append(paths, *blob.Name)
 			} else {
-				paths = append(paths, blob.Name[len(prefix):])
+				name := *blob.Name
+				paths = append(paths, name[len(prefix):])
 			}
-			md5s = append(md5s, fmt.Sprintf("%x", blob.Properties.ContentMD5))
+			b := *blob
-		}
+			md5 := b.Properties.ContentMD5
+			md5s = append(md5s, fmt.Sprintf("%x", md5))
 
+		}
 		if progress != nil {
 			time.Sleep(time.Duration(500) * time.Millisecond)
 			progress.AddBar(1)
@@ -97,28 +105,27 @@ func (az *azContext) internalFilelist(prefix string, progress aptly.Progress) (p
 	return paths, md5s, nil
 }
 
-func (az *azContext) putFile(blob azblob.BlobURL, source io.Reader, sourceMD5 string) error {
+func (az *azContext) putFile(blobName string, source io.Reader, sourceMD5 string) error {
-	uploadOptions := azblob.UploadStreamToBlockBlobOptions{
+	uploadOptions := &azblob.UploadFileOptions{
-		BufferSize: 4 * 1024 * 1024,
+		BlockSize:   4 * 1024 * 1024,
-		MaxBuffers: 8,
+		Concurrency: 8,
 	}
 
+	path := az.blobPath(blobName)
 	if len(sourceMD5) > 0 {
 		decodedMD5, err := hex.DecodeString(sourceMD5)
 		if err != nil {
 			return err
 		}
-		uploadOptions.BlobHTTPHeaders = azblob.BlobHTTPHeaders{
+		uploadOptions.HTTPHeaders = &blob.HTTPHeaders{
-			ContentMD5: decodedMD5,
+			BlobContentMD5: decodedMD5,
 		}
 	}
 
-	_, err := azblob.UploadStreamToBlockBlob(
+	var err error
-		context.Background(),
+	if file, ok := source.(*os.File); ok {
-		source,
+		_, err = az.client.UploadFile(context.TODO(), az.container, path, file, uploadOptions)
-		blob.ToBlockBlobURL(),
+	}
-		uploadOptions,
-	)
 
 	return err
 }

azure/package_pool.go

@@ -5,7 +5,6 @@ import (
 	"os"
 	"path/filepath"
 
-	"github.com/Azure/azure-storage-blob-go/azblob"
 	"github.com/aptly-dev/aptly/aptly"
 	"github.com/aptly-dev/aptly/utils"
 	"github.com/pkg/errors"
@@ -30,7 +29,7 @@ func NewPackagePool(accountName, accountKey, container, prefix, endpoint string)
 	return &PackagePool{az: azctx}, nil
 }
 
-// String returns the storage as string
+// String
 func (pool *PackagePool) String() string {
 	return pool.az.String()
 }
@@ -41,10 +40,7 @@ func (pool *PackagePool) buildPoolPath(filename string, checksums *utils.Checksu
 	return filepath.Join(hash[0:2], hash[2:4], hash[4:32]+"_"+filename)
 }
 
-func (pool *PackagePool) ensureChecksums(
+func (pool *PackagePool) ensureChecksums(poolPath string, checksumStorage aptly.ChecksumStorage) (*utils.ChecksumInfo, error) {
-	poolPath string,
-	checksumStorage aptly.ChecksumStorage,
-) (*utils.ChecksumInfo, error) {
 	targetChecksums, err := checksumStorage.Get(poolPath)
 	if err != nil {
 		return nil, err
@@ -52,8 +48,7 @@ func (pool *PackagePool) ensureChecksums(
 
 	if targetChecksums == nil {
 		// we don't have checksums stored yet for this file
-		blob := pool.az.blobURL(poolPath)
+		download, err := pool.az.client.DownloadStream(context.Background(), pool.az.container, poolPath, nil)
-		download, err := blob.Download(context.Background(), 0, 0, azblob.BlobAccessConditions{}, false, azblob.ClientProvidedKeyOptions{})
 		if err != nil {
 			if isBlobNotFound(err) {
 				return nil, nil
@@ -63,7 +58,7 @@ func (pool *PackagePool) ensureChecksums(
 		}
 
 		targetChecksums = &utils.ChecksumInfo{}
-		*targetChecksums, err = utils.ChecksumsForReader(download.Body(azblob.RetryReaderOptions{}))
+		*targetChecksums, err = utils.ChecksumsForReader(download.Body)
 		if err != nil {
 			return nil, errors.Wrapf(err, "error checksumming blob at %s", poolPath)
 		}
@@ -92,45 +87,49 @@ func (pool *PackagePool) LegacyPath(_ string, _ *utils.ChecksumInfo) (string, er
 }
 
 func (pool *PackagePool) Size(path string) (int64, error) {
-	blob := pool.az.blobURL(path)
+	serviceClient := pool.az.client.ServiceClient()
-	props, err := blob.GetProperties(context.Background(), azblob.BlobAccessConditions{}, azblob.ClientProvidedKeyOptions{})
+	containerClient := serviceClient.NewContainerClient(pool.az.container)
+	blobClient := containerClient.NewBlobClient(path)
+
+	props, err := blobClient.GetProperties(context.TODO(), nil)
 	if err != nil {
 		return 0, errors.Wrapf(err, "error examining %s from %s", path, pool)
 	}
 
-	return props.ContentLength(), nil
+	return *props.ContentLength, nil
 }
 
 func (pool *PackagePool) Open(path string) (aptly.ReadSeekerCloser, error) {
-	blob := pool.az.blobURL(path)
-
 	temp, err := os.CreateTemp("", "blob-download")
 	if err != nil {
-		return nil, errors.Wrap(err, "error creating temporary file for blob download")
+		return nil, errors.Wrapf(err, "error creating tempfile for %s", path)
 	}
-	defer func() { _ = os.Remove(temp.Name()) }()
+	defer os.Remove(temp.Name())
 
-	err = azblob.DownloadBlobToFile(context.Background(), blob, 0, 0, temp, azblob.DownloadFromBlobOptions{})
+	_, err = pool.az.client.DownloadFile(context.TODO(), pool.az.container, path, temp, nil)
 	if err != nil {
-		return nil, errors.Wrapf(err, "error downloading blob at %s", path)
+		return nil, errors.Wrapf(err, "error downloading blob %s", path)
 	}
 
 	return temp, nil
 }
 
 func (pool *PackagePool) Remove(path string) (int64, error) {
-	blob := pool.az.blobURL(path)
+	serviceClient := pool.az.client.ServiceClient()
-	props, err := blob.GetProperties(context.Background(), azblob.BlobAccessConditions{}, azblob.ClientProvidedKeyOptions{})
+	containerClient := serviceClient.NewContainerClient(pool.az.container)
+	blobClient := containerClient.NewBlobClient(path)
+
+	props, err := blobClient.GetProperties(context.TODO(), nil)
 	if err != nil {
-		return 0, errors.Wrapf(err, "error getting props of %s from %s", path, pool)
+		return 0, errors.Wrapf(err, "error examining %s from %s", path, pool)
 	}
 
-	_, err = blob.Delete(context.Background(), azblob.DeleteSnapshotsOptionNone, azblob.BlobAccessConditions{})
+	_, err = pool.az.client.DeleteBlob(context.Background(), pool.az.container, path, nil)
 	if err != nil {
 		return 0, errors.Wrapf(err, "error deleting %s from %s", path, pool)
 	}
 
-	return props.ContentLength(), nil
+	return *props.ContentLength, nil
 }
 
 func (pool *PackagePool) Import(srcPath, basename string, checksums *utils.ChecksumInfo, _ bool, checksumStorage aptly.ChecksumStorage) (string, error) {
@@ -144,7 +143,6 @@ func (pool *PackagePool) Import(srcPath, basename string, checksums *utils.Check
 	}
 
 	path := pool.buildPoolPath(basename, checksums)
-	blob := pool.az.blobURL(path)
 	targetChecksums, err := pool.ensureChecksums(path, checksumStorage)
 	if err != nil {
 		return "", err
@@ -158,9 +156,9 @@ func (pool *PackagePool) Import(srcPath, basename string, checksums *utils.Check
 	if err != nil {
 		return "", err
 	}
-	defer func() { _ = source.Close() }()
+	defer source.Close()
 
-	err = pool.az.putFile(blob, source, checksums.MD5)
+	err = pool.az.putFile(path, source, checksums.MD5)
 	if err != nil {
 		return "", err
 	}

azure/package_pool_test.go

@@ -2,12 +2,12 @@ package azure
 
 import (
 	"context"
-	"io"
+	"io/ioutil"
 	"os"
 	"path/filepath"
 	"runtime"
 
-	"github.com/Azure/azure-storage-blob-go/azblob"
+        "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
 	"github.com/aptly-dev/aptly/aptly"
 	"github.com/aptly-dev/aptly/files"
 	"github.com/aptly-dev/aptly/utils"
@@ -50,8 +50,10 @@ func (s *PackagePoolSuite) SetUpTest(c *C) {
 
 	s.pool, err = NewPackagePool(s.accountName, s.accountKey, container, "", s.endpoint)
 	c.Assert(err, IsNil)
-	cnt := s.pool.az.container
+        publicAccessType := azblob.PublicAccessTypeContainer
-	_, err = cnt.Create(context.Background(), azblob.Metadata{}, azblob.PublicAccessContainer)
+        _, err = s.pool.az.client.CreateContainer(context.TODO(), s.pool.az.container, &azblob.CreateContainerOptions{
+            Access: &publicAccessType,
+        })
 	c.Assert(err, IsNil)
 
 	s.prefixedPool, err = NewPackagePool(s.accountName, s.accountKey, container, prefix, s.endpoint)
@@ -67,8 +69,8 @@ func (s *PackagePoolSuite) TestFilepathList(c *C) {
 	c.Check(err, IsNil)
 	c.Check(list, DeepEquals, []string{})
 
-	_, _ = s.pool.Import(s.debFile, "a.deb", &utils.ChecksumInfo{}, false, s.cs)
+	s.pool.Import(s.debFile, "a.deb", &utils.ChecksumInfo{}, false, s.cs)
-	_, _ = s.pool.Import(s.debFile, "b.deb", &utils.ChecksumInfo{}, false, s.cs)
+	s.pool.Import(s.debFile, "b.deb", &utils.ChecksumInfo{}, false, s.cs)
 
 	list, err = s.pool.FilepathList(nil)
 	c.Check(err, IsNil)
@@ -79,8 +81,8 @@ func (s *PackagePoolSuite) TestFilepathList(c *C) {
 }
 
 func (s *PackagePoolSuite) TestRemove(c *C) {
-	_, _ = s.pool.Import(s.debFile, "a.deb", &utils.ChecksumInfo{}, false, s.cs)
+	s.pool.Import(s.debFile, "a.deb", &utils.ChecksumInfo{}, false, s.cs)
-	_, _ = s.pool.Import(s.debFile, "b.deb", &utils.ChecksumInfo{}, false, s.cs)
+	s.pool.Import(s.debFile, "b.deb", &utils.ChecksumInfo{}, false, s.cs)
 
 	size, err := s.pool.Remove("c7/6b/4bd12fd92e4dfe1b55b18a67a669_a.deb")
 	c.Check(err, IsNil)
@@ -245,7 +247,7 @@ func (s *PackagePoolSuite) TestOpen(c *C) {
 
 	f, err := s.pool.Open(path)
 	c.Assert(err, IsNil)
-	contents, err := io.ReadAll(f)
+	contents, err := ioutil.ReadAll(f)
 	c.Assert(err, IsNil)
 	c.Check(len(contents), Equals, 2738)
 	c.Check(f.Close(), IsNil)

azure/public.go

@@ -3,19 +3,22 @@ package azure
 import (
 	"context"
 	"fmt"
-	"net/http"
 	"os"
 	"path/filepath"
 	"time"
 
-	"github.com/Azure/azure-storage-blob-go/azblob"
+	"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
+	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
+	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/lease"
 	"github.com/aptly-dev/aptly/aptly"
 	"github.com/aptly-dev/aptly/utils"
+	"github.com/google/uuid"
 	"github.com/pkg/errors"
 )
 
 // PublishedStorage abstract file system with published files (actually hosted on Azure)
 type PublishedStorage struct {
+	prefix    string
 	az        *azContext
 	pathCache map[string]map[string]string
 }
@@ -35,7 +38,7 @@ func NewPublishedStorage(accountName, accountKey, container, prefix, endpoint st
 	return &PublishedStorage{az: azctx}, nil
 }
 
-// String returns the storage as string
+// String
 func (storage *PublishedStorage) String() string {
 	return storage.az.String()
 }
@@ -62,9 +65,9 @@ func (storage *PublishedStorage) PutFile(path string, sourceFilename string) err
 	if err != nil {
 		return err
 	}
-	defer func() { _ = source.Close() }()
+	defer source.Close()
 
-	err = storage.az.putFile(storage.az.blobURL(path), source, sourceMD5)
+	err = storage.az.putFile(path, source, sourceMD5)
 	if err != nil {
 		err = errors.Wrap(err, fmt.Sprintf("error uploading %s to %s", sourceFilename, storage))
 	}
@@ -74,14 +77,15 @@ func (storage *PublishedStorage) PutFile(path string, sourceFilename string) err
 
 // RemoveDirs removes directory structure under public path
 func (storage *PublishedStorage) RemoveDirs(path string, _ aptly.Progress) error {
+	path = storage.az.blobPath(path)
 	filelist, err := storage.Filelist(path)
 	if err != nil {
 		return err
 	}
 
 	for _, filename := range filelist {
-		blob := storage.az.blobURL(filepath.Join(path, filename))
+		blob := filepath.Join(path, filename)
-		_, err := blob.Delete(context.Background(), azblob.DeleteSnapshotsOptionNone, azblob.BlobAccessConditions{})
+		_, err := storage.az.client.DeleteBlob(context.Background(), storage.az.container, blob, nil)
 		if err != nil {
 			return fmt.Errorf("error deleting path %s from %s: %s", filename, storage, err)
 		}
@@ -92,8 +96,8 @@ func (storage *PublishedStorage) RemoveDirs(path string, _ aptly.Progress) error
 
 // Remove removes single file under public path
 func (storage *PublishedStorage) Remove(path string) error {
-	blob := storage.az.blobURL(path)
+	path = storage.az.blobPath(path)
-	_, err := blob.Delete(context.Background(), azblob.DeleteSnapshotsOptionNone, azblob.BlobAccessConditions{})
+	_, err := storage.az.client.DeleteBlob(context.Background(), storage.az.container, path, nil)
 	if err != nil {
 		err = errors.Wrap(err, fmt.Sprintf("error deleting %s from %s: %s", path, storage, err))
 	}
@@ -112,9 +116,8 @@ func (storage *PublishedStorage) LinkFromPool(publishedPrefix, publishedRelPath,
 	sourcePath string, sourceChecksums utils.ChecksumInfo, force bool) error {
 
 	relFilePath := filepath.Join(publishedRelPath, fileName)
-	// prefixRelFilePath := filepath.Join(publishedPrefix, relFilePath)
+	prefixRelFilePath := filepath.Join(publishedPrefix, relFilePath)
-	// FIXME: check how to integrate publishedPrefix:
+	poolPath := storage.az.blobPath(prefixRelFilePath)
-	poolPath := storage.az.blobPath(fileName)
 
 	if storage.pathCache == nil {
 		storage.pathCache = make(map[string]map[string]string)
@@ -155,9 +158,9 @@ func (storage *PublishedStorage) LinkFromPool(publishedPrefix, publishedRelPath,
 	if err != nil {
 		return err
 	}
-	defer func() { _ = source.Close() }()
+	defer source.Close()
 
-	err = storage.az.putFile(storage.az.blobURL(relFilePath), source, sourceMD5)
+	err = storage.az.putFile(relFilePath, source, sourceMD5)
 	if err == nil {
 		pathCache[relFilePath] = sourceMD5
 	} else {
@@ -174,57 +177,58 @@ func (storage *PublishedStorage) Filelist(prefix string) ([]string, error) {
 }
 
 // Internal copy or move implementation
-func (storage *PublishedStorage) internalCopyOrMoveBlob(src, dst string, metadata azblob.Metadata, move bool) error {
+func (storage *PublishedStorage) internalCopyOrMoveBlob(src, dst string, metadata map[string]*string, move bool) error {
 	const leaseDuration = 30
+	leaseID := uuid.NewString()
 
-	dstBlobURL := storage.az.blobURL(dst)
+	serviceClient := storage.az.client.ServiceClient()
-	srcBlobURL := storage.az.blobURL(src)
+	containerClient := serviceClient.NewContainerClient(storage.az.container)
-	leaseResp, err := srcBlobURL.AcquireLease(context.Background(), "", leaseDuration, azblob.ModifiedAccessConditions{})
+	srcBlobClient := containerClient.NewBlobClient(src)
-	if err != nil || leaseResp.StatusCode() != http.StatusCreated {
+	blobLeaseClient, err := lease.NewBlobClient(srcBlobClient, &lease.BlobClientOptions{LeaseID: to.Ptr(leaseID)})
-		return fmt.Errorf("error acquiring lease on source blob %s", srcBlobURL)
+	if err != nil {
+		return fmt.Errorf("error acquiring lease on source blob %s", src)
 	}
-	defer func() { _, _ = srcBlobURL.BreakLease(context.Background(), azblob.LeaseBreakNaturally, azblob.ModifiedAccessConditions{}) }()
-	srcBlobLeaseID := leaseResp.LeaseID()
 
-	copyResp, err := dstBlobURL.StartCopyFromURL(
+	_, err = blobLeaseClient.AcquireLease(context.Background(), leaseDuration, nil)
-		context.Background(),
+	if err != nil {
-		srcBlobURL.URL(),
+		return fmt.Errorf("error acquiring lease on source blob %s", src)
-		metadata,
+	}
-		azblob.ModifiedAccessConditions{},
+	defer blobLeaseClient.BreakLease(context.Background(), &lease.BlobBreakOptions{BreakPeriod: to.Ptr(int32(60))})
-		azblob.BlobAccessConditions{},
+
-		azblob.DefaultAccessTier,
+	dstBlobClient := containerClient.NewBlobClient(dst)
-		nil)
+	copyResp, err := dstBlobClient.StartCopyFromURL(context.Background(), srcBlobClient.URL(), &blob.StartCopyFromURLOptions{
+		Metadata: metadata,
+	})
+
 	if err != nil {
 		return fmt.Errorf("error copying %s -> %s in %s: %s", src, dst, storage, err)
 	}
 
-	copyStatus := copyResp.CopyStatus()
+	copyStatus := *copyResp.CopyStatus
 	for {
-		if copyStatus == azblob.CopyStatusSuccess {
+		if copyStatus == blob.CopyStatusTypeSuccess {
 			if move {
-				_, err = srcBlobURL.Delete(
+				_, err := storage.az.client.DeleteBlob(context.Background(), storage.az.container, src, &blob.DeleteOptions{
-					context.Background(),
+					AccessConditions: &blob.AccessConditions{
-					azblob.DeleteSnapshotsOptionNone,
+						LeaseAccessConditions: &blob.LeaseAccessConditions{
-					azblob.BlobAccessConditions{
+							LeaseID: &leaseID,
-						LeaseAccessConditions: azblob.LeaseAccessConditions{LeaseID: srcBlobLeaseID},
+						},
-					})
+					},
+				})
 				return err
 			}
 			return nil
-		} else if copyStatus == azblob.CopyStatusPending {
+		} else if copyStatus == blob.CopyStatusTypePending {
 			time.Sleep(1 * time.Second)
-			blobPropsResp, err := dstBlobURL.GetProperties(
+			getMetadata, err := dstBlobClient.GetProperties(context.TODO(), nil)
-				context.Background(),
-				azblob.BlobAccessConditions{LeaseAccessConditions: azblob.LeaseAccessConditions{LeaseID: srcBlobLeaseID}},
-				azblob.ClientProvidedKeyOptions{})
 			if err != nil {
-				return fmt.Errorf("error getting destination blob properties %s", dstBlobURL)
+				return fmt.Errorf("error getting copy progress %s", dst)
 			}
-			copyStatus = blobPropsResp.CopyStatus()
+			copyStatus = *getMetadata.CopyStatus
 
-			_, err = srcBlobURL.RenewLease(context.Background(), srcBlobLeaseID, azblob.ModifiedAccessConditions{})
+			_, err = blobLeaseClient.RenewLease(context.Background(), nil)
 			if err != nil {
-				return fmt.Errorf("error renewing source blob lease %s", srcBlobURL)
+				return fmt.Errorf("error renewing source blob lease %s", src)
 			}
 		} else {
 			return fmt.Errorf("error copying %s -> %s in %s: %s", dst, src, storage, copyStatus)
@@ -239,7 +243,9 @@ func (storage *PublishedStorage) RenameFile(oldName, newName string) error {
 
 // SymLink creates a copy of src file and adds link information as meta data
 func (storage *PublishedStorage) SymLink(src string, dst string) error {
-	return storage.internalCopyOrMoveBlob(src, dst, azblob.Metadata{"SymLink": src}, false /* move */)
+	metadata := make(map[string]*string)
+	metadata["SymLink"] = &src
+	return storage.internalCopyOrMoveBlob(src, dst, metadata, false /* do not remove src */)
 }
 
 // HardLink using symlink functionality as hard links do not exist
@@ -249,28 +255,33 @@ func (storage *PublishedStorage) HardLink(src string, dst string) error {
 
 // FileExists returns true if path exists
 func (storage *PublishedStorage) FileExists(path string) (bool, error) {
-	blob := storage.az.blobURL(path)
+	serviceClient := storage.az.client.ServiceClient()
-	resp, err := blob.GetProperties(context.Background(), azblob.BlobAccessConditions{}, azblob.ClientProvidedKeyOptions{})
+	containerClient := serviceClient.NewContainerClient(storage.az.container)
+	blobClient := containerClient.NewBlobClient(path)
+	_, err := blobClient.GetProperties(context.Background(), nil)
 	if err != nil {
 		if isBlobNotFound(err) {
 			return false, nil
 		}
-		return false, err
+		return false, fmt.Errorf("error checking if blob %s exists: %v", path, err)
-	} else if resp.StatusCode() == http.StatusOK {
-		return true, nil
 	}
-	return false, fmt.Errorf("error checking if blob %s exists %d", blob, resp.StatusCode())
+	return true, nil
 }
 
 // ReadLink returns the symbolic link pointed to by path.
 // This simply reads text file created with SymLink
 func (storage *PublishedStorage) ReadLink(path string) (string, error) {
-	blob := storage.az.blobURL(path)
+	serviceClient := storage.az.client.ServiceClient()
-	resp, err := blob.GetProperties(context.Background(), azblob.BlobAccessConditions{}, azblob.ClientProvidedKeyOptions{})
+	containerClient := serviceClient.NewContainerClient(storage.az.container)
+	blobClient := containerClient.NewBlobClient(path)
+	props, err := blobClient.GetProperties(context.Background(), nil)
 	if err != nil {
-		return "", err
+		return "", fmt.Errorf("failed to get blob properties: %v", err)
-	} else if resp.StatusCode() != http.StatusOK {
-		return "", fmt.Errorf("error checking if blob %s exists %d", blob, resp.StatusCode())
 	}
-	return resp.NewMetadata()["SymLink"], nil
+
+	metadata := props.Metadata
+	if originalBlob, exists := metadata["original_blob"]; exists {
+		return *originalBlob, nil
+	}
+	return "", fmt.Errorf("error reading link %s: %v", path, err)
 }

azure/public_test.go

@@ -4,11 +4,14 @@ import (
 	"context"
 	"crypto/md5"
 	"crypto/rand"
-	"io"
+	"io/ioutil"
 	"os"
 	"path/filepath"
+        "bytes"
 
-	"github.com/Azure/azure-storage-blob-go/azblob"
+        "github.com/Azure/azure-sdk-for-go/sdk/azcore"
+        "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
+        "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
 	"github.com/aptly-dev/aptly/files"
 	"github.com/aptly-dev/aptly/utils"
 	. "gopkg.in/check.v1"
@@ -33,7 +36,7 @@ func randString(n int) string {
 	}
 	const alphanum = "0123456789abcdefghijklmnopqrstuvwxyz"
 	var bytes = make([]byte, n)
-	_, _ = rand.Read(bytes)
+	rand.Read(bytes)
 	for i, b := range bytes {
 		bytes[i] = alphanum[b%byte(len(alphanum))]
 	}
@@ -66,8 +69,10 @@ func (s *PublishedStorageSuite) SetUpTest(c *C) {
 
 	s.storage, err = NewPublishedStorage(s.accountName, s.accountKey, container, "", s.endpoint)
 	c.Assert(err, IsNil)
-	cnt := s.storage.az.container
+        publicAccessType := azblob.PublicAccessTypeContainer
-	_, err = cnt.Create(context.Background(), azblob.Metadata{}, azblob.PublicAccessContainer)
+        _, err = s.storage.az.client.CreateContainer(context.Background(), s.storage.az.container, &azblob.CreateContainerOptions{
+            Access: &publicAccessType,
+        })
 	c.Assert(err, IsNil)
 
 	s.prefixedStorage, err = NewPublishedStorage(s.accountName, s.accountKey, container, prefix, s.endpoint)
@@ -75,41 +80,39 @@ func (s *PublishedStorageSuite) SetUpTest(c *C) {
 }
 
 func (s *PublishedStorageSuite) TearDownTest(c *C) {
-	cnt := s.storage.az.container
+        _, err := s.storage.az.client.DeleteContainer(context.Background(), s.storage.az.container, nil)
-	_, err := cnt.Delete(context.Background(), azblob.ContainerAccessConditions{})
 	c.Assert(err, IsNil)
 }
 
 func (s *PublishedStorageSuite) GetFile(c *C, path string) []byte {
-	blob := s.storage.az.container.NewBlobURL(path)
+        resp, err := s.storage.az.client.DownloadStream(context.Background(), s.storage.az.container, path, nil)
-	resp, err := blob.Download(context.Background(), 0, azblob.CountToEnd, azblob.BlobAccessConditions{}, false, azblob.ClientProvidedKeyOptions{})
 	c.Assert(err, IsNil)
-	body := resp.Body(azblob.RetryReaderOptions{MaxRetryRequests: 3})
+	data, err := ioutil.ReadAll(resp.Body)
-	data, err := io.ReadAll(body)
 	c.Assert(err, IsNil)
 	return data
 }
 
 func (s *PublishedStorageSuite) AssertNoFile(c *C, path string) {
-	_, err := s.storage.az.container.NewBlobURL(path).GetProperties(
+        serviceClient := s.storage.az.client.ServiceClient()
-		context.Background(), azblob.BlobAccessConditions{}, azblob.ClientProvidedKeyOptions{})
+        containerClient := serviceClient.NewContainerClient(s.storage.az.container)
+        blobClient := containerClient.NewBlobClient(path)
+        _, err := blobClient.GetProperties(context.Background(), nil)
 	c.Assert(err, NotNil)
-	storageError, ok := err.(azblob.StorageError)
+
+        storageError, ok := err.(*azcore.ResponseError)
 	c.Assert(ok, Equals, true)
-	c.Assert(string(storageError.ServiceCode()), Equals, string(string(azblob.StorageErrorCodeBlobNotFound)))
+	c.Assert(storageError.StatusCode, Equals, 404)
 }
 
 func (s *PublishedStorageSuite) PutFile(c *C, path string, data []byte) {
 	hash := md5.Sum(data)
-	_, err := azblob.UploadBufferToBlockBlob(
+        uploadOptions := &azblob.UploadStreamOptions{
-		context.Background(),
+            HTTPHeaders: &blob.HTTPHeaders{
-		data,
+            	BlobContentMD5: hash[:],
-		s.storage.az.container.NewBlockBlobURL(path),
+            },
-		azblob.UploadToBlockBlobOptions{
+        }
-			BlobHTTPHeaders: azblob.BlobHTTPHeaders{
+        reader := bytes.NewReader(data)
-				ContentMD5: hash[:],
+        _, err := s.storage.az.client.UploadStream(context.Background(), s.storage.az.container, path, reader, uploadOptions)
-			},
-		})
 	c.Assert(err, IsNil)
 }
 
@@ -118,7 +121,7 @@ func (s *PublishedStorageSuite) TestPutFile(c *C) {
 	filename := "a/b.txt"
 
 	dir := c.MkDir()
-	err := os.WriteFile(filepath.Join(dir, "a"), content, 0644)
+	err := ioutil.WriteFile(filepath.Join(dir, "a"), content, 0644)
 	c.Assert(err, IsNil)
 
 	err = s.storage.PutFile(filename, filepath.Join(dir, "a"))
@@ -137,7 +140,7 @@ func (s *PublishedStorageSuite) TestPutFilePlus(c *C) {
 	filename := "a/b+c.txt"
 
 	dir := c.MkDir()
-	err := os.WriteFile(filepath.Join(dir, "a"), content, 0644)
+	err := ioutil.WriteFile(filepath.Join(dir, "a"), content, 0644)
 	c.Assert(err, IsNil)
 
 	err = s.storage.PutFile(filename, filepath.Join(dir, "a"))
@@ -255,7 +258,7 @@ func (s *PublishedStorageSuite) TestRemoveDirsPlus(c *C) {
 
 func (s *PublishedStorageSuite) TestRenameFile(c *C) {
 	dir := c.MkDir()
-	err := os.WriteFile(filepath.Join(dir, "a"), []byte("Welcome to Azure!"), 0644)
+	err := ioutil.WriteFile(filepath.Join(dir, "a"), []byte("Welcome to Azure!"), 0644)
 	c.Assert(err, IsNil)
 
 	err = s.storage.PutFile("source.txt", filepath.Join(dir, "a"))
@@ -277,18 +280,18 @@ func (s *PublishedStorageSuite) TestLinkFromPool(c *C) {
 	cs := files.NewMockChecksumStorage()
 
 	tmpFile1 := filepath.Join(c.MkDir(), "mars-invaders_1.03.deb")
-	err := os.WriteFile(tmpFile1, []byte("Contents"), 0644)
+	err := ioutil.WriteFile(tmpFile1, []byte("Contents"), 0644)
 	c.Assert(err, IsNil)
 	cksum1 := utils.ChecksumInfo{MD5: "c1df1da7a1ce305a3b60af9d5733ac1d"}
 
 	tmpFile2 := filepath.Join(c.MkDir(), "mars-invaders_1.03.deb")
-	err = os.WriteFile(tmpFile2, []byte("Spam"), 0644)
+	err = ioutil.WriteFile(tmpFile2, []byte("Spam"), 0644)
 	c.Assert(err, IsNil)
 	cksum2 := utils.ChecksumInfo{MD5: "e9dfd31cc505d51fc26975250750deab"}
 
 	tmpFile3 := filepath.Join(c.MkDir(), "netboot/boot.img.gz")
-	_ = os.MkdirAll(filepath.Dir(tmpFile3), 0777)
+	os.MkdirAll(filepath.Dir(tmpFile3), 0777)
-	err = os.WriteFile(tmpFile3, []byte("Contents"), 0644)
+	err = ioutil.WriteFile(tmpFile3, []byte("Contents"), 0644)
 	c.Assert(err, IsNil)
 	cksum3 := utils.ChecksumInfo{MD5: "c1df1da7a1ce305a3b60af9d5733ac1d"}
 
@@ -330,7 +333,7 @@ func (s *PublishedStorageSuite) TestLinkFromPool(c *C) {
 
 	// 2nd link from pool, providing wrong path for source file
 	//
-	// this test should check that file already exists in S3 and skip upload (which would fail if not skipped)
+	// this test should check that file already exists in Azure and skip upload (which would fail if not skipped)
 	s.prefixedStorage.pathCache = nil
 	err = s.prefixedStorage.LinkFromPool("", filepath.Join("pool", "main", "m/mars-invaders"), "mars-invaders_1.03.deb", pool, "wrong-looks-like-pathcache-doesnt-work", cksum1, false)
 	c.Check(err, IsNil)

cmd/api_serve.go

@@ -46,7 +46,7 @@ func aptlyAPIServe(cmd *commander.Command, args []string) error {
 	}
 	if err == nil && len(listeners) == 1 {
 		listener := listeners[0]
-		defer func() { _ = listener.Close() }()
+		defer listener.Close()
 		fmt.Printf("\nTaking over web server at: %s (press Ctrl+C to quit)...\n", listener.Addr().String())
 		err = http.Serve(listener, api.Router(context))
 		if err != nil {
@@ -67,7 +67,7 @@ func aptlyAPIServe(cmd *commander.Command, args []string) error {
 		if _, ok := <-sigchan; ok {
 			fmt.Printf("\nShutdown signal received, waiting for background tasks...\n")
 			context.TaskList().Wait()
-			_ = server.Shutdown(stdcontext.Background())
+			server.Shutdown(stdcontext.Background())
 		}
 	})()
 	defer close(sigchan)
@@ -75,14 +75,14 @@ func aptlyAPIServe(cmd *commander.Command, args []string) error {
 	listenURL, err := url.Parse(listen)
 	if err == nil && listenURL.Scheme == "unix" {
 		file := listenURL.Path
-		_ = os.Remove(file)
+		os.Remove(file)
 
 		var listener net.Listener
 		listener, err = net.Listen("unix", file)
 		if err != nil {
 			return fmt.Errorf("failed to listen on: %s\n%s", file, err)
 		}
-		defer func() { _ = listener.Close() }()
+		defer listener.Close()
 
 		err = server.Serve(listener)
 	} else {

cmd/cmd.go

@@ -21,7 +21,7 @@ const (
 )
 
 // ListPackagesRefList shows list of packages in PackageRefList
-func ListPackagesRefList(reflist *deb.PackageRefList, collectionFactory *deb.CollectionFactory) (err error) {
+func ListPackagesRefList(reflist deb.AnyRefList, collectionFactory *deb.CollectionFactory) (err error) {
 	fmt.Printf("Packages:\n")
 
 	if reflist == nil {
@@ -97,7 +97,7 @@ package environment to new version.`,
 		Flag: *flag.NewFlagSet("aptly", flag.ExitOnError),
 		Subcommands: []*commander.Command{
 			makeCmdConfig(),
-			makeCmdDB(),
+			makeCmdDb(),
 			makeCmdGraph(),
 			makeCmdMirror(),
 			makeCmdRepo(),

cmd/config_show.go

@@ -5,7 +5,7 @@ import (
 	"fmt"
 
 	"github.com/smira/commander"
-	yaml "gopkg.in/yaml.v3"
+	"gopkg.in/yaml.v3"
 )
 
 func aptlyConfigShow(_ *commander.Command, _ []string) error {

cmd/db.go

@@ -4,13 +4,13 @@ import (
 	"github.com/smira/commander"
 )
 
-func makeCmdDB() *commander.Command {
+func makeCmdDb() *commander.Command {
 	return &commander.Command{
 		UsageLine: "db",
 		Short:     "manage aptly's internal database and package pool",
 		Subcommands: []*commander.Command{
-			makeCmdDBCleanup(),
+			makeCmdDbCleanup(),
-			makeCmdDBRecover(),
+			makeCmdDbRecover(),
 		},
 	}
 }

cmd/db_cleanup.go

@@ -6,13 +6,14 @@ import (
 	"strings"
 
 	"github.com/aptly-dev/aptly/aptly"
+	"github.com/aptly-dev/aptly/database"
 	"github.com/aptly-dev/aptly/deb"
 	"github.com/aptly-dev/aptly/utils"
 	"github.com/smira/commander"
 )
 
 // aptly db cleanup
-func aptlyDBCleanup(cmd *commander.Command, args []string) error {
+func aptlyDbCleanup(cmd *commander.Command, args []string) error {
 	var err error
 
 	if len(args) != 0 {
@@ -24,12 +25,20 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 	dryRun := context.Flags().Lookup("dry-run").Value.Get().(bool)
 	collectionFactory := context.NewCollectionFactory()
 
-	// collect information about references packages...
+	// collect information about references packages and their reflistbuckets...
-	existingPackageRefs := deb.NewPackageRefList()
+	existingPackageRefs := deb.NewSplitRefList()
+	existingBuckets := deb.NewRefListDigestSet()
 
 	// used only in verbose mode to report package use source
 	packageRefSources := map[string][]string{}
 
+	var reflistMigration *deb.RefListMigration
+	if !dryRun {
+		reflistMigration = collectionFactory.RefListCollection().NewMigration()
+	} else {
+		reflistMigration = collectionFactory.RefListCollection().NewMigrationDryRun()
+	}
+
 	context.Progress().ColoredPrintf("@{w!}Loading mirrors, local repos, snapshots and published repos...@|")
 	if verbose {
 		context.Progress().ColoredPrintf("@{y}Loading mirrors:@|")
@@ -39,20 +48,21 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 			context.Progress().ColoredPrintf("- @{g}%s@|", repo.Name)
 		}
 
-		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		sl := deb.NewSplitRefList()
-		if e != nil {
+		e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
+		if e != nil && e != database.ErrNotFound {
 			return e
 		}
-		if repo.RefList() != nil {
-			existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
 
-			if verbose {
+		existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
-				description := fmt.Sprintf("mirror %s", repo.Name)
+		existingBuckets.AddAllInRefList(sl)
-				_ = repo.RefList().ForEach(func(key []byte) error {
+
-					packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
+		if verbose {
-					return nil
+			description := fmt.Sprintf("mirror %s", repo.Name)
-				})
+			sl.ForEach(func(key []byte) error {
-			}
+				packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
+				return nil
+			})
 		}
 
 		return nil
@@ -71,21 +81,23 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 			context.Progress().ColoredPrintf("- @{g}%s@|", repo.Name)
 		}
 
-		e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		sl := deb.NewSplitRefList()
-		if e != nil {
+		e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, repo.RefKey(), reflistMigration)
+		if e != nil && e != database.ErrNotFound {
 			return e
 		}
 
-		if repo.RefList() != nil {
+		existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
-			existingPackageRefs = existingPackageRefs.Merge(repo.RefList(), false, true)
+		existingBuckets.AddAllInRefList(sl)
 
-			if verbose {
+		existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
-				description := fmt.Sprintf("local repo %s", repo.Name)
+
-				_ = repo.RefList().ForEach(func(key []byte) error {
+		if verbose {
-					packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
+			description := fmt.Sprintf("local repo %s", repo.Name)
-					return nil
+			sl.ForEach(func(key []byte) error {
-				})
+				packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
-			}
+				return nil
+			})
 		}
 
 		return nil
@@ -104,16 +116,18 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 			context.Progress().ColoredPrintf("- @{g}%s@|", snapshot.Name)
 		}
 
-		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		sl := deb.NewSplitRefList()
+		e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, snapshot.RefKey(), reflistMigration)
 		if e != nil {
 			return e
 		}
 
-		existingPackageRefs = existingPackageRefs.Merge(snapshot.RefList(), false, true)
+		existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+		existingBuckets.AddAllInRefList(sl)
 
 		if verbose {
 			description := fmt.Sprintf("snapshot %s", snapshot.Name)
-			_ = snapshot.RefList().ForEach(func(key []byte) error {
+			sl.ForEach(func(key []byte) error {
 				packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
 				return nil
 			})
@@ -136,17 +150,21 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 		if published.SourceKind != deb.SourceLocalRepo {
 			return nil
 		}
-		e := collectionFactory.PublishedRepoCollection().LoadComplete(published, collectionFactory)
-		if e != nil {
-			return e
-		}
 
 		for _, component := range published.Components() {
-			existingPackageRefs = existingPackageRefs.Merge(published.RefList(component), false, true)
+			sl := deb.NewSplitRefList()
+			e := collectionFactory.RefListCollection().LoadCompleteAndMigrate(sl, published.RefKey(component), reflistMigration)
+			if e != nil {
+				return e
+			}
+
+			existingPackageRefs = existingPackageRefs.Merge(sl, false, true)
+			existingBuckets.AddAllInRefList(sl)
+
 			if verbose {
 				description := fmt.Sprintf("published repository %s:%s/%s component %s",
 					published.Storage, published.Prefix, published.Distribution, component)
-				_ = published.RefList(component).ForEach(func(key []byte) error {
+				sl.ForEach(func(key []byte) error {
 					packageRefSources[string(key)] = append(packageRefSources[string(key)], description)
 					return nil
 				})
@@ -160,11 +178,29 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 
 	collectionFactory.Flush()
 
+	err = reflistMigration.Flush()
+	if err != nil {
+		return err
+	}
+
+	if verbose {
+		if stats := reflistMigration.Stats(); stats.Reflists > 0 {
+			if !dryRun {
+				context.Progress().ColoredPrintf("@{w!}Split %d reflist(s) into %d bucket(s) (%d segment(s))@|",
+					stats.Reflists, stats.Buckets, stats.Segments)
+			} else {
+				context.Progress().ColoredPrintf(
+					"@{y!}Skipped splitting %d reflist(s) into %d bucket(s) (%d segment(s)), as -dry-run has been requested.@|",
+					stats.Reflists, stats.Buckets, stats.Segments)
+			}
+		}
+	}
+
 	// ... and compare it to the list of all packages
 	context.Progress().ColoredPrintf("@{w!}Loading list of all packages...@|")
 	allPackageRefs := collectionFactory.PackageCollection().AllPackageRefs()
 
-	toDelete := allPackageRefs.Subtract(existingPackageRefs)
+	toDelete := allPackageRefs.Subtract(existingPackageRefs.Flatten())
 
 	// delete packages that are no longer referenced
 	context.Progress().ColoredPrintf("@{r!}Deleting unreferenced packages (%d)...@|", toDelete.Len())
@@ -202,6 +238,32 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 		}
 	}
 
+	bucketsToDelete, err := collectionFactory.RefListCollection().AllBucketDigests()
+	if err != nil {
+		return err
+	}
+
+	bucketsToDelete.RemoveAll(existingBuckets)
+
+	context.Progress().ColoredPrintf("@{r!}Deleting unreferenced reflist buckets (%d)...@|", bucketsToDelete.Len())
+	if bucketsToDelete.Len() > 0 {
+		if !dryRun {
+			batch := db.CreateBatch()
+			err := bucketsToDelete.ForEach(func(digest []byte) error {
+				return collectionFactory.RefListCollection().UnsafeDropBucket(digest, batch)
+			})
+			if err != nil {
+				return err
+			}
+
+			if err := batch.Write(); err != nil {
+				return err
+			}
+		} else {
+			context.Progress().ColoredPrintf("@{y!}Skipped reflist deletion, as -dry-run has been requested.@|")
+		}
+	}
+
 	collectionFactory.Flush()
 
 	// now, build a list of files that should be present in Repository (package pool)
@@ -291,9 +353,9 @@ func aptlyDBCleanup(cmd *commander.Command, args []string) error {
 	return err
 }
 
-func makeCmdDBCleanup() *commander.Command {
+func makeCmdDbCleanup() *commander.Command {
 	cmd := &commander.Command{
-		Run:       aptlyDBCleanup,
+		Run:       aptlyDbCleanup,
 		UsageLine: "cleanup",
 		Short:     "cleanup DB and package pool",
 		Long: `

cmd/db_recover.go

@@ -1,16 +1,13 @@
 package cmd
 
 import (
-	"fmt"
-
-	"github.com/aptly-dev/aptly/deb"
 	"github.com/smira/commander"
 
 	"github.com/aptly-dev/aptly/database/goleveldb"
 )
 
 // aptly db recover
-func aptlyDBRecover(cmd *commander.Command, args []string) error {
+func aptlyDbRecover(cmd *commander.Command, args []string) error {
 	var err error
 
 	if len(args) != 0 {
@@ -19,19 +16,14 @@ func aptlyDBRecover(cmd *commander.Command, args []string) error {
 	}
 
 	context.Progress().Printf("Recovering database...\n")
-	if err = goleveldb.RecoverDB(context.DBPath()); err != nil {
+	err = goleveldb.RecoverDB(context.DBPath())
-		return err
-	}
-
-	context.Progress().Printf("Checking database integrity...\n")
-	err = checkIntegrity()
 
 	return err
 }
 
-func makeCmdDBRecover() *commander.Command {
+func makeCmdDbRecover() *commander.Command {
 	cmd := &commander.Command{
-		Run:       aptlyDBRecover,
+		Run:       aptlyDbRecover,
 		UsageLine: "recover",
 		Short:     "recover DB after crash",
 		Long: `
@@ -46,36 +38,3 @@ Example:
 
 	return cmd
 }
-
-func checkIntegrity() error {
-	return context.NewCollectionFactory().LocalRepoCollection().ForEach(checkRepo)
-}
-
-func checkRepo(repo *deb.LocalRepo) error {
-	collectionFactory := context.NewCollectionFactory()
-	repos := collectionFactory.LocalRepoCollection()
-
-	err := repos.LoadComplete(repo)
-	if err != nil {
-		return fmt.Errorf("load complete repo %q: %s", repo.Name, err)
-	}
-
-	dangling, err := deb.FindDanglingReferences(repo.RefList(), collectionFactory.PackageCollection())
-	if err != nil {
-		return fmt.Errorf("find dangling references: %w", err)
-	}
-
-	if len(dangling.Refs) > 0 {
-		for _, ref := range dangling.Refs {
-			context.Progress().Printf("Removing dangling database reference %q\n", ref)
-		}
-
-		repo.UpdateRefList(repo.RefList().Subtract(dangling))
-
-		if err = repos.Update(repo); err != nil {
-			return fmt.Errorf("update repo: %w", err)
-		}
-	}
-
-	return nil
-}

cmd/graph.go

@@ -38,8 +38,8 @@ func aptlyGraph(cmd *commander.Command, args []string) error {
 	if err != nil {
 		return err
 	}
-	_ = tempfile.Close()
+	tempfile.Close()
-	_ = os.Remove(tempfile.Name())
+	os.Remove(tempfile.Name())
 
 	format := context.Flags().Lookup("format").Value.String()
 	output := context.Flags().Lookup("output").Value.String()

cmd/mirror.go

@@ -20,7 +20,7 @@ func getVerifier(flags *flag.FlagSet) (pgp.Verifier, error) {
 		verifier.AddKeyring(keyRing)
 	}
 
-	err := verifier.InitKeyring(!ignoreSignatures) // be verbose only if verifying signatures is requested
+	err := verifier.InitKeyring(ignoreSignatures == false) // be verbose only if verifying signatures is requested
 	if err != nil {
 		return nil, err
 	}

cmd/mirror_create.go

@@ -69,7 +69,7 @@ func aptlyMirrorCreate(cmd *commander.Command, args []string) error {
 	}
 
 	collectionFactory := context.NewCollectionFactory()
-	err = collectionFactory.RemoteRepoCollection().Add(repo)
+	err = collectionFactory.RemoteRepoCollection().Add(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to add mirror: %s", err)
 	}

cmd/mirror_edit.go

@@ -78,7 +78,7 @@ func aptlyMirrorEdit(cmd *commander.Command, args []string) error {
 		}
 	}
 
-	err = collectionFactory.RemoteRepoCollection().Update(repo)
+	err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to edit: %s", err)
 	}

cmd/mirror_list.go

@@ -32,7 +32,7 @@ func aptlyMirrorListTxt(cmd *commander.Command, _ []string) error {
 
 	repos := make([]string, collectionFactory.RemoteRepoCollection().Len())
 	i := 0
-	_ = collectionFactory.RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
+	collectionFactory.RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
 		if raw {
 			repos[i] = repo.Name
 		} else {
@@ -42,7 +42,7 @@ func aptlyMirrorListTxt(cmd *commander.Command, _ []string) error {
 		return nil
 	})
 
-	_ = context.CloseDatabase()
+	context.CloseDatabase()
 
 	sort.Strings(repos)
 
@@ -70,13 +70,13 @@ func aptlyMirrorListJSON(_ *commander.Command, _ []string) error {
 
 	repos := make([]*deb.RemoteRepo, context.NewCollectionFactory().RemoteRepoCollection().Len())
 	i := 0
-	_ = context.NewCollectionFactory().RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
+	context.NewCollectionFactory().RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
 		repos[i] = repo
 		i++
 		return nil
 	})
 
-	_ = context.CloseDatabase()
+	context.CloseDatabase()
 
 	sort.Slice(repos, func(i, j int) bool {
 		return repos[i].Name < repos[j].Name

cmd/mirror_rename.go

@@ -37,7 +37,7 @@ func aptlyMirrorRename(cmd *commander.Command, args []string) error {
 	}
 
 	repo.Name = newName
-	err = collectionFactory.RemoteRepoCollection().Update(repo)
+	err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to rename: %s", err)
 	}

cmd/mirror_show.go

@@ -38,7 +38,7 @@ func aptlyMirrorShowTxt(_ *commander.Command, args []string) error {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -86,7 +86,7 @@ func aptlyMirrorShowTxt(_ *commander.Command, args []string) error {
 		if repo.LastDownloadDate.IsZero() {
 			fmt.Printf("Unable to show package list, mirror hasn't been downloaded yet.\n")
 		} else {
-			_ = ListPackagesRefList(repo.RefList(), collectionFactory)
+			ListPackagesRefList(repo.RefList(), collectionFactory)
 		}
 	}
 
@@ -98,12 +98,13 @@ func aptlyMirrorShowJSON(_ *commander.Command, args []string) error {
 
 	name := args[0]
 
-	repo, err := context.NewCollectionFactory().RemoteRepoCollection().ByName(name)
+	collectionFactory := context.NewCollectionFactory()
+	repo, err := collectionFactory.RemoteRepoCollection().ByName(name)
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = context.NewCollectionFactory().RemoteRepoCollection().LoadComplete(repo)
+	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -113,13 +114,13 @@ func aptlyMirrorShowJSON(_ *commander.Command, args []string) error {
 	if withPackages {
 		if repo.RefList() != nil {
 			var list *deb.PackageList
-			list, err = deb.NewPackageListFromRefList(repo.RefList(), context.NewCollectionFactory().PackageCollection(), context.Progress())
+			list, err = deb.NewPackageListFromRefList(repo.RefList(), collectionFactory.PackageCollection(), context.Progress())
 			if err != nil {
 				return fmt.Errorf("unable to get package list: %s", err)
 			}
 
 			list.PrepareIndex()
-			_ = list.ForEachIndexed(func(p *deb.Package) error {
+			list.ForEachIndexed(func(p *deb.Package) error {
 				repo.Packages = append(repo.Packages, p.GetFullName())
 				return nil
 			})

cmd/mirror_update.go

@@ -29,7 +29,7 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to update: %s", err)
 	}
 
-	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+	err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to update: %s", err)
 	}
@@ -101,12 +101,12 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
 		err = context.ReOpenDatabase()
 		if err == nil {
 			repo.MarkAsIdle()
-			_ = collectionFactory.RemoteRepoCollection().Update(repo)
+			collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 		}
 	}()
 
 	repo.MarkAsUpdating()
-	err = collectionFactory.RemoteRepoCollection().Update(repo)
+	err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to update: %s", err)
 	}
@@ -173,7 +173,7 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
 						file, e = os.CreateTemp("", task.File.Filename)
 						if e == nil {
 							task.TempDownPath = file.Name()
-							_ = file.Close()
+							file.Close()
 						}
 					}
 					if e != nil {
@@ -261,8 +261,8 @@ func aptlyMirrorUpdate(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to update: download errors:\n  %s", strings.Join(errors, "\n  "))
 	}
 
-	_ = repo.FinalizeDownload(collectionFactory, context.Progress())
+	repo.FinalizeDownload(collectionFactory, context.Progress())
-	err = collectionFactory.RemoteRepoCollection().Update(repo)
+	err = collectionFactory.RemoteRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to update: %s", err)
 	}

cmd/package_search.go

@@ -40,7 +40,7 @@ func aptlyPackageSearch(cmd *commander.Command, args []string) error {
 	}
 
 	format := context.Flags().Lookup("format").Value.String()
-	_ = PrintPackageList(result, format, "")
+	PrintPackageList(result, format, "")
 
 	return err
 }

cmd/package_show.go

@@ -14,7 +14,7 @@ import (
 
 func printReferencesTo(p *deb.Package, collectionFactory *deb.CollectionFactory) (err error) {
 	err = collectionFactory.RemoteRepoCollection().ForEach(func(repo *deb.RemoteRepo) error {
-		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -30,7 +30,7 @@ func printReferencesTo(p *deb.Package, collectionFactory *deb.CollectionFactory)
 	}
 
 	err = collectionFactory.LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
-		e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -46,7 +46,7 @@ func printReferencesTo(p *deb.Package, collectionFactory *deb.CollectionFactory)
 	}
 
 	err = collectionFactory.SnapshotCollection().ForEach(func(snapshot *deb.Snapshot) error {
-		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -84,8 +84,8 @@ func aptlyPackageShow(cmd *commander.Command, args []string) error {
 	result := q.Query(collectionFactory.PackageCollection())
 
 	err = result.ForEach(func(p *deb.Package) error {
-		_ = p.Stanza().WriteTo(w, p.IsSource, false, false)
+		p.Stanza().WriteTo(w, p.IsSource, false, false)
-		_ = w.Flush()
+		w.Flush()
 		fmt.Printf("\n")
 
 		if withFiles {
@@ -109,7 +109,7 @@ func aptlyPackageShow(cmd *commander.Command, args []string) error {
 
 		if withReferences {
 			fmt.Printf("References to package:\n")
-			_ = printReferencesTo(p, collectionFactory)
+			printReferencesTo(p, collectionFactory)
 			fmt.Printf("\n")
 		}
 

cmd/publish.go

@@ -1,8 +1,6 @@
 package cmd
 
 import (
-	"strings"
-
 	"github.com/aptly-dev/aptly/pgp"
 	"github.com/smira/commander"
 	"github.com/smira/flag"
@@ -14,20 +12,7 @@ func getSigner(flags *flag.FlagSet) (pgp.Signer, error) {
 	}
 
 	signer := context.GetSigner()
-
+	signer.SetKey(flags.Lookup("gpg-key").Value.String())
-	var gpgKeys []string
-
-	// CLI args have priority over config
-	cliKeys := flags.Lookup("gpg-key").Value.Get().([]string)
-	if len(cliKeys) > 0 {
-		gpgKeys = cliKeys
-	} else if len(context.Config().GpgKeys) > 0 {
-		gpgKeys = context.Config().GpgKeys
-	}
-
-	for _, gpgKey := range gpgKeys {
-		signer.SetKey(gpgKey)
-	}
 	signer.SetKeyRing(flags.Lookup("keyring").Value.String(), flags.Lookup("secret-keyring").Value.String())
 	signer.SetPassphrase(flags.Lookup("passphrase").Value.String(), flags.Lookup("passphrase-file").Value.String())
 	signer.SetBatch(flags.Lookup("batch").Value.Get().(bool))
@@ -41,23 +26,6 @@ func getSigner(flags *flag.FlagSet) (pgp.Signer, error) {
 
 }
 
-type gpgKeyFlag struct {
-	gpgKeys []string
-}
-
-func (k *gpgKeyFlag) Set(value string) error {
-	k.gpgKeys = append(k.gpgKeys, value)
-	return nil
-}
-
-func (k *gpgKeyFlag) Get() interface{} {
-	return k.gpgKeys
-}
-
-func (k *gpgKeyFlag) String() string {
-	return strings.Join(k.gpgKeys, ",")
-}
-
 func makeCmdPublish() *commander.Command {
 	return &commander.Command{
 		UsageLine: "publish",

cmd/publish_list.go

@@ -53,7 +53,7 @@ func aptlyPublishListTxt(cmd *commander.Command, _ []string) error {
 		return fmt.Errorf("unable to load list of repos: %s", err)
 	}
 
-	_ = context.CloseDatabase()
+	context.CloseDatabase()
 
 	sort.Strings(published)
 
@@ -99,7 +99,7 @@ func aptlyPublishListJSON(_ *commander.Command, _ []string) error {
 		return fmt.Errorf("unable to load list of repos: %s", err)
 	}
 
-	_ = context.CloseDatabase()
+	context.CloseDatabase()
 
 	sort.Slice(repos, func(i, j int) bool {
 		return repos[i].GetPath() < repos[j].GetPath()

cmd/publish_repo.go

@@ -34,7 +34,7 @@ Example:
 	}
 	cmd.Flag.String("distribution", "", "distribution name to publish")
 	cmd.Flag.String("component", "", "component name to publish (for multi-component publishing, separate components with commas)")
-	cmd.Flag.Var(&gpgKeyFlag{}, "gpg-key", "GPG key ID to use when signing the release (flag is repeatable, can be specified multiple times)")
+	cmd.Flag.String("gpg-key", "", "GPG key ID to use when signing the release")
 	cmd.Flag.Var(&keyRingsFlag{}, "keyring", "GPG keyring to use (instead of default)")
 	cmd.Flag.String("secret-keyring", "", "GPG secret keyring to use (instead of default)")
 	cmd.Flag.String("passphrase", "", "GPG passphrase for the key (warning: could be insecure)")

cmd/publish_snapshot.go

@@ -49,7 +49,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
 				return fmt.Errorf("unable to publish: %s", err)
 			}
 
-			err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+			err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 			if err != nil {
 				return fmt.Errorf("unable to publish: %s", err)
 			}
@@ -85,7 +85,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
 				return fmt.Errorf("unable to publish: %s", err)
 			}
 
-			err = collectionFactory.LocalRepoCollection().LoadComplete(localRepo)
+			err = collectionFactory.LocalRepoCollection().LoadComplete(localRepo, collectionFactory.RefListCollection())
 			if err != nil {
 				return fmt.Errorf("unable to publish: %s", err)
 			}
@@ -156,7 +156,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
 
 	duplicate := collectionFactory.PublishedRepoCollection().CheckDuplicate(published)
 	if duplicate != nil {
-		_ = collectionFactory.PublishedRepoCollection().LoadComplete(duplicate, collectionFactory)
+		collectionFactory.PublishedRepoCollection().LoadComplete(duplicate, collectionFactory)
 		return fmt.Errorf("prefix/distribution already used by another published repo: %s", duplicate)
 	}
 
@@ -175,7 +175,7 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to publish: %s", err)
 	}
 
-	err = collectionFactory.PublishedRepoCollection().Add(published)
+	err = collectionFactory.PublishedRepoCollection().Add(published, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to save to DB: %s", err)
 	}
@@ -190,11 +190,9 @@ func aptlyPublishSnapshotOrRepo(cmd *commander.Command, args []string) error {
 
 	context.Progress().Printf("\n%s been successfully published.\n", message)
 
-	if ps, err := context.GetPublishedStorage(storage); err == nil {
+	if localStorage, ok := context.GetPublishedStorage(storage).(aptly.FileSystemPublishedStorage); ok {
-		if localStorage, ok := ps.(aptly.FileSystemPublishedStorage); ok {
+		context.Progress().Printf("Please setup your webserver to serve directory '%s' with autoindexing.\n",
-			context.Progress().Printf("Please setup your webserver to serve directory '%s' with autoindexing.\n",
+			localStorage.PublicPath())
-				localStorage.PublicPath())
-		}
 	}
 
 	context.Progress().Printf("Now you can add following line to apt sources:\n")
@@ -232,7 +230,7 @@ Example:
 	}
 	cmd.Flag.String("distribution", "", "distribution name to publish")
 	cmd.Flag.String("component", "", "component name to publish (for multi-component publishing, separate components with commas)")
-	cmd.Flag.Var(&gpgKeyFlag{}, "gpg-key", "GPG key ID to use when signing the release (flag is repeatable, can be specified multiple times)")
+	cmd.Flag.String("gpg-key", "", "GPG key ID to use when signing the release")
 	cmd.Flag.Var(&keyRingsFlag{}, "keyring", "GPG keyring to use (instead of default)")
 	cmd.Flag.String("secret-keyring", "", "GPG secret keyring to use (instead of default)")
 	cmd.Flag.String("passphrase", "", "GPG passphrase for the key (warning: could be insecure)")

cmd/publish_switch.go

@@ -72,7 +72,7 @@ func aptlyPublishSwitch(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to switch: %s", err)
 		}
 
-		err = snapshotCollection.LoadComplete(snapshot)
+		err = snapshotCollection.LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to switch: %s", err)
 		}
@@ -108,7 +108,7 @@ func aptlyPublishSwitch(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to publish: %s", err)
 	}
 
-	err = collectionFactory.PublishedRepoCollection().Update(published)
+	err = collectionFactory.PublishedRepoCollection().Update(published, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to save to DB: %s", err)
 	}
@@ -151,7 +151,7 @@ This command would switch published repository (with one component) named ppa/wh
 `,
 		Flag: *flag.NewFlagSet("aptly-publish-switch", flag.ExitOnError),
 	}
-	cmd.Flag.Var(&gpgKeyFlag{}, "gpg-key", "GPG key ID to use when signing the release (flag is repeatable, can be specified multiple times)")
+	cmd.Flag.String("gpg-key", "", "GPG key ID to use when signing the release")
 	cmd.Flag.Var(&keyRingsFlag{}, "keyring", "GPG keyring to use (instead of default)")
 	cmd.Flag.String("secret-keyring", "", "GPG secret keyring to use (instead of default)")
 	cmd.Flag.String("passphrase", "", "GPG passphrase for the key (warning: could be insecure)")

cmd/publish_update.go

@@ -69,7 +69,7 @@ func aptlyPublishUpdate(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to publish: %s", err)
 	}
 
-	err = collectionFactory.PublishedRepoCollection().Update(published)
+	err = collectionFactory.PublishedRepoCollection().Update(published, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to save to DB: %s", err)
 	}
@@ -115,7 +115,7 @@ Example:
 `,
 		Flag: *flag.NewFlagSet("aptly-publish-update", flag.ExitOnError),
 	}
-	cmd.Flag.Var(&gpgKeyFlag{}, "gpg-key", "GPG key ID to use when signing the release (flag is repeatable, can be specified multiple times)")
+	cmd.Flag.String("gpg-key", "", "GPG key ID to use when signing the release")
 	cmd.Flag.Var(&keyRingsFlag{}, "keyring", "GPG keyring to use (instead of default)")
 	cmd.Flag.String("secret-keyring", "", "GPG secret keyring to use (instead of default)")
 	cmd.Flag.String("passphrase", "", "GPG passphrase for the key (warning: could be insecure)")

cmd/repo_add.go

@@ -28,7 +28,7 @@ func aptlyRepoAdd(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to add: %s", err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to add: %s", err)
 	}
@@ -58,9 +58,9 @@ func aptlyRepoAdd(cmd *commander.Command, args []string) error {
 
 	processedFiles = append(processedFiles, otherFiles...)
 
-	repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
+	repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
 
-	err = collectionFactory.LocalRepoCollection().Update(repo)
+	err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to save: %s", err)
 	}

cmd/repo_create.go

@@ -36,7 +36,7 @@ func aptlyRepoCreate(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to load source snapshot: %s", err)
 		}
 
-		err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to load source snapshot: %s", err)
 		}
@@ -44,7 +44,7 @@ func aptlyRepoCreate(cmd *commander.Command, args []string) error {
 		repo.UpdateRefList(snapshot.RefList())
 	}
 
-	err = collectionFactory.LocalRepoCollection().Add(repo)
+	err = collectionFactory.LocalRepoCollection().Add(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to add local repo: %s", err)
 	}

cmd/repo_edit.go

@@ -22,7 +22,7 @@ func aptlyRepoEdit(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to edit: %s", err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to edit: %s", err)
 	}
@@ -53,7 +53,7 @@ func aptlyRepoEdit(cmd *commander.Command, args []string) error {
 		}
 	}
 
-	err = collectionFactory.LocalRepoCollection().Update(repo)
+	err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to edit: %s", err)
 	}

cmd/repo_include.go

@@ -67,7 +67,7 @@ func aptlyRepoInclude(cmd *commander.Command, args []string) error {
 	_, failedFiles2, err = deb.ImportChangesFiles(
 		changesFiles, reporter, acceptUnsigned, ignoreSignatures, forceReplace, noRemoveFiles, verifier, repoTemplate,
 		context.Progress(), collectionFactory.LocalRepoCollection(), collectionFactory.PackageCollection(),
-		context.PackagePool(), collectionFactory.ChecksumCollection,
+		collectionFactory.RefListCollection(), context.PackagePool(), collectionFactory.ChecksumCollection,
 		uploaders, query.Parse)
 	failedFiles = append(failedFiles, failedFiles2...)
 

cmd/repo_list.go

@@ -32,11 +32,11 @@ func aptlyRepoListTxt(cmd *commander.Command, _ []string) error {
 	collectionFactory := context.NewCollectionFactory()
 	repos := make([]string, collectionFactory.LocalRepoCollection().Len())
 	i := 0
-	_ = collectionFactory.LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
+	collectionFactory.LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
 		if raw {
 			repos[i] = repo.Name
 		} else {
-			e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
+			e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 			if e != nil {
 				return e
 			}
@@ -47,7 +47,7 @@ func aptlyRepoListTxt(cmd *commander.Command, _ []string) error {
 		return nil
 	})
 
-	_ = context.CloseDatabase()
+	context.CloseDatabase()
 
 	sort.Strings(repos)
 
@@ -76,8 +76,9 @@ func aptlyRepoListJSON(_ *commander.Command, _ []string) error {
 
 	repos := make([]*deb.LocalRepo, context.NewCollectionFactory().LocalRepoCollection().Len())
 	i := 0
-	_ = context.NewCollectionFactory().LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
+	context.NewCollectionFactory().LocalRepoCollection().ForEach(func(repo *deb.LocalRepo) error {
-		e := context.NewCollectionFactory().LocalRepoCollection().LoadComplete(repo)
+		collectionFactory := context.NewCollectionFactory()
+		e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -87,7 +88,7 @@ func aptlyRepoListJSON(_ *commander.Command, _ []string) error {
 		return nil
 	})
 
-	_ = context.CloseDatabase()
+	context.CloseDatabase()
 
 	sort.Slice(repos, func(i, j int) bool {
 		return repos[i].Name < repos[j].Name

cmd/repo_move.go

@@ -25,13 +25,13 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to %s: %s", command, err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(dstRepo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(dstRepo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to %s: %s", command, err)
 	}
 
 	var (
-		srcRefList *deb.PackageRefList
+		srcRefList *deb.SplitRefList
 		srcRepo    *deb.LocalRepo
 	)
 
@@ -45,7 +45,7 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to %s: source and destination are the same", command)
 		}
 
-		err = collectionFactory.LocalRepoCollection().LoadComplete(srcRepo)
+		err = collectionFactory.LocalRepoCollection().LoadComplete(srcRepo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to %s: %s", command, err)
 		}
@@ -59,12 +59,12 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to %s: %s", command, err)
 		}
 
-		err = collectionFactory.RemoteRepoCollection().LoadComplete(srcRemoteRepo)
+		err = collectionFactory.RemoteRepoCollection().LoadComplete(srcRemoteRepo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to %s: %s", command, err)
 		}
 
-		if srcRemoteRepo.RefList() == nil {
+		if srcRemoteRepo.RefList().Len() == 0 {
 			return fmt.Errorf("unable to %s: mirror not updated", command)
 		}
 
@@ -161,17 +161,17 @@ func aptlyRepoMoveCopyImport(cmd *commander.Command, args []string) error {
 	if context.Flags().Lookup("dry-run").Value.Get().(bool) {
 		context.Progress().Printf("\nChanges not saved, as dry run has been requested.\n")
 	} else {
-		dstRepo.UpdateRefList(deb.NewPackageRefListFromPackageList(dstList))
+		dstRepo.UpdateRefList(deb.NewSplitRefListFromPackageList(dstList))
 
-		err = collectionFactory.LocalRepoCollection().Update(dstRepo)
+		err = collectionFactory.LocalRepoCollection().Update(dstRepo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to save: %s", err)
 		}
 
 		if command == "move" { // nolint: goconst
-			srcRepo.UpdateRefList(deb.NewPackageRefListFromPackageList(srcList))
+			srcRepo.UpdateRefList(deb.NewSplitRefListFromPackageList(srcList))
 
-			err = collectionFactory.LocalRepoCollection().Update(srcRepo)
+			err = collectionFactory.LocalRepoCollection().Update(srcRepo, collectionFactory.RefListCollection())
 			if err != nil {
 				return fmt.Errorf("unable to save: %s", err)
 			}

cmd/repo_remove.go

@@ -24,7 +24,7 @@ func aptlyRepoRemove(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to remove: %s", err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to remove: %s", err)
 	}
@@ -54,7 +54,7 @@ func aptlyRepoRemove(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to remove: %s", err)
 	}
 
-	_ = toRemove.ForEach(func(p *deb.Package) error {
+	toRemove.ForEach(func(p *deb.Package) error {
 		list.Remove(p)
 		context.Progress().ColoredPrintf("@r[-]@| %s removed", p)
 		return nil
@@ -63,9 +63,9 @@ func aptlyRepoRemove(cmd *commander.Command, args []string) error {
 	if context.Flags().Lookup("dry-run").Value.Get().(bool) {
 		context.Progress().Printf("\nChanges not saved, as dry run has been requested.\n")
 	} else {
-		repo.UpdateRefList(deb.NewPackageRefListFromPackageList(list))
+		repo.UpdateRefList(deb.NewSplitRefListFromPackageList(list))
 
-		err = collectionFactory.LocalRepoCollection().Update(repo)
+		err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to save: %s", err)
 		}

cmd/repo_rename.go

@@ -32,7 +32,7 @@ func aptlyRepoRename(cmd *commander.Command, args []string) error {
 	}
 
 	repo.Name = newName
-	err = collectionFactory.LocalRepoCollection().Update(repo)
+	err = collectionFactory.LocalRepoCollection().Update(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to rename: %s", err)
 	}

cmd/repo_show.go

@@ -36,7 +36,7 @@ func aptlyRepoShowTxt(_ *commander.Command, args []string) error {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -52,7 +52,7 @@ func aptlyRepoShowTxt(_ *commander.Command, args []string) error {
 
 	withPackages := context.Flags().Lookup("with-packages").Value.Get().(bool)
 	if withPackages {
-		_ = ListPackagesRefList(repo.RefList(), collectionFactory)
+		ListPackagesRefList(repo.RefList(), collectionFactory)
 	}
 
 	return err
@@ -63,12 +63,13 @@ func aptlyRepoShowJSON(_ *commander.Command, args []string) error {
 
 	name := args[0]
 
-	repo, err := context.NewCollectionFactory().LocalRepoCollection().ByName(name)
+	collectionFactory := context.NewCollectionFactory()
+	repo, err := collectionFactory.LocalRepoCollection().ByName(name)
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = context.NewCollectionFactory().LocalRepoCollection().LoadComplete(repo)
+	err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -79,7 +80,7 @@ func aptlyRepoShowJSON(_ *commander.Command, args []string) error {
 	if withPackages {
 		if repo.RefList() != nil {
 			var list *deb.PackageList
-			list, err = deb.NewPackageListFromRefList(repo.RefList(), context.NewCollectionFactory().PackageCollection(), context.Progress())
+			list, err = deb.NewPackageListFromRefList(repo.RefList(), collectionFactory.PackageCollection(), context.Progress())
 			if err == nil {
 				packageList = list.FullNames()
 			}

cmd/serve.go

@@ -97,11 +97,7 @@ func aptlyServe(cmd *commander.Command, args []string) error {
 		}
 	}
 
-	ps, err := context.GetPublishedStorage("")
+	publicPath := context.GetPublishedStorage("").(aptly.FileSystemPublishedStorage).PublicPath()
-	if err != nil {
-		return err
-	}
-	publicPath := ps.(aptly.FileSystemPublishedStorage).PublicPath()
 	ShutdownContext()
 
 	fmt.Printf("\nStarting web server at: %s (press Ctrl+C to quit)...\n", listen)

cmd/snapshot_create.go

@@ -30,7 +30,7 @@ func aptlySnapshotCreate(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}
 
-		err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}
@@ -50,7 +50,7 @@ func aptlySnapshotCreate(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}
 
-		err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}
@@ -71,7 +71,7 @@ func aptlySnapshotCreate(cmd *commander.Command, args []string) error {
 		return commander.ErrCommandError
 	}
 
-	err = collectionFactory.SnapshotCollection().Add(snapshot)
+	err = collectionFactory.SnapshotCollection().Add(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to add snapshot: %s", err)
 	}

cmd/snapshot_diff.go

@@ -23,7 +23,7 @@ func aptlySnapshotDiff(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to load snapshot A: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(snapshotA)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshotA, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to load snapshot A: %s", err)
 	}
@@ -34,13 +34,13 @@ func aptlySnapshotDiff(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to load snapshot B: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(snapshotB)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshotB, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to load snapshot B: %s", err)
 	}
 
 	// Calculate diff
-	diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection())
+	diff, err := snapshotA.RefList().Diff(snapshotB.RefList(), collectionFactory.PackageCollection(), nil)
 	if err != nil {
 		return fmt.Errorf("unable to calculate diff: %s", err)
 	}

cmd/snapshot_filter.go

@@ -27,7 +27,7 @@ func aptlySnapshotFilter(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to filter: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(source)
+	err = collectionFactory.SnapshotCollection().LoadComplete(source, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to filter: %s", err)
 	}
@@ -87,7 +87,7 @@ func aptlySnapshotFilter(cmd *commander.Command, args []string) error {
 	destination := deb.NewSnapshotFromPackageList(args[1], []*deb.Snapshot{source}, result,
 		fmt.Sprintf("Filtered '%s', query was: '%s'", source.Name, strings.Join(args[2:], " ")))
 
-	err = collectionFactory.SnapshotCollection().Add(destination)
+	err = collectionFactory.SnapshotCollection().Add(destination, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to create snapshot: %s", err)
 	}

cmd/snapshot_list.go

@@ -33,7 +33,7 @@ func aptlySnapshotListTxt(cmd *commander.Command, _ []string) error {
 	collection := collectionFactory.SnapshotCollection()
 
 	if raw {
-		_ = collection.ForEachSorted(sortMethodString, func(snapshot *deb.Snapshot) error {
+		collection.ForEachSorted(sortMethodString, func(snapshot *deb.Snapshot) error {
 			fmt.Printf("%s\n", snapshot.Name)
 			return nil
 		})
@@ -68,7 +68,7 @@ func aptlySnapshotListJSON(cmd *commander.Command, _ []string) error {
 
 	jsonSnapshots := make([]*deb.Snapshot, collection.Len())
 	i := 0
-	_ = collection.ForEachSorted(sortMethodString, func(snapshot *deb.Snapshot) error {
+	collection.ForEachSorted(sortMethodString, func(snapshot *deb.Snapshot) error {
 		jsonSnapshots[i] = snapshot
 		i++
 		return nil

cmd/snapshot_merge.go

@@ -24,7 +24,7 @@ func aptlySnapshotMerge(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to load snapshot: %s", err)
 		}
 
-		err = collectionFactory.SnapshotCollection().LoadComplete(sources[i])
+		err = collectionFactory.SnapshotCollection().LoadComplete(sources[i], collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to load snapshot: %s", err)
 		}
@@ -57,7 +57,7 @@ func aptlySnapshotMerge(cmd *commander.Command, args []string) error {
 	destination := deb.NewSnapshotFromRefList(args[0], sources, result,
 		fmt.Sprintf("Merged from sources: %s", strings.Join(sourceDescription, ", ")))
 
-	err = collectionFactory.SnapshotCollection().Add(destination)
+	err = collectionFactory.SnapshotCollection().Add(destination, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to create snapshot: %s", err)
 	}

cmd/snapshot_pull.go

@@ -29,7 +29,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to pull: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to pull: %s", err)
 	}
@@ -40,7 +40,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
 		return fmt.Errorf("unable to pull: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(source)
+	err = collectionFactory.SnapshotCollection().LoadComplete(source, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to pull: %s", err)
 	}
@@ -116,7 +116,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
 
 	alreadySeen := map[string]bool{}
 
-	_ = result.ForEachIndexed(func(pkg *deb.Package) error {
+	result.ForEachIndexed(func(pkg *deb.Package) error {
 		key := pkg.Architecture + "_" + pkg.Name
 		_, seen := alreadySeen[key]
 
@@ -132,7 +132,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
 
 		// If !allMatches, add only first matching name-arch package
 		if !seen || allMatches {
-			_ = packageList.Add(pkg)
+			packageList.Add(pkg)
 			context.Progress().ColoredPrintf("@g[+]@| %s added", pkg)
 		}
 
@@ -149,7 +149,7 @@ func aptlySnapshotPull(cmd *commander.Command, args []string) error {
 		destination := deb.NewSnapshotFromPackageList(args[2], []*deb.Snapshot{snapshot, source}, packageList,
 			fmt.Sprintf("Pulled into '%s' with '%s' as source, pull request was: '%s'", snapshot.Name, source.Name, strings.Join(args[3:], " ")))
 
-		err = collectionFactory.SnapshotCollection().Add(destination)
+		err = collectionFactory.SnapshotCollection().Add(destination, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to create snapshot: %s", err)
 		}

cmd/snapshot_rename.go

@@ -32,7 +32,7 @@ func aptlySnapshotRename(cmd *commander.Command, args []string) error {
 	}
 
 	snapshot.Name = newName
-	err = collectionFactory.SnapshotCollection().Update(snapshot)
+	err = collectionFactory.SnapshotCollection().Update(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to rename: %s", err)
 	}

cmd/snapshot_search.go

@@ -25,7 +25,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 	command := cmd.Parent.Name()
 	collectionFactory := context.NewCollectionFactory()
 
-	var reflist *deb.PackageRefList
+	var reflist *deb.SplitRefList
 
 	if command == "snapshot" { // nolint: goconst
 		var snapshot *deb.Snapshot
@@ -34,7 +34,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 			return fmt.Errorf("unable to search: %s", err)
 		}
 
-		err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to search: %s", err)
 		}
@@ -47,7 +47,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 			return fmt.Errorf("unable to search: %s", err)
 		}
 
-		err = collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		err = collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to search: %s", err)
 		}
@@ -60,7 +60,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 			return fmt.Errorf("unable to search: %s", err)
 		}
 
-		err = collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		err = collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to search: %s", err)
 		}
@@ -123,7 +123,7 @@ func aptlySnapshotMirrorRepoSearch(cmd *commander.Command, args []string) error
 	}
 
 	format := context.Flags().Lookup("format").Value.String()
-	_ = PrintPackageList(result, format, "")
+	PrintPackageList(result, format, "")
 
 	return err
 }

cmd/snapshot_show.go

@@ -35,7 +35,7 @@ func aptlySnapshotShowTxt(_ *commander.Command, args []string) error {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -79,23 +79,24 @@ func aptlySnapshotShowTxt(_ *commander.Command, args []string) error {
 
 	withPackages := context.Flags().Lookup("with-packages").Value.Get().(bool)
 	if withPackages {
-		_ = ListPackagesRefList(snapshot.RefList(), collectionFactory)
+		ListPackagesRefList(snapshot.RefList(), collectionFactory)
 	}
 
 	return err
 }
 
 func aptlySnapshotShowJSON(_ *commander.Command, args []string) error {
+	collectionFactory := context.NewCollectionFactory()
 	var err error
 
 	name := args[0]
 
-	snapshot, err := context.NewCollectionFactory().SnapshotCollection().ByName(name)
+	snapshot, err := collectionFactory.SnapshotCollection().ByName(name)
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
 
-	err = context.NewCollectionFactory().SnapshotCollection().LoadComplete(snapshot)
+	err = collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 	if err != nil {
 		return fmt.Errorf("unable to show: %s", err)
 	}
@@ -105,14 +106,14 @@ func aptlySnapshotShowJSON(_ *commander.Command, args []string) error {
 		for _, sourceID := range snapshot.SourceIDs {
 			if snapshot.SourceKind == deb.SourceSnapshot {
 				var source *deb.Snapshot
-				source, err = context.NewCollectionFactory().SnapshotCollection().ByUUID(sourceID)
+				source, err = collectionFactory.SnapshotCollection().ByUUID(sourceID)
 				if err != nil {
 					continue
 				}
 				snapshot.Snapshots = append(snapshot.Snapshots, source)
 			} else if snapshot.SourceKind == deb.SourceLocalRepo {
 				var source *deb.LocalRepo
-				source, err = context.NewCollectionFactory().LocalRepoCollection().ByUUID(sourceID)
+				source, err = collectionFactory.LocalRepoCollection().ByUUID(sourceID)
 				if err != nil {
 					continue
 				}
@@ -133,13 +134,13 @@ func aptlySnapshotShowJSON(_ *commander.Command, args []string) error {
 	if withPackages {
 		if snapshot.RefList() != nil {
 			var list *deb.PackageList
-			list, err = deb.NewPackageListFromRefList(snapshot.RefList(), context.NewCollectionFactory().PackageCollection(), context.Progress())
+			list, err = deb.NewPackageListFromRefList(snapshot.RefList(), collectionFactory.PackageCollection(), context.Progress())
 			if err != nil {
 				return fmt.Errorf("unable to get package list: %s", err)
 			}
 
 			list.PrepareIndex()
-			_ = list.ForEachIndexed(func(p *deb.Package) error {
+			list.ForEachIndexed(func(p *deb.Package) error {
 				snapshot.Packages = append(snapshot.Packages, p.GetFullName())
 				return nil
 			})

cmd/snapshot_verify.go

@@ -23,7 +23,7 @@ func aptlySnapshotVerify(cmd *commander.Command, args []string) error {
 			return fmt.Errorf("unable to verify: %s", err)
 		}
 
-		err = collectionFactory.SnapshotCollection().LoadComplete(snapshots[i])
+		err = collectionFactory.SnapshotCollection().LoadComplete(snapshots[i], collectionFactory.RefListCollection())
 		if err != nil {
 			return fmt.Errorf("unable to verify: %s", err)
 		}

cmd/task_run.go

@@ -6,7 +6,7 @@ import (
 	"os"
 	"strings"
 
-	shellwords "github.com/mattn/go-shellwords"
+	"github.com/mattn/go-shellwords"
 	"github.com/smira/commander"
 )
 
@@ -31,7 +31,7 @@ func aptlyTaskRun(cmd *commander.Command, args []string) error {
 		if err != nil {
 			return err
 		}
-		defer func() { _ = file.Close() }()
+		defer file.Close()
 
 		scanner := bufio.NewScanner(file)
 

completion.d/aptly

@@ -22,36 +22,34 @@
 
 __aptly_mirror_list()
 {
-  aptly ${aptly_global_opts[@]} mirror list -raw
+  aptly mirror list -raw
 }
 
 __aptly_repo_list()
 {
-  aptly ${aptly_global_opts[@]} repo list -raw
+  aptly repo list -raw
 }
 
 __aptly_snapshot_list()
 {
-  aptly ${aptly_global_opts[@]} snapshot list -raw
+  aptly snapshot list -raw
 }
 
 __aptly_published_distributions()
 {
-  aptly ${aptly_global_opts[@]} publish list -raw | cut -d ' ' -f 2 | sort | uniq
+  aptly publish list -raw | cut -d ' ' -f 2 | sort | uniq
 }
 
 __aptly_published_prefixes()
 {
-  aptly ${aptly_global_opts[@]} publish list -raw | cut -d ' ' -f 1 | sort | uniq
+  aptly publish list -raw | cut -d ' ' -f 1 | sort | uniq
 }
 
 __aptly_prefixes_for_distribution()
 {
-  aptly ${aptly_global_opts[@]} publish list -raw | awk -v dist="$1" '{ if (dist == $2) print $1 }' | sort | uniq
+  aptly publish list -raw | awk -v dist="$1" '{ if (dist == $2) print $1 }' | sort | uniq
 }
 
-
-
 _aptly()
 {
     cur="${COMP_WORDS[COMP_CWORD]}"
@@ -59,12 +57,7 @@ _aptly()
     prevprev="${COMP_WORDS[COMP_CWORD-2]}"
 
     commands="api config db graph mirror package publish repo serve snapshot task version"
-
+    options="-architectures= -config= -db-open-attempts= -dep-follow-all-variants -dep-follow-recommends -dep-follow-source -dep-follow-suggests -dep-verbose-resolve -gpg-provider="
-    options="-architectures -config -db-open-attempts -dep-follow-all-variants -dep-follow-recommends -dep-follow-source -dep-follow-suggests -dep-verbose-resolve -gpg-provider"
-    options_without_arg="-dep-follow-all-variants -dep-follow-recommends -dep-follow-source -dep-follow-suggests -dep-verbose-resolve"
-    options_with_arg="-architectures -db-open-attempts -gpg-provider"
-    options_with_path_arg="-config"
-
     db_subcommands="cleanup recover"
     mirror_subcommands="create drop edit show list rename search update"
     publish_subcommands="drop list repo snapshot switch update source"
@@ -76,41 +69,12 @@ _aptly()
     config_subcommands="show"
     api_subcommands="serve"
 
-    local cmd subcmd numargs numoptions i aptly_global_opts
+    local cmd subcmd numargs numoptions i
 
     numargs=0
     numoptions=0
 
-    for opt in "${options_with_path_arg[@]}"; do
-        [[ "$prev" == "$opt" ]] || continue
-        compopt -o filenames 2>/dev/null
-        _filedir
-        return 0
-    done
-
     for (( i=1; i < $COMP_CWORD; i++ )); do
-      word=${COMP_WORDS[i]}
-      if [[ "$word" == -*=* ]]; then
-        for o in "${options[@]}"; do
-          [[ ${word%%=*} == "$o" ]] && aptly_global_opts+=("$word")
-        done
-      else
-        for o in "${options_with_arg[@]}" ""${options_with_path_arg[@]}"" ; do
-          if [[ "$word" == "$o" ]]; then
-            if (( i + 1 < COMP_CWORD )); then
-                aptly_global_opts+=("$word" "${COMP_WORDS[i+1]}")
-            else
-                aptly_global_opts+=("$word")
-            fi
-            (( i++ ))
-            continue 2
-          fi
-        done
-      fi
-      for o in ${options_without_arg[@]}; do
-          [[ "$word" == "$o" ]] && aptly_global_opts+=("$word")
-      done
-
       if [[ -n "$cmd" ]]; then
         if [[ ! -n "$subcmd" ]]; then
           subcmd=${COMP_WORDS[i]}
@@ -375,7 +339,7 @@ _aptly()
                 if [[ "$cur" == -* ]]; then
                   COMPREPLY=($(compgen -W "-accept-unsigned -force-replace -ignore-signatures -keyring= -no-remove-files -repo= -uploaders-file=" -- ${cur}))
                 else
-                  compopt -o filenames 2>/dev/null
+                  comptopt -o filenames 2>/dev/null
                   COMPREPLY=($(compgen -f -- ${cur}))
                   return 0
                 fi

console/progress.go

@@ -6,7 +6,6 @@ import (
 	"strings"
 
 	"github.com/aptly-dev/aptly/aptly"
-	"github.com/aptly-dev/aptly/utils"
 	"github.com/cheggaaa/pb"
 	"github.com/rs/zerolog/log"
 	"github.com/wsxiaoys/terminal/color"
@@ -79,7 +78,7 @@ func (p *Progress) InitBar(count int64, isBytes bool, _ aptly.BarType) {
 	if p.bar != nil {
 		panic("bar already initialized")
 	}
-	if utils.RunningOnTerminal() {
+	if RunningOnTerminal() {
 		p.bar = pb.New(0)
 		p.bar.Total = count
 		p.bar.NotPrint = true
@@ -142,7 +141,7 @@ func (p *Progress) PrintfStdErr(msg string, a ...interface{}) {
 
 // ColoredPrintf does printf in colored way + newline
 func (p *Progress) ColoredPrintf(msg string, a ...interface{}) {
-	if utils.RunningOnTerminal() {
+	if RunningOnTerminal() {
 		p.queue <- printTask{code: codePrint, message: color.Sprintf(msg, a...) + "\n"}
 	} else {
 		// stip color marks

console/progress_test.go

@@ -11,7 +11,7 @@ func Test(t *testing.T) {
 	TestingT(t)
 }
 
-type ProgressSuite struct{}
+type ProgressSuite struct {}
 
 var _ = Suite(&ProgressSuite{})
 

console/terminal.go

@@ -0,0 +1,12 @@
+package console
+
+import (
+	"syscall"
+
+	"golang.org/x/term"
+)
+
+// RunningOnTerminal checks whether stdout is terminal
+func RunningOnTerminal() bool {
+	return term.IsTerminal(syscall.Stdout)
+}

context/context.go

@@ -100,7 +100,6 @@ func (context *AptlyContext) config() *utils.ConfigStructure {
 			configLocations := []string{homeLocation, "/usr/local/etc/aptly.conf", "/etc/aptly.conf"}
 
 			for _, configLocation := range configLocations {
-				// FIXME: check if exists, check if readable
 				err = utils.LoadConfig(configLocation, &utils.Config)
 				if os.IsPermission(err) || os.IsNotExist(err) {
 					continue
@@ -116,12 +115,7 @@ func (context *AptlyContext) config() *utils.ConfigStructure {
 			if err != nil {
 				fmt.Fprintf(os.Stderr, "Config file not found, creating default config at %s\n\n", homeLocation)
 
-				defaultConfig := aptly.AptlyConf
+				utils.SaveConfigRaw(homeLocation, aptly.AptlyConf)
-				if len(defaultConfig) == 0 {
-					defaultConfig = []byte("root_dir: \"\"")
-				}
-
-				_ = utils.SaveConfigRaw(homeLocation, defaultConfig)
 				err = utils.LoadConfig(homeLocation, &utils.Config)
 				if err != nil {
 					Fatal(fmt.Errorf("error loading config file %s: %s", homeLocation, err))
@@ -129,14 +123,6 @@ func (context *AptlyContext) config() *utils.ConfigStructure {
 			}
 		}
 
-		if utils.Config.LogFormat == "json" {
-			context.StructuredLogging(true)
-			utils.SetupJSONLogger(utils.Config.LogLevel, os.Stdout)
-		} else {
-			context.StructuredLogging(false)
-			utils.SetupDefaultLogger(utils.Config.LogLevel)
-		}
-
 		context.configLoaded = true
 
 	}
@@ -247,7 +233,7 @@ func (context *AptlyContext) newDownloader(progress aptly.Progress) aptly.Downlo
 		// If flag is defined prefer it to global setting
 		maxTries = maxTriesFlag.Value.Get().(int)
 	}
-	var downloader = context.config().Downloader
+	var downloader string = context.config().Downloader
 	downloaderFlag := context.flags.Lookup("downloader")
 	if downloaderFlag != nil {
 		downloader = downloaderFlag.Value.String()
@@ -309,8 +295,8 @@ func (context *AptlyContext) _database() (database.Storage, error) {
 		switch context.config().DatabaseBackend.Type {
 		case "leveldb":
 			dbPath := filepath.Join(context.config().GetRootDir(), "db")
-			if len(context.config().DatabaseBackend.DBPath) != 0 {
+			if len(context.config().DatabaseBackend.DbPath) != 0 {
-				dbPath = context.config().DatabaseBackend.DBPath
+				dbPath = context.config().DatabaseBackend.DbPath
 			}
 			context.database, err = goleveldb.NewDB(dbPath)
 		case "etcd":
@@ -412,8 +398,8 @@ func (context *AptlyContext) PackagePool() aptly.PackagePool {
 	return context.packagePool
 }
 
-// GetPublishedStorage returns instance of PublishedStorage, or an error if the storage is not configured
+// GetPublishedStorage returns instance of PublishedStorage
-func (context *AptlyContext) GetPublishedStorage(name string) (aptly.PublishedStorage, error) {
+func (context *AptlyContext) GetPublishedStorage(name string) aptly.PublishedStorage {
 	context.Lock()
 	defer context.Unlock()
 
@@ -424,14 +410,14 @@ func (context *AptlyContext) GetPublishedStorage(name string) (aptly.PublishedSt
 		} else if strings.HasPrefix(name, "filesystem:") {
 			params, ok := context.config().FileSystemPublishRoots[name[11:]]
 			if !ok {
-				return nil, fmt.Errorf("published local storage %v not configured", name[11:])
+				Fatal(fmt.Errorf("published local storage %v not configured", name[11:]))
 			}
 
 			publishedStorage = files.NewPublishedStorage(params.RootDir, params.LinkMethod, params.VerifyMethod)
 		} else if strings.HasPrefix(name, "s3:") {
 			params, ok := context.config().S3PublishRoots[name[3:]]
 			if !ok {
-				return nil, fmt.Errorf("published S3 storage %v not configured", name[3:])
+				Fatal(fmt.Errorf("published S3 storage %v not configured", name[3:]))
 			}
 
 			var err error
@@ -441,39 +427,39 @@ func (context *AptlyContext) GetPublishedStorage(name string) (aptly.PublishedSt
 				params.EncryptionMethod, params.PlusWorkaround, params.DisableMultiDel,
 				params.ForceSigV2, params.ForceVirtualHostedStyle, params.Debug)
 			if err != nil {
-				return nil, err
+				Fatal(err)
 			}
 		} else if strings.HasPrefix(name, "swift:") {
 			params, ok := context.config().SwiftPublishRoots[name[6:]]
 			if !ok {
-				return nil, fmt.Errorf("published Swift storage %v not configured", name[6:])
+				Fatal(fmt.Errorf("published Swift storage %v not configured", name[6:]))
 			}
 
 			var err error
 			publishedStorage, err = swift.NewPublishedStorage(params.UserName, params.Password,
 				params.AuthURL, params.Tenant, params.TenantID, params.Domain, params.DomainID, params.TenantDomain, params.TenantDomainID, params.Container, params.Prefix)
 			if err != nil {
-				return nil, err
+				Fatal(err)
 			}
 		} else if strings.HasPrefix(name, "azure:") {
 			params, ok := context.config().AzurePublishRoots[name[6:]]
 			if !ok {
-				return nil, fmt.Errorf("published Azure storage %v not configured", name[6:])
+				Fatal(fmt.Errorf("Published Azure storage %v not configured", name[6:]))
 			}
 
 			var err error
 			publishedStorage, err = azure.NewPublishedStorage(
 				params.AccountName, params.AccountKey, params.Container, params.Prefix, params.Endpoint)
 			if err != nil {
-				return nil, err
+				Fatal(err)
 			}
 		} else {
-			return nil, fmt.Errorf("unknown published storage format: %v", name)
+			Fatal(fmt.Errorf("unknown published storage format: %v", name))
 		}
 		context.publishedStorages[name] = publishedStorage
 	}
 
-	return publishedStorage, nil
+	return publishedStorage
 }
 
 // UploadPath builds path to upload storage
@@ -603,17 +589,17 @@ func (context *AptlyContext) Shutdown() {
 
 	if aptly.EnableDebug {
 		if context.fileMemProfile != nil {
-			_ = pprof.WriteHeapProfile(context.fileMemProfile)
+			pprof.WriteHeapProfile(context.fileMemProfile)
-			_ = context.fileMemProfile.Close()
+			context.fileMemProfile.Close()
 			context.fileMemProfile = nil
 		}
 		if context.fileCPUProfile != nil {
 			pprof.StopCPUProfile()
-			_ = context.fileCPUProfile.Close()
+			context.fileCPUProfile.Close()
 			context.fileCPUProfile = nil
 		}
 		if context.fileMemProfile != nil {
-			_ = context.fileMemProfile.Close()
+			context.fileMemProfile.Close()
 			context.fileMemProfile = nil
 		}
 	}
@@ -621,7 +607,7 @@ func (context *AptlyContext) Shutdown() {
 		context.taskList.Stop()
 	}
 	if context.database != nil {
-		_ = context.database.Close()
+		context.database.Close()
 		context.database = nil
 	}
 	if context.downloader != nil {
@@ -666,7 +652,7 @@ func NewContext(flags *flag.FlagSet) (*AptlyContext, error) {
 			if err != nil {
 				return nil, err
 			}
-			_ = pprof.StartCPUProfile(context.fileCPUProfile)
+			pprof.StartCPUProfile(context.fileCPUProfile)
 		}
 
 		memprofile := flags.Lookup("memprofile").Value.String()
@@ -686,7 +672,7 @@ func NewContext(flags *flag.FlagSet) (*AptlyContext, error) {
 				return nil, err
 			}
 
-			_, _ = context.fileMemStats.WriteString("# Time\tHeapSys\tHeapAlloc\tHeapIdle\tHeapReleased\n")
+			context.fileMemStats.WriteString("# Time\tHeapSys\tHeapAlloc\tHeapIdle\tHeapReleased\n")
 
 			go func() {
 				var stats runtime.MemStats
@@ -696,7 +682,7 @@ func NewContext(flags *flag.FlagSet) (*AptlyContext, error) {
 				for {
 					runtime.ReadMemStats(&stats)
 					if context.fileMemStats != nil {
-						_, _ = context.fileMemStats.WriteString(fmt.Sprintf("%d\t%d\t%d\t%d\t%d\n",
+						context.fileMemStats.WriteString(fmt.Sprintf("%d\t%d\t%d\t%d\t%d\n",
 							(time.Now().UnixNano()-start)/1000000, stats.HeapSys, stats.HeapAlloc, stats.HeapIdle, stats.HeapReleased))
 						time.Sleep(interval)
 					} else {

context/context_test.go

@@ -1,6 +1,8 @@
 package context
 
 import (
+	"fmt"
+	"os"
 	"reflect"
 	"testing"
 
@@ -78,9 +80,10 @@ func (s *AptlyContextSuite) SetUpTest(c *C) {
 
 func (s *AptlyContextSuite) TestGetPublishedStorageBadFS(c *C) {
 	// https://github.com/aptly-dev/aptly/issues/711
-	// https://github.com/aptly-dev/aptly/issues/1477
+	// This will fail on account of us not having a config, so the
-	// GetPublishedStorage must return an error (not panic) when the
+	// storage never exists.
-	// requested storage is not configured.
+	c.Assert(func() { s.context.GetPublishedStorage("filesystem:fuji") },
-	_, err := s.context.GetPublishedStorage("filesystem:fuji")
+		FatalErrorPanicMatches,
-	c.Assert(err, NotNil)
+		&FatalError{ReturnCode: 1, Message: fmt.Sprintf("error loading config file %s/.aptly.conf: invalid yaml (EOF) or json (EOF)",
+			os.Getenv("HOME"))})
 }

database/database.go

@@ -48,6 +48,8 @@ type Storage interface {
 
 	CreateTemporary() (Storage, error)
 
+	GetRecommendedMaxKVSize() int
+
 	Open() error
 	Close() error
 	CompactDB() error

database/etcddb/database_test.go

@@ -14,7 +14,8 @@ func Test(t *testing.T) {
 }
 
 type EtcDDBSuite struct {
-	db database.Storage
+	url string
+	db  database.Storage
 }
 
 var _ = Suite(&EtcDDBSuite{})
@@ -66,17 +67,17 @@ func (s *EtcDDBSuite) TestDelete(c *C) {
 func (s *EtcDDBSuite) TestByPrefix(c *C) {
 	//c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{})
 
-	_ = s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
+	s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
-	_ = s.db.Put([]byte{0x80, 0x03}, []byte{0x03})
+	s.db.Put([]byte{0x80, 0x03}, []byte{0x03})
-	_ = s.db.Put([]byte{0x80, 0x02}, []byte{0x02})
+	s.db.Put([]byte{0x80, 0x02}, []byte{0x02})
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x01}, {0x02}, {0x03}})
 	c.Check(s.db.KeysByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x80, 0x01}, {0x80, 0x02}, {0x80, 0x03}})
 
-	_ = s.db.Put([]byte{0x90, 0x01}, []byte{0x04})
+	s.db.Put([]byte{0x90, 0x01}, []byte{0x04})
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x01}, {0x02}, {0x03}})
 	c.Check(s.db.KeysByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x80, 0x01}, {0x80, 0x02}, {0x80, 0x03}})
 
-	_ = s.db.Put([]byte{0x00, 0x01}, []byte{0x05})
+	s.db.Put([]byte{0x00, 0x01}, []byte{0x05})
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x01}, {0x02}, {0x03}})
 	c.Check(s.db.KeysByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x80, 0x01}, {0x80, 0x02}, {0x80, 0x03}})
 
@@ -108,7 +109,7 @@ func (s *EtcDDBSuite) TestHasPrefix(c *C) {
 	//c.Check(s.db.HasPrefix([]byte(nil)), Equals, false)
 	//c.Check(s.db.HasPrefix([]byte{0x80}), Equals, false)
 
-	_ = s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
+	s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
 
 	c.Check(s.db.HasPrefix([]byte(nil)), Equals, true)
 	c.Check(s.db.HasPrefix([]byte{0x80}), Equals, true)
@@ -123,17 +124,15 @@ func (s *EtcDDBSuite) TestTransactionCommit(c *C) {
 		value2 = []byte("value2")
 	)
 	transaction, err := s.db.OpenTransaction()
-	c.Assert(err, IsNil)
 
 	err = s.db.Put(key, value)
 	c.Assert(err, IsNil)
 
 	c.Assert(err, IsNil)
-	_ = transaction.Put(key2, value2)
+	transaction.Put(key2, value2)
 	v, err := s.db.Get(key)
-	c.Assert(err, IsNil)
 	c.Check(v, DeepEquals, value)
-	err = transaction.Delete(key)
+        err = transaction.Delete(key)
 	c.Assert(err, IsNil)
 
 	_, err = transaction.Get(key2)
@@ -156,3 +155,4 @@ func (s *EtcDDBSuite) TestTransactionCommit(c *C) {
 	_, err = transaction.Get(key)
 	c.Assert(err, NotNil)
 }
+

database/etcddb/storage.go

@@ -145,7 +145,7 @@ func (s *EtcDStorage) Close() error {
 	return err
 }
 
-// Open returns the database
+// Reopen tries to open (re-open) the database
 func (s *EtcDStorage) Open() error {
 	if s.db != nil {
 		return nil

database/etcddb/transaction.go

@@ -67,7 +67,8 @@ func (t *transaction) Commit() (err error) {
 // Discard is safe to call after Commit(), it would be no-op
 func (t *transaction) Discard() {
 	t.ops = []clientv3.Op{}
-	_ = t.tmpdb.Drop()
+	t.tmpdb.Drop()
+	return
 }
 
 // transaction should implement database.Transaction

database/goleveldb/database.go

@@ -9,10 +9,13 @@ import (
 	"github.com/aptly-dev/aptly/database"
 )
 
+const blockSize = 4 * 1024
+
 func internalOpen(path string, throttleCompaction bool) (*leveldb.DB, error) {
 	o := &opt.Options{
 		Filter:                 filter.NewBloomFilter(10),
 		OpenFilesCacheCapacity: 256,
+		BlockSize:              blockSize,
 	}
 
 	if throttleCompaction {
@@ -51,8 +54,8 @@ func RecoverDB(path string) error {
 		return err
 	}
 
-	_ = db.Close()
+	db.Close()
-	_ = stor.Close()
+	stor.Close()
 
 	return nil
 }

database/goleveldb/database_test.go

@@ -119,17 +119,17 @@ func (s *LevelDBSuite) TestDelete(c *C) {
 func (s *LevelDBSuite) TestByPrefix(c *C) {
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{})
 
-	_ = s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
+	s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
-	_ = s.db.Put([]byte{0x80, 0x03}, []byte{0x03})
+	s.db.Put([]byte{0x80, 0x03}, []byte{0x03})
-	_ = s.db.Put([]byte{0x80, 0x02}, []byte{0x02})
+	s.db.Put([]byte{0x80, 0x02}, []byte{0x02})
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x01}, {0x02}, {0x03}})
 	c.Check(s.db.KeysByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x80, 0x01}, {0x80, 0x02}, {0x80, 0x03}})
 
-	_ = s.db.Put([]byte{0x90, 0x01}, []byte{0x04})
+	s.db.Put([]byte{0x90, 0x01}, []byte{0x04})
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x01}, {0x02}, {0x03}})
 	c.Check(s.db.KeysByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x80, 0x01}, {0x80, 0x02}, {0x80, 0x03}})
 
-	_ = s.db.Put([]byte{0x00, 0x01}, []byte{0x05})
+	s.db.Put([]byte{0x00, 0x01}, []byte{0x05})
 	c.Check(s.db.FetchByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x01}, {0x02}, {0x03}})
 	c.Check(s.db.KeysByPrefix([]byte{0x80}), DeepEquals, [][]byte{{0x80, 0x01}, {0x80, 0x02}, {0x80, 0x03}})
 
@@ -161,7 +161,7 @@ func (s *LevelDBSuite) TestHasPrefix(c *C) {
 	c.Check(s.db.HasPrefix([]byte(nil)), Equals, false)
 	c.Check(s.db.HasPrefix([]byte{0x80}), Equals, false)
 
-	_ = s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
+	s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
 
 	c.Check(s.db.HasPrefix([]byte(nil)), Equals, true)
 	c.Check(s.db.HasPrefix([]byte{0x80}), Equals, true)
@@ -180,8 +180,8 @@ func (s *LevelDBSuite) TestBatch(c *C) {
 	c.Assert(err, IsNil)
 
 	batch := s.db.CreateBatch()
-	_ = batch.Put(key2, value2)
+	batch.Put(key2, value2)
-	_ = batch.Delete(key)
+	batch.Delete(key)
 
 	v, err := s.db.Get(key)
 	c.Check(err, IsNil)
@@ -202,9 +202,9 @@ func (s *LevelDBSuite) TestBatch(c *C) {
 }
 
 func (s *LevelDBSuite) TestCompactDB(c *C) {
-	_ = s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
+	s.db.Put([]byte{0x80, 0x01}, []byte{0x01})
-	_ = s.db.Put([]byte{0x80, 0x03}, []byte{0x03})
+	s.db.Put([]byte{0x80, 0x03}, []byte{0x03})
-	_ = s.db.Put([]byte{0x80, 0x02}, []byte{0x02})
+	s.db.Put([]byte{0x80, 0x02}, []byte{0x02})
 
 	c.Check(s.db.CompactDB(), IsNil)
 }

database/goleveldb/storage.go

@@ -16,6 +16,17 @@ type storage struct {
 	db   *leveldb.DB
 }
 
+func (s *storage) GetRecommendedMaxKVSize() int {
+	// The block size configured is not actually a *set* block size, but rather a
+	// *minimum*. LevelDB only checks if a block is full after a new key/value pair is
+	// written, meaning that blocks will tend to overflow a bit.
+	// Therefore, using the default block size as the max value size will ensure
+	// that a new block will only contain a single value and that the size will
+	// only ever be as large as around double the block size (if the block was
+	// nearly full before the new items were added).
+	return blockSize
+}
+
 // CreateTemporary creates new DB of the same type in temp dir
 func (s *storage) CreateTemporary() (database.Storage, error) {
 	tempdir, err := os.MkdirTemp("", "aptly")

deb/changes.go

@@ -60,14 +60,14 @@ func (c *Changes) VerifyAndParse(acceptUnsigned, ignoreSignature bool, verifier
 	if err != nil {
 		return err
 	}
-	defer func() { _ = input.Close() }()
+	defer input.Close()
 
 	isClearSigned, err := verifier.IsClearSigned(input)
 	if err != nil {
 		return err
 	}
 
-	_, _ = input.Seek(0, 0)
+	input.Seek(0, 0)
 
 	if !isClearSigned && !acceptUnsigned {
 		return fmt.Errorf(".changes file is not signed and unsigned processing hasn't been enabled")
@@ -79,7 +79,7 @@ func (c *Changes) VerifyAndParse(acceptUnsigned, ignoreSignature bool, verifier
 		if err != nil {
 			return err
 		}
-		_, _ = input.Seek(0, 0)
+		input.Seek(0, 0)
 
 		c.SignatureKeys = keyInfo.GoodKeys
 	}
@@ -91,7 +91,7 @@ func (c *Changes) VerifyAndParse(acceptUnsigned, ignoreSignature bool, verifier
 		if err != nil {
 			return err
 		}
-		defer func() { _ = text.Close() }()
+		defer text.Close()
 	} else {
 		text = input
 	}
@@ -291,7 +291,8 @@ func CollectChangesFiles(locations []string, reporter aptly.ResultReporter) (cha
 // ImportChangesFiles imports referenced files in changes files into local repository
 func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, acceptUnsigned, ignoreSignatures, forceReplace, noRemoveFiles bool,
 	verifier pgp.Verifier, repoTemplate *template.Template, progress aptly.Progress, localRepoCollection *LocalRepoCollection, packageCollection *PackageCollection,
-	pool aptly.PackagePool, checksumStorageProvider aptly.ChecksumStorageProvider, uploaders *Uploaders, parseQuery parseQuery) (processedFiles []string, failedFiles []string, err error) {
+	reflistCollection *RefListCollection, pool aptly.PackagePool, checksumStorageProvider aptly.ChecksumStorageProvider, uploaders *Uploaders,
+	parseQuery parseQuery) (processedFiles []string, failedFiles []string, err error) {
 
 	for _, path := range changesFiles {
 		var changes *Changes
@@ -307,7 +308,7 @@ func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, ac
 		if err != nil {
 			failedFiles = append(failedFiles, path)
 			reporter.Warning("unable to process file %s: %s", changes.ChangesName, err)
-			_ = changes.Cleanup()
+			changes.Cleanup()
 			continue
 		}
 
@@ -315,7 +316,7 @@ func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, ac
 		if err != nil {
 			failedFiles = append(failedFiles, path)
 			reporter.Warning("unable to process file %s: %s", changes.ChangesName, err)
-			_ = changes.Cleanup()
+			changes.Cleanup()
 			continue
 		}
 
@@ -334,7 +335,7 @@ func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, ac
 		if err != nil {
 			failedFiles = append(failedFiles, path)
 			reporter.Warning("unable to process file %s: %s", changes.ChangesName, err)
-			_ = changes.Cleanup()
+			changes.Cleanup()
 			continue
 		}
 
@@ -354,12 +355,12 @@ func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, ac
 				failedFiles = append(failedFiles, path)
 				reporter.Warning("changes file skipped due to uploaders config: %s, keys %#v: %s",
 					changes.ChangesName, changes.SignatureKeys, err)
-				_ = changes.Cleanup()
+				changes.Cleanup()
 				continue
 			}
 		}
 
-		err = localRepoCollection.LoadComplete(repo)
+		err = localRepoCollection.LoadComplete(repo, reflistCollection)
 		if err != nil {
 			return nil, nil, fmt.Errorf("unable to load repo: %s", err)
 		}
@@ -382,9 +383,9 @@ func ImportChangesFiles(changesFiles []string, reporter aptly.ResultReporter, ac
 			return nil, nil, fmt.Errorf("unable to import package files: %s", err)
 		}
 
-		repo.UpdateRefList(NewPackageRefListFromPackageList(list))
+		repo.UpdateRefList(NewSplitRefListFromPackageList(list))
 
-		err = localRepoCollection.Update(repo)
+		err = localRepoCollection.Update(repo, reflistCollection)
 		if err != nil {
 			return nil, nil, fmt.Errorf("unable to save: %s", err)
 		}

deb/changes_test.go

@@ -21,6 +21,7 @@ type ChangesSuite struct {
 	db                  database.Storage
 	localRepoCollection *LocalRepoCollection
 	packageCollection   *PackageCollection
+	reflistCollection   *RefListCollection
 	packagePool         aptly.PackagePool
 	checksumStorage     aptly.ChecksumStorage
 	progress            aptly.Progress
@@ -42,6 +43,7 @@ func (s *ChangesSuite) SetUpTest(c *C) {
 	s.db, _ = goleveldb.NewOpenDB(c.MkDir())
 	s.localRepoCollection = NewLocalRepoCollection(s.db)
 	s.packageCollection = NewPackageCollection(s.db)
+	s.reflistCollection = NewRefListCollection(s.db)
 
 	s.checksumStorage = files.NewMockChecksumStorage()
 	s.packagePool = files.NewPackagePool(s.Dir, false)
@@ -51,7 +53,7 @@ func (s *ChangesSuite) SetUpTest(c *C) {
 
 func (s *ChangesSuite) TearDownTest(c *C) {
 	s.progress.Shutdown()
-	_ = s.db.Close()
+	s.db.Close()
 }
 
 func (s *ChangesSuite) TestParseAndVerify(c *C) {
@@ -88,7 +90,7 @@ func (s *ChangesSuite) TestCollectChangesFiles(c *C) {
 
 func (s *ChangesSuite) TestImportChangesFiles(c *C) {
 	repo := NewLocalRepo("test", "Test Comment")
-	c.Assert(s.localRepoCollection.Add(repo), IsNil)
+	c.Assert(s.localRepoCollection.Add(repo, s.reflistCollection), IsNil)
 
 	origFailedFiles := []string{
 		"testdata/changes/calamares.changes",
@@ -108,13 +110,13 @@ func (s *ChangesSuite) TestImportChangesFiles(c *C) {
 
 	for _, path := range origFailedFiles {
 		filename := filepath.Join(s.Dir, filepath.Base(path))
-		_ = utils.CopyFile(path, filename)
+		utils.CopyFile(path, filename)
 		expectedFailedFiles = append(expectedFailedFiles, filename)
 	}
 
 	for _, path := range origProcessedFiles {
 		filename := filepath.Join(s.Dir, filepath.Base(path))
-		_ = utils.CopyFile(path, filename)
+		utils.CopyFile(path, filename)
 		expectedProcessedFiles = append(expectedProcessedFiles, filename)
 	}
 
@@ -124,7 +126,8 @@ func (s *ChangesSuite) TestImportChangesFiles(c *C) {
 	processedFiles, failedFiles, err := ImportChangesFiles(
 		append(changesFiles, "testdata/changes/notexistent.changes"),
 		s.Reporter, true, true, false, false, &NullVerifier{},
-		template.Must(template.New("test").Parse("test")), s.progress, s.localRepoCollection, s.packageCollection, s.packagePool, func(database.ReaderWriter) aptly.ChecksumStorage { return s.checksumStorage },
+		template.Must(template.New("test").Parse("test")), s.progress, s.localRepoCollection, s.packageCollection, s.reflistCollection, s.packagePool,
+		func(database.ReaderWriter) aptly.ChecksumStorage { return s.checksumStorage },
 		nil, nil)
 	c.Assert(err, IsNil)
 	c.Check(failedFiles, DeepEquals, append(expectedFailedFiles, "testdata/changes/notexistent.changes"))
@@ -133,7 +136,7 @@ func (s *ChangesSuite) TestImportChangesFiles(c *C) {
 
 func (s *ChangesSuite) TestImportDbgsymWithVersionedSourceField(c *C) {
 	repo := NewLocalRepo("test", "Test Comment")
-	c.Assert(s.localRepoCollection.Add(repo), IsNil)
+	c.Assert(s.localRepoCollection.Add(repo, s.reflistCollection), IsNil)
 
 	changesFiles, failedFiles := CollectChangesFiles(
 		[]string{"testdata/dbgsym-with-source-version"}, s.Reporter)
@@ -142,7 +145,8 @@ func (s *ChangesSuite) TestImportDbgsymWithVersionedSourceField(c *C) {
 
 	_, failedFiles, err := ImportChangesFiles(
 		changesFiles, s.Reporter, true, true, false, true, &NullVerifier{},
-		template.Must(template.New("test").Parse("test")), s.progress, s.localRepoCollection, s.packageCollection, s.packagePool, func(database.ReaderWriter) aptly.ChecksumStorage { return s.checksumStorage },
+		template.Must(template.New("test").Parse("test")), s.progress, s.localRepoCollection, s.packageCollection, s.reflistCollection, s.packagePool,
+		func(database.ReaderWriter) aptly.ChecksumStorage { return s.checksumStorage },
 		nil, nil)
 	c.Assert(err, IsNil)
 	c.Check(failedFiles, IsNil)

deb/checksum_collection_test.go

@@ -28,7 +28,7 @@ func (s *ChecksumCollectionSuite) SetUpTest(c *C) {
 }
 
 func (s *ChecksumCollectionSuite) TearDownTest(c *C) {
-	_ = s.db.Close()
+	s.db.Close()
 }
 
 func (s *ChecksumCollectionSuite) TestFlow(c *C) {

deb/collections.go

@@ -16,6 +16,7 @@ type CollectionFactory struct {
 	snapshots      *SnapshotCollection
 	localRepos     *LocalRepoCollection
 	publishedRepos *PublishedRepoCollection
+	reflists       *RefListCollection
 	checksums      *ChecksumCollection
 }
 
@@ -91,6 +92,17 @@ func (factory *CollectionFactory) PublishedRepoCollection() *PublishedRepoCollec
 	return factory.publishedRepos
 }
 
+func (factory *CollectionFactory) RefListCollection() *RefListCollection {
+	factory.Lock()
+	defer factory.Unlock()
+
+	if factory.reflists == nil {
+		factory.reflists = NewRefListCollection(factory.db)
+	}
+
+	return factory.reflists
+}
+
 // ChecksumCollection returns (or creates) new ChecksumCollection
 func (factory *CollectionFactory) ChecksumCollection(db database.ReaderWriter) aptly.ChecksumStorage {
 	factory.Lock()

deb/deb.go

@@ -17,7 +17,7 @@ import (
 	"github.com/aptly-dev/aptly/pgp"
 	"github.com/kjk/lzma"
 	"github.com/klauspost/compress/zstd"
-	xz "github.com/smira/go-xz"
+	"github.com/smira/go-xz"
 )
 
 // Source kinds
@@ -35,7 +35,7 @@ func GetControlFileFromDeb(packageFile string) (Stanza, error) {
 	if err != nil {
 		return nil, err
 	}
-	defer func() { _ = file.Close() }()
+	defer file.Close()
 
 	library := ar.NewReader(file)
 	for {
@@ -66,14 +66,14 @@ func GetControlFileFromDeb(packageFile string) (Stanza, error) {
 				if err != nil {
 					return nil, errors.Wrapf(err, "unable to ungzip %s from %s", header.Name, packageFile)
 				}
-				defer func() { _ = ungzip.Close() }()
+				defer ungzip.Close()
 				tarInput = ungzip
 			case "control.tar.xz":
 				unxz, err := xz.NewReader(bufReader)
 				if err != nil {
 					return nil, errors.Wrapf(err, "unable to unxz %s from %s", header.Name, packageFile)
 				}
-				defer func() { _ = unxz.Close() }()
+				defer unxz.Close()
 				tarInput = unxz
 			case "control.tar.zst":
 				unzstd, err := zstd.NewReader(bufReader)
@@ -116,10 +116,10 @@ func GetControlFileFromDsc(dscFile string, verifier pgp.Verifier) (Stanza, error
 	if err != nil {
 		return nil, err
 	}
-	defer func() { _ = file.Close() }()
+	defer file.Close()
 
 	isClearSigned, err := verifier.IsClearSigned(file)
-	_, _ = file.Seek(0, 0)
+	file.Seek(0, 0)
 
 	if err != nil {
 		return nil, err
@@ -132,7 +132,7 @@ func GetControlFileFromDsc(dscFile string, verifier pgp.Verifier) (Stanza, error
 		if err != nil {
 			return nil, err
 		}
-		defer func() { _ = text.Close() }()
+		defer text.Close()
 	} else {
 		text = file
 	}
@@ -181,7 +181,7 @@ func GetContentsFromDeb(file io.Reader, packageFile string) ([]string, error) {
 					if err != nil {
 						return nil, errors.Wrapf(err, "unable to ungzip data.tar.gz from %s", packageFile)
 					}
-					defer func() { _ = ungzip.Close() }()
+					defer ungzip.Close()
 					tarInput = ungzip
 				}
 			case "data.tar.bz2":
@@ -191,11 +191,11 @@ func GetContentsFromDeb(file io.Reader, packageFile string) ([]string, error) {
 				if err != nil {
 					return nil, errors.Wrapf(err, "unable to unxz data.tar.xz from %s", packageFile)
 				}
-				defer func() { _ = unxz.Close() }()
+				defer unxz.Close()
 				tarInput = unxz
 			case "data.tar.lzma":
 				unlzma := lzma.NewReader(bufReader)
-				defer func() { _ = unlzma.Close() }()
+				defer unlzma.Close()
 				tarInput = unlzma
 			case "data.tar.zst":
 				unzstd, err := zstd.NewReader(bufReader)

deb/find_dangling.go

@@ -1,45 +0,0 @@
-package deb
-
-import (
-	"errors"
-	"fmt"
-
-	"github.com/aptly-dev/aptly/database"
-)
-
-// FindDanglingReferences finds references that exist in the given PackageRefList, but not in the given PackageCollection.
-// It returns all such references, so they can be removed from the database.
-func FindDanglingReferences(reflist *PackageRefList, packages *PackageCollection) (dangling *PackageRefList, err error) {
-	dangling = &PackageRefList{}
-
-	err = reflist.ForEach(func(key []byte) error {
-		ok, err := isDangling(packages, key)
-		if err != nil {
-			return err
-		}
-
-		if ok {
-			dangling.Refs = append(dangling.Refs, key)
-		}
-
-		return nil
-	})
-	if err != nil {
-		return nil, err
-	}
-
-	return dangling, nil
-}
-
-func isDangling(packages *PackageCollection, key []byte) (bool, error) {
-	_, err := packages.ByKey(key)
-	if errors.Is(err, database.ErrNotFound) {
-		return true, nil
-	}
-
-	if err != nil {
-		return false, fmt.Errorf("get reference %q: %w", key, err)
-	}
-
-	return false, nil
-}

deb/find_dangling_test.go

@@ -1,46 +0,0 @@
-package deb_test
-
-import (
-	"bytes"
-	"testing"
-
-	"github.com/aptly-dev/aptly/database/goleveldb"
-	"github.com/aptly-dev/aptly/deb"
-)
-
-func TestFindDanglingReferences(t *testing.T) {
-	reflist := deb.NewPackageRefList()
-	reflist.Refs = [][]byte{[]byte("P existing 1.2.3"), []byte("P dangling 1.2.3")}
-
-	db, _ := goleveldb.NewOpenDB(t.TempDir())
-	packages := deb.NewPackageCollection(db)
-
-	if err := packages.Update(&deb.Package{Name: "existing", Version: "1.2.3"}); err != nil {
-		t.Fatal(err)
-	}
-
-	dangling, err := deb.FindDanglingReferences(reflist, packages)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	exp := &deb.PackageRefList{
-		Refs: [][]byte{[]byte("P dangling 1.2.3")},
-	}
-
-	compareRefs(t, exp, dangling)
-}
-
-func compareRefs(t *testing.T, exp, got *deb.PackageRefList) {
-	t.Helper()
-
-	if len(exp.Refs) != len(got.Refs) {
-		t.Fatalf("refs length mismatch: exp %d, got %d", len(exp.Refs), len(got.Refs))
-	}
-
-	for i := range exp.Refs {
-		if !bytes.Equal(exp.Refs[i], got.Refs[i]) {
-			t.Fatalf("refs do not match: exp %q, got %q", exp.Refs[i], got.Refs[i])
-		}
-	}
-}

deb/format.go

@@ -288,11 +288,8 @@ func (c *ControlFileReader) ReadStanza() (Stanza, error) {
 			lastField = canonicalCase(parts[0])
 			lastFieldMultiline = isMultilineField(lastField, c.isRelease)
 			if lastFieldMultiline {
-				// Trim trailing whitespace from the inline value so that
+				stanza[lastField] = parts[1]
-				// "Package-List: " does not add empty line
+				if parts[1] != "" {
-				inlineVal := strings.TrimRight(parts[1], " \t")
-				stanza[lastField] = inlineVal
-				if inlineVal != "" {
 					stanza[lastField] += "\n"
 				}
 			} else {

deb/format_test.go

@@ -128,35 +128,6 @@ func (s *ControlFileSuite) TestReadWriteStanza(c *C) {
 	c.Assert(strings.HasPrefix(str, "Package: "), Equals, true)
 }
 
-// Sources may contain "Package-List: " with a trailing space.
-// That trailing space must not be preserved and re-emitted
-// as a spurious blank continuation line when the stanza is written back out.
-func (s *ControlFileSuite) TestPackageListTrailingSpace(c *C) {
-	input := "Package-List: \n" +
-		" bash deb shells required arch=any\n" +
-		" bash-doc deb doc optional arch=all\n"
-
-	r := NewControlFileReader(bytes.NewBufferString(input), false, false)
-	stanza, err := r.ReadStanza()
-	c.Assert(err, IsNil)
-
-	c.Check(stanza["Package-List"], Equals,
-		" bash deb shells required arch=any\n"+
-			" bash-doc deb doc optional arch=all\n")
-
-	buf := &bytes.Buffer{}
-	w := bufio.NewWriter(buf)
-	err = stanza.Copy().WriteTo(w, true, false, false)
-	c.Assert(err, IsNil)
-	c.Assert(w.Flush(), IsNil)
-
-	written := buf.String()
-	c.Assert(strings.Contains(written, "Package-List:\n \n"), Equals, false,
-		Commentf("spurious blank continuation line found in written output:\n%s", written))
-	c.Assert(strings.Contains(written, "Package-List:\n bash"), Equals, true,
-		Commentf("expected Package-List entries not found in written output:\n%s", written))
-}
-
 func (s *ControlFileSuite) TestReadWriteInstallerStanza(c *C) {
 	s.reader = bytes.NewBufferString(installerFile)
 	r := NewControlFileReader(s.reader, false, true)
@@ -192,7 +163,7 @@ func (s *ControlFileSuite) TestCanonicalCase(c *C) {
 func (s *ControlFileSuite) TestLongFields(c *C) {
 	f, err := os.Open("long.stanza")
 	c.Assert(err, IsNil)
-	defer func() { _ = f.Close() }()
+	defer f.Close()
 
 	r := NewControlFileReader(f, false, false)
 	stanza, e := r.ReadStanza()

deb/graph.go

@@ -12,15 +12,15 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 	var err error
 
 	graph := gographviz.NewEscape()
-	_ = graph.SetDir(true)
+	graph.SetDir(true)
-	_ = graph.SetName("aptly")
+	graph.SetName("aptly")
 
 	var labelStart string
 	var labelEnd string
 
 	switch layout {
 	case "vertical":
-		_ = graph.AddAttr("aptly", "rankdir", "LR")
+		graph.AddAttr("aptly", "rankdir", "LR")
 		labelStart = ""
 		labelEnd = ""
 	case "horizontal":
@@ -33,12 +33,12 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 	existingNodes := map[string]bool{}
 
 	err = collectionFactory.RemoteRepoCollection().ForEach(func(repo *RemoteRepo) error {
-		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo)
+		e := collectionFactory.RemoteRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
 
-		_ = graph.AddNode("aptly", repo.UUID, map[string]string{
+		graph.AddNode("aptly", repo.UUID, map[string]string{
 			"shape":     "Mrecord",
 			"style":     "filled",
 			"fillcolor": "darkgoldenrod1",
@@ -55,12 +55,12 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 	}
 
 	err = collectionFactory.LocalRepoCollection().ForEach(func(repo *LocalRepo) error {
-		e := collectionFactory.LocalRepoCollection().LoadComplete(repo)
+		e := collectionFactory.LocalRepoCollection().LoadComplete(repo, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
 
-		_ = graph.AddNode("aptly", repo.UUID, map[string]string{
+		graph.AddNode("aptly", repo.UUID, map[string]string{
 			"shape":     "Mrecord",
 			"style":     "filled",
 			"fillcolor": "mediumseagreen",
@@ -75,13 +75,13 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 		return nil, err
 	}
 
-	_ = collectionFactory.SnapshotCollection().ForEach(func(snapshot *Snapshot) error {
+	collectionFactory.SnapshotCollection().ForEach(func(snapshot *Snapshot) error {
 		existingNodes[snapshot.UUID] = true
 		return nil
 	})
 
 	err = collectionFactory.SnapshotCollection().ForEach(func(snapshot *Snapshot) error {
-		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot)
+		e := collectionFactory.SnapshotCollection().LoadComplete(snapshot, collectionFactory.RefListCollection())
 		if e != nil {
 			return e
 		}
@@ -91,7 +91,7 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 			description = "Snapshot from repo"
 		}
 
-		_ = graph.AddNode("aptly", snapshot.UUID, map[string]string{
+		graph.AddNode("aptly", snapshot.UUID, map[string]string{
 			"shape":     "Mrecord",
 			"style":     "filled",
 			"fillcolor": "cadetblue1",
@@ -103,7 +103,7 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 			for _, uuid := range snapshot.SourceIDs {
 				_, exists := existingNodes[uuid]
 				if exists {
-					_ = graph.AddEdge(uuid, snapshot.UUID, true, nil)
+					graph.AddEdge(uuid, snapshot.UUID, true, nil)
 				}
 			}
 		}
@@ -114,8 +114,8 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 		return nil, err
 	}
 
-	_ = collectionFactory.PublishedRepoCollection().ForEach(func(repo *PublishedRepo) error {
+	collectionFactory.PublishedRepoCollection().ForEach(func(repo *PublishedRepo) error {
-		_ = graph.AddNode("aptly", repo.UUID, map[string]string{
+		graph.AddNode("aptly", repo.UUID, map[string]string{
 			"shape":     "Mrecord",
 			"style":     "filled",
 			"fillcolor": "darkolivegreen1",
@@ -127,7 +127,7 @@ func BuildGraph(collectionFactory *CollectionFactory, layout string) (gographviz
 		for _, uuid := range repo.Sources {
 			_, exists := existingNodes[uuid]
 			if exists {
-				_ = graph.AddEdge(uuid, repo.UUID, true, nil)
+				graph.AddEdge(uuid, repo.UUID, true, nil)
 			}
 		}
 

deb/index_files.go

@@ -59,24 +59,24 @@ func (file *indexFile) Finalize(signer pgp.Signer) error {
 		if file.discardable {
 			return nil
 		}
-		_, _ = file.BufWriter()
+		file.BufWriter()
 	}
 
 	err := file.w.Flush()
 	if err != nil {
-		_ = file.tempFile.Close()
+		file.tempFile.Close()
 		return fmt.Errorf("unable to write to index file: %s", err)
 	}
 
 	if file.compressable {
 		err = utils.CompressFile(file.tempFile, file.onlyGzip || file.parent.skipBz2)
 		if err != nil {
-			_ = file.tempFile.Close()
+			file.tempFile.Close()
 			return fmt.Errorf("unable to compress index file: %s", err)
 		}
 	}
 
-	_ = file.tempFile.Close()
+	file.tempFile.Close()
 
 	exts := []string{""}
 	cksumExts := exts
@@ -220,11 +220,11 @@ func packageIndexByHash(file *indexFile, ext string, hash string, sum string) er
 				// If we managed to resolve the link target: delete it. This is the
 				// oldest physical index file we no longer need. Once we drop our
 				// old symlink we'll essentially forget about it existing at all.
-				_ = file.parent.publishedStorage.Remove(linkTarget)
+				file.parent.publishedStorage.Remove(linkTarget)
 			}
-			_ = file.parent.publishedStorage.Remove(oldIndexPath)
+			file.parent.publishedStorage.Remove(oldIndexPath)
 		}
-		_ = file.parent.publishedStorage.RenameFile(indexPath, oldIndexPath)
+		file.parent.publishedStorage.RenameFile(indexPath, oldIndexPath)
 	}
 
 	// create symlink

deb/list.go

@@ -91,7 +91,7 @@ func NewPackageListWithDuplicates(duplicates bool, capacity int) *PackageList {
 }
 
 // NewPackageListFromRefList loads packages list from PackageRefList
-func NewPackageListFromRefList(reflist *PackageRefList, collection *PackageCollection, progress aptly.Progress) (*PackageList, error) {
+func NewPackageListFromRefList(reflist AnyRefList, collection *PackageCollection, progress aptly.Progress) (*PackageList, error) {
 	// empty reflist
 	if reflist == nil {
 		return NewPackageList(), nil
@@ -438,7 +438,7 @@ func (l *PackageList) Scan(q PackageQuery) (result *PackageList) {
 	result = NewPackageListWithDuplicates(l.duplicatesAllowed, 0)
 	for _, pkg := range l.packages {
 		if q.Matches(pkg) {
-			_ = result.Add(pkg)
+			result.Add(pkg)
 		}
 	}
 
@@ -456,7 +456,7 @@ func (l *PackageList) SearchByKey(arch, name, version string) (result *PackageLi
 
 	pkg := l.packages["P"+arch+" "+name+" "+version]
 	if pkg != nil {
-		_ = result.Add(pkg)
+		result.Add(pkg)
 	}
 
 	return
@@ -598,7 +598,6 @@ func (l *PackageList) Filter(options FilterOptions) (*PackageList, error) {
 					//
 					// when follow-all-variants is enabled, we need to try to expand anyway,
 					// as even if dependency is satisfied now, there might be other ways to satisfy dependency
-					// FIXME: do not search twice
 					if result.Search(dep, false, true) != nil {
 						if options.DependencyOptions&DepVerboseResolve == DepVerboseResolve && options.Progress != nil {
 							options.Progress.ColoredPrintf("@{y}Already satisfied dependency@|: %s with %s", &dep, result.Search(dep, true, true))

deb/list_test.go

@@ -96,7 +96,7 @@ func (s *PackageListSuite) SetUpTest(c *C) {
 		{Name: "dpkg", Version: "1.7", Architecture: "source", SourceArchitecture: "any", IsSource: true, deps: &PackageDependencies{}},
 	}
 	for _, p := range s.packages {
-		_ = s.il.Add(p)
+		s.il.Add(p)
 	}
 	s.il.PrepareIndex()
 
@@ -110,7 +110,7 @@ func (s *PackageListSuite) SetUpTest(c *C) {
 		{Name: "app", Version: "3.0", Architecture: "amd64", deps: &PackageDependencies{PreDepends: []string{"dpkg >= 1.6)"}, Depends: []string{"lib (>> 0.9)", "data (>= 1.0)"}}},
 	}
 	for _, p := range s.packages2 {
-		_ = s.il2.Add(p)
+		s.il2.Add(p)
 	}
 	s.il2.PrepareIndex()
 
@@ -202,8 +202,8 @@ func (s *PackageListSuite) TestRemoveWhenIndexed(c *C) {
 }
 
 func (s *PackageListSuite) TestForeach(c *C) {
-	_ = s.list.Add(s.p1)
+	s.list.Add(s.p1)
-	_ = s.list.Add(s.p3)
+	s.list.Add(s.p3)
 
 	Len := 0
 	err := s.list.ForEach(func(*Package) error {
@@ -232,21 +232,21 @@ func (s *PackageListSuite) TestIndex(c *C) {
 }
 
 func (s *PackageListSuite) TestAppend(c *C) {
-	_ = s.list.Add(s.p1)
+	s.list.Add(s.p1)
-	_ = s.list.Add(s.p3)
+	s.list.Add(s.p3)
 
 	err := s.list.Append(s.il)
 	c.Check(err, IsNil)
 	c.Check(s.list.Len(), Equals, 16)
 
 	list := NewPackageList()
-	_ = list.Add(s.p4)
+	list.Add(s.p4)
 
 	err = s.list.Append(list)
 	c.Check(err, ErrorMatches, "package already exists and is different: .*")
 
 	s.list.PrepareIndex()
-	c.Check(func() { _ = s.list.Append(s.il) }, Panics, "Append not supported when indexed")
+	c.Check(func() { s.list.Append(s.il) }, Panics, "Append not supported when indexed")
 }
 
 func (s *PackageListSuite) TestSearch(c *C) {
@@ -312,7 +312,7 @@ func (s *PackageListSuite) TestSearch(c *C) {
 
 func (s *PackageListSuite) TestFilter(c *C) {
 	c.Check(func() {
-		_, _ = s.list.Filter(FilterOptions{
+		s.list.Filter(FilterOptions{
 			Queries: []PackageQuery{&PkgQuery{"abcd", "0.3", "i386"}},
 		})
 	}, Panics, "list not indexed, can't filter")
@@ -479,7 +479,7 @@ func (s *PackageListSuite) TestVerifyDependencies(c *C) {
 		{Pkg: "mail-agent", Relation: VersionDontCare, Version: "", Architecture: "arm"}})
 
 	for _, p := range s.sourcePackages {
-		_ = s.il.Add(p)
+		s.il.Add(p)
 	}
 
 	missing, err = s.il.VerifyDependencies(DepFollowSource, []string{"i386", "amd64"}, s.il, nil)