Files
aptly/deb/local.go
Ryan Gonzalez 19a705f80d Split reflists to share their contents across snapshots
In current aptly, each repository and snapshot has its own reflist in
the database. This brings a few problems with it:

- Given a sufficiently large repositories and snapshots, these lists can
  get enormous, reaching >1MB. This is a problem for LevelDB's overall
  performance, as it tends to prefer values around the confiruged block
  size (defaults to just 4KiB).
- When you take these large repositories and snapshot them, you have a
  full, new copy of the reflist, even if only a few packages changed.
  This means that having a lot of snapshots with a few changes causes
  the database to basically be full of largely duplicate reflists.
- All the duplication also means that many of the same refs are being
  loaded repeatedly, which can cause some slowdown but, more notably,
  eats up huge amounts of memory.
- Adding on more and more new repositories and snapshots will cause the
  time and memory spent on things like cleanup and publishing to grow
  roughly linearly.

At the core, there are two problems here:

- Reflists get very big because there are just a lot of packages.
- Different reflists can tend to duplicate much of the same contents.

*Split reflists* aim at solving this by separating reflists into 64
*buckets*. Package refs are sorted into individual buckets according to
the following system:

- Take the first 3 letters of the package name, after dropping a `lib`
  prefix. (Using only the first 3 letters will cause packages with
  similar prefixes to end up in the same bucket, under the assumption
  that packages with similar names tend to be updated together.)
- Take the 64-bit xxhash of these letters. (xxhash was chosen because it
  relatively good distribution across the individual bits, which is
  important for the next step.)
- Use the first 6 bits of the hash (range [0:63]) as an index into the
  buckets.

Once refs are placed in buckets, a sha256 digest of all the refs in the
bucket is taken. These buckets are then stored in the database, split
into roughly block-sized segments, and all the repositories and
snapshots simply store an array of bucket digests.

This approach means that *repositories and snapshots can share their
reflist buckets*. If a snapshot is taken of a repository, it will have
the same contents, so its split reflist will point to the same buckets
as the base repository, and only one copy of each bucket is stored in
the database. When some packages in the repository change, only the
buckets containing those packages will be modified; all the other
buckets will remain unchanged, and thus their contents will still be
shared. Later on, when these reflists are loaded, each bucket is only
loaded once, short-cutting loaded many megabytes of data. In effect,
split reflists are essentially copy-on-write, with only the changed
buckets stored individually.

Changing the disk format means that a migration needs to take place, so
that task is moved into the database cleanup step, which will migrate
reflists over to split reflists, as well as delete any unused reflist
buckets.

All the reflist tests are also changed to additionally test out split
reflists; although the internal logic is all shared (since buckets are,
themselves, just normal reflists), some special additions are needed to
have native versions of the various reflist helper methods.

In our tests, we've observed the following improvements:

- Memory usage during publish and database cleanup, with
  `GOMEMLIMIT=2GiB`, goes down from ~3.2GiB (larger than the memory
  limit!) to ~0.7GiB, a decrease of ~4.5x.
- Database size decreases from 1.3GB to 367MB.

*In my local tests*, publish times had also decreased down to mere
seconds but the same effect wasn't observed on the server, with the
times staying around the same. My suspicions are that this is due to I/O
performance: my local system is an M1 MBP, which almost certainly has
much faster disk speeds than our DigitalOcean block volumes. Split
reflists include a side effect of requiring more random accesses from
reading all the buckets by their keys, so if your random I/O
performance is slower, it might cancel out the benefits. That being
said, even in that case, the memory usage and database size advantages
still persist.

Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
2025-02-15 23:49:21 +01:00

249 lines
6.0 KiB
Go

package deb
import (
"bytes"
"errors"
"fmt"
"log"
"github.com/aptly-dev/aptly/database"
"github.com/google/uuid"
"github.com/ugorji/go/codec"
)
// LocalRepo is a collection of packages created locally
type LocalRepo struct {
// Permanent internal ID
UUID string `codec:"UUID" json:"-"`
// User-assigned name
Name string
// Comment
Comment string
// DefaultDistribution
DefaultDistribution string `codec:",omitempty"`
// DefaultComponent
DefaultComponent string `codec:",omitempty"`
// Uploaders configuration
Uploaders *Uploaders `codec:"Uploaders,omitempty" json:"-"`
// "Snapshot" of current list of packages
packageRefs *SplitRefList
}
// NewLocalRepo creates new instance of Debian local repository
func NewLocalRepo(name string, comment string) *LocalRepo {
return &LocalRepo{
UUID: uuid.NewString(),
Name: name,
Comment: comment,
}
}
// String interface
func (repo *LocalRepo) String() string {
if repo.Comment != "" {
return fmt.Sprintf("[%s]: %s", repo.Name, repo.Comment)
}
return fmt.Sprintf("[%s]", repo.Name)
}
// NumPackages return number of packages in local repo
func (repo *LocalRepo) NumPackages() int {
return repo.packageRefs.Len()
}
// RefList returns package list for repo
func (repo *LocalRepo) RefList() *SplitRefList {
return repo.packageRefs
}
// UpdateRefList changes package list for local repo
func (repo *LocalRepo) UpdateRefList(sl *SplitRefList) {
repo.packageRefs = sl
}
// Encode does msgpack encoding of LocalRepo
func (repo *LocalRepo) Encode() []byte {
var buf bytes.Buffer
encoder := codec.NewEncoder(&buf, &codec.MsgpackHandle{})
encoder.Encode(repo)
return buf.Bytes()
}
// Decode decodes msgpack representation into LocalRepo
func (repo *LocalRepo) Decode(input []byte) error {
decoder := codec.NewDecoderBytes(input, &codec.MsgpackHandle{})
return decoder.Decode(repo)
}
// Key is a unique id in DB
func (repo *LocalRepo) Key() []byte {
return []byte("L" + repo.UUID)
}
// RefKey is a unique id for package reference list
func (repo *LocalRepo) RefKey() []byte {
return []byte("E" + repo.UUID)
}
// LocalRepoCollection does listing, updating/adding/deleting of LocalRepos
type LocalRepoCollection struct {
db database.Storage
cache map[string]*LocalRepo
}
// NewLocalRepoCollection loads LocalRepos from DB and makes up collection
func NewLocalRepoCollection(db database.Storage) *LocalRepoCollection {
return &LocalRepoCollection{
db: db,
cache: make(map[string]*LocalRepo),
}
}
func (collection *LocalRepoCollection) search(filter func(*LocalRepo) bool, unique bool) []*LocalRepo {
result := []*LocalRepo(nil)
for _, r := range collection.cache {
if filter(r) {
result = append(result, r)
}
}
if unique && len(result) > 0 {
return result
}
collection.db.ProcessByPrefix([]byte("L"), func(_, blob []byte) error {
r := &LocalRepo{}
if err := r.Decode(blob); err != nil {
log.Printf("Error decoding local repo: %s\n", err)
return nil
}
if filter(r) {
if _, exists := collection.cache[r.UUID]; !exists {
collection.cache[r.UUID] = r
result = append(result, r)
if unique {
return errors.New("abort")
}
}
}
return nil
})
return result
}
// Add appends new repo to collection and saves it
func (collection *LocalRepoCollection) Add(repo *LocalRepo, reflistCollection *RefListCollection) error {
_, err := collection.ByName(repo.Name)
if err == nil {
return fmt.Errorf("local repo with name %s already exists", repo.Name)
}
err = collection.Update(repo, reflistCollection)
if err != nil {
return err
}
collection.cache[repo.UUID] = repo
return nil
}
// Update stores updated information about repo in DB
func (collection *LocalRepoCollection) Update(repo *LocalRepo, reflistCollection *RefListCollection) error {
batch := collection.db.CreateBatch()
batch.Put(repo.Key(), repo.Encode())
if repo.packageRefs != nil {
bc := reflistCollection.NewBatch(batch)
reflistCollection.UpdateInBatch(repo.packageRefs, repo.RefKey(), bc)
}
return batch.Write()
}
// LoadComplete loads additional information for local repo
func (collection *LocalRepoCollection) LoadComplete(repo *LocalRepo, reflistCollection *RefListCollection) error {
repo.packageRefs = NewSplitRefList()
err := reflistCollection.LoadComplete(repo.packageRefs, repo.RefKey())
if err == database.ErrNotFound {
return nil
}
return err
}
// ByName looks up repository by name
func (collection *LocalRepoCollection) ByName(name string) (*LocalRepo, error) {
result := collection.search(func(r *LocalRepo) bool { return r.Name == name }, true)
if len(result) == 0 {
return nil, fmt.Errorf("local repo with name %s not found", name)
}
return result[0], nil
}
// ByUUID looks up repository by uuid
func (collection *LocalRepoCollection) ByUUID(uuid string) (*LocalRepo, error) {
if r, ok := collection.cache[uuid]; ok {
return r, nil
}
key := (&LocalRepo{UUID: uuid}).Key()
value, err := collection.db.Get(key)
if err == database.ErrNotFound {
return nil, fmt.Errorf("local repo with uuid %s not found", uuid)
}
if err != nil {
return nil, err
}
r := &LocalRepo{}
err = r.Decode(value)
if err == nil {
collection.cache[r.UUID] = r
}
return r, err
}
// ForEach runs method for each repository
func (collection *LocalRepoCollection) ForEach(handler func(*LocalRepo) error) error {
return collection.db.ProcessByPrefix([]byte("L"), func(_, blob []byte) error {
r := &LocalRepo{}
if err := r.Decode(blob); err != nil {
log.Printf("Error decoding repo: %s\n", err)
return nil
}
return handler(r)
})
}
// Len returns number of remote repos
func (collection *LocalRepoCollection) Len() int {
return len(collection.db.KeysByPrefix([]byte("L")))
}
// Drop removes remote repo from collection
func (collection *LocalRepoCollection) Drop(repo *LocalRepo) error {
if _, err := collection.db.Get(repo.Key()); err != nil {
if err == database.ErrNotFound {
return errors.New("local repo not found")
}
return err
}
delete(collection.cache, repo.UUID)
batch := collection.db.CreateBatch()
batch.Delete(repo.Key())
batch.Delete(repo.RefKey())
return batch.Write()
}