vincent/aptly
1
0
Fork 0
mirror of https://github.com/aptly-dev/aptly.git synced 2026-01-12 03:21:33 +00:00
Code Issues Projects Releases Wiki Activity

Use parallel gzip instead of gzip for compression

Browse Source 
golangs compress/gzip isn't a parallel implementation, so it's quite a
bit slower on most modern servers then pgzip. The below benchmark
run shows that publishing a debian bullseye mirror snapshot (amd64, arm64,
armhf, source) shows a gain of about 35% in publishing time (when skipping
bz2 using MR #1081)

```
 hyperfine -w 1 -m 3 -L aptly aptly-nobz2,aptly-nobz2-pgzip -p "{aptly} -config aptly.conf publish drop bullseye || true" "{aptly} -config aptly.conf  publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye"
Benchmark 1: aptly-nobz2 -config aptly.conf  publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye
  Time (mean ± σ):     35.548 s ±  0.378 s    [User: 39.465 s, System: 10.046 s]
  Range (min … max):   35.149 s … 35.902 s    3 runs

Benchmark 2: aptly-nobz2-pgzip -config aptly.conf  publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye
  Time (mean ± σ):     26.592 s ±  0.069 s    [User: 42.207 s, System: 9.676 s]
  Range (min … max):   26.521 s … 26.660 s    3 runs

Summary
  'aptly-nobz2-pgzip -config aptly.conf  publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye' ran
    1.34 ± 0.01 times faster than 'aptly-nobz2 -config aptly.conf  publish snapshot --skip-bz2=true --skip-contents --skip-signing bullseye'
```

Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>
This commit is contained in:
Sjoerd Simons
2022-06-18 19:58:09 +02:00
committed by Benj Fassbind
parent 26254a0ad8
commit 2aca913e92
3 changed files with 6 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
utils/compress.go
View File

@@ -1,10 +1,11 @@
package utils
import (
"compress/gzip"
"io"
"os"
"os/exec"
"github.com/klauspost/pgzip"
)
// CompressFile compresses file specified by source to .gz & .bz2
@@ -19,7 +20,7 @@ func CompressFile(source *os.File, onlyGzip bool) error {
}
defer gzFile.Close()
gzWriter := gzip.NewWriter(gzFile)
gzWriter := pgzip.NewWriter(gzFile)
defer gzWriter.Close()
source.Seek(0, 0)