Fix S3 concurrent map writes causing pod crashes

PROBLEM:
- Pod crashes with "fatal error: concurrent map writes" during S3 publications
- Root cause: pathCache map in PublishedStorage accessed without synchronization
- Occurs during concurrent LinkFromPool operations in S3 publishing

SOLUTION:
- Add sync.RWMutex to PublishedStorage struct for thread-safe map access
- Implement double-check locking pattern for cache initialization
- Protect all map operations (read/write/delete) with appropriate locks

CHANGES:
- s3/public.go: Add pathCacheMutex field and protect all map operations
  * Cache initialization with double-check locking in LinkFromPool
  * Read operations protected with RLock/RUnlock
  * Write operations protected with Lock/Unlock
  * Delete operations in Remove() and RemoveDirs() protected

IMPACT:
- Eliminates concurrent map writes panic
- Prevents pod crashes during S3 publications
- Maintains performance with minimal synchronization overhead
- Uses read-write locks allowing concurrent reads while serializing writes
This commit is contained in:
Nick Bozhenko
2025-07-15 22:46:37 -04:00
parent 308fb80a6e
commit 1693863499
102 changed files with 867 additions and 28632 deletions
+7 -17
View File
@@ -7,7 +7,6 @@ import (
"sort"
"strconv"
"strings"
"sync"
"sync/atomic"
"github.com/aptly-dev/aptly/aptly"
@@ -101,18 +100,7 @@ type dbRequest struct {
err chan<- error
}
var (
dbRequests chan dbRequest
dbRequestsOnce sync.Once
)
// initDBRequests initializes the database request channel in a thread-safe manner
func initDBRequests() {
dbRequestsOnce.Do(func() {
dbRequests = make(chan dbRequest, 1)
go acquireDatabase()
})
}
var dbRequests chan dbRequest
// Acquire database lock and release it when not needed anymore.
//
@@ -151,8 +139,9 @@ func acquireDatabase() {
// runTaskInBackground to run a task which accquire database.
// Important do not forget to defer to releaseDatabaseConnection
func acquireDatabaseConnection() error {
// Ensure channel is initialized
initDBRequests()
if dbRequests == nil {
return nil
}
errCh := make(chan error)
dbRequests <- dbRequest{acquiredb, errCh}
@@ -162,8 +151,9 @@ func acquireDatabaseConnection() error {
// Release database connection when not needed anymore
func releaseDatabaseConnection() error {
// Ensure channel is initialized
initDBRequests()
if dbRequests == nil {
return nil
}
errCh := make(chan error)
dbRequests <- dbRequest{releasedb, errCh}