mirror of
https://github.com/aptly-dev/aptly.git
synced 2026-06-19 07:40:20 +00:00
Nick Bozhenko
463c34a38e
This commit addresses several critical race conditions and improves the reliability of etcd operations through better timeout and retry handling. ## Race Condition Fixes 1. **Task Resource Management Bug** - Fixed incorrect variable usage in task/list.go:78 - Was using completed task's resources instead of idle task's resources - This caused resource conflicts and potential deadlocks 2. **Database Channel Initialization** - Added sync.Once pattern to ensure thread-safe channel initialization - Prevents panic from concurrent access during startup - Created initDBRequests() function for safe initialization 3. **Published Storage Double-Checked Locking** - Implemented double-checked locking pattern in GetPublishedStorage - Reduces lock contention while preventing concurrent initialization - Improves performance for frequently accessed storage 4. **File Operation Synchronization** - Created FileLockRegistry in utils/filelock.go - Prevents concurrent file operations (create, rename, delete, link) - Implements deadlock prevention for multi-file operations - Critical for preventing file corruption during parallel publishes 5. **WaitGroup Miscount Prevention** - Added defer pattern to ensure Done() is always called - Protects against panics during task execution - Prevents "negative WaitGroup counter" errors ## etcd Improvements 1. **Timeout Protection** - Replaced global context.TODO() with per-operation timeout contexts - Default timeout: 60 seconds (configurable) - Prevents indefinite hangs when etcd is unresponsive 2. **Environment Variable Configuration** - APTLY_ETCD_TIMEOUT: Operation timeout (default: 60s) - APTLY_ETCD_DIAL_TIMEOUT: Connection timeout (default: 60s) - APTLY_ETCD_KEEPALIVE: Keep-alive timeout (default: 7200s) - APTLY_ETCD_MAX_MSG_SIZE: Max message size (default: 50MB) 3. **Retry Logic for Read Operations** - Get operations retry up to 3 times with exponential backoff - Only retries on temporary/network errors - Improves reliability without risking data inconsistency 4. **Enhanced Error Logging** - All etcd errors now logged with operation context - Replaces silent failures with actionable error messages - Improves debugging and monitoring capabilities 5. **Increased Message Size Limits** - Default increased from 10MB to 50MB - Configurable via environment variable - Prevents "message too large" errors for large operations ## Testing - Added comprehensive tests for etcd timeout functionality - Tests verify context timeout, retry logic, and configuration - All existing tests pass with the new implementation ## Documentation - Updated README.rst with etcd configuration section - Documented all environment variables and their defaults - Added examples and feature descriptions These changes significantly improve the reliability and debuggability of aptly when using etcd as the database backend, while also fixing critical race conditions that could cause data corruption or service crashes.
Downloaded packages
SHA-256
For each uploaded Debian package a SHA-256 checksum is computed. This checksum is used to create a file tree where each package will reside, with following hierarchy.
Directory and sub-directories structure
- 1st and 2nd characters of SHA-256 checksum as sub-directory of rootDir/pool directory.
- 3rd and 4th characters of SHA-256 checksum as sub-directory of the former
ex:
sha256sum 476e0cdac6bc757dd2b78bacc1325323b09c45ecb41d4562deec2a1c7c148405 my-package_1.2.3_all.deb
${rootDir}/pool # rootDir defined in aptly.conf
└── 47
└── 6e
Filename
The following items are concatenated to form the filename under which package is stored.
- 5th to the 31st characters of SHA-256 checksum
- "_" (undescore)
- filename of uploaded Debian as defined in Debian package file names
ex:
sha256sum 476e0cdac6bc757dd2b78bacc1325323b09c45ecb41d4562deec2a1c7c148405 my-package_1.2.3_all.deb
0cdac6bc757dd2b78bacc13253_my-package_1.2.3_all.deb
MD5
For each uploaded Debian package a MD5 checksum is computed. This checksum is used to create a file tree where each package will reside, with following hierarchy
Note: MD5 is only legacy layout. Its support is limited to 'read' files from the pool, it never puts files this way for new package files.
Directory and sub-directories structure
- 1st and 2nd characters of MD5 checksum as sub-directory name of rootDir/pool directory
- 3rd and 4th characters of MD5 chacksum as sub-directory name of the former
ex:
md5sum feea3c0c3e823615bf2d417b052a96b4 my-package_1.2.3_all.deb
${rootDir}/pool # rootDir defined in aptly.conf
└── fe
└── ea
Filename
Uploaded Debian is stored as-is and not renamed.
Example
${rootDir}/pool # rootDir defined in aptly.conf
├── 00
│ ├── 25
│ │ └── yet_another_package-0.6.0_all.deb
│ ├── 60
│ ├── 97
│ │ └── 80ced73165f92fea490f2561a7c4_my-package_0.0.1_all.deb
│ ├── 6e
│ │ └── 0cdac6bc757dd2b78bacc13253_my-package_1.2.3_all.deb # sha256sum 476e0cdac6bc757dd2b78bacc1325323b09c45ecb41d4562deec2a1c7c148405
│ └── db
│ └── yet_another_package-0.5.8_all.deb # md5sum 00db7ada61aa28a6931267f1714cbb15
...
├── 2a
│ ├── 10
│ │ └── yet_another_package-0.5.9_all.deb
│ ├── 64
│ │ └── 80ced73165f92fea490f2561a7c4_my-other-package_2.3.2_amd64.deb
│ ├── 4c
│ ├── 5c
│ │ └── yet_another_package-0.6.1_all.deb
│ ├── 77
│ ├── b5
│ │ └── 4b2eb349236cf5c4af7eca68a43b_my-package_0.2.0_amd64.deb
...
└── ff
├── 4c
├── 5a
│ └── 8868dd8661bbe25c51bdd9b2d25c_my-package_0.2.0_amd64.deb
└── dc