- Replace panic with proper error handling in batch Write() method
- Add retry logic with exponential backoff (up to 3 retries by default)
- Implement isRetryableError() to identify transient failures
- Add comprehensive error logging with retry information
- Return formatted errors after exhausting all retries
This prevents pod crashes from etcd timeout errors by gracefully
handling transient failures and returning errors to the caller.
This commit introduces major enhancements to the CI/CD pipeline and testing infrastructure:
CI/CD Improvements:
- Consolidated modern and legacy CI workflows into a single comprehensive pipeline
- Removed all publishing functionality from CI (no longer needed)
- Added 8 new advanced testing jobs for pull requests:
* advanced-coverage: Detailed coverage analysis with base branch comparison
* performance-profile: CPU and memory profiling with benchmarks
* fuzz-test: Automated fuzz testing for supported packages
* deep-analysis: Multiple static analysis tools (shadow, ineffassign, gosec, staticcheck)
* mutation-test: Tests effectiveness of test suite on changed files
* dependency-audit: Security vulnerabilities and outdated dependency checks
* stress-test: Race detection with 100 iterations and parallel testing
* test-report-summary: Aggregates all reports into a single PR comment
- Enabled RUN_LONG_TESTS by default for thorough testing
- Added automatic PR comment generation with all test results
Testing Infrastructure:
- Added comprehensive test files across all packages to improve coverage
- Implemented unit tests for previously untested packages
- Added race condition tests for concurrent operations
- Created integration tests for API endpoints
- Added storage backend tests (etcd, goleveldb)
- Implemented command-line interface tests
Local Testing Support:
- Added act configuration for testing GitHub Actions locally
- Created docker-compose.ci.yml for full CI environment simulation
- Updated CONTRIBUTING.md with detailed local testing instructions
Documentation Updates:
- Added comprehensive CI documentation to CONTRIBUTING.md
- Removed obsolete references to Travis CI
- Updated Go version requirements to 1.24
- Added act usage instructions and examples
Other Improvements:
- Updated .gitignore to exclude coverage reports and build artifacts
- Added test-act.yml workflow for testing act functionality
- Created CI_SUMMARY.md documenting all CI capabilities
These changes transform aptly's CI from a basic testing pipeline into a comprehensive quality assurance system that provides immediate feedback on code quality, performance, security, and test effectiveness.
This commit addresses several critical race conditions and improves the reliability
of etcd operations through better timeout and retry handling.
## Race Condition Fixes
1. **Task Resource Management Bug**
- Fixed incorrect variable usage in task/list.go:78
- Was using completed task's resources instead of idle task's resources
- This caused resource conflicts and potential deadlocks
2. **Database Channel Initialization**
- Added sync.Once pattern to ensure thread-safe channel initialization
- Prevents panic from concurrent access during startup
- Created initDBRequests() function for safe initialization
3. **Published Storage Double-Checked Locking**
- Implemented double-checked locking pattern in GetPublishedStorage
- Reduces lock contention while preventing concurrent initialization
- Improves performance for frequently accessed storage
4. **File Operation Synchronization**
- Created FileLockRegistry in utils/filelock.go
- Prevents concurrent file operations (create, rename, delete, link)
- Implements deadlock prevention for multi-file operations
- Critical for preventing file corruption during parallel publishes
5. **WaitGroup Miscount Prevention**
- Added defer pattern to ensure Done() is always called
- Protects against panics during task execution
- Prevents "negative WaitGroup counter" errors
## etcd Improvements
1. **Timeout Protection**
- Replaced global context.TODO() with per-operation timeout contexts
- Default timeout: 60 seconds (configurable)
- Prevents indefinite hangs when etcd is unresponsive
2. **Environment Variable Configuration**
- APTLY_ETCD_TIMEOUT: Operation timeout (default: 60s)
- APTLY_ETCD_DIAL_TIMEOUT: Connection timeout (default: 60s)
- APTLY_ETCD_KEEPALIVE: Keep-alive timeout (default: 7200s)
- APTLY_ETCD_MAX_MSG_SIZE: Max message size (default: 50MB)
3. **Retry Logic for Read Operations**
- Get operations retry up to 3 times with exponential backoff
- Only retries on temporary/network errors
- Improves reliability without risking data inconsistency
4. **Enhanced Error Logging**
- All etcd errors now logged with operation context
- Replaces silent failures with actionable error messages
- Improves debugging and monitoring capabilities
5. **Increased Message Size Limits**
- Default increased from 10MB to 50MB
- Configurable via environment variable
- Prevents "message too large" errors for large operations
## Testing
- Added comprehensive tests for etcd timeout functionality
- Tests verify context timeout, retry logic, and configuration
- All existing tests pass with the new implementation
## Documentation
- Updated README.rst with etcd configuration section
- Documented all environment variables and their defaults
- Added examples and feature descriptions
These changes significantly improve the reliability and debuggability of aptly
when using etcd as the database backend, while also fixing critical race
conditions that could cause data corruption or service crashes.
goxc fails with:
Error: database/etcddb/database.go:17:25: cannot use 2048 * 1024 * 1024 (untyped int constant 2147483648) as int value in struct literal (overflows)
For any action which is multi-step (requires updating more than 1 DB
key), use transaction to make update atomic.
Also pack big chunks of updates (importing packages for importing and
mirror updates) into single transaction to improve aptly performance and
get some isolation.
Note that still layers up (Collections) provide some level of isolation,
so this is going to shine with the future PRs to remove collection
locks.
Spin-off of #459
This is spin-off of changes from #459.
Transactions are not being used yet, but batches are updated to work
with the new API.
`database/` package was refactored to split abstract interfaces and
implementation via goleveldb. This should make it easier to implement
new database types.
Allow database to be initialized without opening, unify all the
open paths to retry on failure.
In API router make sure open requests are matched with acks in explicit
way.
This also enables re-open attempts in all the aptly commands, so it
should make running aptly CLI much easier now hopefully.
Fix up system tests for oldoldstable ;)
Nick Bozhenko
André Roth
Mikel Olasagasti Uranga
hudeng
Markus Muellner
Oliver Sauder
Andrey Smirnov
Chris Read
Sebastien Binet