This commit introduces major enhancements to the CI/CD pipeline and testing infrastructure:
CI/CD Improvements:
- Consolidated modern and legacy CI workflows into a single comprehensive pipeline
- Removed all publishing functionality from CI (no longer needed)
- Added 8 new advanced testing jobs for pull requests:
* advanced-coverage: Detailed coverage analysis with base branch comparison
* performance-profile: CPU and memory profiling with benchmarks
* fuzz-test: Automated fuzz testing for supported packages
* deep-analysis: Multiple static analysis tools (shadow, ineffassign, gosec, staticcheck)
* mutation-test: Tests effectiveness of test suite on changed files
* dependency-audit: Security vulnerabilities and outdated dependency checks
* stress-test: Race detection with 100 iterations and parallel testing
* test-report-summary: Aggregates all reports into a single PR comment
- Enabled RUN_LONG_TESTS by default for thorough testing
- Added automatic PR comment generation with all test results
Testing Infrastructure:
- Added comprehensive test files across all packages to improve coverage
- Implemented unit tests for previously untested packages
- Added race condition tests for concurrent operations
- Created integration tests for API endpoints
- Added storage backend tests (etcd, goleveldb)
- Implemented command-line interface tests
Local Testing Support:
- Added act configuration for testing GitHub Actions locally
- Created docker-compose.ci.yml for full CI environment simulation
- Updated CONTRIBUTING.md with detailed local testing instructions
Documentation Updates:
- Added comprehensive CI documentation to CONTRIBUTING.md
- Removed obsolete references to Travis CI
- Updated Go version requirements to 1.24
- Added act usage instructions and examples
Other Improvements:
- Updated .gitignore to exclude coverage reports and build artifacts
- Added test-act.yml workflow for testing act functionality
- Created CI_SUMMARY.md documenting all CI capabilities
These changes transform aptly's CI from a basic testing pipeline into a comprehensive quality assurance system that provides immediate feedback on code quality, performance, security, and test effectiveness.
- Upgrade Go version from 1.22 to 1.24 for better performance and security
- Add comprehensive golangci-lint configuration with:
- Enable additional linters: bodyclose, dupl, exportloopref, gocognit, gocritic, gosec, prealloc
- Configure complexity thresholds (cyclomatic: 15, cognitive: 20)
- Set up security scanning with gosec
- Add code quality rules with revive
- Exclude test files from certain strict checks
- Update dependencies to latest stable versions:
- aws-sdk-go-v2: various components updated
- Azure SDK: updated to v1.18.0 for azcore
- etcd client: remains at v3.6.1 for stability
- prometheus client: updated to v1.22.0
- zerolog: updated to v1.34.0
- Enhance logging utilities:
- Add GetLogLevelOrDebug function for flexible log level configuration
- Support "warning" as alias for "warn" level
- Improve error handling for invalid log levels
These changes improve code quality checks, leverage latest Go features,
and ensure dependencies are up-to-date with security patches.
This commit adds extensive test coverage for the recent improvements to the utils
package, achieving 91.5% coverage (up from 76.8%). The tests ensure the reliability
and correctness of critical utility functions.
## Test Files Added
### utils/config_accessor_test.go
Tests for the new safe accessor methods that prevent concurrent map access:
- TestGetFileSystemPublishRoots: Verifies safe copying of FileSystemPublishRoots map
- TestGetS3PublishRoots: Tests S3 publish roots accessor with nil map handling
- TestGetSwiftPublishRoots: Tests Swift publish roots accessor
- TestGetAzurePublishRoots: Tests Azure publish roots accessor
- All tests verify that modifications to returned maps don't affect the original
### utils/sanitize_test.go
Comprehensive tests for the SanitizePath security function:
- TestSanitizePath: Tests various path sanitization scenarios including:
- Path traversal attempts (../)
- Absolute paths (leading /)
- Shell expansion attempts ($, `)
- Environment variable references
- Command injection attempts
- TestSanitizePathSecurity: Focused security tests for malicious inputs
- Ensures dangerous patterns are properly removed
### utils/checksum_extra_test.go
Tests for additional utility functions:
- TestComplete: Verifies ChecksumInfo.Complete() correctly identifies when all
checksums (MD5, SHA1, SHA256, SHA512) are present
- TestSaveConfigRaw: Tests configuration file saving with error handling
- TestPackagePoolStorageUnmarshalJSON: Tests JSON unmarshaling for different
storage types (Azure, Local, S3, Swift)
## Test Characteristics
1. **Comprehensive Coverage**: Tests cover both normal operation and edge cases
2. **Security Focus**: Special attention to security-sensitive functions
3. **Error Handling**: Tests verify proper error handling and edge cases
4. **Concurrency Safety**: Tests ensure thread-safe operations for shared data
5. **Real-world Scenarios**: Test cases based on actual usage patterns
## Testing Approach
All tests use the gopkg.in/check.v1 framework for consistency with the existing
codebase. Tests are designed to be:
- Fast: No external dependencies or network calls
- Deterministic: No random failures or timing dependencies
- Isolated: Each test is independent and can run in any order
- Clear: Test names and assertions clearly indicate what is being tested
These tests provide confidence that the recent race condition fixes and
improvements work correctly and don't introduce regressions.
This commit addresses several critical race conditions and improves the reliability
of etcd operations through better timeout and retry handling.
## Race Condition Fixes
1. **Task Resource Management Bug**
- Fixed incorrect variable usage in task/list.go:78
- Was using completed task's resources instead of idle task's resources
- This caused resource conflicts and potential deadlocks
2. **Database Channel Initialization**
- Added sync.Once pattern to ensure thread-safe channel initialization
- Prevents panic from concurrent access during startup
- Created initDBRequests() function for safe initialization
3. **Published Storage Double-Checked Locking**
- Implemented double-checked locking pattern in GetPublishedStorage
- Reduces lock contention while preventing concurrent initialization
- Improves performance for frequently accessed storage
4. **File Operation Synchronization**
- Created FileLockRegistry in utils/filelock.go
- Prevents concurrent file operations (create, rename, delete, link)
- Implements deadlock prevention for multi-file operations
- Critical for preventing file corruption during parallel publishes
5. **WaitGroup Miscount Prevention**
- Added defer pattern to ensure Done() is always called
- Protects against panics during task execution
- Prevents "negative WaitGroup counter" errors
## etcd Improvements
1. **Timeout Protection**
- Replaced global context.TODO() with per-operation timeout contexts
- Default timeout: 60 seconds (configurable)
- Prevents indefinite hangs when etcd is unresponsive
2. **Environment Variable Configuration**
- APTLY_ETCD_TIMEOUT: Operation timeout (default: 60s)
- APTLY_ETCD_DIAL_TIMEOUT: Connection timeout (default: 60s)
- APTLY_ETCD_KEEPALIVE: Keep-alive timeout (default: 7200s)
- APTLY_ETCD_MAX_MSG_SIZE: Max message size (default: 50MB)
3. **Retry Logic for Read Operations**
- Get operations retry up to 3 times with exponential backoff
- Only retries on temporary/network errors
- Improves reliability without risking data inconsistency
4. **Enhanced Error Logging**
- All etcd errors now logged with operation context
- Replaces silent failures with actionable error messages
- Improves debugging and monitoring capabilities
5. **Increased Message Size Limits**
- Default increased from 10MB to 50MB
- Configurable via environment variable
- Prevents "message too large" errors for large operations
## Testing
- Added comprehensive tests for etcd timeout functionality
- Tests verify context timeout, retry logic, and configuration
- All existing tests pass with the new implementation
## Documentation
- Updated README.rst with etcd configuration section
- Documented all environment variables and their defaults
- Added examples and feature descriptions
These changes significantly improve the reliability and debuggability of aptly
when using etcd as the database backend, while also fixing critical race
conditions that could cause data corruption or service crashes.
This commit addresses critical race conditions that were causing "map write failed"
errors and pod crashes in production environments. The issue occurred when multiple
goroutines accessed shared configuration maps simultaneously without proper synchronization.
Root Cause:
The global utils.Config structure contains several maps (FileSystemPublishRoots,
S3PublishRoots, SwiftPublishRoots, AzurePublishRoots) that were being accessed
directly by concurrent HTTP handlers. While context.Config() uses a mutex, it
returns a pointer to the global config, leaving subsequent map access unprotected.
Changes Made:
1. Added safe accessor methods in utils/config.go:
- GetFileSystemPublishRoots() - returns defensive copy of map
- GetS3PublishRoots() - returns defensive copy of map
- GetSwiftPublishRoots() - returns defensive copy of map
- GetAzurePublishRoots() - returns defensive copy of map
2. Updated API handlers to use safe accessors:
- api/s3.go: apiS3List() now uses GetS3PublishRoots()
- api/router.go: reposListInAPIMode() now uses GetFileSystemPublishRoots()
3. Updated context package storage initialization:
- context/context.go: GetPublishedStorage() now uses safe accessors for all
storage type configurations (filesystem, s3, swift, azure)
Impact:
- Eliminates "concurrent map writes" panics that were causing service instability
- Prevents pod crashes and restarts in Kubernetes environments
- Ensures thread-safe access to configuration maps during concurrent API requests
- Minimal performance overhead (microseconds) from creating map copies
The fix is backward compatible and requires no configuration changes. The defensive
copying approach ensures that even if config maps are modified after initialization
(which shouldn't happen in production), concurrent readers remain safe.
This addresses the production issues observed in lf-aptly-* pods where multiple
parallel publish requests or API calls were triggering race conditions.
When aptly crashes it is possible to get a corrupt database with a dangling key reference.
This results in an error with 'key not found', eg:
ERROR: unable to load package Pall example-package 1.2.3 778cf6f877bf6e2d: key not found
This change makes `db recover` fix this situation by removing the dangling references.
sort both aptly output and gold file. output original output for
debugging on failure.
* Makefile: enable CAPTURE=1 env variable for capturing gold files
* docker-system-test: use AWS env vars for S3 tests
* fix system tests timing issue with order of gpg logs in publish tests
Nick Bozhenko
André Roth
dependabot[bot]
boxjan
Silke Hofstra
Christoph Fiehe
Sébastien Delafond
Mikel Olasagasti Uranga