During sync, `git checkout` can trigger fetch for missing objects in
partial clones. This internal fetch can trigger `git maintenance` or
`git gc` and cause delays during the local checkout phase. Set
maintenance.auto to false and gc.auto to 0 in during `_InitRemote` if
`depth=1` to ensure that implicit fetches spawned by git skip GC.
Bug: 379111283
Change-Id: I6b22a4867f29b6e9598746cb752820a84dc2aeb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/540681
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
While we support running `repo` on clients with older Python versions,
we don't need to hold the runners & release code back. These are only
used by repo devs on their systems to develop & release repo.
Python 3.9 was picked due to its typing changs which we've already
started using in this code.
Change-Id: I6f8885c84298760514c25abeb1fccb0338947bf4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/539801
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This hasn't been working out as well as we'd hope. Tox relies on
the system having Python versions installed which distros don't
tend to carry anymore. Our custom run_tests leverages vpython
when possible to run stable Python 3.8 & 3.11 versions which is
providing an OK level of coverage in practice.
Change-Id: Ida517f7be47ca95703e43bc0af5a24dd70c0467e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/540001
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Most users do not care about the manifest history in .repo/manifests/.
Let's change the default to 1 so things work smoothly for most people
most of the time. For the rare folks who want the full history, they
can add --manifest-depth=0 to their `repo init`.
This has no effect on existing checkouts.
Spot checking Android & CrOS manifests shows significant speedups.
Full history can take O(10's seconds) to O(minutes) while depth of 1
takes constant time of O(~5 seconds).
Bug: 468033850
Change-Id: I4b8ed62a8a636babcc5226552badb69600d0c353
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/535481
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
The gitdir for a project may be left in a state with bare=true due
to a previous failed sync. In this state, during a subsequent sync
attempt, repo will skip initializing the gitdir (since the directory
already exists) and directly attempt to checkout the worktree, which
will fail because the project is bare. To reduce the chance of this
happening, initialize the gitdir in a temp directory and move it once
it is ready.
Bug: 457478027
Change-Id: I4767494a3a54e7734174eae3a0d939fa9d174288
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/524203
Tested-by: Kaushik Lingarkar <kaushikl@qti.qualcomm.com>
Commit-Queue: Kaushik Lingarkar <kaushikl@qti.qualcomm.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This avoids copying/linking the same file/link multiple times if a
copyfile/linkfile element with the same values has been specifed
multiple times. This can happen when including a common manifest that
uses an extend-project element that has a copyfile/linkfile element.
This uses dicts rather than sets to store the copyfiles and linkfiles to
make sure the order they are specified in the manifest is maintained.
For Python 3.7+, maintaining the order that keys are added to dicts is
guaranteed, and for Python 3.6 it happened to be true.
The _CopyFile class and the _LinkFile class are changed to inherit from
NamedTuple to be able to store them in dicts.
Change-Id: I9f5a80298b875251a81c5fe7d353e262d104fae4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/525322
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Commit-Queue: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
This new command allows users to delete projects from the worktree
and from the `.repo` directory. It is a destructive operation.
It handles shared projects by refusing to wipe them unless the
`--force` flag is used. It also checks for uncommitted changes
before wiping.
Bug: 393383056
Change-Id: Ia30d8ffdc781a3f179af56310ce31c9dae331bbe
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/490801
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Interleaved sync didn't save _fetch_times and _local_sync_state to disk.
Phased sync saved them, but incorrectly applied moving average smoothing
repeatedly when fetching submodules, and discarded historical data
during partial syncs.
Move .Save() calls to the end of main sync loops to ensure they run
once. Update _FetchTimes.Save() to merge new data with existing history,
preventing data loss.
Change-Id: I174f98a62ac86859f1eeea1daba65eb35c227852
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/519821
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
'repo sync --rebase' should do a rebase if it encounters local commits
during a 'repo sync'.
This was broken by
https://gerrit-review.git.corp.google.com/c/git-repo/+/437421,
which caused this to execute the '_doff' hook (which stands for
'do fast forward'), which is implemented using 'git merge --no-stat'.
This caused *multiple* actual editor windows to pop up (*) during
'repo sync --rebase', asking the user to enter a commit message for the
merge.
In this CL I explicitly make that code path do a 'git rebase'.
(*) and if you use a terminal editor like 'vim', this means you have 2+ concurrent vim windows rendered in the same terminal, while 'repo sync' keeps on printing other output lines, again in the same terminal. The result is .... not pretty to say the least :(
Bug: b:434565811
Test: Used it myself for over a week.
Change-Id: I0bf3ff181f15b9d5b2e3f85f7f84e302139fdab7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/518602
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jeroen Dhollander <jeroendh@google.com>
Commit-Queue: Jeroen Dhollander <jeroendh@google.com>
With the introduction of interleaved sync mode, the submodule activation
logic broke because the 'has_submodules' attribute was no longer being
populated when needed. With this change, each submodule is initialized
when it enters the Sync_LocalHalf stage, whereas previously all
submodules were initialized at once when the parent repository entered
the Sync_LocalHalf stage. The init is now retried if it fails, as
submodules may concurrently modify the parent’s git config, potentially
causing contention when attempting to obtain a lock on it.
This change makes the submodule activation logic more robust and less
prone to breakage.
Bug: 444366154
Change-Id: I25eca4ea2a6868219045cfa088988eb01ded47d2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/509041
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Kaushik Lingarkar <kaushikl@qti.qualcomm.com>
Reviewed-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
Commit-Queue: Kaushik Lingarkar <kaushikl@qti.qualcomm.com>
Reviewed-by: Scott Lee <ddoman@google.com>
When callback= is used, optparse does not automatically initialize
The destination when a dest= is not specified. Refine the test to
allow dest= options when callback= is used even when it seems like
it is otherwise redundant.
Bug: b/436611422
Change-Id: I5185f95cb857ca6d37357cac77fb117a83db9c0c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/509861
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
When checkout errors occurred in interleaved sync, they were wrapped in
a SyncError with no message, causing blank lines in the final summary.
Refactor _SyncResult to hold a list of exceptions, ensuring the original
error messages are propagated correctly.
Bug: 438178765
Change-Id: Ic25e515068959829cb6290cfd9e4c2d3963bbbea
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/498342
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Failures in deferred sync actions were not recorded because `_Later.Run`
discarded the `GitError` exception. Record the specific error using
`syncbuf.fail()` and propagate it for proper error aggregation and
reporting.
Bug: 438178765
Change-Id: Iad59e389f9677bd6b8d873ee1ea2aa6ce44c86fa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/498141
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
A race condition occurs when sync redirects sys.stderr to capture worker output, while a background progress thread simultaneously calls fileno() on it. This causes an io.UnsupportedOperation error. Fix by caching the original sys.stderr for all progress bar IO.
Change-Id: Idb1f45d707596d31238a19fd373cac3bf669c405
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/498121
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Keep track of finished projects, not just successful ones, when deciding
which projects still need to be synced. Also project errors are already
reported by sync workers so stall detection doesn't need to add failed
projects to the error list.
Bug: 438178765
Change-Id: Ibf15aad009ba7295e70c8df2ff158215085e9732
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/498062
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
The _CheckForImmutableRevision method was modified in commit 0e776a58 to
include upstream branch validation for superproject scenarios. However,
this change inadvertently broke shallow clones when both clone-depth and
upstream attributes are specified in regular (non-superproject)
manifests.
Issue: When upstream is present, _CheckForImmutableRevision performs two
additional checks: 1. git rev-list on the upstream reference 2. git
merge-base --is-ancestor between revision and upstream
In shallow clones, the upstream branch history may not be available
locally, causing these checks to fail. This triggers the retry mechanism
that removes depth limitations, effectively converting shallow clones to
full clones, resulting in excessive disk usage.
Fix: Make upstream validation conditional on superproject usage. This
preserves the original superproject fix while restoring the method's
original behavior for regular scenarios - checking only if the immutable
revision (SHA1/tag) exists locally.
Note: The SetRevisionId method from the same commit 0e776a58 is left
unchanged as it only stores upstream information (no git operations),
which is beneficial for preserving branch context for commands like
'repo start' without causing fetch-related issues.
The fix ensures that manifests with both clone-depth and upstream work
correctly in non-superproject scenarios, maintaining shallow clone
efficiency and reducing disk usage.
Bug: b/427093249
Change-Id: I00acd4c61b179cd2abf796c2fecb7a2f38016a18
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/493883
Tested-by: Krzysztof Wesolowski <krzysztof.wesolowski@volvocars.com>
Commit-Queue: Krzysztof Wesolowski <krzysztof.wesolowski@volvocars.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Kamaljeet Maini <kamaljeet@google.com>
Reviewed-by: Xin Li <delphij@google.com>
