When running on a machine without the `ssh` command, repo sync would fail even if no ssh or ssh proxy was required. Use exception handling inside ssh.ProxyManager to more gracefully handle the case where ssh is not installed.
Bug: 467714011
Change-Id: I602a0819638ead4d02de88b750839bc3d70549ce
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/535141
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Jordan Esh <esh.jordan@gmail.com>
Commit-Queue: Jordan Esh <esh.jordan@gmail.com>
`delete_unused_projects` needs a full list of active projects to figure
out which orphaned .git dirs need to be deleted. Otherwise it thinks
that only the projects specified in args are active.
Bug: 447626164
Change-Id: I02beebf6a01c77742a8db78221452d71cd78ea73
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/550061
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Add a verbose attribute to the EventLog class, defaulting to False.
Error messages printed to sys.stderr within the EventLog.Write method
are now guarded by this verbose flag. In main.py, set EventLog.verbose
to True if the command-line --verbose option is used. This prevents
trace2 logging failures from being printed to stderr unless verbose
output is explicitly requested.
PROMPT=convert all git trace2 logging print messages to verbose only
logging
BUG: b/479811034
Change-Id: I8757ee52117d766f2f3ec47856db64cc4f51143c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/547542
Tested-by: Sam Saccone <samccone@google.com>
Reviewed-by: Julia Tuttle <juliatuttle@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
When extending a project in a local manifest, the project inherits the
`local:` group. This causes the superproject override logic (which omits
projects with `local:` groups) to incorrectly exclude the project from
the override manifest. This leads to "extend-project element specifies
non-existent project" errors during sync reload.
Fix this by stripping `local:` groups from extended projects, ensuring
they remain visible to superproject overrides while still allowing other
inherited groups to persist.
Bug: 470374343
Change-Id: I1a057ebffebc11a19dc14dde7cc13b9f18cdd0a3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/543222
Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
During sync, `git checkout` can trigger fetch for missing objects in
partial clones. This internal fetch can trigger `git maintenance` or
`git gc` and cause delays during the local checkout phase. Set
maintenance.auto to false and gc.auto to 0 in during `_InitRemote` if
`depth=1` to ensure that implicit fetches spawned by git skip GC.
Bug: 379111283
Change-Id: I6b22a4867f29b6e9598746cb752820a84dc2aeb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/540681
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
This avoids copying/linking the same file/link multiple times if a
copyfile/linkfile element with the same values has been specifed
multiple times. This can happen when including a common manifest that
uses an extend-project element that has a copyfile/linkfile element.
This uses dicts rather than sets to store the copyfiles and linkfiles to
make sure the order they are specified in the manifest is maintained.
For Python 3.7+, maintaining the order that keys are added to dicts is
guaranteed, and for Python 3.6 it happened to be true.
The _CopyFile class and the _LinkFile class are changed to inherit from
NamedTuple to be able to store them in dicts.
Change-Id: I9f5a80298b875251a81c5fe7d353e262d104fae4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/525322
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Commit-Queue: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
This new command allows users to delete projects from the worktree
and from the `.repo` directory. It is a destructive operation.
It handles shared projects by refusing to wipe them unless the
`--force` flag is used. It also checks for uncommitted changes
before wiping.
Bug: 393383056
Change-Id: Ia30d8ffdc781a3f179af56310ce31c9dae331bbe
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/490801
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Interleaved sync didn't save _fetch_times and _local_sync_state to disk.
Phased sync saved them, but incorrectly applied moving average smoothing
repeatedly when fetching submodules, and discarded historical data
during partial syncs.
Move .Save() calls to the end of main sync loops to ensure they run
once. Update _FetchTimes.Save() to merge new data with existing history,
preventing data loss.
Change-Id: I174f98a62ac86859f1eeea1daba65eb35c227852
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/519821
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
When callback= is used, optparse does not automatically initialize
The destination when a dest= is not specified. Refine the test to
allow dest= options when callback= is used even when it seems like
it is otherwise redundant.
Bug: b/436611422
Change-Id: I5185f95cb857ca6d37357cac77fb117a83db9c0c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/509861
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
When checkout errors occurred in interleaved sync, they were wrapped in
a SyncError with no message, causing blank lines in the final summary.
Refactor _SyncResult to hold a list of exceptions, ensuring the original
error messages are propagated correctly.
Bug: 438178765
Change-Id: Ic25e515068959829cb6290cfd9e4c2d3963bbbea
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/498342
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Failures in deferred sync actions were not recorded because `_Later.Run`
discarded the `GitError` exception. Record the specific error using
`syncbuf.fail()` and propagate it for proper error aggregation and
reporting.
Bug: 438178765
Change-Id: Iad59e389f9677bd6b8d873ee1ea2aa6ce44c86fa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/498141
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
The logic for checking for repo self-updates lives in _FetchMain, which
is part of the "phased" sync path.
Extract this logic into a new _UpdateRepoProject helper method. Call
this common helper from _ExecuteHelper before either sync mode begins,
so the repo self-update check is always performed.
Bug: 421935613
Change-Id: I9a804f43fbf6239c4146be446040be531f12fc8a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/484041
Reviewed-by: Scott Lee <ddoman@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
For each assigned project, the worker sequentially calls
Sync_NetworkHalf and Sync_LocalHalf, respecting --local-only and
--network-only flags. To prevent scrambled progress bars, all stderr
output from the checkout phase is captured (shown with --verbose).
Result objects now carry status and timing information from the worker
for state updates.
Bug: 421935613
Change-Id: I398602e08a375e974a8914e5fa48ffae673dda9b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/483301
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Introduce the parallel orchestration framework for `repo sync
--interleaved`.
The new logic respects project dependencies by processing them in
hierarchical levels. Projects sharing a git object directory are grouped
and processed serially. Also reuse the familiar fetch progress bar UX.
Bug: 421935613
Change-Id: Ia388a231fa96b3220e343f952f07021bc9817d19
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/483281
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
This will make it easier to add more formats without exploding the
common --xxx space and checking a large set of boolean flags.
Also fill out the test coverage while we're here.
Bug: b/412725063
Change-Id: I754013dc6cb3445f8a0979cefec599d55dafdcff
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/471941
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
After some refactoring earlier, the forall command was
broken briefly, returning after only one run instead
of after all projects.
This test, albeit simple in nature, would have caught that.
Due to the somewhat demanding nature of forall,
a lot more setup was needed than expected but seems
to do its job now so hopefully it catches similar stuff
in the future.
Change-Id: I51e161ff0e7e31a65401211c376f319bda504532
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445461
Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Fredrik de Groot <fredrik.de.groot@haleytek.com>
command should be cmd_name, to match what git is emitting. This also
fixes arguments, so that only relevant arguments are passed instead
of the entire sys.args, which will contain wrapper information
Change-Id: Id436accfff511292ec2c56798fffb2306dda38fc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443741
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
This adds an optional, built-in checker for
guarding against patches hanging on wrong
base revisions, which is useful if a lower layer of
the manifest changes after a patch was done.
When adding a patch with a new revision using
extend-project or remove-project/project:
C---D---E patches in project bla
/
A---B project bla in manifest state 1
<extend-project name="bla" revision="E" base-rev="B">
If project bla gets updated, in a new snap ID
or by a supplier or similar, to a new state:
C---D---E patches in project bla
/
A---B---F---G project bla in manifest state 2
Parsing will fail because revision of bla is now G,
giving the choice to create a new patch branch
from G and updating base-rev, or keeping previous
branch for some reason and only updating base-rev.
Intended for use in a layered manifest with
hashed revisions. Named refs like branches and tags
also work fine when comparing, but will be misleading
if a branch is used as base-rev.
Change-Id: Ic6211550a7d3cc9656057f6a2087c505b40cad2b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436777
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
With 551285fa35, the comment about number
of workers no longer stands - dict is shared among multiprocesses and
real time information is available.
Using 2.7k projects as the baseline, using chunk size of 4 takes close
to 5 minutes. A chunk size of 32 takes this down to 40s - a reduction of
rougly 8 times which matches the increase.
R=gavinmak@google.com
Bug: b/371638995
Change-Id: Ida5fd8f7abc44b3b82c02aa0f7f7ae01dff5eb07
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438523
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
In method _parse the value of this variable 'have_fg ' is always
False, Maybe reassign it to True is lost.
I guess the author’s original intention was:
if set some value in gitconfig file(for ex: text = black red ul),
the first is bg color, the second is fg color, and the last one is attr.
Change-Id: I372698fe625db4c1fdaa94ea7f193a80a850ecb9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/425997
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Bright Ma <mmh1989@foxmail.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
The log message is already formatted before being passed to the colorer.
To avoid the exception "TypeError: not enough arguments for format
string", we should use the `nofmt_colorer` instead.
This bug occurs only when the formatted string still contains '%'
character. The following snippet can reproduce the bug:
```
from repo_logging import RepoLogger
RepoLogger(__name__).error("%s", "100% failed")
```
Change-Id: I4e3977b3d21aec4e0deb95fc1c6dd1e59272d695
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/432017
Tested-by: Shik Chen <shik@google.com>
Commit-Queue: Shik Chen <shik@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
The current logic to create checkout layers doesn't work in all cases.
For example, let's assume there are three projects: "foo", "foo/bar" and
"foo-bar". Sorting lexicographical order is incorrect as foo-bar would
be placed between foo and foo/bar, breaking layering logic.
Instead, we split filepaths based using path delimiter (always /) and
then use lexicographical sort.
BUG=b:325119758
TEST=./run_tests, manual sync on chromiumos repository
Change-Id: I76924c3cc6ba2bb860d7a3e48406a6bba8f58c10
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/412338
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: George Engelbrecht <engeg@google.com>
If a repo manifest is updated so that project B is placed within a
project A, and if project A had content in new B's location in the old
checkout, then repo sync could break depending on checkout order, since
B can't be checked out before A.
This change introduces checkout levels which enforces right sequence of
checkouts while still allowing for parallel checkout. In an example
above, A will always be checked out first before B.
BUG=b:325119758
TEST=./run_tests, manual sync on ChromeOS repository
Change-Id: Ib3b5e4d2639ca56620a1e4c6bf76d7b1ab805250
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/410421
Tested-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Greg Edelston <gredelston@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Before the change, repo normalizes the urls
with a following format only:
git@github.com:foo/bar
It doesn't cover the following case:
<remote name="org" fetch="git@github.com:org/" />
<project name="somerepo" remote="org" />
Results to:
error: Cannot fetch somerepo
from ssh://git@github.com/org/git@github.com:org/somerepo
Current change fixes it by normalizing this format:
git@github.com:foo
Test: ./run_tests tests/test_manifest_xml.py
Change-Id: I1ad0f5df0d52c0b7229ba4c9a4db4eecb5c1a003
Signed-off-by: Vitalii Dmitriev <vitalii.dmitriev@unikie.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/398337
Commit-Queue: Vitalii Dmitriev <dmit.vitalii@gmail.com>
Tested-by: Vitalii Dmitriev <dmit.vitalii@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
In the case of a project being removed from the manifest, and in the
path in which the project used to exist, and symlink is place to another
project repo will start to warn about partial syncs when a partial sync
did not occur.
Repro steps:
1) Create a manifest with two projects. Project a -> a/ and project b -> b/
2) Run `repo sync`
3) Remove project b from the manifest.
4) Use `link` in the manifest to link all of Project a to b/
Bug: 314161804
Change-Id: I4a4ac4f70a7038bc7e0c4e0e51ae9fc942411a34
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/395640
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Matt Schulte <matsch@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
When parsing the manifest XML, the code looks for a top
level DOM node named "manifest". However, it doesn't check
that it's an element type node so if there is also an XML
document type declaration node present (which has the same
name as the root element) then it selects the wrong node
and hence you end up with no projects defined at all.
Change-Id: I8d101caffbbc2a06e56136ff21302e3f09cfc96b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/390357
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Chris Allen <chris.allen@arm.com>
Commit-Queue: Chris Allen <chris.allen@arm.com>
Mike Frysinger
Jeffery Miller
Jordan Esh
Gavin Mak
Sam Saccone
Peter Kjellerstedt
Fredrik de Groot
Josip Sokcevic
Josip Sokcevic
bright
Shik Chen
Sergiy Belozorov
Vitalii Dmitriev
Matt Schulte
Michael Kelly
Jason R. Coombs
Jason R. Coombs
Chris Allen