On Git 2.11 and newer, replace update-index, diff-index, diff-files, and
ls-files fan-out with one porcelain-v2 snapshot for IsDirty,
UncommittedFiles, and HasChanges. Preserve category ordering and
duplicate staged-plus-unstaged paths, and keep the exact legacy path for
older Git or a failed snapshot.
Bug: 543851900
Bug: 553599402
Change-Id: I309e39977e1f1d4ee6115f488d6aabb83ec3d83c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/632143
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Add a strict, NUL-aware parser for porcelain v2 records, including
ordinary changes, renames, conflicts, untracked paths, branch
divergence, and stash metadata. The reader is gated on Git 2.11, opts
into newer flags only when supported, and preserves arbitrary pathname
bytes.
Bug: 543851900
Bug: 553599402
Change-Id: Ibae3e1056fd9866f3cb6d490745d377cda1e8fef
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/632142
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
NUL-delimited Git formats may contain path bytes that are not valid
UTF-8. Add an opt-in byte capture mode so parsers can preserve those
bytes instead of receiving backslash replacements, while continuing to
decode stderr for diagnostics.
Bug: 543851900
Bug: 553599402
Change-Id: Ia9dfcde065a0f6f72c785a4b96896f346e17510a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/632141
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
In partial clone checkouts (e.g. --filter=blob:none or blob:limit=10k),
DownloadPatchSet previously invoked `git fetch <remote> <ref>` without
explicit filter overrides. Consequently, git fetch applied the
configured partial clone filter from the remote and omitted the blobs
for the incoming change.
When repo download subsequently checks out, cherry-picks, or merges the
downloaded commit, Git discovers that the required blobs are missing
locally and triggers on-demand unadvertised object fetches to the
promisor remote (POST /git-upload-pack with `want <blob_sha>`). On large
repositories, server-side reachability validation for unadvertised
objects can result in expensive graph traversals, proxy/gateway timeouts
(such as HTTP 502 or connection drops), and client fetch failures:
fatal: expected 'packfile'
fatal: could not fetch <sha> from promisor remote
Pass --no-filter (when Git >= 2.17.0) and --no-tags when fetching patch
sets in DownloadPatchSet. Because the change ref is an advertised ref
tip, fetching with --no-filter allows the server to generate a complete
packfile containing all required blobs upfront without requiring
unadvertised object reachability checks. All blobs are present in the
local object database prior to checkout or cherry-pick, completely
preventing on-demand promisor queries.
TAG=agy
CONV=bdc489e3-b20b-4ef3-b32e-549dbe21fc6c
Google-Bug-Id: b/532920848
Test: pytest tests/test_project.py
Change-Id: Ic6118e4edeb754f0467e0580fd1d0965a6553153
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/625621
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Sam Saccone <samccone@google.com>
Commit-Queue: Sam Saccone <samccone@google.com>
`FindProjects` gets its candidate projects from `GetProjects` and then
applies regex matching. However, it only forwards `all_manifests`, so
other project filters are lost on regex paths.
As a result, `repo forall -r/-i --groups` falls back to the configured
group selection instead of using the requested groups.
Similarly, `repo list -r --groups` ignores the requested groups, and
`repo list -r --all` still excludes projects without a checkout.
Pass `groups` and `missing_ok` through `FindProjects`, and forward the
existing options from `forall` and `list`. When no group override is
given, keep using each manifest's effective groups.
Add regression tests for `FindProjects` and both command paths.
Change-Id: I5dfc22172fdd81de5312a84433867300a1342aec
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/631281
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Victor Pushkarev <corvinus.v@gmail.com>
Tested-by: Victor Pushkarev <corvinus.v@gmail.com>
The trace2 socket test uses a Condition to wait for its server to start
listening. If the server calls notify() before the test starts waiting,
the notification is lost and the test waits for the full 120-second
timeout before continuing.
Use an Event for the readiness signal instead.
Once the server starts listening, the event stays set, so the test can
continue immediately even if the server becomes ready before the test
starts waiting.
Change-Id: I0e8e192d997c8bd785bcefa5d5b91493f93863e8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/630861
Tested-by: Victor Pushkarev <corvinus.v@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Victor Pushkarev <corvinus.v@gmail.com>
The default shown for `--jobs-checkout` comes from `DEFAULT_LOCAL_JOBS`,
which depends on the host CPU count. As a result, regenerating the sync
and smartsync manpages on a different machine can change their output
even though repo's behavior has not changed.
Use shared CPU-based help formatting for `--jobs` and `--jobs-checkout`.
Interactive help still shows the actual default, while generated
manpages describe it without including a host-specific value.
Runtime job selection is unchanged.
Add coverage for both help modes and regenerate the affected manpages.
Change-Id: Ia60e6fcc17b6f4618c52a76217c128eddead8848
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/625381
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Victor Pushkarev <corvinus.v@gmail.com>
Tested-by: Victor Pushkarev <corvinus.v@gmail.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Add `repo.reprojectcmd`, the checkout counterpart to `repo.fetchcmd`.
When set, `repo sync` runs this command instead of Git to materialize a
project's index and worktree at the target tree, while `repo` handles
ref updates directly via `git update-ref`.
This replaces Git's tree materialization steps: detaching HEAD,
fast-forwarding, and hard-resetting. Rebasing is not delegated, and the
command is skipped if HEAD is already at the target, if HEAD is ahead of
the target during fast-forward, or for MetaProjects.
The command runs in a subshell with project environment variables (such
as `REPO_TREV`). Before running, `repo` ensures no operation is in
progress and no staged changes exist. Worktree collision detection is
delegated to the command (preserving benign unstaged/untracked edits).
Afterward, `repo` verifies that HEAD was untouched and that the index
matches the target tree.
Like `repo.fetchcmd`, this requires `repo.uselocalgitdirs`. Nested
projects and submodules are unsupported; `repo sync` fails if the
manifest contains any while `repo.reprojectcmd` is enabled.
Verified end-to-end with repo init using local-gitdirs, repo.fetchcmd,
and repo.reprojectcmd ('git -C $REPO_PATH read-tree -m -u $REPO_TREV'):
* Verified detached HEAD checkout and correct reflog generation across
projects.
* Verified benign unstaged edits and untracked files survive checkout.
* Verified conflicting untracked files fail with exit 128 without
clobbering worktree.
* Verified staged changes fail upfront before reprojectcmd is executed.
Bug: 513329573
Change-Id: I964d24d22dccffc05a9b991ad69f3a7e93268c01
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/626281
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
"repo status" aborts the whole tree when one project fails, without
saying why. The three callers that refresh the index stat cache pass -q,
which suppresses git's diagnosis when it cannot create the index lock;
git then exits 128 with both streams empty. A stale lock, a permissions
problem on the gitdir and a full disk all reach that same silent exit,
yet each needs a different remedy.
Route the three identical call sites through one helper that keeps -q
for the refresh itself. When that fails, it repeats the command without
-q to obtain git's diagnosis. The command still fails, which is correct,
but the user is now told why.
Keeping -q on the common path matters: without it git exits 1 whenever
paths hold uncommitted changes, which would report a failure to
telemetry for every modified project, even if we treat it as benign in
the code. The retry accepts that 1, since --unmerged makes git skip
conflicted paths rather than count them, so nothing else yields it.
The similar call in subcmds/sync.py is unchanged: it passes different
flags, without --unmerged, so exit 1 cannot be assumed benign there.
Bug: 560289756
Change-Id: I2e1024c84454f86076081f2d7f155728b1aa35c6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/630641
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
When GetProjects walks all manifests without explicit groups, it
resolves the effective groups once from the selected manifest and reuses
them for projects from child manifests. This can ignore a child
manifest's own group selection and include projects that should be
excluded.
When no groups are specified, resolve them from each project's owning
manifest. Explicit groups still apply to all projects in the call, and
single-manifest behavior remains unchanged.
Add regression tests for per-manifest group selection, explicit
overrides, and name lookup across manifests.
Bug: 382086294
Change-Id: Id5c7e52410431b86b395f489cfd494ca05620975
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/624821
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Victor Pushkarev <corvinus.v@gmail.com>
Tested-by: Victor Pushkarev <corvinus.v@gmail.com>
The update-manpages wrapper calls update_manpages.main() unconditionally.
With spawn and forkserver, multiprocessing re-executes the directly run
main script as __mp_main__.
The wrapper then calls main() again and tries to create another Pool
while the child is still starting.
Protect the entrypoint with __main__ guard. On Linux, this was mostly
hidden while fork was the default start method; Python 3.14 now defaults
to forkserver there. Normal command-line behavior is unchanged.
Add a regression test that mirrors multiprocessing's __mp_main__
execution path and checks that it does not call update_manpages.main().
Change-Id: I28fdb18ccf2e4d61605cc072b4de52b6eb561335
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/626841
Tested-by: Victor Pushkarev <corvinus.v@gmail.com>
Commit-Queue: Victor Pushkarev <corvinus.v@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
When uploading across multiple projects with --yes and an explicit
branch specified (via --current-branch / -c or --branch / --br),
repo previously still opened the interactive branch selection editor.
If an explicit branch and --yes are provided, bypass the interactive
editor and upload all matching pending branches directly.
Bug: 553024110
Test: python3 -m pytest tests/
Test: flake8 subcmds/upload.py tests/test_subcmds_upload.py
Test: isort --check subcmds/upload.py tests/test_subcmds_upload.py
Change-Id: I104619603f6e0539d05b01d3eb0292a9145d58db
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/622661
Tested-by: Arif Kasim <arifkasim@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
In _CheckForImmutableRevision, commit d9cc0a15 restricted upstream ref
validation strictly to superprojects (if use_superproject) to prevent
shallow clones with an upstream attribute from failing ancestry checks
and falling back to full clones.
However, restricting this check exclusively to superprojects broke
non-shallow projects with pinned immutable revisions (such as in Smart
Sync, `repo sync -t <BUILD_ID>`, or pinned manifests): When the target
commit already exists in the local Git object store (for instance,
prefetched into refs/prefetch/ by a background daemon or via shared
object dirs), _CheckForImmutableRevision returned True without verifying
that the local tracking ref (refs/remotes/<remote>/<upstream>) is
present and reaches the revision. As a result, _RemoteFetch skipped
fetching the upstream branch, leaving the local tracking ref stale or
missing. Subsequent `repo start` branches tracking that remote branch
diverged, causing `repo upload` to attempt uploading all intermediate
commits between the stale tracking ref and HEAD.
Restore upstream ancestry validation in _CheckForImmutableRevision for
non-shallow projects when upstream is specified. Also pass the sync
depth into _CheckForImmutableRevision call sites so shallow checkouts
continue to skip upstream verification and avoid triggering un-shallow
fallbacks.
Test: PYTHONPATH=. pytest tests/test_project.py
Change-Id: Ib5bdf41810bb06c8ed053447209ddc2e488f3913
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/624361
Tested-by: Rahul Yadav <yadavrah@google.com>
Commit-Queue: Rahul Yadav <yadavrah@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
GIT_DEFAULT_BRANCH support was added to git var in Git 2.35. Skip that
guaranteed failure on older supported clients and read
init.defaultBranch directly, reducing their fallback from two processes
to one.
Bug: 553599402
Change-Id: I079e9e6057b56afa41b685317a106fc9cdbad8a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623907
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
The superproject only needs one branch tip or HEAD OID. Resolve that ref
with one quiet rev-parse instead of constructing GitRefs, which loads
the entire namespace and resolves HEAD in two or three processes on
older clients.
Bug: 553599402
Change-Id: I162d1133ebd1941e9de5aca43929fd5af17386ef
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623906
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Git 2.32 added the %(describe) pretty placeholder. Use one log format to
retrieve both repo's describe string and commit date on newer clients,
with the existing describe-plus-log path retained for older Git and
untagged output.
Bug: 553599402
Change-Id: I108346030677b2b90ef8bfb5acea1c45be2c85f2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623904
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Send a typed revision expression to cat-file --batch over stdin and
parse its object header and exact byte length. This safely resolves the
commit OID and retrieves its raw message in one Git process instead of
separate rev-parse and cat-file calls.
Bug: 553599402
Change-Id: I764894323ec28e134f2523bd1d5694738b6b52d0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623903
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Git --autostash predates repo's minimum supported client. Pass it
directly to rebase instead of probing the index and separately running
stash and stash pop, reducing the dirty path from four Git processes to
one and correctly covering staged-only changes.
Bug: 553599402
Change-Id: I0d6d59b17996576310498e9378e49380d355236e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623902
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Full Git object IDs are exactly 40 hexadecimal digits for SHA-1 or 64
for SHA-256. Stop treating every intermediate length as immutable, which
could bypass normal ref resolution for invalid revision strings.
Bug: 553599402
Change-Id: I41b4ce6b2bfe2a2d8b351c7f040ec0cf469b474b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623901
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Interleaved sync remembers which projects are done and which ones are
still pending by Project.relpath, and lists failing projects by it as
well. That path is relative to the project's own (sub)manifest though,
so once several manifests are synced together two projects can share
it, and the sets no longer tell them apart.
The damage is done when a project turns up in a later pass, e.g. a
submodule that is only derived once its parent has been synced: its
path is already recorded as finished, so it is left out of every pass
that follows and never synced, while sync still reports success. With
an outer manifest holding
<project name="a" path="a" sync-s="true"/>
where a carries a submodule at b, and a submanifest at sub/ holding
<project name="sub-ab" path="a/b"/>
a fresh `repo sync` checks out sub/a/b but never a/b. Failing projects
are reported under the same ambiguous path, and the stall detection
merges them too.
Use RelPath(local=opt.this_manifest_only) instead, which is unique
within the set of projects being synced and is what GetProjects() and
the other subcommands already use.
Spotted during the review of I30395b8a16a9154f60972e1f408a066af64ba77c.
Change-Id: I5fa694b968774667be6060d9722cb57acbd2b579
Signed-off-by: kimhappy <hwanhee.kim@laplacian.cc>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/622641
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Attach the current marker computed by GetBranches to uploadable and
prunable branches. Overview and prune reporting can then reuse that
result instead of resolving HEAD once per displayed branch.
Bug: 553599402
Change-Id: Ie3e23d08d0caf66c47cc12d7295ec4b6a834560c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623184
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Load refs before resolving the current branch, compare cached HEAD and
manifest OIDs instead of walking their symmetric difference, and reuse
the snapshot when preserving bare HEAD. Apply the same HEAD reuse to
abandon and MetaProject checks, and bound the latter revision walk to
one commit.
Bug: 553599402
Change-Id: I715e65dcfa02c0df4aaa08da5b7bfbb04ca17725
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623183
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Resolve commit-ish values through one typed helper using rev-parse
--verify --quiet and, on Git 2.30+, --end-of-options. Reject option-like
revisions on older clients, reuse the helper for project and manifest
resolution, and validate user branch names with check-ref-format
--branch.
Bug: 553599402
Change-Id: I2dda49ff3e781cec14d3b84263a8dace06b4073f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623182
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
When the regular checkout has already loaded GitRefs, derive its current
branch and HEAD object from that snapshot instead of spawning
symbolic-ref or rev-parse again. Keep linked Git worktrees on their
worktree-specific HEAD path because the shared repository HEAD is not
authoritative for them.
Bug: 543851900
Bug: 553599402
Change-Id: Iec456be5f3c279865b4165a75d1c6df36e331dbc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623181
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Use for-each-ref --include-root-refs to resolve attached and detached
HEAD in the same process as refs/* on Git 2.45 and newer. Limit the
patterns to HEAD and refs so volatile pseudo-refs do not enter the
cache, and retain the existing fallback for older Git and unborn HEADs.
Bug: 543851900
Bug: 553599402
Change-Id: Ie091ca0757f21ec6b1e19d49e09d160e055ec639
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/622981
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Benchmark (100 untouched projects):
* GetUploadableBranches: 0.985s -> 0.0004s (2,200x faster)
* repo upload (3,045 projects): saves ~2.5s of redundant ref scans
During `repo upload`, `GetUploadableBranches()` currently scans
`self._allrefs` across all projects in the workspace. In large
manifests (e.g. Android with 3,000+ projects), `self._allrefs`
executes `git for-each-ref`, `git symbolic-ref`, and full filesystem
mtime traversals on thousands of projects where no local branch was
ever started or modified.
Any branch that can be uploaded for review must have upstream tracking
configured via `[branch "..."]` sections in `.git/config`
(otherwise `branch.LocalMerge` is None and `GetUploadableBranch`
returns None).
This patch introduces early-pruning fast paths:
1. When no branch subsections exist in `.git/config`
(or only 2-part keys like `branch.autosetupmerge`), return []
immediately without touching `_allrefs`.
2. When a `selected_branch` is specified (e.g. `repo upload --br=...`),
query `branch.LocalMerge`, `refs/heads/<branch>`, and
`refs/published/<branch>` directly, skipping the whole-tree
ref scan.
3. In `subcmds/upload.py`, optimize `_GetMergeBranch` to resolve the
merge branch in-memory via `project.CurrentBranch` and
`project.GetBranch()` instead of spawning two `GitCommand`
processes.
Test: ./run_tests tests/test_project.py tests/test_subcmds_upload.py
Change-Id: I3c2c9d63a68a393d478e1fb50f548ec8b7d44639
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623021
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: James Hawkins <jhawkins@google.com>
Tested-by: James Hawkins <jhawkins@google.com>
Benchmark:
* repo upload frameworks/base: 2.84s -> 0.50s (-82.3%, 5.6x faster)
* repo upload (3,045 projects): 7.66s -> 5.24s (-31.6%, 2.42s saved)
From repo's inception through v2.56, GetHead() directly read the
`.git/HEAD` file in Python. In commit 52bab0ba ("project: Use git
rev-parse to read HEAD"), this was replaced with git subprocess calls
on the premise that git provides a dedicated command. However, in
large multi-project workspaces (such as Android with 3,000+ projects),
spawning thousands of git processes introduced severe latency
regressions during `repo upload` and `repo status`.
Furthermore, switching to subprocesses broke detached HEADs and
unborn branches (fixed in commits 7f7d70ef and 8c3585f3 by re-adding
the v2.56 file-reading logic as an error recovery fallback).
This patch restores fast in-memory reading as the primary path, while
adding modern defensive safeguards:
* Symbolic refs (`ref: refs/heads/...`): Strips whitespace and tabs
and returns the ref directly in memory.
* Detached HEAD: Validates 40-char SHA-1 and 64-char SHA-256 commit
hashes via git_config.IsId(), normalizing to lowercase.
* Symlinks: Detects filesystem symlinks via os.path.islink() and
safely falls back to git symbolic-ref.
* Fallback: Catches (OSError, AssertionError) and transparently falls
back to native git commands for reftables, unexpected layouts, or
filesystem errors. Unifies recovery fallback parsing with the fast
path (CRLF/tabs, lowercase hashes, and consistent RelPath errors).
In addition, this change substantially expands test coverage in
tests/test_project.py, adding comprehensive unit tests for symbolic
refs, whitespace/tabs, CRLF line endings, SHA-1, SHA-256, uppercase
hash normalization, symlinks, corrupted worktrees, and fallback
robustness.
Test: ./run_tests tests/test_project.py
Change-Id: Ib5c2530117c6939e4b9293feda81aa745c003c6a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623001
Commit-Queue: James Hawkins <jhawkins@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: James Hawkins <jhawkins@google.com>
The submodules to sync are derived from their parent before it is
fetched, so one that the parent's new revision no longer holds a gitlink
for is still synced. Interleaved sync then fails to check it out,
because `git submodule init` no longer knows that path:
error: Cannot checkout a/c
error: pathspec '.../a/c' did not match any file(s) known to git
Report such submodules while the gitlinks are read again, and leave them
out of the fetch and of the checkout, so the usual project list update
removes them from the working tree. Phased sync already left them out of
the checkout, but still fetched them.
Nothing is reported as removed while the gitlinks of a parent cannot be
read, since a missing gitlink cannot be told apart from a fetch that did
not happen.
Bug: 550074864
Change-Id: Ia429b35c12758a68e4df73665e5512a201af658a
Signed-off-by: kimhappy <hwanhee.kim@laplacian.cc>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/621681
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Phased sync fetches every project at once, so a submodule is fetched at
the gitlink that was read before its parent was fetched. With -c that
stale revision is already present, the fetch is skipped as an immutable
revision, and the checkout then fails on the revision the reloaded
manifest resolved:
error: Cannot checkout a/b
fatal: bad object 5556dad99da5df7e53303c5732aacc8782493e03
Fetch in batches, holding a submodule back until the project holding its
gitlink is fetched, and read the gitlinks again between batches. A
manifest without submodules keeps its single batch.
Bug: 550074864
Change-Id: I04c2eb1ca81ab25b4df2c03c5dc048d2e4ecdcb8
Signed-off-by: kimhappy <hwanhee.kim@laplacian.cc>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/621663
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Interleaved sync processes projects in hierarchical levels, so a
submodule is only handled after the project holding its gitlink has been
fetched and checked out. Its revision, however, is still the one read
before that parent was fetched, so the submodule is synced to the
revision of the previous sync and only catches up on the next one.
Read the gitlinks again right before a level is dispatched. The parent
is done by then, so its submodules are fetched and checked out at the
revision the parent now points at. Sibling submodules are spread over
several levels, so the gitlinks read for one level are carried over to
the next ones and every project is only read once per pass.
Bug: 550074864
Change-Id: I30395b8a16a9154f60972e1f408a066af64ba77c
Signed-off-by: kimhappy <hwanhee.kim@laplacian.cc>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/621662
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
https://gerrit-review.googlesource.com/614762 resolves a fallback
upstream for a SHA-1-pinned project that has no explicit upstream, so
--current-branch can narrow the fetch to one branch instead of fetching
all heads.
It drew that fallback from dest-branch and from the manifest <default>
upstream, dest-branch, and revision. dest-branch and tags are not valid
fallback candidates:
* dest-branch is the review destination for upload, not a fetch
source.
* Tags are meant to be immutable snapshots of a single tagged commit,
and fetching it cannot retrieve an arbitrary pinned SHA-1.
Restrict the fallback to branch heads resolved from the manifest
upstream and revision defaults, and drop dest-branch entirely.
Bug: 541240657
Bug: 544041102
Change-Id: I89ae86891ee166f4c340553ac5a9068efcc18461
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/622541
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
This patch intercepts the output during a successful upload
execution to capture the generated CL URLs. It logs a dynamically
constructed "repo.uploadstate" data event to the active trace2
log containing:
- Uploaded CL URLs
- Target remote name
- Source branch
- Modified files
Test:
1. ./run_tests
2. pytest tests/test_project.py
3. Manual verification:
- Created a dummy branch `test_upload_branch` in a project with local file changes.
- Invoked repo upload passing an explicit trace output directory:
`repo --git-trace2-event-log=/tmp/trace2out upload --dry-run --no-verify art`
- Verified the injected payload successfully appeared on disk within the generated log:
```json
{"event":"data",..."key":"repo.uploadstate/cls","value":""}
{"event":"data",..."key":"repo.uploadstate/remote","value":"goog"}
{"event":"data",..."key":"repo.uploadstate/branch","value":"test_upload_branch"}
{"event":"data",..."key":"repo.uploadstate/files","value":"dummy_file.txt"}
```
(cls is correctly blank on --dry-run but populates on real HTTP pushes)
Bug: 543953499
Change-Id: I9c402a32d01d156d42cf24eaa2e60e22710b5e6f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/616581
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Ram Peri <ramperi@google.com>
Commit-Queue: Ram Peri <ramperi@google.com>
A discovered submodule becomes a project whose revision is the gitlink
read from its parent at manifest load time, which is before the parent
has been fetched. Remember which gitlink a submodule was derived from,
and add a way to read those gitlinks again, so that callers can resolve
a submodule revision once its parent is up-to-date.
A revision that is set does not have to be fetched, so verify its
objects are really there. Without them a submodule that was removed
cannot be told apart from one that was never fetched, which the caller
has to know about.
Bug: 550074864
Change-Id: I4f35d53607126508eec6f02158a3fb604917a199
Signed-off-by: kimhappy <hwanhee.kim@laplacian.cc>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/621661
Reviewed-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Avoid disabling --current-branch when syncing a SHA-1 revision without
an explicit project upstream (e.g., smart tags). Resolve a fallback
upstream from dest-branch or manifest defaults so -c only fetches the
target branch.
Bug: 541240657
Change-Id: Ib44b6a732131210e1ec3a3136747d1a19bc5aa18
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/614762
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Introduce a `RESPECT_SMART_SYNC_OVERRIDE` class attribute to the base
`Command` class, defaulting to `True`. This allows subcommands to
automatically respect the smart sync override manifest if it exists.
The override is applied in `CommonValidateOptions` before any
subcommand-specific validation or execution occurs. The `sync` and
`init` commands explicitly opt out.
This ensures all workspace-aware subcommands consistently align with the
active smart sync override manifest. It also fixes a bug in
multi-manifest setups where running a command from a submanifest would
not apply the override to the outer manifest, causing inconsistency when
resolving projects across all manifests.
Bug: 279204331
Change-Id: I9426e90a13a77ce6bd94b4a82efda4d485cbe116
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/585081
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
When the default branch cannot be determined dynamically, repo fell back
to a hardcoded "refs/heads/master". This can point at a branch that does
not exist on the server, since projects increasingly default to "main".
Rather than swap one hardcoded name for another, ask git itself what it
would use.
Bug: 483758905
Change-Id: Ic66712acea98f8e548a2d6d8211865ee5c416b4d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/612845
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Similar to how repo init -g can restrict repo syncs to
a subset of the manifest globally, allow "repo sync -g" to only sync a
subset of projects from the manifest when running that specific
sync command.
Change-Id: I4929aad109de05c73a7db42bb27fd8d47eea32fc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/609481
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Andrew Chant <achant@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Andrew Chant <achant@google.com>
The names for flags --fetch-submodules / --no-fetch-submodules are
misleading, since they impact the full sync operation (fetch and
checkout), not just the fetching.
Introduce new flags --recurse-submodules / --no-recurse-submodules
and treat the old ones as deprecated aliases.
Change-Id: I78339a3e0496a855c222c1869b27b578507886a7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/608881
Commit-Queue: Josef Malmstrom <Josef.Malmstrom@arm.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Josef Malmstrom <Josef.Malmstrom@arm.com>
Pass the -y flag as a keyword argument "yes" to the hook main function.
This allows upload hooks (such as auto-fixers) to automatically
apply fixes when the -y flag is passed, rather than prompting the user.
Bug: 498893733
Change-Id: I12096029aa3af471ba9175314d749deb0ebd1007
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/605261
Commit-Queue: Rahul Yadav <yadavrah@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Rahul Yadav <yadavrah@google.com>
The superproject-gated upstream check in _CheckForImmutableRevision
only applies to user projects listed in the manifest. MetaProjects
(ManifestProject and RepoProject) never participate in a superproject
relationship, so evaluating git_superproject.UseSuperproject(...,
self.manifest) for them serves no purpose and, worse, calls the
manifest.superproject property which forces a manifest load.
During repo init, ManifestProject._ConfigureDependencies calls
self.Sync_NetworkHalf before manifest.xml has been linked into
.repo/. That reaches _CheckForImmutableRevision, which triggered the
manifest load and failed with:
ManifestParseError: .../.repo/manifest.xml: [Errno 2] No such file
or directory
breaking fresh repo init with SHA-based --manifest-branch combined
with --manifest-upstream-branch.
Factor the "should we consult the superproject for upstream?"
decision into a small overridable hook, _UseSuperprojectForUpstream.
Project's default delegates to git_superproject.UseSuperproject;
MetaProject overrides it to return False, localizing the
MetaProject-specific behavior to MetaProject.
Test verifies that calling _CheckForImmutableRevision on a
ManifestProject whose manifest.xml is not yet on disk returns False
without raising and does not create the file.
Change-Id: I22059109243d914036c06c6fe0081a5aba05da89
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574201
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Ajay Gupta <ajagup@qti.qualcomm.com>
Tested-by: Ajay Gupta <ajagup@qti.qualcomm.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
A global setting for disabling fetching of submodules is useful
since this can currently otherwise only be done by modifying
the manifest, or by explicitly providing projects on command line.
Add this setting as --no-fetch-submodules to mirror the existing
--fetch-submodules.
Change-Id: Ic727c54f11a594aa52315751284b87138cf246bb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/607641
Commit-Queue: Josef Malmstrom <Josef.Malmstrom@arm.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josef Malmstrom <Josef.Malmstrom@arm.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
When running `repo rebase -m` (`--onto-manifest`), the command uses the
raw `revisionExpr` from the manifest (e.g. `main`) directly as the
`--onto` target. This can fail or behave incorrectly if it should
reference the local tracking branch (e.g. `refs/remotes/<remote>/main`).
Resolve `project.revisionExpr` to its local tracking branch using
`project.GetRemote().ToLocal()`. Fall back to using the raw
`revisionExpr` value if the resolution fails (raising a `GitError`).
Bug: 532028666
Change-Id: I4c1bca1374a5842688be227f6aa2afffcdad5397
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/604941
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Per the git documentation for color.ui [1], setting color.ui to "true"
(or "yes") should behave identically to "auto", enabling color only
when output is written to a terminal or an active pager. Previously,
repo was equating "true" and "yes" with "always", which caused color
escape codes to be emitted unconditionally, even when output was piped
or redirected.
Replace the duplicated string-matching logic in SetDefaultColoring and
Coloring.__init__ with a single CONFIG_TO_COLOR_SETTING dict that maps
all git color config values to their behavior. This makes the mapping
easy to verify against the git docs and impossible to get out of sync
between the two call sites.
Added tests for SetDefaultColoring and Coloring.__init__ covering
all color mode values (auto, true, yes, always, never, no, false),
case insensitivity, TTY vs pipe behavior, active pager detection,
and unrecognised input.
[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-colorui
Bug: 295841573
Change-Id: I8a04b9c7e4154de37ed7518c010233039e0afdc9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/602981
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Python 3.14 changed the default multiprocessing start method on Linux
from "fork" to "forkserver". The test_forall_all_projects_called_once
test uses mock.patch.object on Project.GetRevisionId, but class-level
mock patches do not survive into forkserver worker processes because
they start from a clean Python interpreter rather than inheriting the
parent's memory.
Replace the mock with setting revisionId directly on each Project
instance so GetRevisionId() short-circuits without touching git.
This works with any multiprocessing start method since the string
attribute is part of the Project objects stored in _parallel_context,
which is properly serialized to workers via initargs.
Bug: 425319437
Change-Id: Icd3bbd010921d7652bb2425fad85974df9198367
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/602941
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Introduce support for `repo.fetchcmd` configuration, allowing users to
specify a custom command to fetch project data instead of
`_RemoteFetch`.
When `repo.fetchcmd` is specified, `repo` will execute it instead of
`_RemoteFetch` during the network half of sync. Note that
`repo.fetchcmd` requires `repo.uselocalgitdirs` to be enabled.
The command is executed in a subshell with project-context environment
variables, including the new `REPO_TREV` (target revision resolved to a
commit hash) and `REPO_PROJECT_FETCH_URL`.
After execution, `repo` verifies that the target commit is available and
that the tracking ref and FETCH_HEAD are correctly updated.
Tested with:
```
> ~/git-repo/repo init -u https://android-review.googlesource.com/platform/manifest \
--repo-url file:///usr/local/google/home/gavinmak/git-repo \
--groups developers \
--no-repo-verify \
--use-local-gitdirs
...
> git config --file .repo/manifests.git/config repo.fetchcmd 'mkdir -p $REPO_PATH && cd $REPO_PATH && if [ ! -d .git ]; then git init && git remote add aosp $REPO_PROJECT_FETCH_URL; fi && git fetch aosp $REPO_TREV && git reset --hard $REPO_TREV && mkdir -p .git/refs/remotes/aosp && echo $REPO_TREV > .git/refs/remotes/aosp/main && echo $REPO_TREV > .git/FETCH_HEAD'
> ~/git-repo/repo sync -j32
warning: repo is not tracking a remote branch, so it will not receive updates; run `repo init --repo-rev=stable` to fix.
You are currently enrolled in Git submodules experiment (go/android-submodules-quickstart). Use --no-use-superproject to override.
Syncing: 100% (4/4), done in 1m15.686s
Finalizing sync state...
repo sync has finished successfully.
```
Bug: 513329573
Change-Id: I754d3f3c78e86fdeee1a72115297a75b571bc497
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/583883
Reviewed-by: Becky Siegel <beckysiegel@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
When viewing `repo status`, it is difficult to distinguish between branches that have active unpushed changes and stale branches that are fully synced. Previously, developers had to run commands like `repo forall -c "git status"` to see their ahead/behind counts.
This change updates Project.PrintWorkTreeStatus to automatically calculate and display the number of commits a branch is ahead and/or behind its upstream tracking branch. We use `git rev-list --left-right --count` to fetch this information natively and efficiently.
If the branch is completely synced with upstream, no extra text is shown.
Added tests for ahead-only, behind-only, diverged, no-tracking, and fully-synced branch states.
Bug: 319412954
Change-Id: I23879b2d472c7a7e11d01b565428a84b1b4f09c1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/602423
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Gavin Mak
Sam Saccone
Victor Pushkarev
Brian Gan
Arif Kasim
Rahul Yadav
kimhappy
James Hawkins
Ram Peri
Andrew Chant
Josef Malmström
Ajay Gupta