After repo.reprojectcmd exits 0, _Reproject ran `git diff-index --cached
--name-only <revid>^{tree}` to verify that the index matches the target
commit's tree.
Peeling <revid>^{tree} and comparing every entry against the index
forces Git to read the target commit and walk every tree object in the
target revision from .git/objects.
reprojectcmd exists for virtualized filesystems such as ABFS whose
checkout command materializes the worktree and writes .git/index from
its own tree metadata without reading .git/objects/pack/*-trees-*.pack.
Running diff-index against <revid>^{tree} afterward forces Git to fault
in the tree packfiles over the network (260 FetchTrees pack downloads /
1.64 GiB across a 3,296-project Android checkout).
Trust reprojectcmd's exit status for index materialization instead of
re-walking the target tree in Git, while keeping the HEAD postcondition
check that verifies the command did not modify refs.
Bug: 543101585
Change-Id: Ie5feea41a4191245232c6763a489d01596370c55
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643004
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
After repo.fetchcmd exits, _CustomFetch verified the tracking ref and
FETCH_HEAD with ResolveCommit, i.e. `git rev-parse --verify
<ref>^{commit}`. Peeling to a commit makes Git look the object up: it
opens every pack index in objects/pack/ until it finds the commit, then
reads the commit from its packfile.
fetchcmd exists for object stores that are populated lazily, such as
ABFS, whose `abfs git fetch` registers packfiles as virtual files and
returns at once. There the peel is the first read of the packs the
command registered, and it blocks until they are downloaded. A project
whose revision did not change needs no objects to sync, yet paid that
download whenever fetchcmd registered new packs for it.
Compare the object IDs that the refs record instead (ResolveRef, which
runs the same rev-parse without the peel). Only when they differ peel
both sides to commits, so an annotated tag still matches the commit it
points to. A commit missing from the object store now fails the checkout
that reads it instead of the network half.
On a first sync the checkout reads the commit anyway, so there the
download only moves to the next command that needs the object.
Bug: 543101585
Change-Id: I51248ff1e9674b16e3ff8cee69572dbc8c7a8e99
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643003
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Skip projects fetched via repo.fetchcmd in _CheckForBloatedProjects,
and expose Project.UseFetchCmd alongside Project.UseReprojectCmd.
The bloat check exists because Project._RemoteFetch passes --no-auto-gc
for clone-depth="1" repositories. Projects using repo.fetchcmd bypass
_RemoteFetch and manage object storage externally, so scanning their
worktrees for bloat is unnecessary and recommending git repack can
materialize virtualized objects locally.
Bug: 543101585
Change-Id: I16510ad3b5b39c3a5e2a1a75df706439252a088e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655083
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Sparse checkout allows Git to check out only a portion of the
repository. This is particularly helpful for monorepository workflows
where CI jobs only need a portion of the repository to run tests.
This speeds up the CI jobs, and puts less load on the Git server as it
relieves it from having to pack up all the objects.
In the Git project, though it is still labeled experimental, the feature
has been stable for a number of years and is unlikely to change at this
point.
This change allows configuring Git to use sparse checkout and provide a
list of directories for the sync command to retrieve.
Projects with sparse paths default to a blob:none partial clone so the
excluded paths are never fetched, unless a filter is already configured
or the project is excluded from partial clones. Both the plain and the
git-worktree checkout flows configure sparse-checkout before populating
the working tree.
Change-Id: I772c7db5209b345306be378771c8f2040832bca1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/535061
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: John Cai <johncai86@gmail.com>
Commit-Queue: John Cai <johncai86@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Read plain files-backend unborn HEAD symrefs directly before spawning
git symbolic-ref, update already-loaded bare_ref snapshots in
_InitAnyMRef when no external ref changes occurred, and reuse warm
bare_ref snapshots in GetRevisionId for non-tag refs.
Also skip redundant rev-parse and empty-index ls-files subprocesses in
_Reproject when HEAD is guaranteed to be an unborn local ref.
Bug: 564817989
Change-Id: I0da10cefb14753ce6e1e22e192d652790d59a5b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649605
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Update .repo_config.json atomically with file metadata and keep cached
sections, remotes, and branches coherent across SetString, Branch.Save,
ClearCache, and Reload. Also reload Project.config in workers when
fetchcmd or reprojectcmd modifies .git/config externally so subsequent
reads hit the JSON cache instead of re-spawning git config --list in
MainProcess.
Also pass --template= to git init when no custom template directory is
configured so Git does not copy sample hooks only to delete them.
Bug: 564817989
Change-Id: I5ff655b0ab0b34a4f77f13617bbe5a1fff488c8c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649604
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
When uploading a branch that merges another remote-tracking branch (such
as an upstream branch into the destination branch), ReviewableBranch
only excludes the destination tracking branch (^base). As a result, git
rev-list walks the merged parent and counts all merged commits even
though they already exist on a remote branch, triggering the unusual
commit count warning in _VerifyPendingCommits.
Add --ignore-remote-commits to `repo upload` to pass --not --remotes to
git rev-list in ReviewableBranch so commits already reachable from
refs/remotes/* are excluded from the pending commit list.
Test: ./run_tests tests/test_project.py tests/test_subcmds_upload.py
Change-Id: Ib1fb864982729e20a621144580fe95daa478eaee
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/651284
Tested-by: Romaric Jodin <rjodin@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Romaric Jodin <rjodin@google.com>
`Project.UploadForReview()` unconditionally passed `--progress` to
`git push`, forcing client and remote progress meters (`Counting
objects`, `remote: Resolving deltas: 3%..100%`) even on non-TTY
stderr or when `-q`/`--quiet` was passed. Meanwhile, `repo upload`
ignored `opt.quiet` completely.
Move `--progress` injection for `push` into `GitCommand` alongside
`fetch` and `clone`, enabling it only when `sys.stderr.isatty()` and
none of `--progress`, `--no-progress`, `--quiet`, or `-q` are set.
Forward `opt.quiet` from `repo upload` to `Project.UploadForReview()`
and pass `--no-progress` to `git push` rather than `--quiet` so
Gerrit `remote: https://...` review URLs are still emitted and parsed
by `_FindGerritUrls()`. Also suppress the non-error `[OK]` summary
banner when `-q`/`--quiet` is set.
Bug: 578114287
Change-Id: I88b7b055dcca0db80dae6962886a693fe717c741
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649663
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Git 2.29+ introduced v3 bundle headers (`# v3 git bundle`) for SHA-256
and filtered bundles. Allow both v2 and v3 bundle headers when
validating downloaded `clone.bundle` files.
Bug: 576377949
Change-Id: Ic5ac94bf535a801d948f8932213e62a6cd132cac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/647123
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Allow users to configure Smart Sync as the default for a checkout:
* Add `ManifestProject.smart_sync` backed by
`self.config.GetBoolean("repo.smartsync")`.
* Add `-s`/`--smart-sync` and `--no-smart-sync`/`--head` to `repo init`
to persist `repo.smartsync` in the manifest project config.
* Add `--head` as an alias for `--no-smart-sync` in `repo sync` and
`repo init`.
* Update `Sync._ResolveSmartSyncOption` so `ManifestProject.smart_sync`
takes precedence over `<default sync-smartsync="true" />` while
remaining overridable by per-invocation `repo sync` CLI flags.
Bug: 569927750
Test: pytest tests/test_project.py tests/test_subcmds_init.py tests/test_subcmds_sync.py
Change-Id: Ib40efc530336efb59a28acfed3d582713e9d67b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643303
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: LaMont Jones <lamontjones@google.com>
Tested-by: LaMont Jones <lamontjones@google.com>
The automated release process deletes the stable branch and then
recreates it. A client that fetches during that window can't find it:
`repo init` fails to resolve --repo-rev, and the daily self-update in
`repo sync` fails with "Cannot fetch repo tool".
In the launcher, if --repo-rev doesn't resolve after the initial fetch,
wait 10 seconds and refetch once before giving up. `resolve_repo_rev`
gains a `report_errors` flag so this first check doesn't print an error
that the refetch then fixes. Bump the launcher version to 2.69 so
existing users pick this up.
For fetches of the repo project itself (sync's self-update and
`repo selfupdate`), wait 10 seconds instead of 4 before retrying. The
delay is now a `RETRY_SLEEP_INITIAL_SEC` class attribute that
`RepoProject` overrides, so other projects keep the 4 second delay. This
replaces the unused `retry_sleep_initial_sec` parameter of
`_RemoteFetch`.
Bug: 566270786
Change-Id: I23cb810258522b5aa9eae9d7013dbb7904ebcad8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643265
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
In Sync_NetworkHalf, clone_bundle defaults to True on initial sync when
self.objdir does not exist yet, and _ApplyCloneBundle is called before
custom_fetch_configured is checked.
For remotes with HTTP/HTTPS URLs that serve /clone.bundle, this causes
repo sync to download and unpack the full clone.bundle into .git/objects
before invoking repo.fetchcmd.
Skip _ApplyCloneBundle when custom_fetch_configured is enabled so
repo.fetchcmd exclusively handles object fetching.
Bug: 543101585
Change-Id: If41a775a75589ad5a23106163f9932e2940b72d2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/641685
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
When all_refs (e.g. self.bare_ref.all in Sync_LocalHalf) is passed to
GetRevisionId, looking up a tag ref in all_refs returns the tag object
SHA for annotated tags rather than the peeled commit SHA.
In Sync_LocalHalf, this causes repo.reprojectcmd to receive a tag SHA in
REPO_TREV and DetachHead to fail with 'trying to write non-commit object
to branch HEAD', and causes standard checkouts to re-checkout on every
subsequent sync because HEAD (the commit SHA) never equals revid (the
tag object SHA).
Skip the all_refs fast path for refs/tags/* so tags always fall through
to bare_git.ResolveCommit, which peels to the underlying commit SHA.
Bug: 543101585
Change-Id: Ie4afa3cee9149e2a8438bd8fb4a8026a210848d8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/641683
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
When a project's revision is an unpinned branch or tag and superprojects
are not used (so self.revisionId is unset), _CustomFetch previously ran
git ls-remote against the Git remote before invoking repo.fetchcmd just
to pre-populate REPO_TREV.
This added a per-project network round-trip before fetchcmd and forced
branch resolution against the Git remote rather than letting fetchcmd
resolve REPO_RREV from its own server (just as standard git fetch does
in _RemoteFetch).
Only populate REPO_TREV before fetchcmd when the target commit SHA is
already known (via self.revisionId from superproject or a SHA-pinned
revisionExpr). When REPO_RREV is an unpinned branch or tag, leave
REPO_TREV empty, let fetchcmd resolve REPO_RREV and update the local
tracking ref and FETCH_HEAD, and verify afterward that the tracking ref
and FETCH_HEAD resolve to the same valid commit in the object store.
Bug: 543101585
Change-Id: I482bf4c022d0e669e5620d4517581ac193983da0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/640423
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
The documented way to roll back a bad release is to force push an older
release to stable, and docs/release-process.md says clients will then
automatically downgrade. _PostRepoFetch already supports switching to an
older commit, but it only runs when `RepoProject.HasChanges` is true,
and that used the `MetaProject` check of whether the remote has commits
that HEAD lacks. After a rewind it doesn't, so clients that had already
upgraded stayed on the bad release.
Make `RepoProject.HasChanges` true whenever the remote revision differs
from HEAD, in either direction. `ManifestProject` keeps the forward-only
check so local manifest commits are still rebased rather than dropped.
Bug: 571204396
Change-Id: I248507f434d2552ce11a9afabe9a6335cb24ddbd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/640130
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Skip stateless prune planning when a project's worktree directory
does not exist, preventing fetch failures on subsequent syncs.
Also run _LsRemote() bare so it does not depend on worktree
existence.
Bug: 565047698
Change-Id: I385cba4e06813d9113ec7ddc963e8499acffaec5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/634426
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Skip shallow clone bloat checks during --network-only syncs and
filter out projects whose worktree directory does not exist on disk.
Also return None quietly from _GetStatusSnapshot() when a project
lacks an existing worktree directory.
Bug: 565047698
Change-Id: Ia4c7db36a2bd78bc89e7b72f7a492b901b50efa7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/634425
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
When a status snapshot fails, fall back directly to _IsDirtyLegacy()
instead of IsDirty() to avoid running git status twice.
Expose Project.GetDirtyAndHead() so sync bloat checks can inspect
worktree state without invoking private Project methods.
Bug: 565047698
Change-Id: I6432ed82da88778eec9d151fd5b70bedb1efc62b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/634424
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Skip the detached-history walk when its count will not be printed, ask
rev-list --count for upstream gain, and cap the published-but-unmerged
existence probe at one commit. These preserve the decisions while
reducing traversal and output work.
Bug: 553599402
Change-Id: I30f769c9aa5269428ece15ffcb45c50993d7c4b8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/634004
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Every caller expects _FastForward to reject a merge commit. Always pass
--ff-only and remove the optional parameter so a violated ancestry
assumption fails instead of silently creating a merge.
Bug: 553599402
Change-Id: I04690b5ca675cb7d3062df08d5e20cd82f5b9e82
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/634003
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
When Git 2.35 or newer supplies a stash header, use the same status
snapshot for stateless dirty and stash checks. Retain rev-parse for
earlier clients or when the status snapshot is unavailable.
Bug: 543851900
Bug: 553599402
Change-Id: I6e310459d9c1a3ba7978f7907ae6b7a9bc611749
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/632146
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Use porcelain-v2 branch and path records to replace the status display
fan-out with one Git process on Git 2.11 and newer. Preserve the legacy
renderer for older clients, force rename and ahead/behind behavior where
supported, and fall back to the old divergence query only when status
cannot supply branch.ab.
Bug: 543851900
Bug: 553599402
Change-Id: Ic760360f4f48ab9320abf69dbbb7310dab4ac649
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/632144
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
On Git 2.11 and newer, replace update-index, diff-index, diff-files, and
ls-files fan-out with one porcelain-v2 snapshot for IsDirty,
UncommittedFiles, and HasChanges. Preserve category ordering and
duplicate staged-plus-unstaged paths, and keep the exact legacy path for
older Git or a failed snapshot.
Bug: 543851900
Bug: 553599402
Change-Id: I309e39977e1f1d4ee6115f488d6aabb83ec3d83c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/632143
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
In partial clone checkouts (e.g. --filter=blob:none or blob:limit=10k),
DownloadPatchSet previously invoked `git fetch <remote> <ref>` without
explicit filter overrides. Consequently, git fetch applied the
configured partial clone filter from the remote and omitted the blobs
for the incoming change.
When repo download subsequently checks out, cherry-picks, or merges the
downloaded commit, Git discovers that the required blobs are missing
locally and triggers on-demand unadvertised object fetches to the
promisor remote (POST /git-upload-pack with `want <blob_sha>`). On large
repositories, server-side reachability validation for unadvertised
objects can result in expensive graph traversals, proxy/gateway timeouts
(such as HTTP 502 or connection drops), and client fetch failures:
fatal: expected 'packfile'
fatal: could not fetch <sha> from promisor remote
Pass --no-filter (when Git >= 2.17.0) and --no-tags when fetching patch
sets in DownloadPatchSet. Because the change ref is an advertised ref
tip, fetching with --no-filter allows the server to generate a complete
packfile containing all required blobs upfront without requiring
unadvertised object reachability checks. All blobs are present in the
local object database prior to checkout or cherry-pick, completely
preventing on-demand promisor queries.
TAG=agy
CONV=bdc489e3-b20b-4ef3-b32e-549dbe21fc6c
Google-Bug-Id: b/532920848
Test: pytest tests/test_project.py
Change-Id: Ic6118e4edeb754f0467e0580fd1d0965a6553153
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/625621
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Sam Saccone <samccone@google.com>
Commit-Queue: Sam Saccone <samccone@google.com>
Add `repo.reprojectcmd`, the checkout counterpart to `repo.fetchcmd`.
When set, `repo sync` runs this command instead of Git to materialize a
project's index and worktree at the target tree, while `repo` handles
ref updates directly via `git update-ref`.
This replaces Git's tree materialization steps: detaching HEAD,
fast-forwarding, and hard-resetting. Rebasing is not delegated, and the
command is skipped if HEAD is already at the target, if HEAD is ahead of
the target during fast-forward, or for MetaProjects.
The command runs in a subshell with project environment variables (such
as `REPO_TREV`). Before running, `repo` ensures no operation is in
progress and no staged changes exist. Worktree collision detection is
delegated to the command (preserving benign unstaged/untracked edits).
Afterward, `repo` verifies that HEAD was untouched and that the index
matches the target tree.
Like `repo.fetchcmd`, this requires `repo.uselocalgitdirs`. Nested
projects and submodules are unsupported; `repo sync` fails if the
manifest contains any while `repo.reprojectcmd` is enabled.
Verified end-to-end with repo init using local-gitdirs, repo.fetchcmd,
and repo.reprojectcmd ('git -C $REPO_PATH read-tree -m -u $REPO_TREV'):
* Verified detached HEAD checkout and correct reflog generation across
projects.
* Verified benign unstaged edits and untracked files survive checkout.
* Verified conflicting untracked files fail with exit 128 without
clobbering worktree.
* Verified staged changes fail upfront before reprojectcmd is executed.
Bug: 513329573
Change-Id: I964d24d22dccffc05a9b991ad69f3a7e93268c01
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/626281
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
"repo status" aborts the whole tree when one project fails, without
saying why. The three callers that refresh the index stat cache pass -q,
which suppresses git's diagnosis when it cannot create the index lock;
git then exits 128 with both streams empty. A stale lock, a permissions
problem on the gitdir and a full disk all reach that same silent exit,
yet each needs a different remedy.
Route the three identical call sites through one helper that keeps -q
for the refresh itself. When that fails, it repeats the command without
-q to obtain git's diagnosis. The command still fails, which is correct,
but the user is now told why.
Keeping -q on the common path matters: without it git exits 1 whenever
paths hold uncommitted changes, which would report a failure to
telemetry for every modified project, even if we treat it as benign in
the code. The retry accepts that 1, since --unmerged makes git skip
conflicted paths rather than count them, so nothing else yields it.
The similar call in subcmds/sync.py is unchanged: it passes different
flags, without --unmerged, so exit 1 cannot be assumed benign there.
Bug: 560289756
Change-Id: I2e1024c84454f86076081f2d7f155728b1aa35c6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/630641
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
In _CheckForImmutableRevision, commit d9cc0a15 restricted upstream ref
validation strictly to superprojects (if use_superproject) to prevent
shallow clones with an upstream attribute from failing ancestry checks
and falling back to full clones.
However, restricting this check exclusively to superprojects broke
non-shallow projects with pinned immutable revisions (such as in Smart
Sync, `repo sync -t <BUILD_ID>`, or pinned manifests): When the target
commit already exists in the local Git object store (for instance,
prefetched into refs/prefetch/ by a background daemon or via shared
object dirs), _CheckForImmutableRevision returned True without verifying
that the local tracking ref (refs/remotes/<remote>/<upstream>) is
present and reaches the revision. As a result, _RemoteFetch skipped
fetching the upstream branch, leaving the local tracking ref stale or
missing. Subsequent `repo start` branches tracking that remote branch
diverged, causing `repo upload` to attempt uploading all intermediate
commits between the stale tracking ref and HEAD.
Restore upstream ancestry validation in _CheckForImmutableRevision for
non-shallow projects when upstream is specified. Also pass the sync
depth into _CheckForImmutableRevision call sites so shallow checkouts
continue to skip upstream verification and avoid triggering un-shallow
fallbacks.
Test: PYTHONPATH=. pytest tests/test_project.py
Change-Id: Ib5bdf41810bb06c8ed053447209ddc2e488f3913
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/624361
Tested-by: Rahul Yadav <yadavrah@google.com>
Commit-Queue: Rahul Yadav <yadavrah@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
GIT_DEFAULT_BRANCH support was added to git var in Git 2.35. Skip that
guaranteed failure on older supported clients and read
init.defaultBranch directly, reducing their fallback from two processes
to one.
Bug: 553599402
Change-Id: I079e9e6057b56afa41b685317a106fc9cdbad8a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623907
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Attach the current marker computed by GetBranches to uploadable and
prunable branches. Overview and prune reporting can then reuse that
result instead of resolving HEAD once per displayed branch.
Bug: 553599402
Change-Id: Ie3e23d08d0caf66c47cc12d7295ec4b6a834560c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623184
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Load refs before resolving the current branch, compare cached HEAD and
manifest OIDs instead of walking their symmetric difference, and reuse
the snapshot when preserving bare HEAD. Apply the same HEAD reuse to
abandon and MetaProject checks, and bound the latter revision walk to
one commit.
Bug: 553599402
Change-Id: I715e65dcfa02c0df4aaa08da5b7bfbb04ca17725
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623183
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Resolve commit-ish values through one typed helper using rev-parse
--verify --quiet and, on Git 2.30+, --end-of-options. Reject option-like
revisions on older clients, reuse the helper for project and manifest
resolution, and validate user branch names with check-ref-format
--branch.
Bug: 553599402
Change-Id: I2dda49ff3e781cec14d3b84263a8dace06b4073f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623182
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
When the regular checkout has already loaded GitRefs, derive its current
branch and HEAD object from that snapshot instead of spawning
symbolic-ref or rev-parse again. Keep linked Git worktrees on their
worktree-specific HEAD path because the shared repository HEAD is not
authoritative for them.
Bug: 543851900
Bug: 553599402
Change-Id: Iec456be5f3c279865b4165a75d1c6df36e331dbc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623181
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Benchmark (100 untouched projects):
* GetUploadableBranches: 0.985s -> 0.0004s (2,200x faster)
* repo upload (3,045 projects): saves ~2.5s of redundant ref scans
During `repo upload`, `GetUploadableBranches()` currently scans
`self._allrefs` across all projects in the workspace. In large
manifests (e.g. Android with 3,000+ projects), `self._allrefs`
executes `git for-each-ref`, `git symbolic-ref`, and full filesystem
mtime traversals on thousands of projects where no local branch was
ever started or modified.
Any branch that can be uploaded for review must have upstream tracking
configured via `[branch "..."]` sections in `.git/config`
(otherwise `branch.LocalMerge` is None and `GetUploadableBranch`
returns None).
This patch introduces early-pruning fast paths:
1. When no branch subsections exist in `.git/config`
(or only 2-part keys like `branch.autosetupmerge`), return []
immediately without touching `_allrefs`.
2. When a `selected_branch` is specified (e.g. `repo upload --br=...`),
query `branch.LocalMerge`, `refs/heads/<branch>`, and
`refs/published/<branch>` directly, skipping the whole-tree
ref scan.
3. In `subcmds/upload.py`, optimize `_GetMergeBranch` to resolve the
merge branch in-memory via `project.CurrentBranch` and
`project.GetBranch()` instead of spawning two `GitCommand`
processes.
Test: ./run_tests tests/test_project.py tests/test_subcmds_upload.py
Change-Id: I3c2c9d63a68a393d478e1fb50f548ec8b7d44639
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623021
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: James Hawkins <jhawkins@google.com>
Tested-by: James Hawkins <jhawkins@google.com>
Benchmark:
* repo upload frameworks/base: 2.84s -> 0.50s (-82.3%, 5.6x faster)
* repo upload (3,045 projects): 7.66s -> 5.24s (-31.6%, 2.42s saved)
From repo's inception through v2.56, GetHead() directly read the
`.git/HEAD` file in Python. In commit 52bab0ba ("project: Use git
rev-parse to read HEAD"), this was replaced with git subprocess calls
on the premise that git provides a dedicated command. However, in
large multi-project workspaces (such as Android with 3,000+ projects),
spawning thousands of git processes introduced severe latency
regressions during `repo upload` and `repo status`.
Furthermore, switching to subprocesses broke detached HEADs and
unborn branches (fixed in commits 7f7d70ef and 8c3585f3 by re-adding
the v2.56 file-reading logic as an error recovery fallback).
This patch restores fast in-memory reading as the primary path, while
adding modern defensive safeguards:
* Symbolic refs (`ref: refs/heads/...`): Strips whitespace and tabs
and returns the ref directly in memory.
* Detached HEAD: Validates 40-char SHA-1 and 64-char SHA-256 commit
hashes via git_config.IsId(), normalizing to lowercase.
* Symlinks: Detects filesystem symlinks via os.path.islink() and
safely falls back to git symbolic-ref.
* Fallback: Catches (OSError, AssertionError) and transparently falls
back to native git commands for reftables, unexpected layouts, or
filesystem errors. Unifies recovery fallback parsing with the fast
path (CRLF/tabs, lowercase hashes, and consistent RelPath errors).
In addition, this change substantially expands test coverage in
tests/test_project.py, adding comprehensive unit tests for symbolic
refs, whitespace/tabs, CRLF line endings, SHA-1, SHA-256, uppercase
hash normalization, symlinks, corrupted worktrees, and fallback
robustness.
Test: ./run_tests tests/test_project.py
Change-Id: Ib5c2530117c6939e4b9293feda81aa745c003c6a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623001
Commit-Queue: James Hawkins <jhawkins@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: James Hawkins <jhawkins@google.com>
https://gerrit-review.googlesource.com/614762 resolves a fallback
upstream for a SHA-1-pinned project that has no explicit upstream, so
--current-branch can narrow the fetch to one branch instead of fetching
all heads.
It drew that fallback from dest-branch and from the manifest <default>
upstream, dest-branch, and revision. dest-branch and tags are not valid
fallback candidates:
* dest-branch is the review destination for upload, not a fetch
source.
* Tags are meant to be immutable snapshots of a single tagged commit,
and fetching it cannot retrieve an arbitrary pinned SHA-1.
Restrict the fallback to branch heads resolved from the manifest
upstream and revision defaults, and drop dest-branch entirely.
Bug: 541240657
Bug: 544041102
Change-Id: I89ae86891ee166f4c340553ac5a9068efcc18461
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/622541
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
This patch intercepts the output during a successful upload
execution to capture the generated CL URLs. It logs a dynamically
constructed "repo.uploadstate" data event to the active trace2
log containing:
- Uploaded CL URLs
- Target remote name
- Source branch
- Modified files
Test:
1. ./run_tests
2. pytest tests/test_project.py
3. Manual verification:
- Created a dummy branch `test_upload_branch` in a project with local file changes.
- Invoked repo upload passing an explicit trace output directory:
`repo --git-trace2-event-log=/tmp/trace2out upload --dry-run --no-verify art`
- Verified the injected payload successfully appeared on disk within the generated log:
```json
{"event":"data",..."key":"repo.uploadstate/cls","value":""}
{"event":"data",..."key":"repo.uploadstate/remote","value":"goog"}
{"event":"data",..."key":"repo.uploadstate/branch","value":"test_upload_branch"}
{"event":"data",..."key":"repo.uploadstate/files","value":"dummy_file.txt"}
```
(cls is correctly blank on --dry-run but populates on real HTTP pushes)
Bug: 543953499
Change-Id: I9c402a32d01d156d42cf24eaa2e60e22710b5e6f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/616581
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Ram Peri <ramperi@google.com>
Commit-Queue: Ram Peri <ramperi@google.com>
A discovered submodule becomes a project whose revision is the gitlink
read from its parent at manifest load time, which is before the parent
has been fetched. Remember which gitlink a submodule was derived from,
and add a way to read those gitlinks again, so that callers can resolve
a submodule revision once its parent is up-to-date.
A revision that is set does not have to be fetched, so verify its
objects are really there. Without them a submodule that was removed
cannot be told apart from one that was never fetched, which the caller
has to know about.
Bug: 550074864
Change-Id: I4f35d53607126508eec6f02158a3fb604917a199
Signed-off-by: kimhappy <hwanhee.kim@laplacian.cc>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/621661
Reviewed-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Avoid disabling --current-branch when syncing a SHA-1 revision without
an explicit project upstream (e.g., smart tags). Resolve a fallback
upstream from dest-branch or manifest defaults so -c only fetches the
target branch.
Bug: 541240657
Change-Id: Ib44b6a732131210e1ec3a3136747d1a19bc5aa18
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/614762
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Mike Frysinger
Gavin Mak
John Cai
hungweichen
Romaric Jodin
Brian Gan
LaMont Jones
Ram Peri
Sam Saccone
Rahul Yadav
James Hawkins
kimhappy