Skip stateless prune planning when a project's worktree directory
does not exist, preventing fetch failures on subsequent syncs.
Also run _LsRemote() bare so it does not depend on worktree
existence.
Bug: 565047698
Change-Id: I385cba4e06813d9113ec7ddc963e8499acffaec5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/634426
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Skip shallow clone bloat checks during --network-only syncs and
filter out projects whose worktree directory does not exist on disk.
Also return None quietly from _GetStatusSnapshot() when a project
lacks an existing worktree directory.
Bug: 565047698
Change-Id: Ia4c7db36a2bd78bc89e7b72f7a492b901b50efa7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/634425
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
When a status snapshot fails, fall back directly to _IsDirtyLegacy()
instead of IsDirty() to avoid running git status twice.
Expose Project.GetDirtyAndHead() so sync bloat checks can inspect
worktree state without invoking private Project methods.
Bug: 565047698
Change-Id: I6432ed82da88778eec9d151fd5b70bedb1efc62b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/634424
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Skip the detached-history walk when its count will not be printed, ask
rev-list --count for upstream gain, and cap the published-but-unmerged
existence probe at one commit. These preserve the decisions while
reducing traversal and output work.
Bug: 553599402
Change-Id: I30f769c9aa5269428ece15ffcb45c50993d7c4b8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/634004
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Every caller expects _FastForward to reject a merge commit. Always pass
--ff-only and remove the optional parameter so a violated ancestry
assumption fails instead of silently creating a merge.
Bug: 553599402
Change-Id: I04690b5ca675cb7d3062df08d5e20cd82f5b9e82
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/634003
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
When Git 2.35 or newer supplies a stash header, use the same status
snapshot for stateless dirty and stash checks. Retain rev-parse for
earlier clients or when the status snapshot is unavailable.
Bug: 543851900
Bug: 553599402
Change-Id: I6e310459d9c1a3ba7978f7907ae6b7a9bc611749
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/632146
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Use porcelain-v2 branch and path records to replace the status display
fan-out with one Git process on Git 2.11 and newer. Preserve the legacy
renderer for older clients, force rename and ahead/behind behavior where
supported, and fall back to the old divergence query only when status
cannot supply branch.ab.
Bug: 543851900
Bug: 553599402
Change-Id: Ic760360f4f48ab9320abf69dbbb7310dab4ac649
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/632144
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
On Git 2.11 and newer, replace update-index, diff-index, diff-files, and
ls-files fan-out with one porcelain-v2 snapshot for IsDirty,
UncommittedFiles, and HasChanges. Preserve category ordering and
duplicate staged-plus-unstaged paths, and keep the exact legacy path for
older Git or a failed snapshot.
Bug: 543851900
Bug: 553599402
Change-Id: I309e39977e1f1d4ee6115f488d6aabb83ec3d83c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/632143
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
In partial clone checkouts (e.g. --filter=blob:none or blob:limit=10k),
DownloadPatchSet previously invoked `git fetch <remote> <ref>` without
explicit filter overrides. Consequently, git fetch applied the
configured partial clone filter from the remote and omitted the blobs
for the incoming change.
When repo download subsequently checks out, cherry-picks, or merges the
downloaded commit, Git discovers that the required blobs are missing
locally and triggers on-demand unadvertised object fetches to the
promisor remote (POST /git-upload-pack with `want <blob_sha>`). On large
repositories, server-side reachability validation for unadvertised
objects can result in expensive graph traversals, proxy/gateway timeouts
(such as HTTP 502 or connection drops), and client fetch failures:
fatal: expected 'packfile'
fatal: could not fetch <sha> from promisor remote
Pass --no-filter (when Git >= 2.17.0) and --no-tags when fetching patch
sets in DownloadPatchSet. Because the change ref is an advertised ref
tip, fetching with --no-filter allows the server to generate a complete
packfile containing all required blobs upfront without requiring
unadvertised object reachability checks. All blobs are present in the
local object database prior to checkout or cherry-pick, completely
preventing on-demand promisor queries.
TAG=agy
CONV=bdc489e3-b20b-4ef3-b32e-549dbe21fc6c
Google-Bug-Id: b/532920848
Test: pytest tests/test_project.py
Change-Id: Ic6118e4edeb754f0467e0580fd1d0965a6553153
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/625621
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Sam Saccone <samccone@google.com>
Commit-Queue: Sam Saccone <samccone@google.com>
Add `repo.reprojectcmd`, the checkout counterpart to `repo.fetchcmd`.
When set, `repo sync` runs this command instead of Git to materialize a
project's index and worktree at the target tree, while `repo` handles
ref updates directly via `git update-ref`.
This replaces Git's tree materialization steps: detaching HEAD,
fast-forwarding, and hard-resetting. Rebasing is not delegated, and the
command is skipped if HEAD is already at the target, if HEAD is ahead of
the target during fast-forward, or for MetaProjects.
The command runs in a subshell with project environment variables (such
as `REPO_TREV`). Before running, `repo` ensures no operation is in
progress and no staged changes exist. Worktree collision detection is
delegated to the command (preserving benign unstaged/untracked edits).
Afterward, `repo` verifies that HEAD was untouched and that the index
matches the target tree.
Like `repo.fetchcmd`, this requires `repo.uselocalgitdirs`. Nested
projects and submodules are unsupported; `repo sync` fails if the
manifest contains any while `repo.reprojectcmd` is enabled.
Verified end-to-end with repo init using local-gitdirs, repo.fetchcmd,
and repo.reprojectcmd ('git -C $REPO_PATH read-tree -m -u $REPO_TREV'):
* Verified detached HEAD checkout and correct reflog generation across
projects.
* Verified benign unstaged edits and untracked files survive checkout.
* Verified conflicting untracked files fail with exit 128 without
clobbering worktree.
* Verified staged changes fail upfront before reprojectcmd is executed.
Bug: 513329573
Change-Id: I964d24d22dccffc05a9b991ad69f3a7e93268c01
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/626281
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
"repo status" aborts the whole tree when one project fails, without
saying why. The three callers that refresh the index stat cache pass -q,
which suppresses git's diagnosis when it cannot create the index lock;
git then exits 128 with both streams empty. A stale lock, a permissions
problem on the gitdir and a full disk all reach that same silent exit,
yet each needs a different remedy.
Route the three identical call sites through one helper that keeps -q
for the refresh itself. When that fails, it repeats the command without
-q to obtain git's diagnosis. The command still fails, which is correct,
but the user is now told why.
Keeping -q on the common path matters: without it git exits 1 whenever
paths hold uncommitted changes, which would report a failure to
telemetry for every modified project, even if we treat it as benign in
the code. The retry accepts that 1, since --unmerged makes git skip
conflicted paths rather than count them, so nothing else yields it.
The similar call in subcmds/sync.py is unchanged: it passes different
flags, without --unmerged, so exit 1 cannot be assumed benign there.
Bug: 560289756
Change-Id: I2e1024c84454f86076081f2d7f155728b1aa35c6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/630641
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
In _CheckForImmutableRevision, commit d9cc0a15 restricted upstream ref
validation strictly to superprojects (if use_superproject) to prevent
shallow clones with an upstream attribute from failing ancestry checks
and falling back to full clones.
However, restricting this check exclusively to superprojects broke
non-shallow projects with pinned immutable revisions (such as in Smart
Sync, `repo sync -t <BUILD_ID>`, or pinned manifests): When the target
commit already exists in the local Git object store (for instance,
prefetched into refs/prefetch/ by a background daemon or via shared
object dirs), _CheckForImmutableRevision returned True without verifying
that the local tracking ref (refs/remotes/<remote>/<upstream>) is
present and reaches the revision. As a result, _RemoteFetch skipped
fetching the upstream branch, leaving the local tracking ref stale or
missing. Subsequent `repo start` branches tracking that remote branch
diverged, causing `repo upload` to attempt uploading all intermediate
commits between the stale tracking ref and HEAD.
Restore upstream ancestry validation in _CheckForImmutableRevision for
non-shallow projects when upstream is specified. Also pass the sync
depth into _CheckForImmutableRevision call sites so shallow checkouts
continue to skip upstream verification and avoid triggering un-shallow
fallbacks.
Test: PYTHONPATH=. pytest tests/test_project.py
Change-Id: Ib5bdf41810bb06c8ed053447209ddc2e488f3913
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/624361
Tested-by: Rahul Yadav <yadavrah@google.com>
Commit-Queue: Rahul Yadav <yadavrah@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
GIT_DEFAULT_BRANCH support was added to git var in Git 2.35. Skip that
guaranteed failure on older supported clients and read
init.defaultBranch directly, reducing their fallback from two processes
to one.
Bug: 553599402
Change-Id: I079e9e6057b56afa41b685317a106fc9cdbad8a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623907
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Attach the current marker computed by GetBranches to uploadable and
prunable branches. Overview and prune reporting can then reuse that
result instead of resolving HEAD once per displayed branch.
Bug: 553599402
Change-Id: Ie3e23d08d0caf66c47cc12d7295ec4b6a834560c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623184
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Load refs before resolving the current branch, compare cached HEAD and
manifest OIDs instead of walking their symmetric difference, and reuse
the snapshot when preserving bare HEAD. Apply the same HEAD reuse to
abandon and MetaProject checks, and bound the latter revision walk to
one commit.
Bug: 553599402
Change-Id: I715e65dcfa02c0df4aaa08da5b7bfbb04ca17725
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623183
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Resolve commit-ish values through one typed helper using rev-parse
--verify --quiet and, on Git 2.30+, --end-of-options. Reject option-like
revisions on older clients, reuse the helper for project and manifest
resolution, and validate user branch names with check-ref-format
--branch.
Bug: 553599402
Change-Id: I2dda49ff3e781cec14d3b84263a8dace06b4073f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623182
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
When the regular checkout has already loaded GitRefs, derive its current
branch and HEAD object from that snapshot instead of spawning
symbolic-ref or rev-parse again. Keep linked Git worktrees on their
worktree-specific HEAD path because the shared repository HEAD is not
authoritative for them.
Bug: 543851900
Bug: 553599402
Change-Id: Iec456be5f3c279865b4165a75d1c6df36e331dbc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623181
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Benchmark (100 untouched projects):
* GetUploadableBranches: 0.985s -> 0.0004s (2,200x faster)
* repo upload (3,045 projects): saves ~2.5s of redundant ref scans
During `repo upload`, `GetUploadableBranches()` currently scans
`self._allrefs` across all projects in the workspace. In large
manifests (e.g. Android with 3,000+ projects), `self._allrefs`
executes `git for-each-ref`, `git symbolic-ref`, and full filesystem
mtime traversals on thousands of projects where no local branch was
ever started or modified.
Any branch that can be uploaded for review must have upstream tracking
configured via `[branch "..."]` sections in `.git/config`
(otherwise `branch.LocalMerge` is None and `GetUploadableBranch`
returns None).
This patch introduces early-pruning fast paths:
1. When no branch subsections exist in `.git/config`
(or only 2-part keys like `branch.autosetupmerge`), return []
immediately without touching `_allrefs`.
2. When a `selected_branch` is specified (e.g. `repo upload --br=...`),
query `branch.LocalMerge`, `refs/heads/<branch>`, and
`refs/published/<branch>` directly, skipping the whole-tree
ref scan.
3. In `subcmds/upload.py`, optimize `_GetMergeBranch` to resolve the
merge branch in-memory via `project.CurrentBranch` and
`project.GetBranch()` instead of spawning two `GitCommand`
processes.
Test: ./run_tests tests/test_project.py tests/test_subcmds_upload.py
Change-Id: I3c2c9d63a68a393d478e1fb50f548ec8b7d44639
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623021
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: James Hawkins <jhawkins@google.com>
Tested-by: James Hawkins <jhawkins@google.com>
Benchmark:
* repo upload frameworks/base: 2.84s -> 0.50s (-82.3%, 5.6x faster)
* repo upload (3,045 projects): 7.66s -> 5.24s (-31.6%, 2.42s saved)
From repo's inception through v2.56, GetHead() directly read the
`.git/HEAD` file in Python. In commit 52bab0ba ("project: Use git
rev-parse to read HEAD"), this was replaced with git subprocess calls
on the premise that git provides a dedicated command. However, in
large multi-project workspaces (such as Android with 3,000+ projects),
spawning thousands of git processes introduced severe latency
regressions during `repo upload` and `repo status`.
Furthermore, switching to subprocesses broke detached HEADs and
unborn branches (fixed in commits 7f7d70ef and 8c3585f3 by re-adding
the v2.56 file-reading logic as an error recovery fallback).
This patch restores fast in-memory reading as the primary path, while
adding modern defensive safeguards:
* Symbolic refs (`ref: refs/heads/...`): Strips whitespace and tabs
and returns the ref directly in memory.
* Detached HEAD: Validates 40-char SHA-1 and 64-char SHA-256 commit
hashes via git_config.IsId(), normalizing to lowercase.
* Symlinks: Detects filesystem symlinks via os.path.islink() and
safely falls back to git symbolic-ref.
* Fallback: Catches (OSError, AssertionError) and transparently falls
back to native git commands for reftables, unexpected layouts, or
filesystem errors. Unifies recovery fallback parsing with the fast
path (CRLF/tabs, lowercase hashes, and consistent RelPath errors).
In addition, this change substantially expands test coverage in
tests/test_project.py, adding comprehensive unit tests for symbolic
refs, whitespace/tabs, CRLF line endings, SHA-1, SHA-256, uppercase
hash normalization, symlinks, corrupted worktrees, and fallback
robustness.
Test: ./run_tests tests/test_project.py
Change-Id: Ib5c2530117c6939e4b9293feda81aa745c003c6a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623001
Commit-Queue: James Hawkins <jhawkins@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: James Hawkins <jhawkins@google.com>
https://gerrit-review.googlesource.com/614762 resolves a fallback
upstream for a SHA-1-pinned project that has no explicit upstream, so
--current-branch can narrow the fetch to one branch instead of fetching
all heads.
It drew that fallback from dest-branch and from the manifest <default>
upstream, dest-branch, and revision. dest-branch and tags are not valid
fallback candidates:
* dest-branch is the review destination for upload, not a fetch
source.
* Tags are meant to be immutable snapshots of a single tagged commit,
and fetching it cannot retrieve an arbitrary pinned SHA-1.
Restrict the fallback to branch heads resolved from the manifest
upstream and revision defaults, and drop dest-branch entirely.
Bug: 541240657
Bug: 544041102
Change-Id: I89ae86891ee166f4c340553ac5a9068efcc18461
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/622541
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
This patch intercepts the output during a successful upload
execution to capture the generated CL URLs. It logs a dynamically
constructed "repo.uploadstate" data event to the active trace2
log containing:
- Uploaded CL URLs
- Target remote name
- Source branch
- Modified files
Test:
1. ./run_tests
2. pytest tests/test_project.py
3. Manual verification:
- Created a dummy branch `test_upload_branch` in a project with local file changes.
- Invoked repo upload passing an explicit trace output directory:
`repo --git-trace2-event-log=/tmp/trace2out upload --dry-run --no-verify art`
- Verified the injected payload successfully appeared on disk within the generated log:
```json
{"event":"data",..."key":"repo.uploadstate/cls","value":""}
{"event":"data",..."key":"repo.uploadstate/remote","value":"goog"}
{"event":"data",..."key":"repo.uploadstate/branch","value":"test_upload_branch"}
{"event":"data",..."key":"repo.uploadstate/files","value":"dummy_file.txt"}
```
(cls is correctly blank on --dry-run but populates on real HTTP pushes)
Bug: 543953499
Change-Id: I9c402a32d01d156d42cf24eaa2e60e22710b5e6f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/616581
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Ram Peri <ramperi@google.com>
Commit-Queue: Ram Peri <ramperi@google.com>
A discovered submodule becomes a project whose revision is the gitlink
read from its parent at manifest load time, which is before the parent
has been fetched. Remember which gitlink a submodule was derived from,
and add a way to read those gitlinks again, so that callers can resolve
a submodule revision once its parent is up-to-date.
A revision that is set does not have to be fetched, so verify its
objects are really there. Without them a submodule that was removed
cannot be told apart from one that was never fetched, which the caller
has to know about.
Bug: 550074864
Change-Id: I4f35d53607126508eec6f02158a3fb604917a199
Signed-off-by: kimhappy <hwanhee.kim@laplacian.cc>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/621661
Reviewed-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Avoid disabling --current-branch when syncing a SHA-1 revision without
an explicit project upstream (e.g., smart tags). Resolve a fallback
upstream from dest-branch or manifest defaults so -c only fetches the
target branch.
Bug: 541240657
Change-Id: Ib44b6a732131210e1ec3a3136747d1a19bc5aa18
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/614762
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
When the default branch cannot be determined dynamically, repo fell back
to a hardcoded "refs/heads/master". This can point at a branch that does
not exist on the server, since projects increasingly default to "main".
Rather than swap one hardcoded name for another, ask git itself what it
would use.
Bug: 483758905
Change-Id: Ic66712acea98f8e548a2d6d8211865ee5c416b4d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/612845
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
When both `git symbolic-ref` and `git rev-parse` fail to resolve HEAD,
GetHead falls back to reading .git/HEAD directly. With the "reftable" ref
backend, .git/HEAD is only a stub pointing at the "refs/heads/.invalid"
placeholder while the real HEAD lives in the reftable stack. Reading the
stub returned that bogus placeholder value.
Detect the placeholder and treat HEAD as unresolvable (raising
NoManifestException) instead of returning the invalid ref.
Bug: 483758905
Change-Id: Ia0a825c0d1874686b98c5ddcdea6dc26c0d46784
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/612882
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
The superproject-gated upstream check in _CheckForImmutableRevision
only applies to user projects listed in the manifest. MetaProjects
(ManifestProject and RepoProject) never participate in a superproject
relationship, so evaluating git_superproject.UseSuperproject(...,
self.manifest) for them serves no purpose and, worse, calls the
manifest.superproject property which forces a manifest load.
During repo init, ManifestProject._ConfigureDependencies calls
self.Sync_NetworkHalf before manifest.xml has been linked into
.repo/. That reaches _CheckForImmutableRevision, which triggered the
manifest load and failed with:
ManifestParseError: .../.repo/manifest.xml: [Errno 2] No such file
or directory
breaking fresh repo init with SHA-based --manifest-branch combined
with --manifest-upstream-branch.
Factor the "should we consult the superproject for upstream?"
decision into a small overridable hook, _UseSuperprojectForUpstream.
Project's default delegates to git_superproject.UseSuperproject;
MetaProject overrides it to return False, localizing the
MetaProject-specific behavior to MetaProject.
Test verifies that calling _CheckForImmutableRevision on a
ManifestProject whose manifest.xml is not yet on disk returns False
without raising and does not create the file.
Change-Id: I22059109243d914036c06c6fe0081a5aba05da89
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574201
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Ajay Gupta <ajagup@qti.qualcomm.com>
Tested-by: Ajay Gupta <ajagup@qti.qualcomm.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
Introduce support for `repo.fetchcmd` configuration, allowing users to
specify a custom command to fetch project data instead of
`_RemoteFetch`.
When `repo.fetchcmd` is specified, `repo` will execute it instead of
`_RemoteFetch` during the network half of sync. Note that
`repo.fetchcmd` requires `repo.uselocalgitdirs` to be enabled.
The command is executed in a subshell with project-context environment
variables, including the new `REPO_TREV` (target revision resolved to a
commit hash) and `REPO_PROJECT_FETCH_URL`.
After execution, `repo` verifies that the target commit is available and
that the tracking ref and FETCH_HEAD are correctly updated.
Tested with:
```
> ~/git-repo/repo init -u https://android-review.googlesource.com/platform/manifest \
--repo-url file:///usr/local/google/home/gavinmak/git-repo \
--groups developers \
--no-repo-verify \
--use-local-gitdirs
...
> git config --file .repo/manifests.git/config repo.fetchcmd 'mkdir -p $REPO_PATH && cd $REPO_PATH && if [ ! -d .git ]; then git init && git remote add aosp $REPO_PROJECT_FETCH_URL; fi && git fetch aosp $REPO_TREV && git reset --hard $REPO_TREV && mkdir -p .git/refs/remotes/aosp && echo $REPO_TREV > .git/refs/remotes/aosp/main && echo $REPO_TREV > .git/FETCH_HEAD'
> ~/git-repo/repo sync -j32
warning: repo is not tracking a remote branch, so it will not receive updates; run `repo init --repo-rev=stable` to fix.
You are currently enrolled in Git submodules experiment (go/android-submodules-quickstart). Use --no-use-superproject to override.
Syncing: 100% (4/4), done in 1m15.686s
Finalizing sync state...
repo sync has finished successfully.
```
Bug: 513329573
Change-Id: I754d3f3c78e86fdeee1a72115297a75b571bc497
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/583883
Reviewed-by: Becky Siegel <beckysiegel@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
When viewing `repo status`, it is difficult to distinguish between branches that have active unpushed changes and stale branches that are fully synced. Previously, developers had to run commands like `repo forall -c "git status"` to see their ahead/behind counts.
This change updates Project.PrintWorkTreeStatus to automatically calculate and display the number of commits a branch is ahead and/or behind its upstream tracking branch. We use `git rev-list --left-right --count` to fetch this information natively and efficiently.
If the branch is completely synced with upstream, no extra text is shown.
Added tests for ahead-only, behind-only, diverged, no-tracking, and fully-synced branch states.
Bug: 319412954
Change-Id: I23879b2d472c7a7e11d01b565428a84b1b4f09c1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/602423
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
When `repo info` runs, the reported "Current revision" is resolved using
the manifest's target branch tracking ref (e.g. refs/remotes/goog/main).
Introduce Project.GetHeadRevisionId(), which gets the checked-out HEAD
commit in the worktree, and use it in `repo info` with a fallback to the
old behavior if the project is not checked out.
Bug: 526685287
Change-Id: I72280ce27daa210cada27d722a94e365644f06e0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/599481
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Repo sync fails when the following conditions are met:
* There are several checkouts of the same project
in different paths.
* The checkouts are using git hashes as revisions
(not branches).
* There is a clone-depth set on these projects.
* sync-c="true" is set in the manifest.
* The revision specified in the manifest
has moved forward since the first repo init.
The sync fails because only the first gitdir gets the "shallow" file,
and subsequent dirs can't be synced.
Do not optimize away the fetch when the conditions above happen.
Simplified the boolean check in Sync_NetworkHalf and _RemoteFetch
using has_shallow, renamed loop variable to avoid shadowing.
Test: create a manifest matching conditions above, repo init,
forward the hash, and repo sync.
Test: added tests in test_project.py
Bug: 505072873
Originally-by: Elvira Khabirova <elvira.khabirova@volvocars.com>
Change-Id: I37c533c382e34fc5ddab489c5593b9e5d3875be2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/601441
Tested-by: Sainath Varanasi <varanasisai@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Sainath Varanasi <varanasisai@google.com>
Add REPO_PROJECT_FETCH_URL to Project.GetEnvVars(), which resolves to
the remote fetch URL of the project. This is useful for exposing the
URL to custom fetch commands or other external scripts.
Bug: 513329573
Change-Id: Ic2b0a83493934d16bb1152366ee4e1a2c35ea2dc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/596121
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Commit 5534f16 made UpdateCopyLinkfileList to retry _CopyAndLinkFiles
for all projects in the manifest to handle directory-to-symlink
transistions. However, for partial sync's (e.g. repo sync <project>),
projects not included in the sync may not have their local directories
populated, attempts to perform the operation on unavailable projects
would result in dangling symlinks, or errors like:
error: Cannot copy file <source> to <destination>
Fix this by adding a check inside _CopyAndLinkFiles in project.py to
so these operations happen only when the project's worktree exists
and is a directory.
TAG=agy
CONV=90f5fca7-2199-4017-8f62-a010b0ab1dbf
Change-Id: I9ae1e8d62d8fd129cb4535e574c38339c87af441
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/587501
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Xin Li <delphij@google.com>
When working with relative submodule paths, The "./" needs special
handling similar to "../".
See information on:
https://git-scm.com/docs/git-submodule
Which currently states:
"<repository> is the URL of the new submodule’s origin repository.
This may be either an absolute URL, or (if it begins with ./ or ../),
the location relative to the superproject’s default remote
repository (Please note that to specify a repository foo.git which is
located right next to a superproject bar.git, you’ll have to use
../foo.git instead of ./foo.git - as one might expect when following
the rules for relative URLs - because the evaluation of relative URLs
in Git is identical to that of relative directories)."
The implementation also was not handling file/directory names
starting with "." or "..". Explicitly look for "./" and "../"
instead.
Change-Id: I8ae68d61fb0cbb1624183b175236e98a36e4afdb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/579182
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josef Malmstrom <Josef.Malmstrom@arm.com>
Tested-by: Josef Malmstrom <Josef.Malmstrom@arm.com>
When a manifest changes from individual linkfiles inside a directory
(e.g. dest=".llms/rules", dest=".llms/skills") to a single linkfile
for the whole directory (e.g. dest=".llms", src="dot-llms"), two
things need to happen:
1. __linkIt must replace a real directory with a symlink. Use
os.rmdir() instead of platform_utils.remove() for real directories.
rmdir only removes empty directories, so user-created content is
never deleted.
2. UpdateCopyLinkfileList must handle the cleanup correctly:
- Use os.rmdir() for directories (safe for non-empty)
- Remove empty parent directories after cleaning old dests
- Retry _CopyAndLinkFiles for all projects, since in interleaved
sync mode _CopyAndLinkFiles runs before cleanup and may have
failed because the directory was not yet empty
Change-Id: I0437b80beab98bce064cea81c11c47d699be91aa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/569243
Tested-by: Carlos Fernandez <carlosfsanz@meta.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Carlos Fernandez <carlosfsanz@meta.com>
The force_checkout parameter was not propagated in all calls to
Checkout in Sync_LocalHalf.
Without this, repo sync --force-checkout can still fail for projects
currently on a local branch with no upstream/tracking configuration,
because the detach-to-manifest checkout was executed without -f,
leaving local modifications or untracked files able to block sync.
Change-Id: I58551388e2f906c4db96e220707a369057a71c24
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/579181
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josef Malmstrom <Josef.Malmstrom@arm.com>
Tested-by: Josef Malmstrom <Josef.Malmstrom@arm.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
When optimizing fetches for projects with immutable revisions, the fetch
should not be skipped if the project is configured for a shallow clone
(depth > 0) but the .git/shallow file is missing. The absence of the
.git/shallow file means the repository is not a shallow clone, or the
shallow clone is incomplete, so a fetch is necessary to ensure the
revision is present.
Bug: 503081454
Change-Id: Ic3549612bcd69050a926652ee4e522c79ad8124c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/573821
Tested-by: Becky Siegel <beckysiegel@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Becky Siegel <beckysiegel@google.com>
repo calls `git update-ref --stdin` when updating multiple refs during
repo init and repo sync. Historically, `--no-deref` was also passed.
Older Git 2.17 which we still support rejects the combination of
`--stdin` and `--no-deref`, emitting a usage error even when the stdin
input is valid.
The `--no-deref` option is only meaningful when updating symbolic refs
such as HEAD. The stdin-based update-ref path only operates on explicit
refs (tags, remote refs, alternates) and never symbolic refs.
Remove the unnecessary option to restore compatibility with Git 2.17
while preserving identical behavior on newer Git versions.
Tested with:
- Git 2.17.1
- Git 2.34.1
Change-Id: I22001de03800f5699b26a40bc1fb1fec002ed048
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/571721
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Enei <miyako.enei@alpsalpine.com>
Tested-by: Enei <miyako.enei@alpsalpine.com>
Implement in-situ shallow re-fetching and garbage collection logic.
Enables repositories with sync-strategy="stateless" to reclaim disk
space by running reflog expire and git gc --prune=now if the working
tree is clean and has no local commits.
Bug: 498730431
Change-Id: I940bdc9b74da29d3f7b13566667dcddea769ebd3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/568463
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
The only supported sync-strategy is "stateless". The intent is to keep
the local workspace as small as possible by not keeping history during
syncs. This prevents disk space waste for projects with large binaries
where we only care about the current version.
A follow up change will implement the logic.
Bug: 498730431
Change-Id: I84a436a9ca2492893163c6cfda6c28dc62a568f0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/568462
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
If a user has manually unshallowed a repo (e.g. via
`git fetch --unshallow`), the absence of the `shallow` file in the
gitdir indicates a full clone. Re-applying depth during a subsequent
sync would undo the user's intent. Skip re-shallowing in this case
by clearing depth when the project is not new and no shallow file
is present.
Change-Id: I4ee0e78018de9078fe1bd77a9615613ef0c40d33
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/558743
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Carlos Fernandez <carlosfsanz@meta.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Lucas Alves <ltanure@gmail.com>
Tested-by: Lucas Alves <ltanure@gmail.com>
Reviewed-by: Lucas Alves <ltanure@gmail.com>
Currently somewhere use git_superproject.UseSuperproject(), which checks
both the manifest config and user's config, and otherwhere use
manifest.manifestProject.use_superproject, which only checks the
manifest config. This causes Inconsistent behaviors for users who do not
set --use-superproject when doing repo init but have
repo.superprojectChoice in their git config.
Replace where using manifest.manifestProject.use_superproject with
git_superproject.UseSuperproject() to respect user's config and avoid
inconsistency.
Bug: 454514213
Change-Id: I1f734235cdd67b8a6915f1d05967d1aaa4d03f2a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/561801
Commit-Queue: Jacky Liu <qsliu@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jacky Liu <qsliu@google.com>
Gavin Mak
Sam Saccone
Brian Gan
Rahul Yadav
James Hawkins
Ram Peri
kimhappy
Ajay Gupta
Sainath Varanasi
Xin Li
Josef Malmström
Carlos Fernandez
Becky Siegel
Miyako.Enei
Lucas Tanure
Jacky Liu