Compare commits

...
14 Commits
Author SHA1 Message Date
Gavin Mak 0039e39000 project: carry current state from the ref snapshot
Attach the current marker computed by GetBranches to uploadable and
prunable branches. Overview and prune reporting can then reuse that
result instead of resolving HEAD once per displayed branch.

Bug: 553599402
Change-Id: Ie3e23d08d0caf66c47cc12d7295ec4b6a834560c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623184
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2026-08-31 13:26:22 -07:00
Gavin Mak 5f378458d2 project: reuse ref OIDs in branch maintenance
Load refs before resolving the current branch, compare cached HEAD and
manifest OIDs instead of walking their symmetric difference, and reuse
the snapshot when preserving bare HEAD. Apply the same HEAD reuse to
abandon and MetaProject checks, and bound the latter revision walk to
one commit.

Bug: 553599402
Change-Id: I715e65dcfa02c0df4aaa08da5b7bfbb04ca17725
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623183
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2026-08-31 13:25:40 -07:00
Gavin Mak d27034bf62 project: centralize safe branch and commit resolution
Resolve commit-ish values through one typed helper using rev-parse
--verify --quiet and, on Git 2.30+, --end-of-options. Reject option-like
revisions on older clients, reuse the helper for project and manifest
resolution, and validate user branch names with check-ref-format
--branch.

Bug: 553599402
Change-Id: I2dda49ff3e781cec14d3b84263a8dace06b4073f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623182
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-08-31 12:34:53 -07:00
Gavin Mak 6541729a18 project: reuse loaded refs when resolving HEAD
When the regular checkout has already loaded GitRefs, derive its current
branch and HEAD object from that snapshot instead of spawning
symbolic-ref or rev-parse again. Keep linked Git worktrees on their
worktree-specific HEAD path because the shared repository HEAD is not
authoritative for them.

Bug: 543851900
Bug: 553599402
Change-Id: Iec456be5f3c279865b4165a75d1c6df36e331dbc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623181
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2026-08-31 12:33:39 -07:00
Gavin Mak b85e76a86a git_refs: load HEAD with the ref snapshot on Git 2.45
Use for-each-ref --include-root-refs to resolve attached and detached
HEAD in the same process as refs/* on Git 2.45 and newer. Limit the
patterns to HEAD and refs so volatile pseudo-refs do not enter the
cache, and retain the existing fallback for older Git and unborn HEADs.

Bug: 543851900
Bug: 553599402
Change-Id: Ie091ca0757f21ec6b1e19d49e09d160e055ec639
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/622981
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-08-31 11:57:39 -07:00
James Hawkins e5bbb5c9e6 upload: prune ref scans for untouched projects and specific branches
Benchmark (100 untouched projects):
* GetUploadableBranches: 0.985s -> 0.0004s (2,200x faster)
* repo upload (3,045 projects): saves ~2.5s of redundant ref scans

During `repo upload`, `GetUploadableBranches()` currently scans
`self._allrefs` across all projects in the workspace. In large
manifests (e.g. Android with 3,000+ projects), `self._allrefs`
executes `git for-each-ref`, `git symbolic-ref`, and full filesystem
mtime traversals on thousands of projects where no local branch was
ever started or modified.

Any branch that can be uploaded for review must have upstream tracking
configured via `[branch "..."]` sections in `.git/config`
(otherwise `branch.LocalMerge` is None and `GetUploadableBranch`
returns None).

This patch introduces early-pruning fast paths:
1. When no branch subsections exist in `.git/config`
   (or only 2-part keys like `branch.autosetupmerge`), return []
   immediately without touching `_allrefs`.
2. When a `selected_branch` is specified (e.g. `repo upload --br=...`),
   query `branch.LocalMerge`, `refs/heads/<branch>`, and
   `refs/published/<branch>` directly, skipping the whole-tree
   ref scan.
3. In `subcmds/upload.py`, optimize `_GetMergeBranch` to resolve the
   merge branch in-memory via `project.CurrentBranch` and
   `project.GetBranch()` instead of spawning two `GitCommand`
   processes.

Test: ./run_tests tests/test_project.py tests/test_subcmds_upload.py
Change-Id: I3c2c9d63a68a393d478e1fb50f548ec8b7d44639
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623021
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: James Hawkins <jhawkins@google.com>
Tested-by: James Hawkins <jhawkins@google.com>
2026-08-28 16:07:01 -07:00
James Hawkins 4fe87617ff project: read HEAD directly in-memory to avoid subprocesses
Benchmark:
* repo upload frameworks/base: 2.84s -> 0.50s (-82.3%, 5.6x faster)
* repo upload (3,045 projects): 7.66s -> 5.24s (-31.6%, 2.42s saved)

From repo's inception through v2.56, GetHead() directly read the
`.git/HEAD` file in Python. In commit 52bab0ba ("project: Use git
rev-parse to read HEAD"), this was replaced with git subprocess calls
on the premise that git provides a dedicated command. However, in
large multi-project workspaces (such as Android with 3,000+ projects),
spawning thousands of git processes introduced severe latency
regressions during `repo upload` and `repo status`.

Furthermore, switching to subprocesses broke detached HEADs and
unborn branches (fixed in commits 7f7d70ef and 8c3585f3 by re-adding
the v2.56 file-reading logic as an error recovery fallback).

This patch restores fast in-memory reading as the primary path, while
adding modern defensive safeguards:
* Symbolic refs (`ref: refs/heads/...`): Strips whitespace and tabs
  and returns the ref directly in memory.
* Detached HEAD: Validates 40-char SHA-1 and 64-char SHA-256 commit
  hashes via git_config.IsId(), normalizing to lowercase.
* Symlinks: Detects filesystem symlinks via os.path.islink() and
  safely falls back to git symbolic-ref.
* Fallback: Catches (OSError, AssertionError) and transparently falls
  back to native git commands for reftables, unexpected layouts, or
  filesystem errors. Unifies recovery fallback parsing with the fast
  path (CRLF/tabs, lowercase hashes, and consistent RelPath errors).

In addition, this change substantially expands test coverage in
tests/test_project.py, adding comprehensive unit tests for symbolic
refs, whitespace/tabs, CRLF line endings, SHA-1, SHA-256, uppercase
hash normalization, symlinks, corrupted worktrees, and fallback
robustness.

Test: ./run_tests tests/test_project.py
Change-Id: Ib5c2530117c6939e4b9293feda81aa745c003c6a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/623001
Commit-Queue: James Hawkins <jhawkins@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: James Hawkins <jhawkins@google.com>
2026-08-28 16:03:06 -07:00
kimhappy d7299422ae sync: skip submodules removed from their parent
The submodules to sync are derived from their parent before it is
fetched, so one that the parent's new revision no longer holds a gitlink
for is still synced. Interleaved sync then fails to check it out,
because `git submodule init` no longer knows that path:

  error: Cannot checkout a/c
  error: pathspec '.../a/c' did not match any file(s) known to git

Report such submodules while the gitlinks are read again, and leave them
out of the fetch and of the checkout, so the usual project list update
removes them from the working tree. Phased sync already left them out of
the checkout, but still fetched them.

Nothing is reported as removed while the gitlinks of a parent cannot be
read, since a missing gitlink cannot be told apart from a fetch that did
not happen.

Bug: 550074864
Change-Id: Ia429b35c12758a68e4df73665e5512a201af658a
Signed-off-by: kimhappy <hwanhee.kim@laplacian.cc>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/621681
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-08-27 16:22:54 -07:00
kimhappy 3f087a8dd9 sync: fetch submodules after their parent repository
Phased sync fetches every project at once, so a submodule is fetched at
the gitlink that was read before its parent was fetched. With -c that
stale revision is already present, the fetch is skipped as an immutable
revision, and the checkout then fails on the revision the reloaded
manifest resolved:

  error: Cannot checkout a/b
  fatal: bad object 5556dad99da5df7e53303c5732aacc8782493e03

Fetch in batches, holding a submodule back until the project holding its
gitlink is fetched, and read the gitlinks again between batches. A
manifest without submodules keeps its single batch.

Bug: 550074864
Change-Id: I04c2eb1ca81ab25b4df2c03c5dc048d2e4ecdcb8
Signed-off-by: kimhappy <hwanhee.kim@laplacian.cc>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/621663
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-08-27 16:22:49 -07:00
kimhappy 3a6e25af75 sync: check out submodules at the parent's new revision
Interleaved sync processes projects in hierarchical levels, so a
submodule is only handled after the project holding its gitlink has been
fetched and checked out. Its revision, however, is still the one read
before that parent was fetched, so the submodule is synced to the
revision of the previous sync and only catches up on the next one.

Read the gitlinks again right before a level is dispatched. The parent
is done by then, so its submodules are fetched and checked out at the
revision the parent now points at. Sibling submodules are spread over
several levels, so the gitlinks read for one level are carried over to
the next ones and every project is only read once per pass.

Bug: 550074864
Change-Id: I30395b8a16a9154f60972e1f408a066af64ba77c
Signed-off-by: kimhappy <hwanhee.kim@laplacian.cc>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/621662
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-08-27 16:22:43 -07:00
Gavin Mak 41c2597509 project: don't use dest-branch or tags as SHA-1 upstream fallback
https://gerrit-review.googlesource.com/614762 resolves a fallback
upstream for a SHA-1-pinned project that has no explicit upstream, so
--current-branch can narrow the fetch to one branch instead of fetching
all heads.

It drew that fallback from dest-branch and from the manifest <default>
upstream, dest-branch, and revision. dest-branch and tags are not valid
fallback candidates:

* dest-branch is the review destination for upload, not a fetch
  source.
* Tags are meant to be immutable snapshots of a single tagged commit,
  and fetching it cannot retrieve an arbitrary pinned SHA-1.

Restrict the fallback to branch heads resolved from the manifest
upstream and revision defaults, and drop dest-branch entirely.

Bug: 541240657
Bug: 544041102
Change-Id: I89ae86891ee166f4c340553ac5a9068efcc18461
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/622541
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-08-27 12:43:00 -07:00
Rahul Yadav e6ad708009 hooks: add --fix option to auto-apply hook fixes
Pass the --fix flag as a keyword argument "fix" to the hook main
function. This allows hooks (such as git-repohooks) to decouple
automated fix application from the -y/--yes flag so that -y can
answer yes to upload confirmation prompts without triggering file
mutations.

Companion change in git-repohooks:
https://gerrit-review.googlesource.com/c/git-repohooks/+/621761

Bug: 546510319
Test: python3 -m pytest tests/test_hooks.py

Change-Id: If0288d4791fc0a2aba6e88854e3aa81b0923b664
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/619281
Commit-Queue: Rahul Yadav <yadavrah@google.com>
Tested-by: Rahul Yadav <yadavrah@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-08-27 03:28:58 -07:00
Ram Peri 09914bcab7 Add detailed upload context to Trace2 telemetry
This patch intercepts the output during a successful upload
execution to capture the generated CL URLs. It logs a dynamically
constructed "repo.uploadstate" data event to the active trace2
log containing:
- Uploaded CL URLs
- Target remote name
- Source branch
- Modified files

Test:
1. ./run_tests
2. pytest tests/test_project.py
3. Manual verification:
- Created a dummy branch `test_upload_branch` in a project with local file changes.
- Invoked repo upload passing an explicit trace output directory:
  `repo --git-trace2-event-log=/tmp/trace2out upload --dry-run --no-verify art`
- Verified the injected payload successfully appeared on disk within the generated log:
  ```json
  {"event":"data",..."key":"repo.uploadstate/cls","value":""}
  {"event":"data",..."key":"repo.uploadstate/remote","value":"goog"}
  {"event":"data",..."key":"repo.uploadstate/branch","value":"test_upload_branch"}
  {"event":"data",..."key":"repo.uploadstate/files","value":"dummy_file.txt"}
  ```
  (cls is correctly blank on --dry-run but populates on real HTTP pushes)

Bug: 543953499
Change-Id: I9c402a32d01d156d42cf24eaa2e60e22710b5e6f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/616581
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Ram Peri <ramperi@google.com>
Commit-Queue: Ram Peri <ramperi@google.com>
2026-08-26 10:19:42 -07:00
kimhappy 3f1775607f project: allow re-reading submodule gitlinks
A discovered submodule becomes a project whose revision is the gitlink
read from its parent at manifest load time, which is before the parent
has been fetched. Remember which gitlink a submodule was derived from,
and add a way to read those gitlinks again, so that callers can resolve
a submodule revision once its parent is up-to-date.

A revision that is set does not have to be fetched, so verify its
objects are really there. Without them a submodule that was removed
cannot be told apart from one that was never fetched, which the caller
has to know about.

Bug: 550074864
Change-Id: I4f35d53607126508eec6f02158a3fb604917a199
Signed-off-by: kimhappy <hwanhee.kim@laplacian.cc>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/621661
Reviewed-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-08-25 19:39:51 -07:00
20 changed files with 2013 additions and 133 deletions
+1
View File
@@ -400,6 +400,7 @@ _repo() {
'--no-verify[Do not verify]' \
'--verify[Verify]' \
'--ignore-hooks[Ignore hooks]' \
'--fix[Automatically fix]' \
'*: :->project'
;;
version)
+10 -2
View File
@@ -88,7 +88,14 @@ be useful when deploying automatic fixes.
If the repo command that triggered the hook supports a "yes" option (e.g.,
`repo upload --yes`), this option is propagated to the hook's `main` function
as `yes` parameter (defaulting to `False`). Hooks can use this to bypass
interactive confirmation prompts when they can automatically fix issues.
interactive confirmation prompts for safe non-modifying operations.
### Automated Fixes
If the repo command that triggered the hook supports a "fix" option (e.g.,
`repo upload --fix`), this option is propagated to the hook's `main` function
as `fix` parameter (defaulting to `False`). Hooks can use this to automatically
apply fixes without prompting the user.
### Shebang Handling
@@ -126,7 +133,7 @@ This hook runs when people run `repo upload`.
The `pre-upload.py` file should be defined like:
```py
def main(project_list, worktree_list=None, yes=False, **kwargs):
def main(project_list, worktree_list=None, fix=False, yes=False, **kwargs):
"""Main function invoked directly by repo.
We must use the name "main" as that is what repo requires.
@@ -137,6 +144,7 @@ def main(project_list, worktree_list=None, yes=False, **kwargs):
project_list, so that each entry in project_list matches with a
directory in worktree_list. If None, we will attempt to calculate
the directories automatically.
fix: Whether to automatically apply fixes without prompting.
yes: Whether to answer yes to all safe prompts (see
[Safe Prompts](#safe-prompts)).
kwargs: Leave this here for forward-compatibility.
+13
View File
@@ -70,6 +70,19 @@ class _GitCall:
git = _GitCall()
def IsValidBranchName(name: str) -> bool:
"""Return whether |name| is valid where Git expects a branch name."""
p = GitCommand(
None,
["check-ref-format", "--branch", name],
capture_stdout=True,
capture_stderr=True,
add_event_log=False,
log_as_error=False,
)
return p.Wait() == 0
def RepoSourceVersion():
"""Return the version of the repo.git tree."""
ver = getattr(RepoSourceVersion, "version", None)
+37 -6
View File
@@ -14,6 +14,7 @@
import os
from git_command import git_require
from git_command import GitCommand
import platform_utils
from repo_trace import Trace
@@ -41,6 +42,17 @@ class GitRefs:
self._EnsureLoaded()
return self._phyref
@property
def head(self) -> str:
"""Return HEAD's symbolic target or detached object ID."""
self._EnsureLoaded()
return self._symref.get(HEAD) or self._phyref.get(HEAD, "")
@property
def is_loaded(self) -> bool:
"""Whether a ref snapshot has already been loaded."""
return self._phyref is not None
def get(self, name):
try:
return self.all[name]
@@ -87,8 +99,12 @@ class GitRefs:
self._symref = {}
self._mtime = {}
self._ReadRefs()
self._ReadSymbolicRef(HEAD)
root_refs_loaded = self._ReadRefs()
if not root_refs_loaded or (
HEAD not in self._phyref and HEAD not in self._symref
):
# --include-root-refs does not report an unborn HEAD.
self._ReadSymbolicRef(HEAD)
scan = self._symref
attempts = 0
@@ -113,18 +129,32 @@ class GitRefs:
"""Check if a ref_id is a null object ID."""
return ref_id and all(ch == "0" for ch in ref_id)
def _ReadRefs(self) -> None:
"""Read all references using git for-each-ref."""
def _ReadRefs(self) -> bool:
"""Read all references using git for-each-ref.
Returns:
Whether root refs, including HEAD when it exists, were loaded.
"""
include_root_refs = git_require((2, 45, 0))
cmd = [
"for-each-ref",
"--format=%(objectname)%00%(refname)%00%(symref)",
]
if include_root_refs:
cmd.insert(1, "--include-root-refs")
# Avoid caching volatile root refs such as ORIG_HEAD. HEAD and
# refs/* are the only namespaces GitRefs exposes to callers.
cmd.extend([HEAD, "refs"])
p = GitCommand(
None,
["for-each-ref", "--format=%(objectname)%00%(refname)%00%(symref)"],
cmd,
capture_stdout=True,
capture_stderr=True,
bare=True,
gitdir=self._gitdir,
)
if p.Wait() != 0:
return
return False
for line in p.stdout.splitlines():
ref_id, name, symref = line.split("\0")
@@ -132,6 +162,7 @@ class GitRefs:
self._symref[name] = symref
elif ref_id and not self._IsNullRef(ref_id):
self._phyref[name] = ref_id
return include_root_refs
def _ReadSymbolicRef(self, name: str) -> None:
"""Read a symbolic reference."""
+18 -1
View File
@@ -12,6 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import optparse
import os
import re
import sys
@@ -69,6 +70,7 @@ class RepoHook:
ignore_hooks=False,
abort_if_user_denies=False,
yes=False,
fix=False,
):
"""RepoHook constructor.
@@ -91,6 +93,7 @@ class RepoHook:
abort_if_user_denies: If True, we'll abort running the hook if the
user doesn't allow us to run the hook.
yes: If True, then 'Yes' is assumed for any prompts.
fix: If True, then 'Fix' is assumed for any fixup prompts.
"""
self._hook_type = hook_type
self._hooks_project = hooks_project
@@ -102,6 +105,7 @@ class RepoHook:
self._ignore_hooks = ignore_hooks
self._abort_if_user_denies = abort_if_user_denies
self._yes = yes
self._fix = fix
# Store the full path to the script for convenience.
self._script_fullpath = None
@@ -380,6 +384,7 @@ class RepoHook:
kwargs = {
**kwargs,
"hook_should_take_kwargs": True,
"fix": self._fix,
"yes": self._yes,
}
@@ -504,12 +509,17 @@ class RepoHook:
).url,
"bug_url": manifest.contactinfo.bugurl,
"yes": getattr(opt, "yes", False),
"fix": getattr(opt, "fix", False),
}
)
return cls(*args, **kwargs)
@staticmethod
def AddOptionGroup(parser, name):
def AddOptionGroup(
parser: optparse.OptionParser,
name: str,
allow_fix: bool = False,
) -> None:
"""Help options relating to the various hooks."""
# Note that verify and no-verify are NOT opposites of each other, which
@@ -533,3 +543,10 @@ class RepoHook:
action="store_true",
help="Do not abort if %s hooks fail." % name,
)
if allow_fix:
group.add_option(
"--fix",
action="store_true",
default=False,
help="Automatically apply %s fixes without prompting." % name,
)
+4 -1
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "June 2026" "repo upload" "Repo Manual"
.TH REPO "1" "August 2026" "repo upload" "Repo Manual"
.SH NAME
repo \- repo upload - manual page for repo upload
.SH SYNOPSIS
@@ -112,6 +112,9 @@ Run the pre\-upload hook without prompting.
.TP
\fB\-\-ignore\-hooks\fR
Do not abort if pre\-upload hooks fail.
.TP
\fB\-\-fix\fR
Automatically apply pre\-upload fixes without prompting.
.PP
Run `repo help upload` to view the detailed manual.
.SH DETAILS
+2 -2
View File
@@ -692,9 +692,9 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
e.setAttribute("remote", remoteName)
if peg_rev:
if self.IsMirror:
value = p.bare_git.rev_parse(p.revisionExpr + "^0")
value = p.bare_git.ResolveCommit(p.revisionExpr)
else:
value = p.work_git.rev_parse(HEAD + "^0")
value = p.work_git.ResolveCommit(HEAD)
e.setAttribute("revision", value)
if peg_rev_upstream:
if p.upstream:
+242 -58
View File
@@ -195,6 +195,10 @@ class ReviewableBranch:
def name(self):
return self.branch.name
@property
def current(self) -> bool:
return getattr(self.branch, "current", False)
@property
def commits(self):
if self._commit_cache is None:
@@ -239,6 +243,12 @@ class ReviewableBranch:
"--pretty=format:%cd", "-n", "1", R_HEADS + self.name, "--"
)
@property
def modified_files(self) -> List[str]:
return self.project.bare_git.diff(
"--name-only", f"{self.base}...{R_HEADS}{self.name}"
).splitlines()
@property
def base_exists(self):
"""Whether the branch we're tracking exists.
@@ -271,7 +281,8 @@ class ReviewableBranch:
validate_certs=True,
push_options=None,
patchset_description=None,
):
git_event_log: Optional[EventLog] = None,
) -> None:
self.project.UploadForReview(
branch=self.name,
people=people,
@@ -287,6 +298,7 @@ class ReviewableBranch:
validate_certs=validate_certs,
push_options=push_options,
patchset_description=patchset_description,
git_event_log=git_event_log,
)
def GetPublishedRefs(self):
@@ -568,6 +580,7 @@ class Project:
parent=None,
use_git_worktrees=False,
is_derived=False,
gitlink_path: Optional[str] = None,
dest_branch=None,
optimized_fetch=False,
retry_fetches=0,
@@ -597,6 +610,8 @@ class Project:
use_git_worktrees: Whether to use `git worktree` for this project.
is_derived: False if the project was explicitly defined in the
manifest; True if the project is a discovered submodule.
gitlink_path: For a discovered submodule, its path inside the
parent project.
dest_branch: The branch to which to push changes for review by
default.
optimized_fetch: If True, when a project is set to a sha1 revision,
@@ -624,6 +639,7 @@ class Project:
# See the XmlManifest init code for more info.
self.use_git_worktrees = use_git_worktrees
self.is_derived = is_derived
self.gitlink_path = gitlink_path
self.optimized_fetch = optimized_fetch
self.retry_fetches = max(0, retry_fetches)
self.subprojects = []
@@ -758,15 +774,28 @@ class Project:
work_git is otheriwse inaccessible (e.g. an incomplete sync).
"""
try:
b = self.work_git.GetHead()
b = self._GetHead()
except NoManifestException:
# If the local checkout is in a bad state, don't barf. Let the
# callers process this like the head is unreadable.
return None
if b.startswith(R_HEADS):
if b and b.startswith(R_HEADS):
return b[len(R_HEADS) :]
return None
def _GetHead(self) -> Optional[str]:
"""Return worktree HEAD, reusing a compatible loaded ref snapshot."""
if not self.work_git:
return None
# Git worktrees keep the checkout's HEAD in the worktree admin dir,
# while bare_ref reads the shared repository. Its HEAD is not the
# checked-out worktree's HEAD and must not be reused here.
if not self.use_git_worktrees and self.bare_ref.is_loaded:
head = self.bare_ref.head
if head:
return head
return self.work_git.GetHead()
def IsRebaseInProgress(self):
"""Returns true if a rebase or "am" is in progress"""
# "rebase-apply" is used for "git rebase".
@@ -865,8 +894,8 @@ class Project:
def GetBranches(self):
"""Get all existing local branches."""
current = self.CurrentBranch
all_refs = self._allrefs
current = self.CurrentBranch
heads = {}
for name, ref_id in all_refs.items():
@@ -1142,24 +1171,37 @@ class Project:
def GetUploadableBranches(self, selected_branch=None):
"""List any branches which can be uploaded for review."""
heads = {}
pubed = {}
if selected_branch:
branch = self.GetBranch(selected_branch)
if not branch.LocalMerge:
return []
head_id = self.bare_ref.get(R_HEADS + selected_branch)
if not head_id:
return []
pub_id = self.bare_ref.get(R_PUB + selected_branch)
if pub_id and pub_id == head_id:
return []
rb = self.GetUploadableBranch(selected_branch)
if rb:
rb.branch.current = selected_branch == self.CurrentBranch
return [rb]
return []
for name, ref_id in self._allrefs.items():
if name.startswith(R_HEADS):
heads[name[len(R_HEADS) :]] = ref_id
elif name.startswith(R_PUB):
pubed[name[len(R_PUB) :]] = ref_id
# Optimization: Skip scanning _allrefs (which spawns git processes)
# if no local branches with upstream tracking exist in .git/config.
if not any(self.config.GetSubSections("branch")):
return []
branches = self.GetBranches()
ready = []
for branch, ref_id in heads.items():
if branch in pubed and pubed[branch] == ref_id:
continue
if selected_branch and branch != selected_branch:
for branch, branch_config in branches.items():
if branch_config.published == branch_config.revision:
continue
rb = self.GetUploadableBranch(branch)
if rb:
rb.branch.current = branch_config.current
ready.append(rb)
return ready
@@ -1189,7 +1231,8 @@ class Project:
validate_certs=True,
push_options=None,
patchset_description=None,
):
git_event_log: Optional[EventLog] = None,
) -> None:
"""Uploads the named branch for code review."""
if branch is None:
branch = self.CurrentBranch
@@ -1278,14 +1321,47 @@ class Project:
ref_spec = ref_spec + "%" + ",".join(opts)
cmd.append(ref_spec)
GitCommand(self, cmd, bare=True, verify_command=True).Wait()
push_cmd = GitCommand(
self,
cmd,
bare=True,
verify_command=True,
)
push_cmd.Wait()
cls_urls = self._FindGerritUrls(push_cmd.stderr)
try:
rb = ReviewableBranch(self, branch, branch.LocalMerge)
modified_files_list = rb.modified_files
if git_event_log:
git_event_log.LogDataConfigEvents(
{
"cls": ",".join(cls_urls),
"remote": branch.remote.name,
"branch": branch.name,
"files": ",".join(modified_files_list),
},
"repo.uploadstate",
)
except Exception as e:
logger.error("Tracing failed: %s", str(e))
if not dryrun:
msg = f"posted to {branch.remote.review} for {dest_branch}"
self.bare_git.UpdateRef(
R_PUB + branch.name, R_HEADS + branch.name, message=msg
)
@staticmethod
def _FindGerritUrls(stderr: Optional[str]) -> List[str]:
"""Extracts Gerrit review URLs from git push output."""
if not stderr:
return []
return [
match.group(1)
for match in re.finditer(r"(https?://[^/]+/c/.+?/\+/\d+)", stderr)
]
@staticmethod
def _encode_patchset_description(original):
"""Applies percent-encoding for strings sent as patchset description.
@@ -1679,7 +1755,9 @@ class Project:
for linkfile in self.linkfiles:
linkfile._Link()
def GetCommitRevisionId(self):
def GetCommitRevisionId(
self, all_refs: Optional[Dict[str, str]] = None
) -> str:
"""Get revisionId of a commit.
Use this method instead of GetRevisionId to get the id of the commit
@@ -1689,10 +1767,12 @@ class Project:
if self.revisionId:
return self.revisionId
if not self.revisionExpr.startswith(R_TAGS):
return self.GetRevisionId(self._allrefs)
if all_refs is None:
all_refs = self._allrefs
return self.GetRevisionId(all_refs)
try:
return self.bare_git.rev_list(self.revisionExpr, "-1")[0]
return self.bare_git.ResolveCommit(self.revisionExpr)
except GitError:
raise ManifestInvalidRevisionError(
f"revision {self.revisionExpr} in {self.name} not found"
@@ -1704,6 +1784,10 @@ class Project:
Returns None if worktree is not checked out or HEAD cannot be resolved.
"""
if self.work_git:
if not self.use_git_worktrees and self.bare_ref.is_loaded:
head = self.bare_ref.get(HEAD)
if head:
return head
try:
return self.work_git.rev_parse("HEAD")
except GitError:
@@ -1721,7 +1805,7 @@ class Project:
return all_refs[rev]
try:
return self.bare_git.rev_parse("--verify", "%s^0" % rev)
return self.bare_git.ResolveCommit(rev)
except GitError:
raise ManifestInvalidRevisionError(
f"revision {self.revisionExpr} in {self.name} not found"
@@ -1817,8 +1901,8 @@ class Project:
if p.Wait() != 0:
logger.warning("warn: %s: stateless gc failed", self.name)
head = self.work_git.GetHead()
if head.startswith(R_HEADS):
head = self._GetHead()
if head and head.startswith(R_HEADS):
branch = head[len(R_HEADS) :]
try:
head = all_refs[head]
@@ -2349,7 +2433,7 @@ class Project:
# Doesn't exist
return None
head = self.work_git.GetHead()
head = self._GetHead()
if head == rev:
# We can't destroy the branch while we are sitting
# on it. Switch to a detached HEAD.
@@ -2371,9 +2455,9 @@ class Project:
def PruneHeads(self):
"""Prune any topic branches already merged into upstream."""
cb = self.CurrentBranch
kill = []
left = self._allrefs
cb = self.CurrentBranch
for name in left.keys():
if name.startswith(R_HEADS):
name = name[len(R_HEADS) :]
@@ -2385,17 +2469,25 @@ class Project:
if not kill and not cb:
return []
rev = self.GetRevisionId(left)
rev = self.GetCommitRevisionId(left)
head = left.get(R_HEADS + cb) if cb is not None else None
if (
cb is not None
and not self._revlist(HEAD + "..." + rev)
and head == rev
and not self.IsDirty(consider_untracked=False)
):
self.work_git.DetachHead(HEAD)
kill.append(cb)
if kill:
old = self.bare_git.GetHead()
if not self.use_git_worktrees:
old = (
head
if cb in kill
else (self.bare_ref.head or self.bare_git.GetHead())
)
else:
old = self.bare_git.GetHead()
try:
self.bare_git.DetachHead(rev)
@@ -2410,7 +2502,11 @@ class Project:
if IsId(old):
self.bare_git.DetachHead(old)
else:
self.bare_git.SetHead(old)
branch = (
old[len(R_HEADS) :] if old.startswith(R_HEADS) else old
)
if branch not in kill:
self.bare_git.SetHead(old)
left = self._allrefs
for branch in kill:
@@ -2426,6 +2522,7 @@ class Project:
for branch in kill:
if R_HEADS + branch in left:
branch = self.GetBranch(branch)
branch.current = branch.name == cb
base = branch.LocalMerge
if not base:
base = rev
@@ -2573,6 +2670,37 @@ class Project:
return []
return get_submodules(self.gitdir, rev)
def GetSubmoduleRevisions(self) -> Optional[Dict[str, str]]:
"""Read the gitlinks of our submodules at our current revision.
Discovered submodules are derived from the revision their parent was
at when the manifest was loaded, which is before it gets fetched.
Once it is up-to-date, its gitlinks have to be read again, otherwise
the submodules would be synced to the revisions of the previous sync.
Returns:
The revision of every submodule, keyed by its path inside this
project, or None if our revision is not available locally. A
revision that is merely set does not have to be fetched yet, and
without its objects a removed submodule cannot be told apart from
one that was never fetched.
"""
try:
rev = self.GetRevisionId()
self.bare_git.rev_list(
"-1",
"--missing=allow-any",
f"{rev}^0",
"--",
log_as_error=False,
)
except (GitError, ManifestInvalidRevisionError):
return None
return {
path: sha for sha, path, _url, _shallow in self._GetSubmodules()
}
def GetDerivedSubprojects(self):
result = []
if not self.Exists:
@@ -2620,6 +2748,7 @@ class Project:
parent=self,
clone_depth=clone_depth,
is_derived=True,
gitlink_path=path,
)
result.append(subproject)
result.extend(subproject.GetDerivedSubprojects())
@@ -2858,19 +2987,16 @@ class Project:
return True
def _GetUpstreamFallback(self) -> Optional[str]:
"""Resolve a fallback upstream ref when revisionExpr is a SHA-1."""
"""Resolve a fallback upstream branch when revisionExpr is a SHA-1.
Returns manifest default upstream or revision if it names a branch,
or None to fall back to fetching all heads.
"""
default = self.manifest.default
candidates = [self.dest_branch]
if default:
candidates.extend(
(
default.upstreamExpr,
default.destBranchExpr,
default.revisionExpr,
)
)
for cand in candidates:
if cand and not IsId(cand):
if not default:
return None
for cand in (default.upstreamExpr, default.revisionExpr):
if cand and not IsId(cand) and not cand.startswith(R_TAGS):
return cand
return None
@@ -4389,8 +4515,52 @@ class Project:
return dotgit if subpath is None else os.path.join(dotgit, subpath)
@staticmethod
def _ParseHead(line: str) -> Optional[str]:
"""Parse the content of a .git/HEAD file.
Handles both symbolic refs (e.g. 'ref: refs/heads/...') and raw
commit IDs (40-hex SHA-1 or 64-hex SHA-256).
Returns:
The ref name (e.g. 'refs/heads/main') or lowercase commit hash
if valid, or None if empty or invalid.
"""
line = line.strip()
if line.startswith("ref:"):
ref = line[4:].strip()
# Ensure the ref is not empty, pure whitespace, or the
# "refs/heads/.invalid" placeholder used for unborn branches,
# empty repositories, or when the reftables backend is used
# (which will be the default in Git 3.0).
if not ref or ref == R_HEADS + ".invalid":
return None
return ref
else:
# Normalize commit IDs to canonical lowercase hexadecimal,
# matching the output format of `git rev-parse`.
line_lower = line.lower()
if IsId(line_lower):
return line_lower
return None
def GetHead(self):
"""Return the ref that HEAD points to."""
path = None
try:
# Catch AssertionError raised by GetDotgitPath when worktree
# .git pointer file is malformed (e.g. missing 'gitdir:').
path = self.GetDotgitPath(subpath=HEAD)
if not platform_utils.islink(path):
with open(
path, "r", encoding="utf-8", errors="replace"
) as fd:
ref = self._ParseHead(fd.readline())
if ref:
return ref
except (OSError, AssertionError):
pass
try:
return self.symbolic_ref("-q", HEAD, log_as_error=False)
except GitError:
@@ -4410,24 +4580,38 @@ class Project:
# Fallback to direct file reading for compatibility with broken
# repos, e.g. if HEAD points to an unborn branch.
path = self.GetDotgitPath(subpath=HEAD)
if not path:
raise NoManifestException(
self._project.RelPath(local=False), str(e)
)
try:
with open(path) as fd:
line = fd.readline()
with open(
path, "r", encoding="utf-8", errors="replace"
) as fd:
ref = self._ParseHead(fd.readline())
except OSError:
raise NoManifestException(path, str(e))
try:
line = line.decode()
except AttributeError:
pass
if line.startswith("ref: "):
ref = line[5:-1]
else:
ref = line[:-1]
if ref == R_HEADS + ".invalid":
raise NoManifestException(path, str(e))
raise NoManifestException(
self._project.RelPath(local=False), str(e)
)
if not ref:
raise NoManifestException(
self._project.RelPath(local=False), str(e)
)
return ref
def ResolveCommit(self, revision: str) -> str:
"""Resolve |revision| to a commit without option ambiguity."""
cmdv = ["--verify", "--quiet"]
if git_require((2, 30, 0)):
cmdv.append("--end-of-options")
elif revision.startswith("-"):
raise GitError(
f"invalid revision: {revision}",
project=self._project.name,
)
cmdv.append(f"{revision}^{{commit}}")
return self.rev_parse(*cmdv, log_as_error=False)
def SetHead(self, ref, message=None):
cmdv = []
if message is not None:
@@ -4764,8 +4948,8 @@ class MetaProject(Project):
all_refs = self.bare_ref.all
revid = self.GetRevisionId(all_refs)
head = self.work_git.GetHead()
if head.startswith(R_HEADS):
head = self._GetHead()
if head and head.startswith(R_HEADS):
try:
head = all_refs[head]
except KeyError:
@@ -4773,7 +4957,7 @@ class MetaProject(Project):
if revid == head:
return False
elif self._revlist(not_rev(HEAD), revid):
elif self._revlist("-1", not_rev(HEAD), revid):
return True
return False
+2 -4
View File
@@ -20,7 +20,7 @@ from command import Command
from command import DEFAULT_LOCAL_JOBS
from error import RepoError
from error import RepoExitError
from git_command import git
from git_command import IsValidBranchName
from progress import Progress
from repo_logging import RepoLogger
@@ -58,9 +58,7 @@ It is equivalent to "git branch -D <branchname>".
if not opt.all:
branches = args[0].split()
invalid_branches = [
x for x in branches if not git.check_ref_format(f"heads/{x}")
]
invalid_branches = [x for x in branches if not IsValidBranchName(x)]
if invalid_branches:
self.OptionParser.error(
+9 -4
View File
@@ -59,10 +59,15 @@ are displayed.
for project in self.GetProjects(
args, all_manifests=not opt.this_manifest_only
):
br = [project.GetUploadableBranch(x) for x in project.GetBranches()]
br = [x for x in br if x]
local_branches = project.GetBranches()
br = []
for name, branch in local_branches.items():
uploadable = project.GetUploadableBranch(name)
if uploadable:
uploadable.branch.current = branch.current
br.append(uploadable)
if opt.current_branch:
br = [x for x in br if x.name == project.CurrentBranch]
br = [x for x in br if x.current]
all_branches.extend(br)
if not all_branches:
@@ -97,7 +102,7 @@ are displayed.
print(
"%s %-33s (%2d commit%s, %s)"
% (
branch.name == project.CurrentBranch and "*" or " ",
branch.current and "*" or " ",
branch.name,
len(commits),
len(commits) != 1 and "s" or " ",
+1 -1
View File
@@ -80,7 +80,7 @@ class Prune(PagedCommand):
print(
"%s %-33s "
% (
branch.name == project.CurrentBranch and "*" or " ",
branch.current and "*" or " ",
branch.name,
),
end="",
+2 -2
View File
@@ -18,7 +18,7 @@ from typing import NamedTuple
from command import Command
from command import DEFAULT_LOCAL_JOBS
from error import RepoExitError
from git_command import git
from git_command import IsValidBranchName
from git_config import IsImmutable
from progress import Progress
from repo_logging import RepoLogger
@@ -75,7 +75,7 @@ revision specified in the manifest.
self.Usage()
nb = args[0]
if not git.check_ref_format("heads/%s" % nb):
if not IsValidBranchName(nb):
self.OptionParser.error("'%s' is not a valid name" % nb)
@classmethod
+128 -7
View File
@@ -28,7 +28,7 @@ import subprocess
import sys
import tempfile
import time
from typing import List, NamedTuple, Optional, Set, Tuple, Union
from typing import Dict, List, NamedTuple, Optional, Set, Tuple, Union
import urllib.error
import urllib.parse
import urllib.request
@@ -156,6 +156,83 @@ def _SafeCheckoutOrder(checkouts: List[Project]) -> List[List[Project]]:
return res
def _ParentFirstBatches(projects: List[Project]) -> List[List[Project]]:
"""Group |projects| so that a parent is fetched before its submodules.
A discovered submodule can only be fetched at the right revision once the
project holding its gitlink has been fetched, so it is held back to a later
batch than its parent. Projects that are not discovered submodules all end
up in the first batch, which keeps manifests without submodules on a single
batch.
"""
batches = collections.defaultdict(list)
for project in projects:
depth = 0
ancestor = project
while ancestor.Derived and ancestor.parent:
depth += 1
ancestor = ancestor.parent
batches[depth].append(project)
return [batches[depth] for depth in sorted(batches)]
def _RefreshDerivedRevisions(
projects: List[Project],
submodule_revisions: Optional[Dict[Project, Dict[str, str]]] = None,
) -> List[Project]:
"""Re-resolve the gitlinks of the discovered submodules in |projects|.
The revision of a discovered submodule is read from its parent when the
manifest is loaded, so it is stale as soon as the parent gets fetched. It
has to be resolved again once the parent is up-to-date, and before the
submodule itself is fetched and checked out.
Args:
projects: The projects whose discovered submodules to resolve.
submodule_revisions: Gitlinks already read, keyed by the project
holding them. Passing the same dict for project sets that follow
the same fetches, e.g. the levels of one checkout order, keeps a
project from being read more than once.
Returns:
The submodules that their parent no longer holds a gitlink for.
"""
if submodule_revisions is None:
submodule_revisions = {}
subprojects_by_parent = collections.defaultdict(list)
for project in projects:
if project.Derived and project.parent:
subprojects_by_parent[project.parent].append(project)
removed = []
for parent, subprojects in subprojects_by_parent.items():
revisions = submodule_revisions.get(parent)
if revisions is None:
revisions = parent.GetSubmoduleRevisions()
if revisions is None:
# Leave the submodules of a parent we cannot read alone.
continue
submodule_revisions[parent] = revisions
for subproject in subprojects:
rev = revisions.get(subproject.gitlink_path)
if rev:
subproject.SetRevision(rev, revisionId=rev)
else:
removed.append(subproject)
return removed
def _WithoutProjects(
projects: List[Project], unwanted: List[Project]
) -> List[Project]:
"""Return |projects| without the projects in |unwanted|."""
if not unwanted:
return projects
dropped = set(unwanted)
return [p for p in projects if p not in dropped]
def _chunksize(projects: int, jobs: int) -> int:
"""Calculate chunk size for the given number of projects and jobs."""
return min(max(1, projects // jobs), WORKER_BATCH_SIZE)
@@ -1067,6 +1144,40 @@ later is required to fix a server side protocol bug.
return _FetchResult(ret, fetched)
def _FetchParentFirst(
self,
projects: List[Project],
opt: optparse.Values,
err_event: _threading.Event,
ssh_proxy: ssh.ProxyManager,
errors: List[Exception],
) -> _FetchResult:
"""Fetch |projects|, holding submodules back until their parent is done.
Args:
projects: Projects to fetch.
opt: Program options returned from optparse. See _Options().
err_event: Whether an error was hit while processing.
ssh_proxy: SSH manager for clients & masters.
errors: A list to accumulate errors.
Returns:
_FetchResult for all the batches combined.
"""
success = True
fetched = set()
for batch in _ParentFirstBatches(projects):
batch = _WithoutProjects(batch, _RefreshDerivedRevisions(batch))
if not batch:
continue
batch.sort(key=self._fetch_times.Get, reverse=True)
result = self._Fetch(batch, opt, err_event, ssh_proxy, errors)
success = success and result.success
fetched.update(result.projects)
if not success and opt.fail_fast:
break
return _FetchResult(success, fetched)
def _FetchMain(
self, opt, args, all_projects, err_event, ssh_proxy, manifest, errors
):
@@ -1083,12 +1194,10 @@ later is required to fix a server side protocol bug.
Returns:
List of all projects that should be checked out.
"""
to_fetch = []
to_fetch.extend(all_projects)
to_fetch.sort(key=self._fetch_times.Get, reverse=True)
try:
result = self._Fetch(to_fetch, opt, err_event, ssh_proxy, errors)
result = self._FetchParentFirst(
all_projects, opt, err_event, ssh_proxy, errors
)
success = result.success
fetched = result.projects
if not success:
@@ -1131,7 +1240,9 @@ later is required to fix a server side protocol bug.
if previously_missing_set == missing_set:
break
previously_missing_set = missing_set
result = self._Fetch(missing, opt, err_event, ssh_proxy, errors)
result = self._FetchParentFirst(
missing, opt, err_event, ssh_proxy, errors
)
success = result.success
new_fetched = result.projects
if not success:
@@ -2977,12 +3088,22 @@ later is required to fix a server side protocol bug.
# projects in one level can be processed in
# parallel, but we must wait for a level to complete
# before starting the next.
submodule_revisions = {}
for level_projects in _SafeCheckoutOrder(
projects_to_sync
):
if not level_projects:
continue
level_projects = _WithoutProjects(
level_projects,
_RefreshDerivedRevisions(
level_projects, submodule_revisions
),
)
if not level_projects:
continue
objdir_project_map = collections.defaultdict(
list
)
+13 -19
View File
@@ -25,7 +25,6 @@ from editor import Editor
from error import GitError
from error import SilentRepoExitError
from error import UploadError
from git_command import GitCommand
from git_refs import R_HEADS
import git_superproject
from hooks import RepoHook
@@ -379,7 +378,7 @@ Gerrit Code Review: https://www.gerritcodereview.com/
default=True,
help="disable verifying ssl certs (unsafe)",
)
RepoHook.AddOptionGroup(p, "pre-upload")
RepoHook.AddOptionGroup(p, "pre-upload", allow_fix=True)
def _SingleBranch(self, opt, branch, people):
project = branch.project
@@ -649,6 +648,7 @@ Gerrit Code Review: https://www.gerritcodereview.com/
validate_certs=opt.validate_certs,
push_options=push_options,
patchset_description=opt.patchset_description,
git_event_log=self.git_event_log,
)
branch.uploaded = True
@@ -704,24 +704,18 @@ Gerrit Code Review: https://www.gerritcodereview.com/
raise UploadExitError(aggregate_errors=aggregate_errors)
def _GetMergeBranch(self, project, local_branch=None):
"""Get the merge branch name for a local branch.
Resolves the merge branch in-memory via project configuration to
avoid git subprocess overhead during upload.
"""
if local_branch is None:
p = GitCommand(
project,
["rev-parse", "--abbrev-ref", "HEAD"],
capture_stdout=True,
capture_stderr=True,
)
p.Wait()
local_branch = p.stdout.strip()
p = GitCommand(
project,
["config", "--get", "branch.%s.merge" % local_branch],
capture_stdout=True,
capture_stderr=True,
)
p.Wait()
merge_branch = p.stdout.strip()
return merge_branch
local_branch = project.CurrentBranch
if local_branch:
branch = project.GetBranch(local_branch)
if branch.merge:
return branch.merge
return ""
@classmethod
def _GatherOne(cls, opt, project_idx):
+18
View File
@@ -231,6 +231,24 @@ class GitCommandStreamLogsTest(unittest.TestCase):
class GitCallUnitTest(unittest.TestCase):
"""Tests the _GitCall class (via git_command.git)."""
def test_valid_branch_name_uses_branch_mode(self) -> None:
"""Branch validation applies Git's branch-specific restrictions."""
command = mock.MagicMock()
command.Wait.return_value = 1
with mock.patch.object(
git_command, "GitCommand", return_value=command
) as check:
self.assertFalse(git_command.IsValidBranchName("-topic"))
check.assert_called_once_with(
None,
["check-ref-format", "--branch", "-topic"],
capture_stdout=True,
capture_stderr=True,
add_event_log=False,
log_as_error=False,
)
def test_version_tuple(self):
"""Check git.version_tuple() handling."""
ver = git_command.git.version_tuple()
+148 -2
View File
@@ -17,6 +17,8 @@
import os
from pathlib import Path
import subprocess
from typing import Any, List
from unittest import mock
import pytest
import utils_for_test
@@ -24,7 +26,7 @@ import utils_for_test
import git_refs
def _run(repo, *args):
def _run(repo: str, *args: str) -> str:
return subprocess.run(
["git", "-C", repo, *args],
stdout=subprocess.PIPE,
@@ -34,7 +36,7 @@ def _run(repo, *args):
).stdout.strip()
def _init_repo(tmp_path, reftable=False):
def _init_repo(tmp_path: Path, reftable: bool = False) -> str:
repo = os.path.join(tmp_path, "repo")
ref_format = "reftable" if reftable else "files"
utils_for_test.init_git_tree(repo, ref_format=ref_format)
@@ -57,10 +59,154 @@ def test_reads_refs(tmp_path, reftable):
branch = _run(repo, "symbolic-ref", "--short", "HEAD")
head = _run(repo, "rev-parse", "HEAD")
assert refs.symref("HEAD") == f"refs/heads/{branch}"
assert refs.head == f"refs/heads/{branch}"
assert refs.get("HEAD") == head
assert refs.get(f"refs/heads/{branch}") == head
@pytest.mark.parametrize("reftable", [False, True])
def test_reads_detached_head(tmp_path: Path, reftable: bool) -> None:
if reftable and not utils_for_test.supports_reftable():
pytest.skip("reftable not supported")
repo = _init_repo(tmp_path, reftable=reftable)
head = _run(repo, "rev-parse", "HEAD")
_run(repo, "checkout", "--detach", head)
refs = git_refs.GitRefs(os.path.join(repo, ".git"))
assert refs.symref("HEAD") == ""
assert refs.head == head
assert refs.get("HEAD") == head
def test_reads_head_with_root_refs_in_one_command(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Git 2.45 and newer include HEAD in the ref snapshot."""
head = "1" * 40
commands = []
class FakeGitCommand:
def __init__(
self, _project: Any, cmdv: List[str], **_kwargs: Any
) -> None:
commands.append(cmdv)
self.stdout = (
f"{head}\0HEAD\0refs/heads/main\n"
f"{head}\0refs/heads/main\0\n"
)
def Wait(self) -> int:
return 0
monkeypatch.setattr(git_refs, "GitCommand", FakeGitCommand)
monkeypatch.setattr(git_refs, "git_require", lambda _version: True)
refs = git_refs.GitRefs("/nonexistent")
with mock.patch.object(refs, "_ReadSymbolicRef") as read_head:
assert refs.get("HEAD") == head
assert commands == [
[
"for-each-ref",
"--include-root-refs",
"--format=%(objectname)%00%(refname)%00%(symref)",
"HEAD",
"refs",
]
]
read_head.assert_not_called()
def test_root_ref_snapshot_falls_back_for_unborn_head(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""An unborn HEAD still uses symbolic-ref after the ref snapshot."""
class FakeGitCommand:
def __init__(
self, _project: Any, _cmdv: List[str], **_kwargs: Any
) -> None:
self.stdout = ""
def Wait(self) -> int:
return 0
monkeypatch.setattr(git_refs, "GitCommand", FakeGitCommand)
monkeypatch.setattr(git_refs, "git_require", lambda _version: True)
refs = git_refs.GitRefs("/nonexistent")
def read_head(name: str) -> None:
assert name == "HEAD"
refs._symref[name] = "refs/heads/main"
monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head)
assert refs.symref("HEAD") == "refs/heads/main"
def test_old_git_keeps_separate_head_fallback(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Git before 2.45 uses the original for-each-ref and HEAD calls."""
head = "1" * 40
commands = []
class FakeGitCommand:
def __init__(
self, _project: Any, cmdv: List[str], **_kwargs: Any
) -> None:
commands.append(cmdv)
self.stdout = f"{head}\0refs/heads/main\0\n"
def Wait(self) -> int:
return 0
monkeypatch.setattr(git_refs, "GitCommand", FakeGitCommand)
monkeypatch.setattr(git_refs, "git_require", lambda _version: False)
refs = git_refs.GitRefs("/nonexistent")
def read_head(name: str) -> None:
assert name == "HEAD"
refs._symref[name] = "refs/heads/main"
monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head)
assert refs.get("HEAD") == head
assert commands == [
[
"for-each-ref",
"--format=%(objectname)%00%(refname)%00%(symref)",
]
]
def test_for_each_ref_failure_falls_back_to_symbolic_ref(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""When for-each-ref fails, HEAD is still resolved via symbolic-ref."""
class FakeGitCommand:
def __init__(
self, _project: Any, _cmdv: List[str], **_kwargs: Any
) -> None:
self.stdout = ""
def Wait(self) -> int:
return 1
monkeypatch.setattr(git_refs, "GitCommand", FakeGitCommand)
monkeypatch.setattr(git_refs, "git_require", lambda _version: True)
refs = git_refs.GitRefs("/nonexistent")
def read_head(name: str) -> None:
assert name == "HEAD"
refs._symref[name] = "refs/heads/main"
monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head)
assert refs.symref("HEAD") == "refs/heads/main"
@pytest.mark.parametrize("reftable", [False, True])
def test_updates_when_refs_change(tmp_path, reftable):
if reftable and not utils_for_test.supports_reftable():
+64 -2
View File
@@ -15,6 +15,7 @@
"""Unittests for the hooks.py module."""
from io import StringIO
from pathlib import Path
import sys
import pytest
@@ -108,11 +109,11 @@ def test_post_sync_argument_validation() -> None:
@pytest.mark.parametrize("yes_val", (True, False))
def test_repo_upload_yes_arg(tmp_path, yes_val: bool) -> None:
def test_repo_upload_yes_arg(tmp_path: Path, yes_val: bool) -> None:
"""Test that yes is passed in kwargs during hook execution."""
class FakeProject:
def __init__(self, worktree):
def __init__(self, worktree: str) -> None:
self.worktree = worktree
self.enabled_repo_hooks = ["pre-upload"]
self.config = None
@@ -139,3 +140,64 @@ def main(project_list, **kwargs):
assert res is True
assert project_list == [yes_val]
@pytest.mark.parametrize("fix_val", (True, False))
def test_repo_upload_fix_arg(tmp_path: Path, fix_val: bool) -> None:
"""Test that fix is passed in kwargs during hook execution."""
class FakeProject:
def __init__(self, worktree: str) -> None:
self.worktree = worktree
self.enabled_repo_hooks = ["pre-upload"]
self.config = None
hook_file = tmp_path / "pre-upload.py"
hook_content = """
def main(project_list, **kwargs):
project_list.append(kwargs.get("fix"))
"""
hook_file.write_text(hook_content)
hook = hooks.RepoHook(
hook_type="pre-upload",
hooks_project=FakeProject(str(tmp_path)),
repo_topdir=str(tmp_path),
manifest_url="https://gerrit",
allow_all_hooks=True,
fix=fix_val,
)
project_list = []
res = hook.Run(project_list=project_list, worktree_list=[])
assert res is True
assert project_list == [fix_val]
def test_from_subcmd_without_fix_option() -> None:
"""Test that FromSubcmd works when opt does not have fix attribute."""
class Remote:
url = "https://gerrit"
class FakeManifest:
repo_hooks_project = None
topdir = "/fake/topdir"
class manifestProject:
@staticmethod
def GetRemote(name: str) -> "Remote":
return Remote()
class contactinfo:
bugurl = "https://bugs"
class FakeOpt:
bypass_hooks = False
allow_all_hooks = False
ignore_hooks = False
hook = hooks.RepoHook.FromSubcmd(FakeManifest(), FakeOpt(), "post-sync")
assert hook._fix is False
+979 -20
View File
File diff suppressed because it is too large Load Diff
+288 -2
View File
@@ -19,6 +19,7 @@ from pathlib import Path
import shutil
import tempfile
import time
from typing import Dict, List, Optional
import unittest
from unittest import mock
@@ -491,12 +492,24 @@ class LocalSyncState(unittest.TestCase):
class FakeProject:
def __init__(self, relpath, name=None, objdir=None):
def __init__(
self,
relpath: str,
name: Optional[str] = None,
objdir: Optional[str] = None,
parent: Optional["FakeProject"] = None,
is_derived: bool = False,
revisionId: Optional[str] = None,
gitlink_path: Optional[str] = None,
) -> None:
self.relpath = relpath
self.name = name or relpath
self.objdir = objdir or relpath
self.worktree = relpath
self.parent = None
self.parent = parent
self.is_derived = is_derived
self.revisionId = revisionId
self.gitlink_path = gitlink_path
self.use_git_worktrees = False
self.UseAlternates = False
@@ -505,6 +518,16 @@ class FakeProject:
self.config = mock.MagicMock()
self.EnableRepositoryExtension = mock.MagicMock()
@property
def Derived(self) -> bool:
return self.is_derived
def SetRevision(
self, revisionExpr: str, revisionId: Optional[str] = None
) -> None:
self.revisionExpr = revisionExpr
self.revisionId = revisionId or revisionExpr
def RelPath(self, local=None):
return self.relpath
@@ -614,6 +637,159 @@ class SafeCheckoutOrder(unittest.TestCase):
)
class ParentFirstBatches(unittest.TestCase):
def test_no_submodules(self) -> None:
p_a = FakeProject("a")
p_a_b = FakeProject("a/b")
out = sync._ParentFirstBatches([p_a, p_a_b])
self.assertEqual(out, [[p_a, p_a_b]])
def test_submodules_follow_their_parent(self) -> None:
p_a = FakeProject("a")
p_a_b = FakeProject("a/b", parent=p_a, is_derived=True)
p_a_b_c = FakeProject("a/b/c", parent=p_a_b, is_derived=True)
out = sync._ParentFirstBatches([p_a_b_c, p_a, p_a_b])
self.assertEqual(out, [[p_a], [p_a_b], [p_a_b_c]])
class RefreshDerivedRevisions(unittest.TestCase):
def _parent_with_submodules(self, **gitlinks: str) -> FakeProject:
p_a = FakeProject("a")
p_a.GetSubmoduleRevisions = mock.Mock(return_value=gitlinks)
return p_a
def _submodule(self, parent: FakeProject, path: str) -> FakeProject:
return FakeProject(
f"a/{path}", parent=parent, is_derived=True, gitlink_path=path
)
def test_reads_each_parent_once(self) -> None:
p_a = self._parent_with_submodules(b="beef1234", c="cafe1234")
p_a_b = self._submodule(p_a, "b")
p_a_c = self._submodule(p_a, "c")
sync._RefreshDerivedRevisions([p_a, p_a_b, p_a_c])
p_a.GetSubmoduleRevisions.assert_called_once_with()
self.assertEqual(p_a_b.revisionId, "beef1234")
self.assertEqual(p_a_c.revisionId, "cafe1234")
def test_reuses_gitlinks_read_for_an_earlier_level(self) -> None:
p_a = self._parent_with_submodules(b="beef1234", c="cafe1234")
p_a_b = self._submodule(p_a, "b")
p_a_c = self._submodule(p_a, "c")
submodule_revisions = {}
sync._RefreshDerivedRevisions([p_a_b], submodule_revisions)
sync._RefreshDerivedRevisions([p_a_c], submodule_revisions)
p_a.GetSubmoduleRevisions.assert_called_once_with()
self.assertEqual(p_a_c.revisionId, "cafe1234")
def test_tells_apart_projects_with_the_same_path(self) -> None:
# Paths are relative to their own (sub)manifest, so two projects can
# share one.
first = self._parent_with_submodules(b="beef1234")
second = self._parent_with_submodules(b="cafe1234")
first_sub = self._submodule(first, "b")
second_sub = self._submodule(second, "b")
submodule_revisions = {}
sync._RefreshDerivedRevisions([first_sub], submodule_revisions)
sync._RefreshDerivedRevisions([second_sub], submodule_revisions)
self.assertEqual(first_sub.revisionId, "beef1234")
self.assertEqual(second_sub.revisionId, "cafe1234")
def test_ignores_projects_from_the_manifest(self) -> None:
p_a = self._parent_with_submodules()
sync._RefreshDerivedRevisions([p_a])
p_a.GetSubmoduleRevisions.assert_not_called()
def test_reports_submodules_removed_from_their_parent(self) -> None:
p_a = self._parent_with_submodules(b="beef1234")
p_a_c = self._submodule(p_a, "c")
removed = sync._RefreshDerivedRevisions([p_a, p_a_c])
self.assertEqual(removed, [p_a_c])
def test_keeps_submodules_of_an_unreadable_parent(self) -> None:
p_a = FakeProject("a")
p_a.GetSubmoduleRevisions = mock.Mock(return_value=None)
p_a_b = FakeProject(
"a/b",
parent=p_a,
is_derived=True,
revisionId="stale",
gitlink_path="b",
)
removed = sync._RefreshDerivedRevisions([p_a, p_a_b])
self.assertEqual(removed, [])
self.assertEqual(p_a_b.revisionId, "stale")
class FetchParentFirst(unittest.TestCase):
def test_submodules_are_fetched_after_their_parent(self) -> None:
cmd = sync.Sync()
cmd._fetch_times = mock.Mock()
cmd._fetch_times.Get = mock.Mock(return_value=0)
calls = []
p_a = FakeProject("a")
def fake_read() -> Dict[str, str]:
calls.append(("read gitlinks of", p_a.relpath))
return {"b": "beef1234"}
p_a.GetSubmoduleRevisions = mock.Mock(side_effect=fake_read)
p_a_b = FakeProject(
"a/b", parent=p_a, is_derived=True, gitlink_path="b"
)
def fake_fetch(
projects: List[FakeProject], *_args: object
) -> sync._FetchResult:
calls.append(("fetch", [p.relpath for p in projects]))
return sync._FetchResult(True, {p.objdir for p in projects})
opt = mock.Mock(fail_fast=False)
with mock.patch.object(cmd, "_Fetch", side_effect=fake_fetch):
result = cmd._FetchParentFirst([p_a_b, p_a], opt, None, None, [])
self.assertTrue(result.success)
self.assertEqual(result.projects, {"a", "a/b"})
self.assertEqual(
calls,
[
("fetch", ["a"]),
("read gitlinks of", "a"),
("fetch", ["a/b"]),
],
)
class WithoutProjects(unittest.TestCase):
def test_drops_the_unwanted_projects(self) -> None:
p_a = FakeProject("a")
p_a_b = FakeProject("a/b")
self.assertEqual(sync._WithoutProjects([p_a, p_a_b], [p_a_b]), [p_a])
self.assertEqual(sync._WithoutProjects([p_a, p_a_b], []), [p_a, p_a_b])
def test_keeps_projects_with_the_same_path(self) -> None:
# Paths are relative to their own (sub)manifest, so two projects can
# share one.
first = FakeProject("a/b")
second = FakeProject("a/b")
self.assertEqual(
sync._WithoutProjects([first, second], [second]), [first]
)
class Chunksize(unittest.TestCase):
"""Tests for _chunksize."""
@@ -1158,6 +1334,116 @@ class InterleavedSyncTest(unittest.TestCase):
execute_mock.assert_called_once()
def test_interleaved_refreshes_submodule_revision(self) -> None:
"""Test submodules are synced at the revision of the fetched parent."""
opt, args = self.cmd.OptionParser.parse_args(["--interleaved", "-j4"])
opt.quiet = True
submodule = FakeProject(
"projA/sub",
name="projA_sub",
objdir="objA_sub",
parent=self.projA,
is_derived=True,
revisionId="stale",
gitlink_path="sub",
)
all_projects = [self.projA, submodule]
mock.patch.object(
self.cmd, "GetProjects", return_value=all_projects
).start()
self.projA.GetSubmoduleRevisions = mock.Mock(
return_value={"sub": "fetched"}
)
synced = []
def execute_side_effect(
jobs: int,
target: object,
work_items: List[List[int]],
**kwargs: object,
) -> bool:
synced_relpaths_set = kwargs["callback"].args[0]
projects_in_pass = self.cmd.get_parallel_context()["projects"]
for item in work_items:
for project_idx in item:
project = projects_in_pass[project_idx]
synced.append((project.relpath, project.revisionId))
synced_relpaths_set.add(project.relpath)
return True
mock.patch.object(
self.cmd, "ExecuteInParallel", side_effect=execute_side_effect
).start()
self.cmd._SyncInterleaved(
opt,
args,
[],
self.manifest,
self.manifest.manifestProject,
all_projects,
{},
)
self.assertIn(("projA/sub", "fetched"), synced)
def test_interleaved_skips_removed_submodule(self) -> None:
"""Test submodules dropped by their parent are not checked out."""
opt, args = self.cmd.OptionParser.parse_args(["--interleaved", "-j4"])
opt.quiet = True
submodule = FakeProject(
"projA/sub",
name="projA_sub",
objdir="objA_sub",
parent=self.projA,
is_derived=True,
revisionId="stale",
gitlink_path="sub",
)
# The parent no longer holds a gitlink for the submodule.
self.projA.GetSubmoduleRevisions = mock.Mock(return_value={})
# The reloaded manifest no longer derives the removed submodule.
mock.patch.object(
self.cmd, "GetProjects", return_value=[self.projA]
).start()
synced = []
def execute_side_effect(
jobs: int,
target: object,
work_items: List[List[int]],
**kwargs: object,
) -> bool:
synced_relpaths_set = kwargs["callback"].args[0]
projects_in_pass = self.cmd.get_parallel_context()["projects"]
for item in work_items:
for project_idx in item:
project = projects_in_pass[project_idx]
synced.append(project.relpath)
synced_relpaths_set.add(project.relpath)
return True
mock.patch.object(
self.cmd, "ExecuteInParallel", side_effect=execute_side_effect
).start()
self.cmd._SyncInterleaved(
opt,
args,
[],
self.manifest,
self.manifest.manifestProject,
[self.projA, submodule],
{},
)
self.assertEqual(synced, ["projA"])
def test_interleaved_shared_objdir_serial(self):
"""Test that projects with shared objdir are processed serially."""
opt, args = self.cmd.OptionParser.parse_args(["--interleaved", "-j4"])
+34
View File
@@ -63,3 +63,37 @@ def test_UploadAndReport_UnhandledError(cmd: upload.Upload) -> None:
with mock.patch.object(cmd, "_UploadBranch", side_effect=UnexpectedError):
with pytest.raises(UnexpectedError):
cmd._UploadAndReport(opt, [mock.MagicMock()], _STUB_PEOPLE)
def test_GetMergeBranch_explicit_branch(cmd: upload.Upload) -> None:
"""Verify _GetMergeBranch reads branch.merge for explicit local_branch."""
mock_project = mock.MagicMock()
mock_branch = mock.MagicMock()
mock_branch.merge = "refs/heads/main"
mock_project.GetBranch.return_value = mock_branch
res = cmd._GetMergeBranch(mock_project, local_branch="feature")
assert res == "refs/heads/main"
mock_project.GetBranch.assert_called_once_with("feature")
def test_GetMergeBranch_current_branch(cmd: upload.Upload) -> None:
"""Verify _GetMergeBranch falls back to project.CurrentBranch."""
mock_project = mock.MagicMock()
mock_project.CurrentBranch = "auto-cbr"
mock_branch = mock.MagicMock()
mock_branch.merge = "refs/heads/upstream-main"
mock_project.GetBranch.return_value = mock_branch
res = cmd._GetMergeBranch(mock_project, local_branch=None)
assert res == "refs/heads/upstream-main"
mock_project.GetBranch.assert_called_once_with("auto-cbr")
def test_GetMergeBranch_none_when_no_branch(cmd: upload.Upload) -> None:
"""Verify _GetMergeBranch returns empty string when detached HEAD."""
mock_project = mock.MagicMock()
mock_project.CurrentBranch = None
res = cmd._GetMergeBranch(mock_project, local_branch=None)
assert res == ""