Compare commits

..
30 Commits
Author SHA1 Message Date
Gavin Mak a816b88d23 repo: bump launcher version to 2.74
https://gerrit-review.git.corp.google.com/c/git-repo/+/655063 bumped
VERSION to 2.72, but the next release tag is v2.74.

Change-Id: Ic2c0bcb6c1ccc4e553ba7d3f55eb557c47ebb71c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/657024
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-13 14:18:57 -08:00
Gavin Mak 032f2341ec init, sync: support per-branch smart sync branch config
Add `repo init --smart-sync-branch=BRANCH` to persist a server-side
Smart Sync branch override in `repo.<manifest-branch>.smartsyncbranch`
inside .repo/manifests.git/config (with an empty string clearing it).

During Smart Sync, non-empty SYNC_BRANCH still takes top precedence,
while SYNC_BRANCH="" forces the raw manifest branch over checkout
config.

Bug: 543101585
Bug: 585378823
Change-Id: I933a48d03ed5ebd83aa5aba9ddde761dfac23243
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655063
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-11-13 13:55:47 -08:00
Gavin Mak af53524261 sync: sort interleaved sync slowest-first
Phased sync orders each fetch batch slowest-first using
`.repo_fetchtimes.json`, whereas interleaved sync dispatched each
hierarchical level in path order, letting slow projects near the end of
a level start late and extend the tail of the sync.

Combining fetch and checkout times into `.repo_fetchtimes.json` would
mix units with phased mode and older repo versions, and cause `-n` or
`-l` runs to either skew full-sync estimates or sort by the wrong
phase.

Keep fetch and checkout estimates separate instead:

- Parameterize `_SyncTimes` by filename and track fetch durations in
  `.repo_fetchtimes.json` and checkout durations in
  `.repo_checkouttimes.json`, recording a phase only when it succeeds.
- Sort each interleaved level slowest-first by the phases active in
  `opt` (fetch when not `-l`, checkout when not `-n` and the project
  has a worktree).
- Sort each `_SafeCheckoutOrder` level in phased `_Checkout` by
  `_checkout_times` and save in `finally` so phased and interleaved
  runs maintain and use the same per-phase estimates.

Bug: 543101585
Change-Id: Ie8b15a95ab2d8019b058d5f839a93501d0d85979
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/657085
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-11-13 13:44:18 -08:00
Gavin Mak 2259e50864 sync: add --force-jobs flag to override manifest sync-j-max
Checkouts backed by a local daemon via `repo.fetchcmd` and
`repo.reprojectcmd` do not hit remote Git servers during `repo sync` and
benefit from higher `-j` concurrency than a manifest's `sync-j-max` cap
allows.

Add `--force-jobs` to `repo sync` (and `repo smartsync`) to bypass the
manifest `sync-j-max` limit and mention `--force-jobs` in the clamp
warning message.

Bug: 585378822
Change-Id: Id0310f85e5a169207f7acfbe004744e77df2780f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/657543
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-13 13:18:29 -08:00
Gavin Mak 1702d7fb55 project: avoid git spawns in custom fetch ref checks
During _CustomFetch, GetEnvVars resolves REPO_LREV via GetRevisionId
before fetchcmd runs, and postcondition checks afterward run ResolveRef
on the tracking ref and FETCH_HEAD. On cold clones, missing branches, or
packed-refs tracking refs, GetRevisionId fell back to ResolveCommit,
spawning `git rev-parse <rev>^{commit}` and peeling objects from lazily
populated object stores such as ABFS.

Resolve non-tag refs/* in GetRevisionId via GitRefs.TryResolveRef and
authoritative files snapshots so loose, packed, and missing refs resolve
without spawning git or reading .git/objects. In _CustomFetch,
invalidate bare_ref after fetchcmd runs, resolve tracking refs through
bare_ref to warm the snapshot for _InitMRef and Sync_LocalHalf, and read
FETCH_HEAD's leading object ID directly when unambiguous.

Bug: 543101585
Change-Id: Ifb5c0f182e539a2836672a47da348fbeb41e5068
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655964
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-11-12 16:10:31 -08:00
Gavin Mak 7cbd9d1762 project: only unset core.bare when it is true
When creating a gitdir, _InitGitDir spawned `git config --unset-all
core.bare` for every non-mirror project. That is needed when `git init`
wrote `core.bare = true`, as it does for a gitdir under
.repo/project-objects. For a local worktree gitdir (<worktree>/.git, as
with --use-local-gitdirs) `git init` already writes `core.bare = false`,
so the spawn only removed a setting that was already correct. Checked
on git 1.9.1 (the oldest version repo supports), 2.7.4, 2.20.1 and
2.56.

Unset core.bare on non-mirror repositories only when it is currently
true. Local worktree gitdirs now keep the `core.bare = false` that
`git init` wrote.

Bug: 543101585
Change-Id: I662ffce61dc68ebca921941c1a5e5ebce2d22338
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655965
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-12 16:07:48 -08:00
Hung-Wei Chen f01eb8774f git_config, project: coalesce SetString writes in BatchWrites
This change attempts to skip sequential git config calls in order to
further speed up git-repo's performance for VFS integration. On
git-main we currently call git config 26K+ times, causing roughly 5~6
seconds of latency (see b/543101585#comment23's TRACE_FILE for data).
Given the expected time of VFS integration is targeting 30s, 6 seconds
makes a huge difference.

During an initial `repo sync`, initializing each project (`_InitGitDir`
and `_InitRemote`) invokes `SetString` multiple times per project, and
`SyncAnalysisState` writes `repo.syncstate.*` keys once per sync.
Because the `git config` CLI has no batch-write mode to set multiple
keys in a single invocation, each `SetString` call previously spawned a
separate `git config` subprocess and rewrote `.repo_config.json` per key
(see b/543101585#comment23 and b/543101585#comment25 for trace profiles
and benchmark data).

Bug: 543101585
Bug: 564817989
Change-Id: Ia1d4613266fb6b7742e8db637039e72ed26a13a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/657083
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-12 15:37:33 -08:00
Gavin Mak 2aca835f1f sync: warm init.templateDir cache for sync workers
When initializing missing project object directories in parallel, each
sync worker cold-misses the init.templateDir lookup cache and spawns
`git config --get init.templatedir`. Relying on parent process memory
alone also misses workers started via `spawn` or `forkserver`.

Resolve the init.templateDir lookup once in the parent process when
parallel sync workers may initialize an object directory (either
because an objdir is missing or `--force-sync` is active), pass the
cached entry through `parallel_context`, and seed it in `InitWorker`.

Bug: 543101585
Change-Id: I6bac2352a9fd4b62d25ed2a89b670ac21aa70f4e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/657084
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-11-12 15:33:07 -08:00
Gavin Mak c9cfb541aa sync: trim preciousObjects work
Only set preciousObjects when objdir differs from gitdir, check actual
objdir sharing in _GetPreciousObjectsState, and skip post-sync
preciousObjects scans on unshared --use-local-gitdirs projects.

Bug: 543101585
Change-Id: I5077aed128f22cf19f520c7866cfd62f0c2724be
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/652266
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-12 14:51:41 -08:00
Mike Frysinger c85e5ff8a5 pyrefly: initial support
This suppresses a lot of findings in most files so we can run pyrefly
by default and prevent new findings from showing up.  This will need
time to slowly chip away file by file.

Concretely, it's enabled for ~60% of files, and disabled for the other
~40% files.  Not the worst start.

I've opted for disabling files completely at this point rather than
using a baseline because the generated baseline is >200KiB in size.
I think we need to get through a good chunk of that before we switch.

We're going with pyrefly because it's what Google has adopted, and it
evaluated alternatives including mypy, pyright, and its own pytype.

Change-Id: Ib73752dd240a9a84ce0f56059a82292141b6af66
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656364
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-12 11:33:10 -08:00
Mike Frysinger ba7bc573ad typing: fix various errors caught by pyrefly
These should be fairly straightforward and not require much thought.
I've punted the more complicated typing changes.

Change-Id: I79167f07e0ec259586520104f07e4cee096a382e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656663
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-11-12 11:23:08 -08:00
Mike Frysinger f74f219954 project: fix invalid logger call
This used to be a print() call, but the migration was incomplete.

Change-Id: If17e5108aa3d2a2948c4834e9c6ed895d3809f91
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656724
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-12 09:17:39 -08:00
Mike Frysinger f9a1895400 project: fix Project.HasChanges API mismatches
The base Project class defined HasChanges as a function, but the
MetaProject & RepoProject subclasses defined it as a property.
Obviously this is bad & confusing, so standardize it.

Change-Id: Ia3ac9d0592307366296b8ef991dc5d7bb2204490
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656723
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-12 09:17:27 -08:00
Brian Gan 13fe1b9d84 pytest: only keep temp dirs of failed tests
By default, pytest keeps the temp dirs of every test for the last 3
sessions.  Each e2e test clones & checks out a snapshot of the tree
under its `tmp_path`, so these add up, and /tmp is often a tmpfs.

Set `tmp_path_retention_policy` to `failed` so a test's `tmp_path` is
removed as soon as the test passes, and the session's temp dir is
removed when all tests pass.  The `tmp_path` of a failed test is still
kept for debugging.  Pass `-o tmp_path_retention_policy=all` to keep
the dirs of passing tests too.

Test: vpython3 run_tests
Change-Id: I2cf464b063d62f155ffdbfed24a12ea9774086d6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656093
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
2026-11-12 08:43:05 -08:00
Mike Frysinger b1a1cc603a tests: switch to mock.patch.object
Python style reviewers often point people to use mock.patch.object
since it can be statically checked better.  Migrate code, and update
the SKILL file to cover this too.

Change-Id: If7aa8e3ddcdb3d739187b12e3034de1de990a7df
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656246
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-11-11 18:19:24 -08:00
Mike Frysinger 2b7969d1cf run_tests: turn into multicall program
Add shortcuts to easily run individual tools while re-using the embedded
vpython/UV script settings.

Change-Id: I4c63e7fd402666628a18cc77187c958d93910f80
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656245
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-11-11 15:34:09 -08:00
Gavin Mak bb9b4c2b60 git_refs, project: read files-backend refs in Python
GitRefs._LoadAll spawns `git for-each-ref` to load HEAD and refs/*, and
Project._InitAnyMRef leaves warm snapshots stale after direct UpdateRef
writes. On virtualized checkouts with thousands of repositories, these
spawns add avoidable process overhead.

Read HEAD, packed-refs, and loose refs directly in GitRefs when the
repository uses a plain files ref backend, recording (mode, mtime, size,
inode) stat tuples during the single directory pass and rechecking
packed-refs afterward to guard against concurrent pack-refs updates.
The tuples leave out ctime, which backup agents and file system crawlers
move without changing a ref. Fall back to `git for-each-ref` when
GIT_DIR, GIT_COMMON_DIR, reftable, commondir, custom refStorage config,
symlinks, or lockfiles are present.

Also add GitRefs.TryResolveRef for single-ref lookups without scanning
all of refs/, GitRefs.Invalidate, and GitRefs.UpdateRef to keep cached
snapshots warm after _InitAnyMRef writes a direct ref.

Bug: 543101585
Change-Id: I804ac45712208e9f9e6437012661bc1de58b9825
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655963
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-11-11 14:56:42 -08:00
Brian Gan 33aaf275be sync: handle smart sync when the manifest checkout has no branch
If the manifest checkout in `.repo/manifests` is on a detached HEAD, or
on a local branch without an upstream, `Sync._GetBranch` read a missing
`merge` config and crashed with `'NoneType' object has no attribute
'startswith'`. When smart sync is enabled by default, this broke plain
`repo sync` for such checkouts because `AttributeError` bypassed the
fallback to a regular ToT sync in `_ExecuteHelper`, which only catches
`SmartSyncError` and `ManifestParseError`. Pinning the manifest via
`repo init -b <sha>` or `repo init -b refs/tags/<tag>` also sent the
raw revision to the manifest server as a branch name.

Add `ManifestProject.GetMergeBranch(heads_only=...)` (shared with
`XmlManifest._Load`) to resolve the upstream ref of the current
manifest branch, returning `None` under `heads_only=True` unless it
tracks `refs/heads/*`, and raise `SmartSyncError` in `_SmartSyncSetup`
before running the remote helper or contacting the manifest server.
Implicit smart sync now falls back to a regular ToT sync, and explicit
smart sync fails early with an actionable error message.

Bug: 580004259
Test: vpython3 run_tests
Change-Id: Ice1f9bfe7ea8c923a283bcbd0ac7576cf33e42f6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655123
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-11-11 12:02:47 -08:00
Mike Frysinger f6523426b9 style: use from subcmds import sync everywhere
Align with Google style guides and `from subcmds import sync` as a
module rather than import symbols out of it.

Change-Id: Iac4ae43fc8f43304b6ffe381cf290378423cd133
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655847
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-11-11 09:45:00 -08:00
Mike Frysinger d519ee40a9 style: use import hooks everywhere
Align with Google style guides and `import hooks` as a module
rather than import symbols out of it.

Change-Id: I85a071ee73f8904ac33eae437d7853785762ba71
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655846
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-11 09:40:37 -08:00
Mike Frysinger 9b774577cb pytest: enable xdist usage
The testsuite has grown in size quite a bit.  It takes ~3 min on
my cloudtop with ~48 cores.  Running it in parallel makes it take
~20 seconds.

I've capped it to 32 jobs because it seems to complete in the same
amount of time as 128 jobs, and there's startup cost to spinning
up each node.

Change-Id: I6daa456a92e9db213118c9d16c9dee646db448e0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655887
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-11-11 09:14:21 -08:00
Mike Frysinger 0714b7e434 run_tests: regen uv.lock
Since vpython no longer does this automatically, I upgraded flake8
incorrectly by only updating the script file.  Add a note to remind
people they need to regen the lock file themselves.

Change-Id: I7884714a110bbf843c76aca6e3a2ff14b932b943
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655845
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-11-11 05:56:58 -08:00
Mike Frysinger 96363629bd flake8: fix warnings with newer versions
The flake8 version hasn't updated properly due the uv.lock file
being out of date.  Fix newer warnings before unwedging that.

We have a lot of B042 errors that probably should be fixed by reworking
the code, but for now suppress them since we've been doing this for a
long time in repo.
error.py:25:5: B042 Exception class with `__init__` should pass all args to `super().__init__()` to work in edge cases of `pickle` and `copy.copy()`. It should also not take any kwargs.

The only other warning is fairly simple:
subcmds/sync.py:1154:9: B018 Found useless Call expression. Consider either assigning it to a variable or removing it.

Change-Id: I532000a66a5d8493a490fa30080b137aff5f706e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655844
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-11-10 17:59:34 -08:00
Mike Frysinger 11bf826a10 git_command: move _GetBasicEnv to module level
RepoSourceVersion() and _build_env() are module-level functions that
called GitCommand._GetBasicEnv(), while GitCommand itself never called
it directly.  When tests mock git_command.GitCommand before
RepoSourceVersion() has cached its result (such as running
test_upload_for_review_real_event_log in isolation), calling
GitCommand._GetBasicEnv() returned a MagicMock instead of an env dict,
causing subprocess.run to fail with ValueError.

Change-Id: Ib8b866b012602c549ce2d59105526d847b0d7d51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656383
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-10 17:58:35 -08:00
Brian Gan 1e88f21cb1 sync: remove obsolete projects before interleaved checkouts
In `_SyncInterleaved`, `_UpdateManifestLists` only ran after fetching
and checking out all projects. When a manifest replaced a parent project
`foo` with a nested child project `foo/bar`, `foo/bar` was checked out
inside the still-existing obsolete `foo` worktree before
`UpdateProjectList` deleted `foo`. Because `bar/` was tracked by `foo`,
`git status` in `foo` saw `foo/bar`'s newly checked-out files as
modified and untracked files belonging to `foo`, causing
`DeleteWorktree` to fail with `DeleteDirtyWorktreeError` (and similarly,
when `foo/bar` was merged back into `foo`, `foo`'s checkout collided
with the not-yet-deleted `foo/bar` worktree):

  error: foo: Cannot remove project: uncommitted changes are present.
  error: Unable to fully sync the tree
  error: Updating local project lists failed.

Split `_UpdateManifestLists` into `_UpdateProjectLists` and
`_UpdateCopyLinkfileLists`, and run `_UpdateProjectLists` before the
sync loop in `_SyncInterleaved` so obsolete manifest projects are
deleted before new project checkouts begin (and `--fail-fast` aborts
before fetching if an obsolete project cannot be removed). Run
`_UpdateProjectLists` again after the sync loop (passing the set of
paths that already failed removal pre-sync so they are not retried or
reported twice) together with `_UpdateCopyLinkfileLists` so derived
submodules discovered or dropped when `_ReloadManifest` runs during the
loop are updated in `project.list` and copyfiles/linkfiles are updated.

Because the pre-sync `_UpdateProjectLists` pass in `_SyncInterleaved`
runs before projects are fetched (unlike `_SyncPhased`, which runs it
after `_FetchMain`), a `sync-s="true"` parent project whose manifest
revision moved to an unfetched commit cannot derive its submodules yet
(`GetSubmoduleRevisions() is None`). Retain existing submodule checkouts
under such parents until a post-fetch pass can determine whether their
submodules were removed. Also continue past individual
`DeleteWorktreeError` failures in `UpdateProjectList` (skipping obsolete
ancestors of any retained child worktree) so remaining clean obsolete
projects are still removed while failed paths stay tracked in
`project.list`, and skip project list and copy/linkfile updates when
`--network-only` is set so `-n` does not mutate local worktrees.

Bug: 574048485
Test: vpython3 run_tests
Change-Id: I530337977f063173a9a8a48df3250627cd4bc57f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656283
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
2026-11-10 15:25:06 -08:00
Mike Frysinger 4bb901eab0 style: use import command everywhere
Align with Google style guides and `import command` as a module
rather than import symbols out of it.

The help subcmd needed a minor rewrite as it had a "command"
variable already.

Change-Id: I9b465c85246670d63827df0744b00e59033e87b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656243
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-10 12:59:38 -08:00
Mike Frysinger b2e30cef91 tests: delete buggy git_command tests
The Popen mocking is incomplete & buggy:
* It uses a nested class which hinders pickling.
* Nested classes should rarely be used in the first place, stylewise.
* It does not mock the full Popen API causing failures.

For example, this fails:
$ pytest -vvv tests/test_git_command.py::GitCommandWaitTest
...
FAILED tests/test_git_command.py::GitCommandWaitTest::test_default_returns_non_zero_result - AttributeError: 'MockPopen' object has no attribute 'returncode'
FAILED tests/test_git_command.py::GitCommandWaitTest::test_raises_when_verify_non_zero_result - AttributeError: 'MockPopen' object has no attribute 'returncode'
FAILED tests/test_git_command.py::GitCommandWaitTest::test_returns_when_no_verify_non_zero_result - AttributeError: 'MockPopen' object has no attribute 'returncode'
...

Delete the tests since they don't seem to add much value currently.

Bug: b/293344017
Change-Id: I3d613a84657e30cbc7eac0b363ebfcb46b95eb49
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655886
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-10 11:17:32 -08:00
Mike Frysinger 27402ddb25 style: use import main everywhere
Align with Google style guides and `import main` as a module
rather than import symbols out of it.

Change-Id: Iecfb3c89a14d01b24ebfb0aa4584d8d6a94f70fd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654821
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-10 11:08:42 -08:00
Mike Frysinger 5b04e6e7c5 style: use import subcmds everywhere
Align with Google style guides and `import subcmds` as a module
rather than import symbols out of it.

Change-Id: If19503b977798f8a30e5d54cea9cd63a5c7b81d1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654820
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-10 11:01:44 -08:00
Mike Frysinger eb668be30d style: use import command in unittests
Align with Google style guides and `import command` as a module
rather than import symbols out of it in the tests tree.

Change-Id: I5c151d22c9ed8f8c7ea762823c8b2e25225a2ac4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654819
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-10 10:45:58 -08:00
89 changed files with 4573 additions and 942 deletions

No files matched your search

+3 -26
View File
@@ -10,32 +10,10 @@ per-file-ignores =
git_status.py: IMR241, git_status.py: IMR241,
main.py: IMR241, main.py: IMR241,
manifest_xml.py: IMR241, manifest_xml.py: IMR241,
subcmds/abandon.py: IMR241,
subcmds/branches.py: IMR241,
subcmds/checkout.py: IMR241,
subcmds/cherry_pick.py: IMR241,
subcmds/diffmanifests.py: IMR241,
subcmds/diff.py: IMR241,
subcmds/download.py: IMR241,
subcmds/forall.py: IMR241,
subcmds/gc.py: IMR241, subcmds/gc.py: IMR241,
subcmds/grep.py: IMR241,
subcmds/help.py: IMR241,
subcmds/info.py: IMR241,
subcmds/init.py: IMR241,
subcmds/list.py: IMR241,
subcmds/manifest.py: IMR241,
subcmds/overview.py: IMR241,
subcmds/prune.py: IMR241,
subcmds/rebase.py: IMR241, subcmds/rebase.py: IMR241,
subcmds/selfupdate.py: IMR241,
subcmds/smartsync.py: IMR241,
subcmds/stage.py: IMR241,
subcmds/start.py: IMR241,
subcmds/status.py: IMR241,
subcmds/sync.py: IMR241, subcmds/sync.py: IMR241,
subcmds/upload.py: IMR241, subcmds/upload.py: IMR241,
subcmds/version.py: IMR241,
subcmds/wipe.py: IMR241, subcmds/wipe.py: IMR241,
tests/e2e/test_failures.py: IMR241, tests/e2e/test_failures.py: IMR241,
tests/e2e/test_groups.py: IMR241, tests/e2e/test_groups.py: IMR241,
@@ -44,12 +22,11 @@ per-file-ignores =
tests/e2e/test_partial_clone.py: IMR241, tests/e2e/test_partial_clone.py: IMR241,
tests/e2e/test_sync_modes.py: IMR241, tests/e2e/test_sync_modes.py: IMR241,
tests/e2e/test_workflows.py: IMR241, tests/e2e/test_workflows.py: IMR241,
tests/test_command.py: IMR241,
tests/test_error.py: IMR241,
tests/test_main.py: IMR241,
tests/test_subcmds.py: IMR241,
tests/test_subcmds_sync.py: IMR241, tests/test_subcmds_sync.py: IMR241,
extend-ignore = extend-ignore =
# B042: Exception class with __init__ should pass all args to super().__init__()
# We should fix this by reworking our exception classes.
B042,
# E203: Whitespace before ':' # E203: Whitespace before ':'
# See https://github.com/PyCQA/pycodestyle/issues/373 # See https://github.com/PyCQA/pycodestyle/issues/373
E203, E203,
+51 -8
View File
@@ -3471,6 +3471,12 @@ standard streams and filesystem paths.
: : : : asserted directly : : : : : asserted directly :
: : : : in the test : : : : : in the test :
: : : : docstring. : : : : : docstring. :
| **T5-29** | Object-Referenced Mocking via | High | Passing a dotted |
: : `mock.patch.object` : : string target to :
: : : : `mock.patch` :
: : : : instead of using :
: : : : `mock.patch.object` :
: : : : on the object. :
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
@@ -3903,13 +3909,13 @@ file where it was locally imported.**
**Don't:** **Don't:**
```python ```python
@mock.patch("main.time.sleep") @mock.patch.object(main.time, "sleep")
``` ```
**Do:** **Do:**
```python ```python
@mock.patch("time.sleep") @mock.patch.object(time, "sleep")
``` ```
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
@@ -3939,7 +3945,7 @@ LRU cache intact across tests.**
```python ```python
class SshTests(unittest.TestCase): class SshTests(unittest.TestCase):
def test_version(self): def test_version(self):
with mock.patch("ssh._run_ssh_version", return_value="OpenSSH_1.2\n"): with mock.patch.object(ssh, "_run_ssh_version", return_value="OpenSSH_1.2\n"):
self.assertEqual(ssh.version(), (1, 2)) self.assertEqual(ssh.version(), (1, 2))
``` ```
@@ -3951,7 +3957,7 @@ class SshTests(unittest.TestCase):
ssh.version.cache_clear() ssh.version.cache_clear()
def test_version(self): def test_version(self):
with mock.patch("ssh._run_ssh_version", return_value="OpenSSH_1.2\n"): with mock.patch.object(ssh, "_run_ssh_version", return_value="OpenSSH_1.2\n"):
self.assertEqual(ssh.version(), (1, 2)) self.assertEqual(ssh.version(), (1, 2))
``` ```
@@ -4015,15 +4021,15 @@ with contextlib.redirect_stderr(f):
**Don't:** **Don't:**
```python ```python
with mock.patch("sys.stderr") as mock_stderr: with mock.patch.object(time, "sleep") as mock_sleep:
with mock.patch("socket.socket") as ms: with mock.patch.object(socket, "socket") as ms:
run_test() run_test()
``` ```
**Do:** **Do:**
```python ```python
with mock.patch("sys.stderr") as mock_stderr, mock.patch("socket.socket") as ms: with mock.patch.object(time, "sleep") as mock_sleep, mock.patch.object(socket, "socket") as ms:
run_test() run_test()
``` ```
@@ -4075,7 +4081,7 @@ mock_socket.connect.side_effect = OSError("Mock error")
**Do:** **Do:**
```python ```python
mock.patch("socket.socket", side_effect=OSError) mock.patch.object(socket, "socket", side_effect=OSError)
``` ```
**Exceptions:** Tests explicitly verifying exact formatting serialization (e.g., **Exceptions:** Tests explicitly verifying exact formatting serialization (e.g.,
@@ -4613,6 +4619,43 @@ def test_Color_Parse_first_parameter_none(self):
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
#### T5-29: Object-Referenced Mocking via `mock.patch.object`
> **Rule:** Never use `mock.patch`; always use `mock.patch.object`.
>
> **What:** All attribute and function patching in unit tests must use
> `mock.patch.object(target, "attribute", ...)` against an explicitly imported
> module or class object rather than passing dotted string paths to
> `mock.patch("module.attribute", ...)`.
>
> **Applies To:** All unit tests utilizing `unittest.mock`.
>
> **Why:** String-based `mock.patch` paths bypass static analysis and module
> import validation, making tests brittle during refactoring and prone to silent
> resolution errors or import ordering bugs. Referencing the imported object
> directly via `mock.patch.object` ensures the target module is explicitly
> imported and validated at load time. Failing to adhere to this typically
> results in **Brittle Patch Routing / Unvalidated String Targets**.
**Trap 1: Passing a dotted string target to `mock.patch` instead of using
`mock.patch.object` on the object.**
**Don't:**
```python
with mock.patch("ssh._run_ssh_version", return_value="OpenSSH_1.2\n"):
...
```
**Do:**
```python
with mock.patch.object(ssh, "_run_ssh_version", return_value="OpenSSH_1.2\n"):
...
```
--------------------------------------------------------------------------------
### Cross-Domain Dependencies ### Cross-Domain Dependencies
* **Upstream:** T2 | Filesystem Atomicity & Worktree Layout - *Managing safe * **Upstream:** T2 | Filesystem Atomicity & Worktree Layout - *Managing safe
Symlink
+1
View File
@@ -0,0 +1 @@
run_tests
+1 -1
View File
@@ -82,7 +82,7 @@ def _Color(fg=None, bg=None, attr=None):
return code return code
DEFAULT = None DEFAULT: Optional[str] = None
class _CheckConsoleSentinel: class _CheckConsoleSentinel:
+7 -4
View File
@@ -43,7 +43,7 @@ WORKER_BATCH_SIZE = 32
# How many jobs to run in parallel by default? This assumes the jobs are # How many jobs to run in parallel by default? This assumes the jobs are
# largely I/O bound and do not hit the network. # largely I/O bound and do not hit the network.
DEFAULT_LOCAL_JOBS = min(os.cpu_count(), 8) DEFAULT_LOCAL_JOBS = min(os.cpu_count() or 1, 8)
class UsageError(error.RepoExitError): class UsageError(error.RepoExitError):
@@ -105,7 +105,8 @@ class Command:
# Cache for the OptionParser property. # Cache for the OptionParser property.
self._optparse = None self._optparse = None
def WantPager(self, _opt): def WantPager(self, opt) -> bool:
del opt # Unused.
return False return False
@staticmethod @staticmethod
@@ -623,7 +624,8 @@ class InteractiveCommand(Command):
within a pager, even if the user asks to. within a pager, even if the user asks to.
""" """
def WantPager(self, _opt): def WantPager(self, opt) -> bool:
del opt # Unused.
return False return False
@@ -632,7 +634,8 @@ class PagedCommand(Command):
larger than one screen full. larger than one screen full.
""" """
def WantPager(self, _opt): def WantPager(self, opt) -> bool:
del opt # Unused.
return True return True
+2
View File
@@ -249,6 +249,7 @@ _repo() {
'(-s --smart-sync)'{-s,--smart-sync}'[Default to smart sync]' \ '(-s --smart-sync)'{-s,--smart-sync}'[Default to smart sync]' \
'--no-smart-sync[Do not default to smart sync]' \ '--no-smart-sync[Do not default to smart sync]' \
'--head[Do not default to smart sync]' \ '--head[Do not default to smart sync]' \
'--smart-sync-branch=[Smart sync branch]:branch:' \
'--clone-bundle[Use clone bundle]' \ '--clone-bundle[Use clone bundle]' \
'--no-clone-bundle[Do not use clone bundle]' \ '--no-clone-bundle[Do not use clone bundle]' \
'--git-lfs[Use git lfs]' \ '--git-lfs[Use git lfs]' \
@@ -315,6 +316,7 @@ _repo() {
${common_opts} \ ${common_opts} \
'--jobs-network=[Number of network jobs]:jobs:' \ '--jobs-network=[Number of network jobs]:jobs:' \
'--jobs-checkout=[Number of checkout jobs]:jobs:' \ '--jobs-checkout=[Number of checkout jobs]:jobs:' \
'--force-jobs[Force jobs even if manifest caps it]' \
'(-f --force-broken)'{-f,--force-broken}'[Continue sync even if a project fails]' \ '(-f --force-broken)'{-f,--force-broken}'[Continue sync even if a project fails]' \
'--fail-fast[Fail fast]' \ '--fail-fast[Fail fast]' \
'--force-sync[Overwrite existing git directories]' \ '--force-sync[Overwrite existing git directories]' \
+6 -2
View File
@@ -42,8 +42,11 @@ For example, if you want to change the manifest branch, you can simply run
change the git URL/branch that this tracks, re-run `repo init` with the new change the git URL/branch that this tracks, re-run `repo init` with the new
settings. settings.
* `.repo_fetchtimes.json`: Used by `repo sync` to record fetch times when * `.repo_fetchtimes.json`: Used by `repo sync` to record per-project fetch
syncing the various projects. times so subsequent syncs schedule the slowest projects first.
* `.repo_checkouttimes.json`: Used by `repo sync` to record per-project
checkout times so subsequent syncs schedule the slowest projects first.
* `.repo_localsyncstate.json`: Used by `repo sync` to detect and warn on * `.repo_localsyncstate.json`: Used by `repo sync` to detect and warn on
on partial tree syncs. Partial syncs are allowed by `repo` itself, but are on partial tree syncs. Partial syncs are allowed by `repo` itself, but are
@@ -177,6 +180,7 @@ User controlled settings are initialized when running `repo init`.
| repo.partialcloneexclude | `--partial-clone-exclude` | Comma-delimited list of project names (not paths) to exclude while using [partial git clones] | | repo.partialcloneexclude | `--partial-clone-exclude` | Comma-delimited list of project names (not paths) to exclude while using [partial git clones] |
| repo.reference | `--reference` | Reference repo client checkout | | repo.reference | `--reference` | Reference repo client checkout |
| repo.smartsync | `--smart-sync` | Default to [Smart Sync](./smart-sync.md) when syncing | | repo.smartsync | `--smart-sync` | Default to [Smart Sync](./smart-sync.md) when syncing |
| repo.\<branch\>.smartsyncbranch | `--smart-sync-branch` | Server branch to query when [Smart Sync](./smart-sync.md)ing `<branch>` |
| repo.submodules | `--submodules` | Sync git submodules | | repo.submodules | `--submodules` | Sync git submodules |
| repo.superproject | `--use-superproject` | Sync [superproject] | | repo.superproject | `--use-superproject` | Sync [superproject] |
| repo.worktree | `--worktree` | Use [git worktree] for checkouts | | repo.worktree | `--worktree` | Use [git worktree] for checkouts |
+50 -20
View File
@@ -200,9 +200,27 @@ following precedence order (highest to lowest):
Running `repo init --no-smart-sync` (or `repo init --head`) disables default Running `repo init --no-smart-sync` (or `repo init --head`) disables default
Smart Sync for the checkout even when enabled by the user setting or manifest. Smart Sync for the checkout even when enabled by the user setting or manifest.
As with the [manifest setting](#sync-smartsync-manifest-attribute), a As with the [manifest setting](#sync-smartsync-manifest-attribute), a
checkout-enabled default Smart Sync falls back to a regular ToT sync if the checkout-enabled default Smart Sync falls back to a regular ToT sync if Smart
manifest does not define a `<manifest-server>` or the manifest server cannot Sync fails (e.g. the manifest checkout is not tracking a remote branch,
provide a usable manifest. `<manifest-server>` is not defined, or the server cannot provide a usable
manifest).
### `repo init --smart-sync-branch=BRANCH`
Users can configure a server-side Smart Sync branch override for the initialized
manifest branch by passing `--smart-sync-branch=BRANCH` to `repo init`:
```bash
repo init -u <url> -b main -s --smart-sync-branch=git_main-warmup
```
This persists `repo.<manifest-branch>.smartsyncbranch` in
`.repo/manifests.git/config`. Because the setting is scoped to the manifest
branch, switching branches with `repo init -b <other>` will not send the
override branch when syncing `<other>`. Passing `--smart-sync-branch=""` clears
the override for the target manifest branch. Setting a non-empty override
requires the manifest checkout to track a remote branch (not a tag or commit
SHA).
### sync-smartsync (Manifest Attribute) ### sync-smartsync (Manifest Attribute)
@@ -215,16 +233,17 @@ by setting `sync-smartsync="true"` on the `<default>` element:
When enabled in the manifest, `repo sync` will automatically perform a Smart When enabled in the manifest, `repo sync` will automatically perform a Smart
Sync unless explicitly overridden on the command line. Since the user did not Sync unless explicitly overridden on the command line. Since the user did not
ask for a Smart Sync, any failure to obtain a manifest from the manifest server ask for a Smart Sync, any failure to smart sync (e.g. the manifest checkout is
(e.g. the server is unreachable, or it returns a manifest that cannot be parsed) not tracking a remote branch, the server is unreachable, or it returns a
is silently ignored and `repo sync` falls back to a regular ToT sync; pass manifest that cannot be parsed) is silently ignored and `repo sync` falls back
`--verbose` to see the reason. Repo marks these requests as `best_effort` (see to a regular ToT sync; pass `--verbose` to see the reason. Repo marks these
[GetApprovedManifestV2](#GetApprovedManifestV2)), so the server may also decline requests as `best_effort` (see [GetApprovedManifestV2](#GetApprovedManifestV2)),
them, e.g. during a staged rollout. This is only a fallback for the so the server may also decline them, e.g. during a staged rollout. This is only
manifest-driven default: an explicit `-s`/`--smart-sync` or `-t`/`--smart-tag` a fallback for the manifest-driven default: an explicit `-s`/`--smart-sync` or
still fails the sync, as before. Options that specify an explicit target or `-t`/`--smart-tag` still fails the sync, as before. Options that specify an
manifest source (such as `-t`/`--smart-tag`, `-m`/`--manifest-name` or explicit target or manifest source (such as `-t`/`--smart-tag`,
`--superproject-revision`) will also disable the default Smart Sync behavior. `-m`/`--manifest-name` or `--superproject-revision`) will also disable the
default Smart Sync behavior.
Users can override this manifest default on their workstation with the Users can override this manifest default on their workstation with the
`sync.smartsync` user setting; see `sync.smartsync` user setting; see
@@ -236,11 +255,22 @@ Explicitly enables Smart Sync. Repo will call
`GetApprovedManifestV2({branch[, target]})`, or `GetApprovedManifestV2({branch[, target]})`, or
`GetApprovedManifest(branch[, target])` if the server does not implement it. `GetApprovedManifest(branch[, target])` if the server does not implement it.
The `branch` is determined by the current manifest branch as specified by The `branch` sent to the manifest server is resolved in the following order:
`--manifest-branch=BRANCH` when running `repo init`. If `${SYNC_BRANCH}` is
defined and non-empty, its value is sent instead. This allows selecting a 1. `${SYNC_BRANCH}`: If defined and non-empty, its value is sent directly.
branch other than the one the checkout tracks, e.g. a server-side (CI) branch If defined as an empty string (`SYNC_BRANCH=""`), repo sends the checkout's
name that builds from the same manifest branch. raw manifest branch, bypassing `repo.<manifest-branch>.smartsyncbranch`.
2. `repo.<manifest-branch>.smartsyncbranch`: If configured via
`repo init --smart-sync-branch=BRANCH`, its value is sent for that manifest
branch.
3. The current manifest branch (as selected by `--manifest-branch=BRANCH` when
running `repo init`).
Overriding the branch allows selecting a server-side (CI) branch name that
builds or warms caches from the same manifest branch. If `${SYNC_BRANCH}` is
unset or empty and the manifest checkout is not tracking a remote branch (e.g.
it is on a detached HEAD, or was initialized with `-b <sha>` or
`-b refs/tags/<tag>`), Smart Sync fails without contacting the manifest server.
The `target` is defined by environment variables in the order below. If none The `target` is defined by environment variables in the order below. If none
of them match, then `target` is omitted. These variables were decided as they of them match, then `target` is omitted. These variables were decided as they
@@ -266,8 +296,8 @@ Smart Sync by default by setting the boolean `sync.smartsync` key in their
Command line flags (`-s`, `--no-smart-sync`, `-t`) override the user setting, Command line flags (`-s`, `--no-smart-sync`, `-t`) override the user setting,
which in turn overrides the manifest's `sync-smartsync` default. A Smart Sync which in turn overrides the manifest's `sync-smartsync` default. A Smart Sync
enabled this way is treated as an explicit request, so it is subject to the enabled this way is treated as an explicit request, so it is subject to the
same restrictions as `-s` and fails the sync if no manifest can be obtained same restrictions as `-s` and fails the sync if Smart Sync fails instead of
from the server instead of silently falling back to ToT. silently falling back to ToT.
### --smart-tag=TAG / -t ### --smart-tag=TAG / -t
+3 -3
View File
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from typing import List from typing import Optional, Sequence
class BaseRepoError(Exception): class BaseRepoError(Exception):
@@ -22,7 +22,7 @@ class BaseRepoError(Exception):
class RepoError(BaseRepoError): class RepoError(BaseRepoError):
"""Exceptions thrown inside repo that can be handled.""" """Exceptions thrown inside repo that can be handled."""
def __init__(self, *args, project: str = None) -> None: def __init__(self, *args, project: Optional[str] = None) -> None:
super().__init__(*args) super().__init__(*args)
self.project = project self.project = project
@@ -36,7 +36,7 @@ class RepoExitError(BaseRepoError):
self, self,
*args, *args,
exit_code: int = 1, exit_code: int = 1,
aggregate_errors: List[Exception] = None, aggregate_errors: Optional[Sequence[Exception]] = None,
**kwargs, **kwargs,
) -> None: ) -> None:
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
Symlink
+1
View File
@@ -0,0 +1 @@
run_tests
+22 -22
View File
@@ -18,7 +18,7 @@ import os
import re import re
import subprocess import subprocess
import sys import sys
from typing import Any, Optional from typing import Any, Dict, Optional
import error import error
import git_trace2_event_log_base import git_trace2_event_log_base
@@ -87,7 +87,7 @@ def RepoSourceVersion():
# We avoid GitCommand so we don't run into circular deps -- GitCommand needs # We avoid GitCommand so we don't run into circular deps -- GitCommand needs
# to initialize version info we provide. # to initialize version info we provide.
if ver is None: if ver is None:
env = GitCommand._GetBasicEnv() env = _get_basic_env()
proj = os.path.dirname(os.path.abspath(__file__)) proj = os.path.dirname(os.path.abspath(__file__))
env[GIT_DIR] = os.path.join(proj, ".git") env[GIT_DIR] = os.path.join(proj, ".git")
@@ -220,6 +220,25 @@ def git_require(min_version, fail=False, msg=""):
return False return False
def _get_basic_env() -> Dict[str, str]:
"""Return a basic env for running git under.
This is guaranteed to be side-effect free.
"""
env = os.environ.copy()
for key in (
repo_trace.REPO_TRACE,
GIT_DIR,
"GIT_ALTERNATE_OBJECT_DIRECTORIES",
"GIT_OBJECT_DIRECTORY",
"GIT_WORK_TREE",
"GIT_GRAFT_FILE",
"GIT_INDEX_FILE",
):
env.pop(key, None)
return env
def _build_env( def _build_env(
_kwargs_only=(), _kwargs_only=(),
bare: Optional[bool] = False, bare: Optional[bool] = False,
@@ -232,7 +251,7 @@ def _build_env(
assert _kwargs_only == (), "_build_env only accepts keyword arguments." assert _kwargs_only == (), "_build_env only accepts keyword arguments."
env = GitCommand._GetBasicEnv() env = _get_basic_env()
if disable_editor: if disable_editor:
env["GIT_EDITOR"] = ":" env["GIT_EDITOR"] = ":"
@@ -542,25 +561,6 @@ class GitCommand:
return buffer return buffer
@staticmethod
def _GetBasicEnv():
"""Return a basic env for running git under.
This is guaranteed to be side-effect free.
"""
env = os.environ.copy()
for key in (
repo_trace.REPO_TRACE,
GIT_DIR,
"GIT_ALTERNATE_OBJECT_DIRECTORIES",
"GIT_OBJECT_DIRECTORY",
"GIT_WORK_TREE",
"GIT_GRAFT_FILE",
"GIT_INDEX_FILE",
):
env.pop(key, None)
return env
def VerifyCommand(self) -> None: def VerifyCommand(self) -> None:
if self.rc == 0: if self.rc == 0:
return None return None
+326 -52
View File
@@ -24,7 +24,7 @@ import ssl
import subprocess import subprocess
import sys import sys
import tempfile import tempfile
from typing import Optional, Tuple, Union from typing import Any, Dict, Iterator, List, Optional, Tuple, Union
import urllib.error import urllib.error
import urllib.request import urllib.request
@@ -40,6 +40,8 @@ import repo_trace
SYNC_STATE_PREFIX = "repo.syncstate." SYNC_STATE_PREFIX = "repo.syncstate."
ID_RE = re.compile(r"^(?:[0-9a-f]{40}|[0-9a-f]{64})$") ID_RE = re.compile(r"^(?:[0-9a-f]{40}|[0-9a-f]{64})$")
_INI_SECTION_RE = re.compile(r"^[A-Za-z0-9-]+$")
_INI_KEY_RE = re.compile(r"^[A-Za-z][A-Za-z0-9-]*$")
REVIEW_CACHE = {} REVIEW_CACHE = {}
@@ -47,6 +49,85 @@ REVIEW_CACHE = {}
_JSON_META_KEY = "__repo_meta__" _JSON_META_KEY = "__repo_meta__"
def _SplitIniName(name: str) -> Optional[Tuple[str, str, str]]:
"""Split a config key into INI components when plain-append safe.
Args:
name: Dotted git config key name (e.g. "remote.origin.url").
Returns:
A (section, subsection, var) tuple if the key can be written as a
plain INI entry, or None if it requires git config CLI handling.
"""
parts = name.split(".")
if len(parts) < 2:
return None
section = parts[0]
var = parts[-1]
subsection = ".".join(parts[1:-1]) if len(parts) > 2 else ""
if not _INI_SECTION_RE.match(section) or not _INI_KEY_RE.match(var):
return None
if any(
ord(ch) < 0x20 or ord(ch) > 0x7E or ch in ('"', "\\")
for ch in subsection
):
return None
return (section, subsection, var)
def _IsPlainIniValue(val: Any) -> bool:
"""Check whether a config value needs no INI quoting or escaping.
Args:
val: Candidate configuration value passed to SetString.
Returns:
True if val is a string that can be written verbatim without quotes
or backslash escapes; False otherwise.
"""
if not isinstance(val, str):
return False
if val and (val[0] in (" ", "\t") or val[-1] in (" ", "\t")):
return False
return not any(
ord(ch) < 0x20 or ord(ch) > 0x7E or ch in ('"', "\\", ";", "#")
for ch in val
)
def _FormatIniBatch(ops: List[Tuple[str, List[str]]], existing: bytes) -> bytes:
"""Format queued key/value pairs as UTF-8 INI bytes to append.
Args:
ops: Ordered list of (name, values) pairs to append.
existing: Existing raw bytes of the target git config file.
Returns:
UTF-8 encoded INI stanza bytes ready to append after existing.
"""
lines: List[str] = []
if existing and not existing.endswith(b"\n"):
lines.append("\n")
last_header = None
for name, vals in ops:
parsed = _SplitIniName(name)
assert parsed is not None
section, subsection, var = parsed
header_key = (section.lower(), subsection)
if header_key != last_header:
if subsection:
lines.append(f'[{section} "{subsection}"]\n')
else:
lines.append(f"[{section}]\n")
last_header = header_key
for val in vals:
if val == "":
lines.append(f"\t{var} =\n")
else:
lines.append(f"\t{var} = {val}\n")
return "".join(lines).encode("utf-8")
def IsChange(rev): def IsChange(rev):
return rev.startswith(git_refs.R_CHANGES) return rev.startswith(git_refs.R_CHANGES)
@@ -168,6 +249,9 @@ class GitConfig:
self._cache_dict = None self._cache_dict = None
self._cache_fingerprint = None self._cache_fingerprint = None
self._section_dict = None self._section_dict = None
self._batch_depth = 0
self._batch_ops: Dict[str, Tuple[str, List[str]]] = {}
self._batch_json_dirty = False
self._remotes = {} self._remotes = {}
self._branches = {} self._branches = {}
@@ -190,9 +274,36 @@ class GitConfig:
with contextlib.suppress(OSError): with contextlib.suppress(OSError):
platform_utils.remove(self._json, missing_ok=True) platform_utils.remove(self._json, missing_ok=True)
@contextlib.contextmanager
def BatchWrites(self) -> Iterator[None]:
"""Coalesce compatible SetString writes under Git's config.lock.
Yields:
None while SetString writes are queued for batch flushing on exit.
"""
self._batch_depth += 1
try:
yield
except BaseException:
self._batch_depth -= 1
if self._batch_depth == 0:
try:
self._FlushBatch(save_json=True)
except Exception:
self._batch_ops.clear()
self._batch_json_dirty = False
self.ClearCache()
self._RemoveJson()
raise
else:
self._batch_depth -= 1
if self._batch_depth == 0:
self._FlushBatch(save_json=True)
def Reload(self) -> None: def Reload(self) -> None:
"""Clear cached state and reload configuration from disk.""" """Clear cached state and reload configuration from disk."""
self.ClearCache() self.ClearCache()
self._batch_json_dirty = False
self._RemoveJson() self._RemoveJson()
_ = self._cache _ = self._cache
@@ -314,6 +425,7 @@ class GitConfig:
if ( if (
self._cache_dict is not None self._cache_dict is not None
and not self._batch_ops
and self._ReadFileFingerprint() != self._cache_fingerprint and self._ReadFileFingerprint() != self._cache_fingerprint
): ):
self.ClearCache() self.ClearCache()
@@ -333,7 +445,15 @@ class GitConfig:
if value is None: if value is None:
if old: if old:
self._do("--unset-all", name, allow_missing=True) if key in self._batch_ops:
del self._batch_ops[key]
else:
if self._batch_ops and (
key == "include.path" or key.startswith("includeif.")
):
self._FlushBatch(save_json=False)
self._do("--unset-all", name, allow_missing=True)
self._cache_fingerprint = self._ReadFileFingerprint()
del self._cache[key] del self._cache[key]
self._SyncCacheAfterSet(key) self._SyncCacheAfterSet(key)
@@ -345,22 +465,171 @@ class GitConfig:
self.SetString(name, value[0]) self.SetString(name, value[0])
elif old != value: elif old != value:
self._do("--replace-all", name, value[0]) vals = list(value)
try: if self._CanQueueInBatch(key, name, vals, old):
for i in range(1, len(value)): self._cache[key] = vals
self._do("--add", name, value[i]) self._batch_ops[key] = (name, vals)
except BaseException: else:
self.ClearCache() if self._batch_ops:
self._RemoveJson() self._batch_ops.pop(key, None)
raise self._FlushBatch(save_json=False)
self._cache[key] = list(value) self._do("--replace-all", name, vals[0])
try:
for i in range(1, len(vals)):
self._do("--add", name, vals[i])
except BaseException:
self.ClearCache()
self._RemoveJson()
raise
self._cache[key] = vals
self._cache_fingerprint = self._ReadFileFingerprint()
self._SyncCacheAfterSet(key) self._SyncCacheAfterSet(key)
elif len(old) != 1 or old[0] != value: elif len(old) != 1 or old[0] != value:
self._do("--replace-all", name, value) vals = [value]
self._cache[key] = [value] if self._CanQueueInBatch(key, name, vals, old):
self._cache[key] = vals
self._batch_ops[key] = (name, vals)
else:
if self._batch_ops:
self._batch_ops.pop(key, None)
self._FlushBatch(save_json=False)
self._do("--replace-all", name, value)
self._cache[key] = vals
self._cache_fingerprint = self._ReadFileFingerprint()
self._SyncCacheAfterSet(key) self._SyncCacheAfterSet(key)
def _HasIncludes(self) -> bool:
"""Return True if cached config contains include or includeIf keys."""
return self._cache_dict is not None and (
"include.path" in self._cache_dict
or any(k.startswith("includeif.") for k in self._cache_dict)
)
def _CanQueueInBatch(
self,
key: str,
name: str,
vals: List[Any],
old: List[Optional[str]],
) -> bool:
"""Check whether a SetString write can be queued for lockfile append.
Args:
key: Lowercased canonical key used in self._cache.
name: Original dotted key name preserving subsection case.
vals: New value list requested by SetString.
old: Existing cached values for key before this write.
Returns:
True if the write can be appended at batch flush time; False if it
must be flushed immediately through git config.
"""
if self._batch_depth <= 0 or self.file == self._SYSTEM_CONFIG:
return False
if (
key == "include.path"
or key.startswith("includeif.")
or self._HasIncludes()
):
return False
if old and key not in self._batch_ops:
return False
return _SplitIniName(name) is not None and all(
_IsPlainIniValue(v) for v in vals
)
def _FlushBatch(self, save_json: bool = True) -> None:
"""Flush queued SetString writes via config.lock or git config CLI.
Args:
save_json: Whether to persist a dirty .repo_config.json cache
after flushing queued config updates.
"""
ops = list(self._batch_ops.values())
self._batch_ops.clear()
if ops and not self._TryWriteBatchLocked(ops):
self._ApplyBatchViaGit(ops)
if save_json and self._batch_json_dirty:
self._batch_json_dirty = False
if self._cache_dict is not None:
self._SaveJson(self._cache_dict)
def _TryWriteBatchLocked(self, ops: List[Tuple[str, List[str]]]) -> bool:
"""Atomically append queued entries under <file>.lock if unmodified.
Args:
ops: Ordered list of (name, values) pairs to append.
Returns:
True if the lockfile was written and renamed into place; False if
the lock was busy, the file changed externally, or an I/O error
occurred.
"""
if self.file == self._SYSTEM_CONFIG or os.path.islink(self.file):
return False
lock_path = f"{self.file}.lock"
try:
fd = os.open(lock_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o666)
except OSError:
return False
replaced = False
try:
existing = b""
curr_fp = None
if os.path.exists(self.file):
with open(self.file, "rb") as src_fp:
existing = src_fp.read()
st = os.fstat(src_fp.fileno())
curr_fp = (
st.st_mtime_ns,
st.st_ctime_ns,
st.st_size,
st.st_ino,
hashlib.sha1(existing).hexdigest(),
)
if hasattr(os, "fchmod") and not platform_utils.isWindows():
os.fchmod(fd, st.st_mode & 0o777)
if curr_fp != self._cache_fingerprint:
return False
payload = _FormatIniBatch(ops, existing)
with os.fdopen(fd, "wb") as lock_fp:
fd = -1
if existing:
lock_fp.write(existing)
lock_fp.write(payload)
os.replace(lock_path, self.file)
replaced = True
self._cache_fingerprint = self._ReadFileFingerprint()
return True
except OSError:
return False
finally:
if fd >= 0:
os.close(fd)
if not replaced:
platform_utils.remove(lock_path, missing_ok=True)
def _ApplyBatchViaGit(self, ops: List[Tuple[str, List[str]]]) -> None:
"""Apply queued batch operations via the git config CLI fallback.
Args:
ops: Ordered list of (name, values) pairs to write via git config.
"""
try:
for name, vals in ops:
self._do("--replace-all", name, vals[0])
for extra in vals[1:]:
self._do("--add", name, extra)
except BaseException:
self._batch_json_dirty = False
self.ClearCache()
self._RemoveJson()
raise
self.Reload()
def _SyncCacheAfterSet(self, key: str) -> None: def _SyncCacheAfterSet(self, key: str) -> None:
"""Keep the in-memory section index and JSON cache coherent. """Keep the in-memory section index and JSON cache coherent.
@@ -374,17 +643,16 @@ class GitConfig:
elif key.startswith("branch."): elif key.startswith("branch."):
self._branches = {} self._branches = {}
has_includes = self._cache_dict is not None and (
"include.path" in self._cache_dict
or any(k.startswith("includeif.") for k in self._cache_dict)
)
if ( if (
key == "include.path" key == "include.path"
or key.startswith("includeif.") or key.startswith("includeif.")
or has_includes or self._HasIncludes()
): ):
self.ClearCache() self.ClearCache()
self._RemoveJson() self._RemoveJson()
self._batch_json_dirty = False
elif self._batch_depth > 0:
self._batch_json_dirty = True
else: else:
self._SaveJson(self._cache) self._SaveJson(self._cache)
@@ -476,7 +744,10 @@ class GitConfig:
d = self._ReadJson() d = self._ReadJson()
if d is None: if d is None:
d = self._ReadGit() d = self._ReadGit()
self._SaveJson(d) if self._batch_depth > 0:
self._batch_json_dirty = True
else:
self._SaveJson(d)
return d return d
def _ReadFileFingerprint(self) -> Optional[Tuple[int, int, int, int, str]]: def _ReadFileFingerprint(self) -> Optional[Tuple[int, int, int, int, str]]:
@@ -916,17 +1187,18 @@ class Remote:
def Save(self): def Save(self):
"""Save this remote to the configuration.""" """Save this remote to the configuration."""
self._Set("url", self.url) with self._config.BatchWrites():
# projectname is initialized for projects listed in the manifest, but self._Set("url", self.url)
# not for others (e.g. the manifest project). This class is used for # projectname is initialized for projects listed in the manifest,
# all of them. # but not for others (e.g. the manifest project). This class is
if self.pushUrl is not None and self.projectname is not None: # used for all of them.
self._Set("pushurl", self.pushUrl + "/" + self.projectname) if self.pushUrl is not None and self.projectname is not None:
else: self._Set("pushurl", self.pushUrl + "/" + self.projectname)
self._Set("pushurl", self.pushUrl) else:
self._Set("review", self.review) self._Set("pushurl", self.pushUrl)
self._Set("projectname", self.projectname) self._Set("review", self.review)
self._Set("fetch", list(map(str, self.fetch))) self._Set("projectname", self.projectname)
self._Set("fetch", list(map(str, self.fetch)))
def _Set(self, key, value): def _Set(self, key, value):
key = f"remote.{self.name}.{key}" key = f"remote.{self.name}.{key}"
@@ -960,11 +1232,12 @@ class Branch:
def Save(self): def Save(self):
"""Save this branch back into the configuration.""" """Save this branch back into the configuration."""
if self.remote: with self._config.BatchWrites():
self._Set("remote", self.remote.name) if self.remote:
else: self._Set("remote", self.remote.name)
self._Set("remote", None) else:
self._Set("merge", self.merge) self._Set("remote", None)
self._Set("merge", self.merge)
def _Set(self, key, value): def _Set(self, key, value):
key = f"branch.{self.name}.{key}" key = f"branch.{self.name}.{key}"
@@ -1001,23 +1274,24 @@ class SyncAnalysisState:
""" """
self._config = config self._config = config
now = datetime.datetime.now(datetime.timezone.utc) now = datetime.datetime.now(datetime.timezone.utc)
self._Set("main.synctime", now.isoformat(timespec="microseconds")) with self._config.BatchWrites():
self._Set("main.version", "1") self._Set("main.synctime", now.isoformat(timespec="microseconds"))
self._Set("sys.argv", sys.argv) self._Set("main.version", "1")
for key, value in superproject_logging_data.items(): self._Set("sys.argv", sys.argv)
self._Set(f"superproject.{key}", value) for key, value in superproject_logging_data.items():
for key, value in options.__dict__.items(): self._Set(f"superproject.{key}", value)
self._Set(f"options.{key}", value) for key, value in options.__dict__.items():
config_items = config.DumpConfigDict().items() self._Set(f"options.{key}", value)
EXTRACT_NAMESPACES = {"repo", "branch", "remote"} config_items = config.DumpConfigDict().items()
self._SetDictionary( EXTRACT_NAMESPACES = {"repo", "branch", "remote"}
{ self._SetDictionary(
k: v {
for k, v in config_items k: v
if not k.startswith(SYNC_STATE_PREFIX) for k, v in config_items
and k.split(".", 1)[0] in EXTRACT_NAMESPACES if not k.startswith(SYNC_STATE_PREFIX)
} and k.split(".", 1)[0] in EXTRACT_NAMESPACES
) }
)
def _SetDictionary(self, data): def _SetDictionary(self, data):
"""Save all key/value pairs of |data| dictionary. """Save all key/value pairs of |data| dictionary.
+424 -39
View File
@@ -13,6 +13,9 @@
# limitations under the License. # limitations under the License.
import os import os
import re
import stat
from typing import Dict, Optional, Set, Tuple
import git_command import git_command
import platform_utils import platform_utils
@@ -29,6 +32,26 @@ R_WORKTREE_M = R_WORKTREE + "m/"
R_M = "refs/remotes/m/" R_M = "refs/remotes/m/"
# Maximum symbolic-ref indirection depth (matches Git SYMREF_MAXDEPTH). # Maximum symbolic-ref indirection depth (matches Git SYMREF_MAXDEPTH).
_SYMREF_MAXDEPTH = 5 _SYMREF_MAXDEPTH = 5
_HEX_OID_RE = re.compile(r"^(?:[0-9a-f]{40}|[0-9a-f]{64})$")
_BAD_REF_CHARS = frozenset(" ~^:?*[\\")
_StatKey = Tuple[int, int, int, int]
def IsValidRefName(name: str) -> bool:
"""Validate a Git refname per git-check-ref-format rules."""
if not name or name.endswith(".") or ".." in name or "@{" in name:
return False
if any(
ch in _BAD_REF_CHARS or ord(ch) < 0x20 or ord(ch) == 0x7F for ch in name
):
return False
parts = name.split("/")
if len(parts) < 2:
return False
for part in parts:
if not part or part.startswith(".") or part.endswith(".lock"):
return False
return True
class GitRefs: class GitRefs:
@@ -36,7 +59,8 @@ class GitRefs:
self._gitdir = gitdir self._gitdir = gitdir
self._phyref = None self._phyref = None
self._symref = None self._symref = None
self._mtime = {} self._mtime: Dict[str, Optional[_StatKey]] = {}
self._files_authoritative = False
@property @property
def all(self): def all(self):
@@ -54,6 +78,18 @@ class GitRefs:
"""Whether a ref snapshot has already been loaded.""" """Whether a ref snapshot has already been loaded."""
return self._phyref is not None return self._phyref is not None
@property
def is_files_authoritative(self) -> bool:
"""Whether the loaded snapshot came from a complete files scan."""
return self._phyref is not None and self._files_authoritative
def Invalidate(self) -> None:
"""Discard any cached ref snapshot so the next access reloads."""
self._phyref = None
self._symref = None
self._mtime = {}
self._files_authoritative = False
def get(self, name): def get(self, name):
try: try:
return self.all[name] return self.all[name]
@@ -70,6 +106,9 @@ class GitRefs:
if name in self._mtime: if name in self._mtime:
del self._mtime[name] del self._mtime[name]
norm_os = os.path.join(*name.strip("/").split("/"))
if norm_os in self._mtime:
del self._mtime[norm_os]
def symref(self, name): def symref(self, name):
try: try:
@@ -78,6 +117,80 @@ class GitRefs:
except KeyError: except KeyError:
return "" return ""
def TryResolveRef(self, name: str) -> Optional[str]:
"""Resolve a single ref directly on the files backend without git.
Args:
name: Reference name ("HEAD" or "refs/...").
Returns:
The resolved hex object ID if present, "" if the repository uses a
clean files backend and the ref is authoritatively absent or
unborn, or None if git must be invoked as a fallback.
"""
if name != HEAD and (
not name.startswith("refs/") or not IsValidRefName(name)
):
return None
if self._phyref is not None and not self._NeedUpdate():
if name in self._phyref:
return self._phyref[name]
if self._files_authoritative:
return ""
return None
if self._CheckFilesBackendConfig() is None:
return None
refs_dir = os.path.join(self._gitdir, "refs")
if platform_utils.islink(refs_dir) or not os.path.isdir(refs_dir):
return None
head_file = os.path.join(self._gitdir, HEAD)
if platform_utils.islink(head_file) or not os.path.isfile(head_file):
return None
cur = name
for _ in range(_SYMREF_MAXDEPTH + 1):
parts = cur.split("/")
for idx in range(1, len(parts) + 1):
if platform_utils.islink(
os.path.join(self._gitdir, *parts[:idx])
):
return None
ref_path = os.path.join(self._gitdir, *parts)
if os.path.lexists(ref_path + ".lock"):
return None
if os.path.lexists(ref_path):
if os.path.isdir(ref_path):
return None
parsed = self._ParseLooseRefFile(ref_path)
if parsed is None:
return None
is_sym, val, _ = parsed
if not is_sym:
return val
cur = val
continue
if cur == HEAD:
return None
packed_path = os.path.join(self._gitdir, "packed-refs")
if os.path.lexists(packed_path):
packed_res = self._ReadPackedRefs(packed_path)
if packed_res is None:
return None
packed_map, _ = packed_res
else:
packed_map = {}
if os.path.lexists(ref_path):
return None
return packed_map.get(cur, "")
return ""
def UpdateSymref(self, name: str, target: str) -> None: def UpdateSymref(self, name: str, target: str) -> None:
"""Update a symbolic reference in an already-loaded ref snapshot. """Update a symbolic reference in an already-loaded ref snapshot.
@@ -88,7 +201,7 @@ class GitRefs:
if self._phyref is None: if self._phyref is None:
return return
if self._HasExternalRefChanges(name): if self._HasExternalRefChanges(name):
self._phyref = None self.Invalidate()
return return
self._symref[name] = target self._symref[name] = target
@@ -100,38 +213,70 @@ class GitRefs:
self._TrackMtime(anc) self._TrackMtime(anc)
self._TrackMtime(os.path.join(*parts)) self._TrackMtime(os.path.join(*parts))
def _HasExternalRefChanges(self, updated_symref: str) -> bool: def UpdateRef(self, name: str) -> None:
"""Check whether refs outside |updated_symref| changed on disk. """Refresh a direct reference in an already-loaded ref snapshot.
Args: Args:
updated_symref: Symbolic ref path just updated on disk. name: Full reference name just updated on disk (e.g.
"refs/remotes/m/main").
"""
if self._phyref is None:
return
if self._HasExternalRefChanges(name):
self.Invalidate()
return
parts = name.strip("/").split("/")
ref_rel = os.path.join(*parts)
parsed = self._ParseLooseRefFile(os.path.join(self._gitdir, ref_rel))
if parsed is None or parsed[0]:
self.Invalidate()
return
_, oid, ref_stat = parsed
self._symref.pop(name, None)
self._phyref[name] = oid
self._ResolveSymrefs()
for idx in range(1, len(parts)):
self._TrackMtime(os.path.join(*parts[:idx]))
self._mtime[ref_rel] = ref_stat
def _HasExternalRefChanges(self, updated_ref: str) -> bool:
"""Check whether refs outside |updated_ref| changed on disk.
Args:
updated_ref: Reference path just updated on disk.
Returns: Returns:
True if reftable is in use or any unrelated ref path changed. True if files backend is unsafe or any unrelated ref path changed.
""" """
if os.path.exists(os.path.join(self._gitdir, "reftable")): if self._CheckFilesBackendConfig() is None:
return True return True
if "packed-refs" not in self._mtime and os.path.exists( if "packed-refs" not in self._mtime and os.path.lexists(
os.path.join(self._gitdir, "packed-refs") os.path.join(self._gitdir, "packed-refs")
): ):
return True return True
parts = updated_symref.strip("/").split("/") parts = updated_ref.strip("/").split("/")
expected_child = { expected_child = {
"/".join(parts[:idx]): parts[idx] for idx in range(1, len(parts)) "/".join(parts[:idx]): parts[idx] for idx in range(1, len(parts))
} }
for tracked, mtime in self._mtime.items(): for tracked, prev_stat in self._mtime.items():
norm = tracked.replace(os.sep, "/") norm = tracked.replace(os.sep, "/")
if norm == updated_symref: if norm == updated_ref:
continue continue
full_path = os.path.join(self._gitdir, tracked) full_path = os.path.join(self._gitdir, tracked)
try: curr_stat = self._StatPath(full_path)
curr_mtime = os.path.getmtime(full_path) if prev_stat is None:
except OSError: if curr_stat is not None:
return True
continue
if curr_stat is None:
return True return True
if norm in expected_child: if norm in expected_child:
if curr_mtime != mtime: if curr_stat != prev_stat:
try: try:
children = platform_utils.listdir(full_path) children = platform_utils.listdir(full_path)
except OSError: except OSError:
@@ -140,14 +285,15 @@ class GitRefs:
for child in children: for child in children:
if child == next_comp: if child == next_comp:
continue continue
if os.path.join(tracked, child) not in self._mtime: child_key = os.path.join(tracked, child)
if self._mtime.get(child_key) is None:
return True return True
elif curr_mtime != mtime: elif curr_stat != prev_stat:
return True return True
for anc, next_comp in expected_child.items(): for anc, next_comp in expected_child.items():
anc_os = anc.replace("/", os.sep) anc_os = anc.replace("/", os.sep)
if anc_os not in self._mtime: if self._mtime.get(anc_os) is None:
anc_full = os.path.join(self._gitdir, anc_os) anc_full = os.path.join(self._gitdir, anc_os)
try: try:
if platform_utils.listdir(anc_full) != [next_comp]: if platform_utils.listdir(anc_full) != [next_comp]:
@@ -177,20 +323,27 @@ class GitRefs:
if self._phyref is None or self._NeedUpdate(): if self._phyref is None or self._NeedUpdate():
self._LoadAll() self._LoadAll()
def _NeedUpdate(self): def _NeedUpdate(self) -> bool:
with repo_trace.Trace(": scan refs %s", self._gitdir): with repo_trace.Trace(": scan refs %s", self._gitdir):
for name, mtime in self._mtime.items(): if self._files_authoritative and (
try: "GIT_DIR" in os.environ or "GIT_COMMON_DIR" in os.environ
if mtime != os.path.getmtime( ):
os.path.join(self._gitdir, name) return True
): for name, prev_stat in self._mtime.items():
return True if (
except OSError: self._StatPath(os.path.join(self._gitdir, name))
!= prev_stat
):
return True return True
return False return False
def _LoadAll(self): def _LoadAll(self):
with repo_trace.Trace(": load refs %s", self._gitdir): with repo_trace.Trace(": load refs %s", self._gitdir):
if self._TryReadFilesBackend():
self._ResolveSymrefs()
return
self._files_authoritative = False
self._phyref = {} self._phyref = {}
self._symref = {} self._symref = {}
self._mtime = {} self._mtime = {}
@@ -205,19 +358,238 @@ class GitRefs:
self._ResolveSymrefs() self._ResolveSymrefs()
self._RefreshMtimes() self._RefreshMtimes()
def _CheckFilesBackendConfig(self) -> Optional[_StatKey]:
"""Verify standard files ref backend preconditions and stat config."""
if "GIT_DIR" in os.environ or "GIT_COMMON_DIR" in os.environ:
return None
if not os.path.isdir(self._gitdir):
return None
for special in ("reftable", "commondir"):
if os.path.lexists(os.path.join(self._gitdir, special)):
return None
cfg_path = os.path.join(self._gitdir, "config")
if platform_utils.islink(cfg_path):
return None
try:
with open(cfg_path, "rb") as cfg_fd:
if b"refstorage" in cfg_fd.read().lower():
return None
return self._FormatStat(os.fstat(cfg_fd.fileno()))
except OSError:
return None
@classmethod
def _ParseLooseRefFile(
cls, path: str
) -> Optional[Tuple[bool, str, _StatKey]]:
"""Parse a loose ref or HEAD file into (is_symref, val, stat)."""
if platform_utils.islink(path):
return None
try:
with open(path, "r", encoding="utf-8") as fd:
raw = fd.read(1024)
st = cls._FormatStat(os.fstat(fd.fileno()))
except (OSError, UnicodeDecodeError):
return None
line = raw.rstrip("\r\n")
if not line or "\n" in line or "\r" in line:
return None
if line.startswith("ref:"):
target = line[4:].strip()
if not target or target == R_HEADS + ".invalid":
return None
if target != HEAD and (
not target.startswith("refs/") or not IsValidRefName(target)
):
return None
return (True, target, st)
oid = line.strip()
if not _HEX_OID_RE.match(oid) or cls._IsNullRef(oid):
return None
return (False, oid, st)
@classmethod
def _ReadPackedRefs(
cls, packed_path: str
) -> Optional[Tuple[Dict[str, str], _StatKey]]:
"""Parse a packed-refs file into (ref_map, stat)."""
if platform_utils.islink(packed_path):
return None
packed: Dict[str, str] = {}
try:
with open(packed_path, "r", encoding="utf-8") as packed_fd:
for raw_line in packed_fd:
if raw_line.startswith("#"):
continue
line = raw_line.rstrip("\r\n")
if line.startswith("^"):
if not _HEX_OID_RE.match(line[1:]):
return None
continue
parts = line.split(" ", 1)
if len(parts) != 2:
return None
oid, ref_name = parts
if (
not _HEX_OID_RE.match(oid)
or cls._IsNullRef(oid)
or not ref_name.startswith("refs/")
or not IsValidRefName(ref_name)
):
return None
packed[ref_name] = oid
st = cls._FormatStat(os.fstat(packed_fd.fileno()))
except (OSError, UnicodeDecodeError):
return None
return (packed, st)
def _TryReadFilesBackend(self) -> bool:
"""Load HEAD, packed-refs, and loose refs directly when safe.
Returns:
True if the repository uses a standard files ref backend and all
refs and stat fingerprints were read cleanly without invoking git.
"""
cfg_stat = self._CheckFilesBackendConfig()
if cfg_stat is None:
return False
mtime: Dict[str, Optional[_StatKey]] = {
"config": cfg_stat,
"reftable": None,
}
phyref: Dict[str, str] = {}
symref: Dict[str, str] = {}
parsed_head = self._ParseLooseRefFile(os.path.join(self._gitdir, HEAD))
if parsed_head is None:
return False
head_is_sym, head_val, head_stat = parsed_head
mtime[HEAD] = head_stat
if head_is_sym:
symref[HEAD] = head_val
else:
phyref[HEAD] = head_val
packed_path = os.path.join(self._gitdir, "packed-refs")
packed_stat: Optional[_StatKey] = None
if os.path.lexists(packed_path):
packed_res = self._ReadPackedRefs(packed_path)
if packed_res is None:
return False
packed_map, packed_stat = packed_res
phyref.update(packed_map)
mtime["packed-refs"] = packed_stat
refs_dir = os.path.join(self._gitdir, "refs")
if platform_utils.islink(refs_dir):
return False
refs_stat = self._StatPath(refs_dir)
if refs_stat is None or not stat.S_ISDIR(refs_stat[0]):
return False
mtime["refs"] = refs_stat
try:
to_scan = [("refs", "refs", refs_dir)]
while to_scan:
ref_prefix, rel_os_prefix, abs_dir = to_scan.pop()
with os.scandir(abs_dir) as entries:
for entry in entries:
if entry.is_symlink():
return False
ref_name = f"{ref_prefix}/{entry.name}"
if not IsValidRefName(ref_name):
return False
rel_os = os.path.join(rel_os_prefix, entry.name)
if entry.is_dir(follow_symlinks=False):
if ref_name in phyref:
return False
# DirEntry.stat() leaves st_ino == 0 on Windows,
# whereas _StatPath() uses os.lstat() and populates
# st_ino. Stat via _StatPath() so _NeedUpdate() and
# _HasExternalRefChanges() compare matching keys.
d_stat = self._StatPath(entry.path)
if d_stat is None or not stat.S_ISDIR(d_stat[0]):
return False
mtime[rel_os] = d_stat
to_scan.append((ref_name, rel_os, entry.path))
elif entry.is_file(follow_symlinks=False):
parsed = self._ParseLooseRefFile(entry.path)
if parsed is None:
return False
is_sym, val, f_stat = parsed
mtime[rel_os] = f_stat
if is_sym:
phyref.pop(ref_name, None)
symref[ref_name] = val
else:
symref.pop(ref_name, None)
phyref[ref_name] = val
else:
return False
except OSError:
return False
# Guard against a concurrent `git pack-refs --prune` writing
# packed-refs and unlinking loose refs mid-scan.
if self._StatPath(packed_path) != packed_stat:
return False
self._phyref = phyref
self._symref = symref
self._mtime = mtime
self._files_authoritative = True
return True
def _RefreshMtimes(self) -> None: def _RefreshMtimes(self) -> None:
"""Record current modification times for ref storage paths.""" """Record current modification times for ref storage paths."""
self._mtime = {} self._mtime = {}
self._TrackMtime(HEAD) self._TrackMtime(HEAD, track_missing=True)
self._TrackMtime("config") self._TrackMtime("config", track_missing=True)
self._TrackMtime("packed-refs") self._TrackMtime("packed-refs", track_missing=True)
self._TrackMtime("refs", track_missing=True)
self._TrackMtime("reftable", track_missing=True)
self._TrackTreeMtimes("refs") self._TrackTreeMtimes("refs")
self._TrackTreeMtimes("reftable") self._TrackTreeMtimes("reftable")
@staticmethod
def _FormatStat(st: os.stat_result) -> _StatKey:
# ctime is left out: tools that only touch metadata, such as backup
# agents and file system crawlers, move it without changing a ref.
return (st.st_mode, st.st_mtime_ns, st.st_size, st.st_ino)
@classmethod
def _StatPath(cls, path: str) -> Optional[_StatKey]:
try:
st = os.lstat(path)
except OSError:
return None
mount_tag = getattr(stat, "IO_REPARSE_TAG_MOUNT_POINT", -1)
is_reparse_dir = getattr(st, "st_reparse_tag", 0) == mount_tag
if stat.S_ISLNK(st.st_mode) or is_reparse_dir:
# Track the symlink target so mutations to symlinked refs or
# packed-refs invalidate the fallback snapshot, while keeping
# S_IFLNK in mode and negative size when dangling.
link_mode = stat.S_IFLNK | stat.S_IMODE(st.st_mode)
try:
target_st = os.stat(path)
except OSError:
return (link_mode, st.st_mtime_ns, -1 - st.st_size, st.st_ino)
return (
link_mode,
target_st.st_mtime_ns,
target_st.st_size,
target_st.st_ino,
)
return cls._FormatStat(st)
@staticmethod @staticmethod
def _IsNullRef(ref_id: str) -> bool: def _IsNullRef(ref_id: str) -> bool:
"""Check if a ref_id is a null object ID.""" """Check if a ref_id is a null object ID."""
return ref_id and all(ch == "0" for ch in ref_id) return bool(ref_id) and all(ch == "0" for ch in ref_id)
def _ReadRefs(self) -> bool: def _ReadRefs(self) -> bool:
"""Read all references using git for-each-ref. """Read all references using git for-each-ref.
@@ -299,32 +671,45 @@ class GitRefs:
if ref_id: if ref_id:
self._phyref[name] = ref_id self._phyref[name] = ref_id
def _TrackMtime(self, name: str) -> None: def _TrackMtime(self, name: str, track_missing: bool = False) -> None:
"""Track the modification time of a specific gitdir path.""" """Track the stat fingerprint of a specific gitdir path."""
path = os.path.join(self._gitdir, name) st = self._StatPath(os.path.join(self._gitdir, name))
try: if st is not None or track_missing:
self._mtime[name] = os.path.getmtime(path) self._mtime[name] = st
except OSError:
return
def _TrackTreeMtimes(self, root: str) -> None: def _TrackTreeMtimes(self, root: str) -> None:
"""Recursively track modification times for a directory tree.""" """Recursively track modification times for a directory tree."""
root_path = os.path.join(self._gitdir, root) root_path = os.path.join(self._gitdir, root)
try: try:
if not platform_utils.isdir(root_path): if not platform_utils.isdir(root_path):
self._TrackMtime(root)
return return
except OSError: except OSError:
return return
visited: Set[Tuple[int, int]] = set()
to_scan = [root] to_scan = [root]
while to_scan: while to_scan:
name = to_scan.pop() name = to_scan.pop()
self._TrackMtime(name) self._TrackMtime(name)
path = os.path.join(self._gitdir, name) path = os.path.join(self._gitdir, name)
if not platform_utils.isdir(path): try:
dir_st = os.stat(path)
except OSError:
continue continue
if not stat.S_ISDIR(dir_st.st_mode):
continue
if dir_st.st_ino:
dir_key = (dir_st.st_dev, dir_st.st_ino)
if dir_key in visited:
continue
visited.add(dir_key)
for child in platform_utils.listdir(path): try:
children = platform_utils.listdir(path)
except OSError:
continue
for child in children:
child_name = os.path.join(name, child) child_name = os.path.join(name, child)
child_path = os.path.join(self._gitdir, child_name) child_path = os.path.join(self._gitdir, child_name)
if platform_utils.isdir(child_path): if platform_utils.isdir(child_path):
+3 -3
View File
@@ -29,7 +29,7 @@ import os
import sys import sys
import tempfile import tempfile
import time import time
from typing import NamedTuple from typing import NamedTuple, Optional
import urllib.parse import urllib.parse
import git_command import git_command
@@ -54,7 +54,7 @@ class CommitIdsResult(NamedTuple):
"""Return the commit ids and whether caller should exit.""" """Return the commit ids and whether caller should exit."""
# A dictionary with the projects/commit ids on success, otherwise None. # A dictionary with the projects/commit ids on success, otherwise None.
commit_ids: dict commit_ids: Optional[dict]
# Whether the caller should exit. # Whether the caller should exit.
fatal: bool fatal: bool
@@ -63,7 +63,7 @@ class UpdateProjectsResult(NamedTuple):
"""Return the overriding manifest file and whether caller should exit.""" """Return the overriding manifest file and whether caller should exit."""
# Path name of the overriding manifest file if successful, otherwise None. # Path name of the overriding manifest file if successful, otherwise None.
manifest_path: str manifest_path: Optional[str]
# Whether the caller should exit. # Whether the caller should exit.
fatal: bool fatal: bool
+2
View File
@@ -98,6 +98,8 @@ class BaseEventLog:
try: try:
env[KEY] = self._full_sid env[KEY] = self._full_sid
except UnicodeEncodeError: except UnicodeEncodeError:
# This is a specific fallback for Windows for string-vs-bytes.
# pyrefly: ignore
env[KEY] = self._full_sid.encode() env[KEY] = self._full_sid.encode()
if repo_source_version is not None: if repo_source_version is not None:
+3
View File
@@ -323,6 +323,7 @@ class RepoHook:
""" """
# Exec, storing global context in the context dict. We catch exceptions # Exec, storing global context in the context dict. We catch exceptions
# and convert to a HookError w/ just the failing traceback. # and convert to a HookError w/ just the failing traceback.
assert self._script_fullpath is not None
try: try:
exec(compile(data, self._script_fullpath, "exec"), context) exec(compile(data, self._script_fullpath, "exec"), context)
except Exception: except Exception:
@@ -367,6 +368,7 @@ class RepoHook:
# Always run hooks with CWD as topdir. # Always run hooks with CWD as topdir.
os.chdir(self._repo_topdir) os.chdir(self._repo_topdir)
assert self._script_fullpath is not None
# Put the hook dir as the first item of sys.path so hooks can do # Put the hook dir as the first item of sys.path so hooks can do
# relative imports. We want to replace the repo dir as [0] so # relative imports. We want to replace the repo dir as [0] so
# hooks can't import repo files. # hooks can't import repo files.
@@ -407,6 +409,7 @@ class RepoHook:
def _CheckHook(self): def _CheckHook(self):
# Bail with a nice error if we can't find the hook. # Bail with a nice error if we can't find the hook.
assert self._script_fullpath is not None
if not os.path.isfile(self._script_fullpath): if not os.path.isfile(self._script_fullpath):
raise error.HookError( raise error.HookError(
"Couldn't find repo hook: %s" % self._script_fullpath "Couldn't find repo hook: %s" % self._script_fullpath
Symlink
+1
View File
@@ -0,0 +1 @@
run_tests
+12 -6
View File
@@ -37,13 +37,14 @@ import repo_logging
try: try:
# Optional third-party module.
# pyrefly: ignore
import kerberos import kerberos
except ImportError: except ImportError:
kerberos = None kerberos = None
import color import color
from command import InteractiveCommand import command
from command import MirrorSafeCommand
import editor import editor
import error import error
import event_log import event_log
@@ -53,7 +54,7 @@ import git_trace2_event_log
import manifest_xml import manifest_xml
import pager import pager
import repo_trace import repo_trace
from subcmds import all_commands import subcmds
from subcmds.version import Version from subcmds.version import Version
import wrapper import wrapper
@@ -181,7 +182,7 @@ global_options.add_option(
class _Repo: class _Repo:
def __init__(self, repodir): def __init__(self, repodir):
self.repodir = repodir self.repodir = repodir
self.commands = all_commands self.commands = subcmds.all_commands
def _PrintHelp(self, short: bool = False, all_commands: bool = False): def _PrintHelp(self, short: bool = False, all_commands: bool = False):
"""Show --help screen.""" """Show --help screen."""
@@ -415,7 +416,10 @@ class _Repo:
editor.Editor.globalConfig = cmd.client.globalConfig editor.Editor.globalConfig = cmd.client.globalConfig
if not isinstance(cmd, MirrorSafeCommand) and cmd.manifest.IsMirror: if (
not isinstance(cmd, command.MirrorSafeCommand)
and cmd.manifest.IsMirror
):
logger.error("fatal: '%s' requires a working directory", name) logger.error("fatal: '%s' requires a working directory", name)
return 1 return 1
@@ -432,7 +436,9 @@ class _Repo:
cmd.CommonValidateOptions(copts, cargs) cmd.CommonValidateOptions(copts, cargs)
git_trace2_event_log.verbose = copts.verbose git_trace2_event_log.verbose = copts.verbose
if gopts.pager is not False and not isinstance(cmd, InteractiveCommand): if gopts.pager is not False and not isinstance(
cmd, command.InteractiveCommand
):
config = cmd.client.globalConfig config = cmd.client.globalConfig
if gopts.pager: if gopts.pager:
use_pager = True use_pager = True
+4 -1
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man. .\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "June 2026" "repo init" "Repo Manual" .TH REPO "1" "November 2026" "repo init" "Repo Manual"
.SH NAME .SH NAME
repo \- repo init - manual page for repo init repo \- repo init - manual page for repo init
.SH SYNOPSIS .SH SYNOPSIS
@@ -109,6 +109,9 @@ default to smart sync when syncing projects
\fB\-\-no\-smart\-sync\fR, \fB\-\-head\fR \fB\-\-no\-smart\-sync\fR, \fB\-\-head\fR
do not default to smart sync when syncing projects do not default to smart sync when syncing projects
.TP .TP
\fB\-\-smart\-sync\-branch\fR=\fI\,BRANCH\/\fR
server branch to query when smart syncing the manifest branch (empty string to clear)
.TP
\fB\-\-clone\-bundle\fR \fB\-\-clone\-bundle\fR
enable use of \fI\,/clone.bundle\/\fP on HTTP/HTTPS (default if not \fB\-\-partial\-clone\fR) enable use of \fI\,/clone.bundle\/\fP on HTTP/HTTPS (default if not \fB\-\-partial\-clone\fR)
.TP .TP
+4 -1
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man. .\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "September 2026" "repo smartsync" "Repo Manual" .TH REPO "1" "November 2026" "repo smartsync" "Repo Manual"
.SH NAME .SH NAME
repo \- repo smartsync - manual page for repo smartsync repo \- repo smartsync - manual page for repo smartsync
.SH SYNOPSIS .SH SYNOPSIS
@@ -23,6 +23,9 @@ number of network jobs to run in parallel (defaults to \fB\-\-jobs\fR or 1). Ign
\fB\-\-jobs\-checkout\fR=\fI\,JOBS\/\fR \fB\-\-jobs\-checkout\fR=\fI\,JOBS\/\fR
number of local checkout jobs to run in parallel (defaults to \fB\-\-jobs\fR or based on number of CPU cores). Ignored unless \fB\-\-no\-interleaved\fR is set number of local checkout jobs to run in parallel (defaults to \fB\-\-jobs\fR or based on number of CPU cores). Ignored unless \fB\-\-no\-interleaved\fR is set
.TP .TP
\fB\-\-force\-jobs\fR
force jobs to be the value of \fB\-j\fR even if the manifest caps it
.TP
\fB\-f\fR, \fB\-\-force\-broken\fR \fB\-f\fR, \fB\-\-force\-broken\fR
obsolete option (to be deleted in the future) obsolete option (to be deleted in the future)
.TP .TP
+4 -1
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man. .\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "October 2026" "repo sync" "Repo Manual" .TH REPO "1" "November 2026" "repo sync" "Repo Manual"
.SH NAME .SH NAME
repo \- repo sync - manual page for repo sync repo \- repo sync - manual page for repo sync
.SH SYNOPSIS .SH SYNOPSIS
@@ -23,6 +23,9 @@ number of network jobs to run in parallel (defaults to \fB\-\-jobs\fR or 1). Ign
\fB\-\-jobs\-checkout\fR=\fI\,JOBS\/\fR \fB\-\-jobs\-checkout\fR=\fI\,JOBS\/\fR
number of local checkout jobs to run in parallel (defaults to \fB\-\-jobs\fR or based on number of CPU cores). Ignored unless \fB\-\-no\-interleaved\fR is set number of local checkout jobs to run in parallel (defaults to \fB\-\-jobs\fR or based on number of CPU cores). Ignored unless \fB\-\-no\-interleaved\fR is set
.TP .TP
\fB\-\-force\-jobs\fR
force jobs to be the value of \fB\-j\fR even if the manifest caps it
.TP
\fB\-f\fR, \fB\-\-force\-broken\fR \fB\-f\fR, \fB\-\-force\-broken\fR
obsolete option (to be deleted in the future) obsolete option (to be deleted in the future)
.TP .TP
+1 -5
View File
@@ -1211,11 +1211,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
self.manifestFile = override self.manifestFile = override
try: try:
m = self.manifestProject self.branch = self.manifestProject.GetMergeBranch()
b = m.GetBranch(m.CurrentBranch).merge
if b is not None and b.startswith(git_refs.R_HEADS):
b = b[len(git_refs.R_HEADS) :]
self.branch = b
parent_groups = self.parent_groups.copy() parent_groups = self.parent_groups.copy()
if self.path_prefix: if self.path_prefix:
+2 -1
View File
@@ -17,6 +17,7 @@ import os
import platform import platform
import shutil import shutil
import stat import stat
from typing import Union
def isWindows(): def isWindows():
@@ -224,7 +225,7 @@ def rmdir(path):
os.rmdir(_makelongpath(path)) os.rmdir(_makelongpath(path))
def removedirs(path: str) -> None: def removedirs(path: Union[str, os.PathLike]) -> None:
"""Remove a directory tree of symlinks and empty directories. """Remove a directory tree of symlinks and empty directories.
Walks |path| bottom-up without following symlinks. Removes symlinks Walks |path| bottom-up without following symlinks. Removes symlinks
+12 -2
View File
@@ -17,6 +17,8 @@ from ctypes import wintypes
import errno import errno
# Windows-only ctypes APIs are not in typeshed when checking on non-Windows.
# pyrefly: ignore
kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
UCHAR = ctypes.c_ubyte UCHAR = ctypes.c_ubyte
@@ -164,7 +166,9 @@ def _create_symlink(source, link_name, dwFlags):
# "the unprivileged create flag is unsupported below Windows 10 (1703, # "the unprivileged create flag is unsupported below Windows 10 (1703,
# v10.0.14972). retry without it." # v10.0.14972). retry without it."
if not CreateSymbolicLinkW(link_name, source, dwFlags): if not CreateSymbolicLinkW(link_name, source, dwFlags):
# pyrefly: ignore
code = ctypes.get_last_error() code = ctypes.get_last_error()
# pyrefly: ignore
error_desc = ctypes.FormatError(code).strip() error_desc = ctypes.FormatError(code).strip()
if code == ERROR_PRIVILEGE_NOT_HELD: if code == ERROR_PRIVILEGE_NOT_HELD:
raise OSError(errno.EPERM, error_desc, link_name) raise OSError(errno.EPERM, error_desc, link_name)
@@ -190,7 +194,9 @@ def readlink(path):
) )
if reparse_point_handle == INVALID_HANDLE_VALUE: if reparse_point_handle == INVALID_HANDLE_VALUE:
_raise_winerror( _raise_winerror(
ctypes.get_last_error(), f'Error opening symbolic link "{path}"' # pyrefly: ignore
ctypes.get_last_error(),
f'Error opening symbolic link "{path}"',
) )
target_buffer = ctypes.c_buffer(MAXIMUM_REPARSE_DATA_BUFFER_SIZE) target_buffer = ctypes.c_buffer(MAXIMUM_REPARSE_DATA_BUFFER_SIZE)
n_bytes_returned = wintypes.DWORD() n_bytes_returned = wintypes.DWORD()
@@ -207,7 +213,9 @@ def readlink(path):
CloseHandle(reparse_point_handle) CloseHandle(reparse_point_handle)
if not io_result: if not io_result:
_raise_winerror( _raise_winerror(
ctypes.get_last_error(), f'Error reading symbolic link "{path}"' # pyrefly: ignore
ctypes.get_last_error(),
f'Error reading symbolic link "{path}"',
) )
rdb = REPARSE_DATA_BUFFER.from_buffer(target_buffer) rdb = REPARSE_DATA_BUFFER.from_buffer(target_buffer)
if rdb.ReparseTag == IO_REPARSE_TAG_SYMLINK: if rdb.ReparseTag == IO_REPARSE_TAG_SYMLINK:
@@ -221,6 +229,8 @@ def readlink(path):
def _raise_winerror(code, error_desc): def _raise_winerror(code, error_desc):
# pyrefly: ignore
win_error_desc = ctypes.FormatError(code).strip() win_error_desc = ctypes.FormatError(code).strip()
error_desc = f"{error_desc}: {win_error_desc}" error_desc = f"{error_desc}: {win_error_desc}"
# pyrefly: ignore
raise ctypes.WinError(code, error_desc) raise ctypes.WinError(code, error_desc)
+2
View File
@@ -20,6 +20,8 @@ import time
try: try:
import threading as _threading import threading as _threading
except ImportError: except ImportError:
# This module was removed in Python 3.9, but we still support Python 3.6.
# pyrefly: ignore
import dummy_threading as _threading import dummy_threading as _threading
import repo_trace import repo_trace
+257 -107
View File
@@ -29,7 +29,7 @@ import sys
import tarfile import tarfile
import tempfile import tempfile
import time import time
from typing import Any, Dict, List, NamedTuple, Optional, Tuple from typing import Any, Dict, List, NamedTuple, Optional, Tuple, Union
import urllib.parse import urllib.parse
import color import color
@@ -120,7 +120,9 @@ def _GitTemplateEnvKey() -> Tuple[Any, ...]:
return base + extra return base + extra
@functools.lru_cache(maxsize=None) _INIT_TEMPLATE_CACHE: Dict[Tuple[Any, ...], bool] = {}
def _HasCustomGitTemplateDir(env_key: Tuple[Any, ...]) -> bool: def _HasCustomGitTemplateDir(env_key: Tuple[Any, ...]) -> bool:
"""Return whether Git has a custom template directory configured. """Return whether Git has a custom template directory configured.
@@ -130,19 +132,26 @@ def _HasCustomGitTemplateDir(env_key: Tuple[Any, ...]) -> bool:
Returns: Returns:
True if a non-empty GIT_TEMPLATE_DIR or init.templatedir is configured. True if a non-empty GIT_TEMPLATE_DIR or init.templatedir is configured.
""" """
cached = _INIT_TEMPLATE_CACHE.get(env_key)
if cached is not None:
return cached
git_template_dir = env_key[0] git_template_dir = env_key[0]
if git_template_dir is not None: if git_template_dir is not None:
return bool(git_template_dir) result = bool(git_template_dir)
else:
p = git_command.GitCommand(
None,
["config", "--get", "init.templatedir"],
capture_stdout=True,
capture_stderr=True,
bare=True,
add_event_log=False,
)
result = p.Wait() == 0 and bool(p.stdout.strip())
p = git_command.GitCommand( _INIT_TEMPLATE_CACHE[env_key] = result
None, return result
["config", "--get", "init.templatedir"],
capture_stdout=True,
capture_stderr=True,
bare=True,
add_event_log=False,
)
return p.Wait() == 0 and bool(p.stdout.strip())
def _FirstLines(lines: List[str], limit: int = 10) -> str: def _FirstLines(lines: List[str], limit: int = 10) -> str:
@@ -1211,6 +1220,7 @@ class Project:
"""Returns a list of strings, untracked files in the git tree.""" """Returns a list of strings, untracked files in the git tree."""
return self.work_git.LsOthers() return self.work_git.LsOthers()
@property
def HasChanges(self) -> bool: def HasChanges(self) -> bool:
"""Returns true if there are uncommitted changes.""" """Returns true if there are uncommitted changes."""
return bool(self.UncommittedFiles(get_all=False)) return bool(self.UncommittedFiles(get_all=False))
@@ -1916,24 +1926,26 @@ class Project:
if is_new is None: if is_new is None:
is_new = not self.Exists is_new = not self.Exists
if is_new: with self.config.BatchWrites():
self._InitGitDir(force_sync=force_sync, quiet=quiet) if is_new:
else: self._InitGitDir(force_sync=force_sync, quiet=quiet)
try: else:
# At this point, it's possible that gitdir points to an old try:
# objdir (e.g. name changed, but objdir exists). Check # At this point, it's possible that gitdir points to an old
# references to ensure that's not the case. See # objdir (e.g. name changed, but objdir exists). Check
# https://issues.gerritcodereview.com/40013418 for more # references to ensure that's not the case. See
# details. # https://issues.gerritcodereview.com/40013418 for more
self._CheckDirReference(self.objdir, self.gitdir) # details.
self._CheckDirReference(self.objdir, self.gitdir)
self._UpdateHooks(quiet=quiet) self._UpdateHooks(quiet=quiet)
except error.GitError as e: except error.GitError as e:
if not force_sync: if not force_sync:
raise e raise e
# Let _InitGitDir fix the issue, force_sync is always True here. # Let _InitGitDir fix the issue, force_sync is always True
self._InitGitDir(force_sync=True, quiet=quiet) # here.
self._InitRemote() self._InitGitDir(force_sync=True, quiet=quiet)
self._InitRemote()
if self.UseAlternates: if self.UseAlternates:
# If gitdir/objects is a symlink, migrate it from the old layout. # If gitdir/objects is a symlink, migrate it from the old layout.
@@ -2157,8 +2169,26 @@ class Project:
is_tag = rev.startswith( is_tag = rev.startswith(
git_refs.R_TAGS git_refs.R_TAGS
) or self.revisionExpr.startswith(git_refs.R_TAGS) ) or self.revisionExpr.startswith(git_refs.R_TAGS)
if all_refs is None and not is_tag and self.bare_ref.is_loaded: if all_refs is None and not is_tag:
all_refs = self.bare_ref.all if self.bare_ref.is_loaded:
all_refs = self.bare_ref.all
if (
rev not in all_refs
and rev.startswith("refs/")
and git_refs.IsValidRefName(rev)
and self.bare_ref.is_files_authoritative
):
raise error.ManifestInvalidRevisionError(
f"revision {self.revisionExpr} in {self.name} not found"
)
elif rev.startswith("refs/"):
resolved = self.bare_ref.TryResolveRef(rev)
if resolved is not None:
if resolved:
return resolved
raise error.ManifestInvalidRevisionError(
f"revision {self.revisionExpr} in {self.name} not found"
)
if all_refs is not None and not is_tag and rev in all_refs: if all_refs is not None and not is_tag and rev in all_refs:
return all_refs[rev] return all_refs[rev]
@@ -3443,24 +3473,41 @@ class Project:
return False return False
finally: finally:
self._RefreshConfigIfModified(config_stat) self._RefreshConfigIfModified(config_stat)
self.bare_ref.Invalidate()
# Verify postconditions against the object IDs that the refs record. # Verify postconditions against the object IDs that the refs record.
# Peeling them would read the object store, and on one that is # Peeling them would read the object store, and on one that is
# populated lazily that downloads the objects fetchcmd just registered # populated lazily that downloads the objects fetchcmd just registered
# before anything needs them. The checkout reads them when it does. # before anything needs them. The checkout reads them when it does.
# With GIT_DIR or GIT_COMMON_DIR set, leave the lookups to git.
read_gitdir = (
"GIT_DIR" not in os.environ and "GIT_COMMON_DIR" not in os.environ
)
if not git_config.IsId(self.revisionExpr): if not git_config.IsId(self.revisionExpr):
try: try:
ref_name = self.GetRemote().ToLocal(self.revisionExpr) ref_name = self.GetRemote().ToLocal(self.revisionExpr)
except error.GitError as e: except error.GitError as e:
logger.error("error: Failed to resolve tracking ref: %s", e) logger.error("error: Failed to resolve tracking ref: %s", e)
return False return False
try: resolved_ref = self.bare_ref.get(ref_name) if read_gitdir else ""
resolved_ref = self.bare_git.ResolveRef(ref_name) if not resolved_ref:
except error.GitError: if (
logger.error( read_gitdir
"error: Postcondition failed: %s not found", ref_name and ref_name.startswith("refs/")
) and git_refs.IsValidRefName(ref_name)
return False and self.bare_ref.is_files_authoritative
):
logger.error(
"error: Postcondition failed: %s not found", ref_name
)
return False
try:
resolved_ref = self.bare_git.ResolveRef(ref_name)
except error.GitError:
logger.error(
"error: Postcondition failed: %s not found", ref_name
)
return False
if repo_trev and not self._IsSameCommit(resolved_ref, repo_trev): if repo_trev and not self._IsSameCommit(resolved_ref, repo_trev):
logger.error( logger.error(
"error: Postcondition failed: %s is %s, expected %s", "error: Postcondition failed: %s is %s, expected %s",
@@ -3472,11 +3519,15 @@ class Project:
if not repo_trev: if not repo_trev:
repo_trev = resolved_ref repo_trev = resolved_ref
try: fetch_head = self._ReadSimpleFetchHead() if read_gitdir else None
fetch_head = self.bare_git.ResolveRef("FETCH_HEAD") if not fetch_head:
except error.GitError: try:
logger.error("error: Postcondition failed: FETCH_HEAD not found") fetch_head = self.bare_git.ResolveRef("FETCH_HEAD")
return False except error.GitError:
logger.error(
"error: Postcondition failed: FETCH_HEAD not found"
)
return False
if not self._IsSameCommit(fetch_head, repo_trev): if not self._IsSameCommit(fetch_head, repo_trev):
logger.error( logger.error(
"error: Postcondition failed: FETCH_HEAD is %s, expected %s", "error: Postcondition failed: FETCH_HEAD is %s, expected %s",
@@ -3487,6 +3538,26 @@ class Project:
return True return True
def _ReadSimpleFetchHead(self) -> Optional[str]:
"""Read FETCH_HEAD's leading object ID directly when unambiguous."""
if os.path.lexists(os.path.join(self.gitdir, "commondir")):
return None
fh_path = os.path.join(self.gitdir, "FETCH_HEAD")
try:
if platform_utils.islink(fh_path):
return None
with open(fh_path, "r", encoding="utf-8") as fd:
raw_line = fd.readline(4096)
except (OSError, UnicodeDecodeError):
return None
if not raw_line:
return None
line = raw_line.rstrip("\r\n")
oid = line.split("\t", 1)[0]
if git_config.IsId(oid) and not all(ch == "0" for ch in oid):
return oid
return None
def _IsSameCommit(self, oid: str, expected: str) -> bool: def _IsSameCommit(self, oid: str, expected: str) -> bool:
"""Whether object ID |oid| names the same commit as |expected|. """Whether object ID |oid| names the same commit as |expected|.
@@ -4367,6 +4438,20 @@ class Project:
project=self.name, project=self.name,
) )
@staticmethod
def WarmInitTemplateCache() -> Tuple[Tuple[Any, ...], bool]:
"""Resolve and cache the init.templateDir lookup for sync workers."""
env_key = _GitTemplateEnvKey()
return env_key, _HasCustomGitTemplateDir(env_key)
@staticmethod
def SeedInitTemplateCache(
cache_entry: Tuple[Tuple[Any, ...], bool],
) -> None:
"""Seed the init.templateDir lookup cache in a worker process."""
env_key, has_custom = cache_entry
_INIT_TEMPLATE_CACHE[env_key] = has_custom
def _InitGitDir(self, mirror_git=None, force_sync=False, quiet=False): def _InitGitDir(self, mirror_git=None, force_sync=False, quiet=False):
# Prefix for temporary directories created during gitdir initialization. # Prefix for temporary directories created during gitdir initialization.
TMP_GITDIR_PREFIX = ".tmp-project-initgitdir-" TMP_GITDIR_PREFIX = ".tmp-project-initgitdir-"
@@ -4472,24 +4557,29 @@ class Project:
) )
m = self.manifest.manifestProject.config m = self.manifest.manifestProject.config
for key in ["user.name", "user.email"]: with curr_config.BatchWrites():
if m.Has(key, include_defaults=False): for key in ["user.name", "user.email"]:
curr_config.SetString(key, m.GetString(key)) if m.Has(key, include_defaults=False):
if not self.manifest.EnableGitLfs: curr_config.SetString(key, m.GetString(key))
curr_config.SetString( if not self.manifest.EnableGitLfs:
"filter.lfs.smudge", "git-lfs smudge --skip -- %f" curr_config.SetString(
) "filter.lfs.smudge",
curr_config.SetString( "git-lfs smudge --skip -- %f",
"filter.lfs.process", "git-lfs filter-process --skip" )
) curr_config.SetString(
curr_config.SetBoolean( "filter.lfs.process",
"core.bare", True if self.manifest.IsMirror else None "git-lfs filter-process --skip",
) )
if self.manifest.IsMirror:
curr_config.SetBoolean("core.bare", True)
elif curr_config.GetBoolean("core.bare"):
curr_config.SetBoolean("core.bare", None)
if tmp_gitdir: if tmp_gitdir:
platform_utils.rename(tmp_gitdir, self.gitdir) platform_utils.rename(tmp_gitdir, self.gitdir)
tmp_gitdir = None tmp_gitdir = None
if not init_obj_dir: self.config.ClearCache()
if not init_obj_dir and self.objdir != self.gitdir:
# The project might be shared (obj_dir already initialized), but # The project might be shared (obj_dir already initialized), but
# such information is not available here. Instead of passing it, # such information is not available here. Instead of passing it,
# set it as shared, and rely to be unset down the execution # set it as shared, and rely to be unset down the execution
@@ -4590,27 +4680,28 @@ class Project:
raise raise
def _InitRemote(self): def _InitRemote(self):
if self.remote.url: with self.config.BatchWrites():
remote = self.GetRemote() if self.remote.url:
remote.url = self.remote.url remote = self.GetRemote()
remote.pushUrl = self.remote.pushUrl remote.url = self.remote.url
remote.review = self.remote.review remote.pushUrl = self.remote.pushUrl
remote.projectname = self.name remote.review = self.remote.review
remote.projectname = self.name
if self.worktree: if self.worktree:
remote.ResetFetch(mirror=False) remote.ResetFetch(mirror=False)
else: else:
remote.ResetFetch(mirror=True) remote.ResetFetch(mirror=True)
remote.Save() remote.Save()
# Disable auto-gc for depth=1 to prevent hangs during lazy fetches # Disable auto-gc for depth=1 to prevent hangs during lazy fetches
# inside git checkout for partial clones. # inside git checkout for partial clones.
effective_depth = ( effective_depth = (
self.clone_depth or self.manifest.manifestProject.depth self.clone_depth or self.manifest.manifestProject.depth
) )
if effective_depth == 1: if effective_depth == 1:
self.config.SetBoolean("maintenance.auto", False) self.config.SetBoolean("maintenance.auto", False)
self.config.SetInt("gc.auto", 0) self.config.SetInt("gc.auto", 0)
def _InitMRef(self): def _InitMRef(self):
"""Initialize the pseudo m/<manifest branch> ref.""" """Initialize the pseudo m/<manifest branch> ref."""
@@ -4664,6 +4755,8 @@ class Project:
msg = "manifest set to %s" % self.revisionId msg = "manifest set to %s" % self.revisionId
dst = self.revisionId + "^0" dst = self.revisionId + "^0"
active_git.UpdateRef(ref, dst, message=msg, detach=True) active_git.UpdateRef(ref, dst, message=msg, detach=True)
if active_git is self.bare_git:
self.bare_ref.UpdateRef(ref)
else: else:
remote = self.GetRemote() remote = self.GetRemote()
dst = remote.ToLocal(self.revisionExpr) dst = remote.ToLocal(self.revisionExpr)
@@ -4671,6 +4764,8 @@ class Project:
msg = "manifest set to %s" % self.revisionExpr msg = "manifest set to %s" % self.revisionExpr
if detach: if detach:
active_git.UpdateRef(ref, dst, message=msg, detach=True) active_git.UpdateRef(ref, dst, message=msg, detach=True)
if active_git is self.bare_git:
self.bare_ref.UpdateRef(ref)
else: else:
active_git.symbolic_ref("-m", msg, ref, dst) active_git.symbolic_ref("-m", msg, ref, dst)
if active_git is self.bare_git: if active_git is self.bare_git:
@@ -5822,7 +5917,7 @@ class MetaProject(Project):
return False return False
@property @property
def HasChanges(self): def HasChanges(self) -> bool:
"""Has the remote received new commits not yet checked out?""" """Has the remote received new commits not yet checked out?"""
if not self.remote or not self.revisionExpr: if not self.remote or not self.revisionExpr:
return False return False
@@ -5874,6 +5969,24 @@ class RepoProject(MetaProject):
class ManifestProject(MetaProject): class ManifestProject(MetaProject):
"""The MetaProject for manifests.""" """The MetaProject for manifests."""
def GetMergeBranch(self, heads_only: bool = False) -> Optional[str]:
"""Return the upstream merge ref of the current branch.
The `refs/heads/` prefix is stripped when present.
Args:
heads_only: If True, return None unless the upstream merge ref is
under `refs/heads/` (i.e. tracking a remote branch rather than
pinned to a tag or commit via `repo init -b`).
"""
current_branch = self.CurrentBranch
if not current_branch:
return None
merge = self.GetBranch(current_branch).merge or ""
if merge.startswith(git_refs.R_HEADS):
return merge[len(git_refs.R_HEADS) :] or None
return None if heads_only else (merge or None)
def MetaBranchSwitch(self, submodules=False, verbose=False): def MetaBranchSwitch(self, submodules=False, verbose=False):
"""Prepare for manifest branch switch.""" """Prepare for manifest branch switch."""
@@ -5968,6 +6081,27 @@ class ManifestProject(MetaProject):
"""Whether we use smart sync by default.""" """Whether we use smart sync by default."""
return self.config.GetBoolean("repo.smartsync") return self.config.GetBoolean("repo.smartsync")
@staticmethod
def _SmartSyncBranchKey(branch: str) -> str:
"""Return the git config key for a branch's smart sync branch."""
return f"repo.{branch}.smartsyncbranch"
def GetSmartSyncBranch(self, branch: Optional[str] = None) -> Optional[str]:
"""Return the configured smart sync branch for |branch|."""
if branch is None:
branch = self.GetMergeBranch(heads_only=True)
if not branch:
return None
return self.config.GetString(self._SmartSyncBranchKey(branch))
def SetSmartSyncBranch(
self, branch: str, smart_sync_branch: Optional[str]
) -> None:
"""Set or clear the smart sync branch for |branch|."""
self.config.SetString(
self._SmartSyncBranchKey(branch), smart_sync_branch or None
)
@property @property
def partial_clone(self): def partial_clone(self):
"""Whether this is a partial clone.""" """Whether this is a partial clone."""
@@ -6076,36 +6210,37 @@ class ManifestProject(MetaProject):
def Sync( def Sync(
self, self,
_kwargs_only=(), _kwargs_only: Tuple[Any, ...] = (),
manifest_url="", manifest_url: Optional[str] = "",
manifest_branch=None, manifest_branch: Optional[str] = None,
standalone_manifest=False, standalone_manifest: Union[bool, str, None] = False,
groups="", groups: Optional[str] = "",
mirror=False, mirror: Optional[bool] = False,
reference="", reference: Optional[str] = "",
dissociate=False, dissociate: Optional[bool] = False,
worktree=False, worktree: Optional[bool] = False,
submodules=False, submodules: Optional[bool] = False,
archive=False, archive: Optional[bool] = False,
partial_clone=None, partial_clone: Optional[bool] = None,
depth=None, depth: Optional[int] = None,
clone_filter="blob:none", clone_filter: Optional[str] = "blob:none",
partial_clone_exclude=None, partial_clone_exclude: Optional[str] = None,
clone_bundle=None, clone_bundle: Optional[bool] = None,
git_lfs=None, git_lfs: Optional[bool] = None,
use_superproject=None, use_superproject: Optional[bool] = None,
smart_sync: Optional[bool] = None, smart_sync: Optional[bool] = None,
verbose=False, smart_sync_branch: Optional[str] = None,
current_branch_only=False, verbose: Optional[bool] = False,
git_event_log=None, current_branch_only: Optional[bool] = False,
platform="", git_event_log: Optional[git_trace2_event_log.EventLog] = None,
manifest_name="default.xml", platform: Optional[str] = "",
tags="", manifest_name: Optional[str] = "default.xml",
this_manifest_only=False, tags: Union[bool, str, None] = "",
outer_manifest=True, this_manifest_only: Optional[bool] = False,
clone_filter_for_depth=None, outer_manifest: bool = True,
use_local_gitdirs=False, clone_filter_for_depth: Optional[str] = None,
): use_local_gitdirs: Optional[bool] = False,
) -> bool:
"""Sync the manifest and all submanifests. """Sync the manifest and all submanifests.
Args: Args:
@@ -6137,6 +6272,8 @@ class ManifestProject(MetaProject):
use_superproject: a boolean, whether to use the manifest use_superproject: a boolean, whether to use the manifest
superproject to sync projects. superproject to sync projects.
smart_sync: a boolean, whether to default to smart sync. smart_sync: a boolean, whether to default to smart sync.
smart_sync_branch: a string, server branch to query when smart
syncing the manifest branch (empty string to clear).
verbose: a boolean, whether to show all output, rather than only verbose: a boolean, whether to show all output, rather than only
errors. errors.
current_branch_only: a boolean, whether to only fetch the current current_branch_only: a boolean, whether to only fetch the current
@@ -6185,6 +6322,7 @@ class ManifestProject(MetaProject):
git_lfs=git_lfs, git_lfs=git_lfs,
use_superproject=use_superproject, use_superproject=use_superproject,
smart_sync=smart_sync, smart_sync=smart_sync,
smart_sync_branch=smart_sync_branch,
verbose=verbose, verbose=verbose,
current_branch_only=current_branch_only, current_branch_only=current_branch_only,
tags=tags, tags=tags,
@@ -6308,7 +6446,7 @@ class ManifestProject(MetaProject):
elif platform in all_platforms: elif platform in all_platforms:
groups.append(platformize(platform)) groups.append(platformize(platform))
elif platform != "none": elif platform != "none":
logger.error("fatal: invalid platform flag", file=sys.stderr) logger.error("fatal: invalid platform flag: %s", platform)
return False return False
self.config.SetString("manifest.platform", platform) self.config.SetString("manifest.platform", platform)
@@ -6506,6 +6644,18 @@ class ManifestProject(MetaProject):
) )
return False return False
if smart_sync_branch is not None:
target_branch = self.GetMergeBranch(heads_only=True)
if target_branch:
self.SetSmartSyncBranch(target_branch, smart_sync_branch)
elif smart_sync_branch:
logger.error(
"fatal: --smart-sync-branch requires the manifest "
"checkout to track a branch; run `repo init -b "
"<branch>`"
)
return False
if not manifest_name: if not manifest_name:
logger.error("fatal: manifest name (-m) is required.") logger.error("fatal: manifest name (-m) is required.")
return False return False
+73
View File
@@ -41,7 +41,80 @@ extend_skip_glob = '*_pb2.py'
single_line_exclusions = ['abc', 'collections.abc', 'typing'] single_line_exclusions = ['abc', 'collections.abc', 'typing']
[tool.pytest.ini_options] [tool.pytest.ini_options]
addopts = [
"--durations=5",
"--durations-min=1.0",
"--strict-markers",
"--numprocesses=auto",
# Currently the test suite doesn't seem to get faster with more threads,
# so cap it here for now since there's a startup cost for every one.
"--maxprocesses=32",
]
markers = """ markers = """
skip_cq: Skip tests in the CQ. Should be rarely used! skip_cq: Skip tests in the CQ. Should be rarely used!
e2e: Hermetic end-to-end tests (tests/e2e/). Deselect with -m 'not e2e'. e2e: Hermetic end-to-end tests (tests/e2e/). Deselect with -m 'not e2e'.
""" """
# Only keep the temp dirs of failed tests. By default, pytest keeps the temp
# dirs of all tests for the last 3 runs, and each e2e test has its own checkout.
# Use `-o tmp_path_retention_policy=all` to inspect the dirs of passing tests.
tmp_path_retention_policy = "failed"
# https://pyrefly.org/en/docs/configuration/
[tool.pyrefly]
project-includes = [
# go/keep-sorted start
"**/*.py*",
"release/update-hooks",
"release/update-manpages",
"repo",
"run_tests",
# go/keep-sorted end
]
# We use relative module imports for our own modules.
enable-fallback-search-path = true
# NB: Do not add more modules here. We need to shrink this list.
project-excludes = [
"command.py",
"git_command.py",
"git_config.py",
"git_refs.py",
"git_status.py",
"git_superproject.py",
"main.py",
"manifest_xml.py",
"project.py",
"subcmds/branches.py",
"subcmds/cherry_pick.py",
"subcmds/config.py",
"subcmds/diffmanifests.py",
"subcmds/forall.py",
"subcmds/gc.py",
"subcmds/grep.py",
"subcmds/help.py",
"subcmds/info.py",
"subcmds/init.py",
"subcmds/manifest.py",
"subcmds/rebase.py",
"subcmds/selfupdate.py",
"subcmds/smartsync.py",
"subcmds/stage.py",
"subcmds/start.py",
"subcmds/status.py",
"subcmds/sync.py",
"subcmds/upload.py",
"subcmds/version.py",
"subcmds/wipe.py",
"tests/e2e/harness.py",
"tests/e2e/test_groups.py",
"tests/test_command.py",
"tests/test_git_command.py",
"tests/test_git_config.py",
"tests/test_git_superproject.py",
"tests/test_git_trace2_event_log.py",
"tests/test_manifest_xml.py",
"tests/test_project.py",
"tests/test_subcmds_info.py",
"tests/test_subcmds_sync.py",
"tests/test_subcmds_wipe.py",
"tests/test_wrapper.py",
]
Symlink
+1
View File
@@ -0,0 +1 @@
run_tests
Symlink
+1
View File
@@ -0,0 +1 @@
run_tests
+16 -10
View File
@@ -26,7 +26,7 @@ import platform
import shlex import shlex
import subprocess import subprocess
import sys import sys
from typing import NamedTuple, Tuple from typing import List, NamedTuple, Tuple
# These should never be newer than the main.py version since this needs to be a # These should never be newer than the main.py version since this needs to be a
@@ -119,17 +119,15 @@ if __name__ == "__main__":
# repo default configuration # repo default configuration
# #
REPO_URL = os.environ.get("REPO_URL", None) REPO_URL = (
if not REPO_URL: os.environ.get("REPO_URL") or "https://gerrit.googlesource.com/git-repo"
REPO_URL = "https://gerrit.googlesource.com/git-repo" )
REPO_REV = os.environ.get("REPO_REV") REPO_REV = os.environ.get("REPO_REV") or "stable"
if not REPO_REV:
REPO_REV = "stable"
# URL to file bug reports for repo tool issues. # URL to file bug reports for repo tool issues.
BUG_URL = "https://issues.gerritcodereview.com/issues/new?component=1370071" BUG_URL = "https://issues.gerritcodereview.com/issues/new?component=1370071"
# increment this whenever we make important changes to this script # increment this whenever we make important changes to this script
VERSION = (2, 71) VERSION = (2, 74)
# increment this if the MAINTAINER_KEYS block is modified # increment this if the MAINTAINER_KEYS block is modified
KEYRING_VERSION = (2, 3) KEYRING_VERSION = (2, 3)
@@ -251,7 +249,7 @@ def GetParser():
return parser return parser
def InitParser(parser): def InitParser(parser: optparse.OptionParser) -> optparse.OptionParser:
"""Setup the CLI parser.""" """Setup the CLI parser."""
# NB: Keep in sync with command.py:_CommonOptions(). # NB: Keep in sync with command.py:_CommonOptions().
@@ -457,6 +455,14 @@ def InitParser(parser):
dest="smart_sync", dest="smart_sync",
help="do not default to smart sync when syncing projects", help="do not default to smart sync when syncing projects",
) )
group.add_option(
"--smart-sync-branch",
action="store",
default=None,
metavar="BRANCH",
help="server branch to query when smart syncing the manifest branch "
"(empty string to clear)",
)
group.add_option( group.add_option(
"--clone-bundle", "--clone-bundle",
action="store_true", action="store_true",
@@ -902,7 +908,7 @@ def _GetRepoConfig(name):
def _InitHttp(): def _InitHttp():
handlers = [] handlers: List[urllib.request.BaseHandler] = []
mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm() mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
try: try:
+5 -1
View File
@@ -25,6 +25,7 @@ import os
import sys import sys
import tempfile import tempfile
import time import time
from typing import Optional
import platform_utils import platform_utils
@@ -35,7 +36,7 @@ REPO_TRACE = "REPO_TRACE"
# Temporarily set tracing to always on unless user expicitly sets to 0. # Temporarily set tracing to always on unless user expicitly sets to 0.
_TRACE = os.environ.get(REPO_TRACE) != "0" _TRACE = os.environ.get(REPO_TRACE) != "0"
_TRACE_TO_STDERR = False _TRACE_TO_STDERR = False
_TRACE_FILE = None _TRACE_FILE: Optional[str] = None
_TRACE_FILE_NAME = "TRACE_FILE" _TRACE_FILE_NAME = "TRACE_FILE"
_MAX_SIZE = 70 # in MiB _MAX_SIZE = 70 # in MiB
_NEW_COMMAND_SEP = "+++++++++++++++NEW COMMAND+++++++++++++++++++" _NEW_COMMAND_SEP = "+++++++++++++++NEW COMMAND+++++++++++++++++++"
@@ -104,6 +105,7 @@ class Trace(contextlib.ContextDecorator):
f"PID: {os.getpid()} START: {self._time()} :{self._trace_msg}\n" f"PID: {os.getpid()} START: {self._time()} :{self._trace_msg}\n"
) )
assert _TRACE_FILE is not None
with open(_TRACE_FILE, "a") as f: with open(_TRACE_FILE, "a") as f:
print(print_msg, file=f) print(print_msg, file=f)
@@ -120,6 +122,7 @@ class Trace(contextlib.ContextDecorator):
f"PID: {os.getpid()} END: {self._time()} :{self._trace_msg}\n" f"PID: {os.getpid()} END: {self._time()} :{self._trace_msg}\n"
) )
assert _TRACE_FILE is not None
with open(_TRACE_FILE, "a") as f: with open(_TRACE_FILE, "a") as f:
print(print_msg, file=f) print(print_msg, file=f)
@@ -141,6 +144,7 @@ def _GetTraceFile(quiet):
def _ClearOldTraces(): def _ClearOldTraces():
"""Clear the oldest commands if trace file is too big.""" """Clear the oldest commands if trace file is too big."""
assert _TRACE_FILE is not None
try: try:
with open(_TRACE_FILE, errors="ignore") as f: with open(_TRACE_FILE, errors="ignore") as f:
if os.path.getsize(f.name) / (1024 * 1024) <= _MAX_SIZE: if os.path.getsize(f.name) / (1024 * 1024) <= _MAX_SIZE:
+62 -4
View File
@@ -13,16 +13,20 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
# NB: Regen uv.lock whenever changing this script by running:
# $ vpython3 -vpython-tool upgrade run_tests
# /// script # /// script
# requires-python = '>=3.11,<3.12' # requires-python = '>=3.11,<3.12'
# dependencies = [ # dependencies = [
# 'pytest==8.3.4', # 'pytest==8.3.4',
# 'pytest-xdist==3.8.0',
# 'black==25.1.0', # 'black==25.1.0',
# 'flake8==7.4.1', # 'flake8==7.4.1',
# 'flake8-bugbear==26.9.30', # 'flake8-bugbear==26.9.30',
# 'flake8-comprehensions==3.17.0', # 'flake8-comprehensions==3.17.0',
# 'flake8-import-restrictions==2.1', # 'flake8-import-restrictions==2.1',
# 'isort==5.10.1' # 'isort==5.10.1',
# 'pyrefly==1.4.0',
# ] # ]
# /// # ///
@@ -31,10 +35,15 @@
import functools import functools
import os import os
import shlex import shlex
import shutil
import subprocess import subprocess
import sys import sys
import black
import flake8.main.cli
import isort.main
import pyrefly.__main__
import pytest
# NB: While tests/* support Python >=3.6 to match requirements.json for `repo`, # NB: While tests/* support Python >=3.6 to match requirements.json for `repo`,
# the higher level runner logic does not need to be held back. # the higher level runner logic does not need to be held back.
@@ -162,6 +171,30 @@ def run_isort():
).returncode ).returncode
def run_pyrefly() -> int:
"""Returns the exit code from pyrefly."""
argv = ["--version"]
log_cmd("pyrefly", argv)
subprocess.run(
[pyrefly.__main__.get_pyrefly_bin()] + argv,
check=True,
cwd=ROOT_DIR,
)
argv = [
"check",
"--verbose",
"--output-format=full-text",
"--progress-bar=simple",
]
log_cmd("pyrefly", argv)
return subprocess.run(
[pyrefly.__main__.get_pyrefly_bin()] + argv,
check=False,
cwd=ROOT_DIR,
).returncode
def run_check_metadata(): def run_check_metadata():
"""Returns the exit code from check-metadata.""" """Returns the exit code from check-metadata."""
argv = [] argv = []
@@ -199,8 +232,8 @@ def cipd_bootstrap() -> None:
return return
def main(argv): def main_run_tests(argv: list[str]) -> int:
"""The main entry.""" """The run_tests program."""
cipd_bootstrap() cipd_bootstrap()
checks = ( checks = (
functools.partial(run_pytest, argv), functools.partial(run_pytest, argv),
@@ -208,6 +241,7 @@ def main(argv):
run_black, run_black,
run_flake8, run_flake8,
run_isort, run_isort,
run_pyrefly,
run_check_metadata, run_check_metadata,
run_update_manpages, run_update_manpages,
) )
@@ -216,5 +250,29 @@ def main(argv):
return 1 if sum(c() for c in checks) else 0 return 1 if sum(c() for c in checks) else 0
def main_pyrefly(argv: list[str]) -> int:
"""The pyrefly program."""
result = subprocess.run(
[pyrefly.__main__.get_pyrefly_bin()] + argv, check=False
)
return result.returncode
main_black = black.main
main_flake8 = flake8.main.cli.main
main_isort = isort.main.main
main_pytest = pytest.main
def main(argv: list[str]) -> int:
"""The main dispatcher."""
# Figure out what script we're running as and dispatch to its main.
# This makes it easier for people to use the same env as run_tests.
argv0 = os.path.basename(sys.argv[0])
funcname = f"main_{argv0}"
func = globals().get(funcname, main_run_tests)
return func(argv)
if __name__ == "__main__": if __name__ == "__main__":
sys.exit(main(sys.argv[1:])) sys.exit(main(sys.argv[1:]))
+37 -27
View File
@@ -1,43 +1,46 @@
attrs==25.4.0 ; python_full_version < '3.12' \ attrs==25.4.0 \
--hash=sha256:3e2be3ccc931d6c0cda7d37122b992d6d4b905968a437da7843cff016e2c41e8 \ --hash=sha256:3e2be3ccc931d6c0cda7d37122b992d6d4b905968a437da7843cff016e2c41e8 \
--hash=sha256:adcf7e2a1fb3b36ac48d97835bb6d8ade15b8dcce26aba8bf1d14847b57a3373 --hash=sha256:adcf7e2a1fb3b36ac48d97835bb6d8ade15b8dcce26aba8bf1d14847b57a3373
# via flake8-bugbear # via flake8-bugbear
black==25.1.0 ; python_full_version < '3.12' \ black==25.1.0 \
--hash=sha256:95e8176dae143ba9097f351d174fdaf0ccd29efb414b362ae3fd72bf0f710717 \ --hash=sha256:95e8176dae143ba9097f351d174fdaf0ccd29efb414b362ae3fd72bf0f710717 \
--hash=sha256:c9ed033e41998dd45d6f19bf9923e2444cb401185ccb26d1a2101a9049637c1f --hash=sha256:c9ed033e41998dd45d6f19bf9923e2444cb401185ccb26d1a2101a9049637c1f
click==8.4.1 ; python_full_version < '3.12' \ click==8.4.1 \
--hash=sha256:482be17c6991b8c19c5429a1e995d9b0efdbb63172824c41f99965dc0ade8ec2 --hash=sha256:482be17c6991b8c19c5429a1e995d9b0efdbb63172824c41f99965dc0ade8ec2
# via black # via black
colorama==0.4.6 ; python_full_version < '3.12' and sys_platform == 'win32' \ colorama==0.4.6 ; sys_platform == 'win32' \
--hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 \ --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 \
--hash=sha256:8a5a9494346d7ae2d8f599e4661738007d2df56dbbf4ccbbf7a6a23679a3a302 --hash=sha256:8a5a9494346d7ae2d8f599e4661738007d2df56dbbf4ccbbf7a6a23679a3a302
# via # via
# click # click
# pytest # pytest
flake8==6.0.0 ; python_full_version < '3.12' \ execnet==2.1.1 \
--hash=sha256:9b5bf0ebd06aff1601e38dfc4e91f9b99ba3cdf446e74ee25a9cbbe01aef81ee \ --hash=sha256:26dee51f1b80cebd6d0ca8e74dd8745419761d3bef34163928cbebbdc4749fdc \
--hash=sha256:f34b5cb8bf908ff49e4673d3dbc557b2381a14132b637d53e9d3a59816f4b6ba --hash=sha256:58d6d58362e52bfab78f40ebfed9575a3334c41593bee4eb5fcda518290018dc
# via pytest-xdist
flake8==7.4.1 \
--hash=sha256:7ff3b14668bfec70da5ce3eeaf1756098330598a9a69f1ee312ed315d27c6508
# via # via
# flake8-bugbear # flake8-bugbear
# flake8-comprehensions # flake8-comprehensions
flake8-bugbear==24.12.12 ; python_full_version < '3.12' \ flake8-bugbear==26.9.30 \
--hash=sha256:1b6967436f65ca22a42e5373aaa6f2d87966ade9aa38d4baf2a1be550767545e --hash=sha256:d0fec65b92eb79adde488515c925c33afabbf85d194bd881e845c71444960fc5
flake8-comprehensions==3.16.0 ; python_full_version < '3.12' \ flake8-comprehensions==3.17.0 \
--hash=sha256:7c1eadc9d22e765f39857798febe7766b4d9c519793c6c149e3e13bf99693f70 --hash=sha256:3943a9c6f2593c3bc5cc64106c2f89d63c6ecd49c8343597f8257b8fcfc8b0a2
flake8-import-restrictions==2.1 ; python_full_version < '3.12' \ flake8-import-restrictions==2.1 \
--hash=sha256:77522e76b82a6ccb25820b1e195825a695094ceded23a18c1b5ce026c23e1071 --hash=sha256:77522e76b82a6ccb25820b1e195825a695094ceded23a18c1b5ce026c23e1071
iniconfig==2.3.0 ; python_full_version < '3.12' \ iniconfig==2.3.0 \
--hash=sha256:e6ae3625a739878eeb4baf25f17a0b4e9d985421e45b897d1332ed7478f822d9 \ --hash=sha256:e6ae3625a739878eeb4baf25f17a0b4e9d985421e45b897d1332ed7478f822d9 \
--hash=sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12 --hash=sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12
# via pytest # via pytest
isort==5.10.1 ; python_full_version < '3.12' \ isort==5.10.1 \
--hash=sha256:6f62d78e2f89b4500b080fe3a81690850cd254227f27f75c3a0c491a1f351ba7 \ --hash=sha256:6f62d78e2f89b4500b080fe3a81690850cd254227f27f75c3a0c491a1f351ba7 \
--hash=sha256:ec0da0d88470e89ade75bf1e2ea7b4349ec906f95c84aff08e4d41dea6864d25 --hash=sha256:ec0da0d88470e89ade75bf1e2ea7b4349ec906f95c84aff08e4d41dea6864d25
mccabe==0.7.0 ; python_full_version < '3.12' \ mccabe==0.7.0 \
--hash=sha256:6c2d30ab6be0e4a46919781807b4f0d834ebdd6c6e3dca0bda5a15f863427b6e \ --hash=sha256:6c2d30ab6be0e4a46919781807b4f0d834ebdd6c6e3dca0bda5a15f863427b6e \
--hash=sha256:b85abd0a321cdd357d0be34adbf49cf01453a857a23e94d44612ec6f159cb261 --hash=sha256:b85abd0a321cdd357d0be34adbf49cf01453a857a23e94d44612ec6f159cb261
# via flake8 # via flake8
mypy-extensions==1.1.0 ; python_full_version < '3.12' \ mypy-extensions==1.1.0 \
--hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \ --hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \
--hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \ --hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \
--hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \ --hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \
@@ -48,31 +51,38 @@ mypy-extensions==1.1.0 ; python_full_version < '3.12' \
--hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \ --hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \
--hash=sha256:4ec363d206d93f233330fa7f08eefe9a382dd1e326112a94244f95ec2f22f10e --hash=sha256:4ec363d206d93f233330fa7f08eefe9a382dd1e326112a94244f95ec2f22f10e
# via black # via black
packaging==26.2 ; python_full_version < '3.12' \ packaging==26.2 \
--hash=sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e --hash=sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e
# via # via
# black # black
# pytest # pytest
pathspec==1.0.4 ; python_full_version < '3.12' \ pathspec==1.0.4 \
--hash=sha256:eeaf9fc7cbc0a4492b666be276f38956baa67cd3faca745462db83d98f1cef87 \ --hash=sha256:eeaf9fc7cbc0a4492b666be276f38956baa67cd3faca745462db83d98f1cef87 \
--hash=sha256:fb6ae2fd4e7c921a165808a552060e722767cfa526f99ca5156ed2ce45a5c723 --hash=sha256:fb6ae2fd4e7c921a165808a552060e722767cfa526f99ca5156ed2ce45a5c723
# via black # via black
platformdirs==4.9.2 ; python_full_version < '3.12' \ platformdirs==4.9.2 \
--hash=sha256:9170634f126f8efdae22fb58ae8a0eaa86f38365bc57897a6c4f781d1f5875bd \ --hash=sha256:9170634f126f8efdae22fb58ae8a0eaa86f38365bc57897a6c4f781d1f5875bd \
--hash=sha256:ef245e78eb7a2ba57599bdb39c4ff323cfae3de356a47207b58bb29e7f479f46 --hash=sha256:ef245e78eb7a2ba57599bdb39c4ff323cfae3de356a47207b58bb29e7f479f46
# via black # via black
pluggy==1.6.0 ; python_full_version < '3.12' \ pluggy==1.6.0 \
--hash=sha256:88349c39e440567519b15b27a781bcd3430de6bae17c44c37e758795be96dc5a \ --hash=sha256:88349c39e440567519b15b27a781bcd3430de6bae17c44c37e758795be96dc5a \
--hash=sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746 --hash=sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746
# via pytest # via pytest
pycodestyle==2.10.0 ; python_full_version < '3.12' \ pycodestyle==2.15.0 \
--hash=sha256:8a4eaf0d0495c7395bdab3589ac2db602797d76207242c17d470186815706610 \ --hash=sha256:12fd2f73c7b8ee8845a0431111df8faf4c1a07d6e64e2ee7f0c74014dab14181
--hash=sha256:aacf8d28ecea10fad31b2d14943a42d85d007f4d2fee8f212fc05751c8460647
# via flake8 # via flake8
pyflakes==3.0.1 ; python_full_version < '3.12' \ pyflakes==4.0.3 \
--hash=sha256:a968b8baa43926124b692c7e76279bc5fa8c6ef9a2281e4a2a475be69f954bdf \ --hash=sha256:330ba92b8c1db2eb0b8f4068f6c58674e2649a99e334769aa50e3e9c5b11c23a
--hash=sha256:ec55bf7fe21fff7f1ad2f7da62363d749e2a470500eab1b555334b67aa1ef8cf
# via flake8 # via flake8
pytest==8.3.4 ; python_full_version < '3.12' \ pyrefly==1.4.0 \
--hash=sha256:460cec39bc8743915e9c2c8c10ba926ad0b29b099d89c1859fb97e0059f42fa7 \
--hash=sha256:5a6370d686b6cbbac8869832f09a52012888371956fbc68d446a349bc85ccd43 \
--hash=sha256:7aff37c2b711c8240ed650da0c6148237b327bf7e5dfacee65884f1adea22305 \
--hash=sha256:b6951f51687304caebfedf13dd1a943aeafdbc95abcc4e24a683b97467a77007
pytest==8.3.4 \
--hash=sha256:00cba7845d2dccfe5403025a688d60ceae228b1ddaa866d2eb86b2edb6923e73 \ --hash=sha256:00cba7845d2dccfe5403025a688d60ceae228b1ddaa866d2eb86b2edb6923e73 \
--hash=sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6 --hash=sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6
# via pytest-xdist
pytest-xdist==3.8.0 \
--hash=sha256:202ca578cfeb7370784a8c33d6d05bc6e13b4f25b5053c30a152269fd10f0b88 \
--hash=sha256:9e659fa2eacbed776070a08a075eff2d9c8e010bab4eea169640c22a1b079f2c
+5
View File
@@ -65,3 +65,8 @@ wheel: <
name: "infra/python/wheels/exceptiongroup-py3" name: "infra/python/wheels/exceptiongroup-py3"
version: "version:1.1.2" version: "version:1.1.2"
> >
wheel: <
name: "infra/python/wheels/pytest-xdist-py2_py3"
version: "version:1.31.0"
>
+2
View File
@@ -17,6 +17,8 @@
import os import os
# Not installed in the test virtualenv.
# pyrefly: ignore
import setuptools import setuptools
+3 -4
View File
@@ -16,8 +16,7 @@ import collections
import functools import functools
import itertools import itertools
from command import Command import command
from command import DEFAULT_LOCAL_JOBS
import error import error
import git_command import git_command
import progress import progress
@@ -31,7 +30,7 @@ class AbandonError(error.RepoExitError):
"""Exit error when abandon command fails.""" """Exit error when abandon command fails."""
class Abandon(Command): class Abandon(command.Command):
COMMON = True COMMON = True
helpSummary = "Permanently abandon a development branch" helpSummary = "Permanently abandon a development branch"
helpUsage = """ helpUsage = """
@@ -42,7 +41,7 @@ deleting it (and all its history) from your local repository.
It is equivalent to "git branch -D <branchname>". It is equivalent to "git branch -D <branchname>".
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
def _Options(self, p): def _Options(self, p):
p.add_option( p.add_option(
+3 -4
View File
@@ -16,8 +16,7 @@ import itertools
import sys import sys
import color import color
from command import Command import command
from command import DEFAULT_LOCAL_JOBS
class BranchColoring(color.Coloring): class BranchColoring(color.Coloring):
@@ -62,7 +61,7 @@ class BranchInfo:
return self.published_equal == len(self.projects) return self.published_equal == len(self.projects)
class Branches(Command): class Branches(command.Command):
COMMON = True COMMON = True
helpSummary = "View current topic branches" helpSummary = "View current topic branches"
helpUsage = """ helpUsage = """
@@ -96,7 +95,7 @@ the branch appears in, or does not appear in. If no project list
is shown, then the branch appears in all projects. is shown, then the branch appears in all projects.
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
@classmethod @classmethod
def _ExpandProjectToBranches(cls, project_idx): def _ExpandProjectToBranches(cls, project_idx):
+6 -7
View File
@@ -13,10 +13,9 @@
# limitations under the License. # limitations under the License.
import functools import functools
from typing import NamedTuple from typing import NamedTuple, Optional
from command import Command import command
from command import DEFAULT_LOCAL_JOBS
import error import error
import progress import progress
import repo_logging import repo_logging
@@ -27,9 +26,9 @@ logger = repo_logging.RepoLogger(__file__)
class CheckoutBranchResult(NamedTuple): class CheckoutBranchResult(NamedTuple):
# Whether the Project is on the branch (i.e. branch exists and no errors) # Whether the Project is on the branch (i.e. branch exists and no errors)
result: bool result: Optional[bool]
project_idx: int project_idx: int
error: Exception error: Optional[Exception]
class CheckoutCommandError(error.RepoExitError): class CheckoutCommandError(error.RepoExitError):
@@ -40,7 +39,7 @@ class MissingBranchError(error.RepoExitError):
"""Exception thrown when no project has specified branch.""" """Exception thrown when no project has specified branch."""
class Checkout(Command): class Checkout(command.Command):
COMMON = True COMMON = True
helpSummary = "Checkout a branch for development" helpSummary = "Checkout a branch for development"
helpUsage = """ helpUsage = """
@@ -54,7 +53,7 @@ The command is equivalent to:
repo forall [<project>...] -c git checkout <branchname> repo forall [<project>...] -c git checkout <branchname>
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
def ValidateOptions(self, opt, args): def ValidateOptions(self, opt, args):
if not args: if not args:
+2 -2
View File
@@ -16,7 +16,7 @@ import re
import sys import sys
from typing import Tuple from typing import Tuple
from command import Command import command
import error import error
import git_command import git_command
import repo_logging import repo_logging
@@ -26,7 +26,7 @@ CHANGE_ID_RE = re.compile(r"^\s*Change-Id: I([0-9a-f]{40})\s*$")
logger = repo_logging.RepoLogger(__file__) logger = repo_logging.RepoLogger(__file__)
class CherryPick(Command): class CherryPick(command.Command):
COMMON = True COMMON = True
helpSummary = "Cherry-pick a change." helpSummary = "Cherry-pick a change."
helpUsage = """ helpUsage = """
+3 -4
View File
@@ -15,11 +15,10 @@
import functools import functools
import io import io
from command import DEFAULT_LOCAL_JOBS import command
from command import PagedCommand
class Diff(PagedCommand): class Diff(command.PagedCommand):
COMMON = True COMMON = True
helpSummary = "Show changes between commit and working tree" helpSummary = "Show changes between commit and working tree"
helpUsage = """ helpUsage = """
@@ -29,7 +28,7 @@ The -u option causes '%prog' to generate diff output with file paths
relative to the repository root, so the output can be applied relative to the repository root, so the output can be applied
to the Unix 'patch' command. to the Unix 'patch' command.
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
def _Options(self, p): def _Options(self, p):
p.add_option( p.add_option(
+2 -2
View File
@@ -13,7 +13,7 @@
# limitations under the License. # limitations under the License.
import color import color
from command import PagedCommand import command
import manifest_xml import manifest_xml
@@ -22,7 +22,7 @@ class _Coloring(color.Coloring):
super().__init__(config, "status") super().__init__(config, "status")
class Diffmanifests(PagedCommand): class Diffmanifests(command.PagedCommand):
"""A command to see logs in projects represented by manifests """A command to see logs in projects represented by manifests
This is used to see deeper differences between manifests. Where a simple This is used to see deeper differences between manifests. Where a simple
+2 -2
View File
@@ -15,7 +15,7 @@
import re import re
import sys import sys
from command import Command import command
import error import error
import repo_logging import repo_logging
@@ -28,7 +28,7 @@ class DownloadCommandError(error.RepoExitError):
"""Error raised when download command fails.""" """Error raised when download command fails."""
class Download(Command): class Download(command.Command):
COMMON = True COMMON = True
helpSummary = "Download and checkout a change" helpSummary = "Download and checkout a change"
helpUsage = """ helpUsage = """
+4 -6
View File
@@ -22,9 +22,7 @@ import subprocess
import sys import sys
import color import color
from command import Command import command
from command import DEFAULT_LOCAL_JOBS
from command import MirrorSafeCommand
import repo_logging import repo_logging
@@ -43,7 +41,7 @@ class ForallColoring(color.Coloring):
self.project = self.printer("project", attr="bold") self.project = self.printer("project", attr="bold")
class Forall(Command, MirrorSafeCommand): class Forall(command.Command, command.MirrorSafeCommand):
COMMON = False COMMON = False
helpSummary = "Run a shell command in each project" helpSummary = "Run a shell command in each project"
helpUsage = """ helpUsage = """
@@ -136,7 +134,7 @@ terminal and are not redirected.
If -e is used, when a command exits unsuccessfully, '%prog' will abort If -e is used, when a command exits unsuccessfully, '%prog' will abort
without iterating through the remaining projects. without iterating through the remaining projects.
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
@staticmethod @staticmethod
def _cmd_option(option, _opt_str, _value, parser): def _cmd_option(option, _opt_str, _value, parser):
@@ -197,7 +195,7 @@ without iterating through the remaining projects.
"--interactive", action="store_true", help="force interactive usage" "--interactive", action="store_true", help="force interactive usage"
) )
def WantPager(self, opt): def WantPager(self, opt) -> bool:
return opt.project_header and opt.jobs == 1 return opt.project_header and opt.jobs == 1
def ValidateOptions(self, opt, args): def ValidateOptions(self, opt, args):
+2 -2
View File
@@ -15,14 +15,14 @@
import os import os
from typing import List, Set from typing import List, Set
from command import Command import command
import git_command import git_command
import platform_utils import platform_utils
import progress import progress
from project import Project from project import Project
class Gc(Command): class Gc(command.Command):
COMMON = True COMMON = True
helpSummary = "Cleaning up internal repo and Git state." helpSummary = "Cleaning up internal repo and Git state."
helpUsage = """ helpUsage = """
+5 -5
View File
@@ -17,8 +17,7 @@ import sys
from typing import NamedTuple from typing import NamedTuple
import color import color
from command import DEFAULT_LOCAL_JOBS import command
from command import PagedCommand
import error import error
import git_command import git_command
import repo_logging import repo_logging
@@ -50,7 +49,7 @@ class GrepCommandError(error.SilentRepoExitError):
aggregate errors exit silently.""" aggregate errors exit silently."""
class Grep(PagedCommand): class Grep(command.PagedCommand):
COMMON = True COMMON = True
helpSummary = "Print lines matching a pattern" helpSummary = "Print lines matching a pattern"
helpUsage = """ helpUsage = """
@@ -84,7 +83,7 @@ contain a line that matches both expressions:
repo grep --all-match -e NODE -e Unexpected repo grep --all-match -e NODE -e Unexpected
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
@staticmethod @staticmethod
def _carry_option(_option, opt_str, value, parser): def _carry_option(_option, opt_str, value, parser):
@@ -103,8 +102,9 @@ contain a line that matches both expressions:
if value is not None: if value is not None:
pt.append(value) pt.append(value)
def _CommonOptions(self, p): def _CommonOptions(self, p, opt_v=True):
"""Override common options slightly.""" """Override common options slightly."""
del opt_v # Unused.
super()._CommonOptions(p, opt_v=False) super()._CommonOptions(p, opt_v=False)
def _Options(self, p): def _Options(self, p):
+18 -18
View File
@@ -15,12 +15,12 @@
import re import re
import sys import sys
import textwrap import textwrap
from typing import Sequence
import color import color
from command import MirrorSafeCommand import command
from command import PagedCommand
import error import error
from subcmds import all_commands import subcmds
import wrapper import wrapper
@@ -28,7 +28,7 @@ class InvalidHelpCommand(error.RepoExitError):
"""Invalid command passed into help.""" """Invalid command passed into help."""
class Help(PagedCommand, MirrorSafeCommand): class Help(command.PagedCommand, command.MirrorSafeCommand):
COMMON = False COMMON = False
helpSummary = "Display detailed help on a command" helpSummary = "Display detailed help on a command"
helpUsage = """ helpUsage = """
@@ -38,17 +38,17 @@ class Help(PagedCommand, MirrorSafeCommand):
Displays detailed usage information about a command. Displays detailed usage information about a command.
""" """
def _PrintCommands(self, commandNames): def _PrintCommands(self, names: Sequence[str]) -> None:
"""Helper to display |commandNames| summaries.""" """Helper to display |names| summaries."""
maxlen = 0 maxlen = 0
for name in commandNames: for name in names:
maxlen = max(maxlen, len(name)) maxlen = max(maxlen, len(name))
fmt = " %%-%ds %%s" % maxlen fmt = " %%-%ds %%s" % maxlen
for name in commandNames: for name in names:
command = all_commands[name]() cmd = subcmds.all_commands[name]()
try: try:
summary = command.helpSummary.strip() summary = cmd.helpSummary.strip()
except AttributeError: except AttributeError:
summary = "" summary = ""
print(fmt % (name, summary)) print(fmt % (name, summary))
@@ -59,8 +59,8 @@ Displays detailed usage information about a command.
def PrintAllCommandsBody(self): def PrintAllCommandsBody(self):
print("The complete list of recognized repo commands is:") print("The complete list of recognized repo commands is:")
commandNames = sorted(all_commands) names = sorted(subcmds.all_commands)
self._PrintCommands(commandNames) self._PrintCommands(names)
print( print(
"See 'repo help <command>' for more information on a " "See 'repo help <command>' for more information on a "
"specific command." "specific command."
@@ -74,10 +74,10 @@ Displays detailed usage information about a command.
def PrintCommonCommandsBody(self): def PrintCommonCommandsBody(self):
print("The most commonly used repo commands are:") print("The most commonly used repo commands are:")
commandNames = sorted( names = sorted(
name for name, command in all_commands.items() if command.COMMON name for name, cmd in subcmds.all_commands.items() if cmd.COMMON
) )
self._PrintCommands(commandNames) self._PrintCommands(names)
print( print(
"See 'repo help <command>' for more information on a specific " "See 'repo help <command>' for more information on a specific "
@@ -146,8 +146,8 @@ Displays detailed usage information about a command.
out._PrintSection("Description", "helpDescription") out._PrintSection("Description", "helpDescription")
def _PrintAllCommandHelp(self): def _PrintAllCommandHelp(self):
for name in sorted(all_commands): for name in sorted(subcmds.all_commands):
cmd = all_commands[name](manifest=self.manifest) cmd = subcmds.all_commands[name](manifest=self.manifest)
self._PrintCommandHelp(cmd, header_prefix=f"[{name}] ") self._PrintCommandHelp(cmd, header_prefix=f"[{name}] ")
def _Options(self, p): def _Options(self, p):
@@ -178,7 +178,7 @@ Displays detailed usage information about a command.
name = args[0] name = args[0]
try: try:
cmd = all_commands[name](manifest=self.manifest) cmd = subcmds.all_commands[name](manifest=self.manifest)
except KeyError: except KeyError:
print( print(
"repo: '%s' is not a repo command." % name, file=sys.stderr "repo: '%s' is not a repo command." % name, file=sys.stderr
+4 -5
View File
@@ -21,8 +21,7 @@ import sys
from typing import Any, Dict, List, NamedTuple, Optional, Tuple from typing import Any, Dict, List, NamedTuple, Optional, Tuple
import color import color
from command import DEFAULT_LOCAL_JOBS import command
from command import PagedCommand
import git_refs import git_refs
@@ -51,9 +50,9 @@ class _Coloring(color.Coloring):
super().__init__(config, "status") super().__init__(config, "status")
class Info(PagedCommand): class Info(command.PagedCommand):
COMMON = True COMMON = True
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
helpSummary = ( helpSummary = (
"Get info on the manifest branch, current branch or unmerged branches" "Get info on the manifest branch, current branch or unmerged branches"
) )
@@ -131,7 +130,7 @@ class Info(PagedCommand):
help=f"output format: {', '.join(formats)} (default: %default)", help=f"output format: {', '.join(formats)} (default: %default)",
) )
def WantPager(self, opt): def WantPager(self, opt) -> bool:
return OutputFormat[opt.format.upper()] == OutputFormat.TEXT return OutputFormat[opt.format.upper()] == OutputFormat.TEXT
def ValidateOptions(self, opt, args): def ValidateOptions(self, opt, args):
+13 -6
View File
@@ -12,12 +12,13 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import optparse
import os import os
import sys import sys
from typing import List
import color import color
from command import InteractiveCommand import command
from command import MirrorSafeCommand
import error import error
import git_command import git_command
import repo_logging import repo_logging
@@ -29,7 +30,7 @@ logger = repo_logging.RepoLogger(__file__)
_REPO_ALLOW_SHALLOW = os.environ.get("REPO_ALLOW_SHALLOW") _REPO_ALLOW_SHALLOW = os.environ.get("REPO_ALLOW_SHALLOW")
class Init(InteractiveCommand, MirrorSafeCommand): class Init(command.InteractiveCommand, command.MirrorSafeCommand):
COMMON = True COMMON = True
RESPECT_SMART_SYNC_OVERRIDE = False RESPECT_SMART_SYNC_OVERRIDE = False
MULTI_MANIFEST_SUPPORT = True MULTI_MANIFEST_SUPPORT = True
@@ -90,7 +91,7 @@ manifest, a subsequent `repo sync` (or `repo sync -d`) is necessary
to update the working directory files. to update the working directory files.
""" """
def _CommonOptions(self, p): def _CommonOptions(self, p, opt_v=True):
"""Disable due to re-use of Wrapper().""" """Disable due to re-use of Wrapper()."""
def _Options(self, p): def _Options(self, p):
@@ -129,7 +130,7 @@ to update the working directory files.
"REPO_GIT_LFS": "git_lfs", "REPO_GIT_LFS": "git_lfs",
} }
def _SyncManifest(self, opt): def _SyncManifest(self, opt: optparse.Values) -> None:
"""Call manifestProject.Sync with arguments from opt. """Call manifestProject.Sync with arguments from opt.
Args: Args:
@@ -163,6 +164,7 @@ to update the working directory files.
git_lfs=opt.git_lfs, git_lfs=opt.git_lfs,
use_superproject=opt.use_superproject, use_superproject=opt.use_superproject,
smart_sync=opt.smart_sync, smart_sync=opt.smart_sync,
smart_sync_branch=opt.smart_sync_branch,
verbose=opt.verbose, verbose=opt.verbose,
current_branch_only=opt.current_branch_only, current_branch_only=opt.current_branch_only,
tags=opt.tags, tags=opt.tags,
@@ -295,7 +297,7 @@ to update the working directory files.
print(" rm -r %s" % os.path.join(self.manifest.topdir, ".repo")) print(" rm -r %s" % os.path.join(self.manifest.topdir, ".repo"))
print("and try again.") print("and try again.")
def ValidateOptions(self, opt, args): def ValidateOptions(self, opt: optparse.Values, args: List[str]) -> None:
if opt.reference: if opt.reference:
opt.reference = os.path.expanduser(opt.reference) opt.reference = os.path.expanduser(opt.reference)
@@ -324,6 +326,11 @@ to update the working directory files.
" be used with --standalone-manifest." " be used with --standalone-manifest."
) )
if opt.standalone_manifest and opt.smart_sync_branch is not None:
self.OptionParser.error(
"--smart-sync-branch cannot be used with --standalone-manifest."
)
if opt.manifest_upstream_branch and opt.manifest_branch is None: if opt.manifest_upstream_branch and opt.manifest_branch is None:
self.OptionParser.error( self.OptionParser.error(
"--manifest-upstream-branch cannot be used without " "--manifest-upstream-branch cannot be used without "
+2 -3
View File
@@ -14,11 +14,10 @@
import os import os
from command import Command import command
from command import MirrorSafeCommand
class List(Command, MirrorSafeCommand): class List(command.Command, command.MirrorSafeCommand):
COMMON = True COMMON = True
helpSummary = "List projects and their associated directories" helpSummary = "List projects and their associated directories"
helpUsage = """ helpUsage = """
+2 -2
View File
@@ -18,7 +18,7 @@ import optparse
import os import os
import sys import sys
from command import PagedCommand import command
import repo_logging import repo_logging
@@ -35,7 +35,7 @@ class OutputFormat(enum.Enum):
JSON = enum.auto() JSON = enum.auto()
class Manifest(PagedCommand): class Manifest(command.PagedCommand):
COMMON = False COMMON = False
helpSummary = "Manifest inspection utility" helpSummary = "Manifest inspection utility"
helpUsage = """ helpUsage = """
+2 -2
View File
@@ -15,10 +15,10 @@
import optparse import optparse
import color import color
from command import PagedCommand import command
class Overview(PagedCommand): class Overview(command.PagedCommand):
COMMON = True COMMON = True
helpSummary = "Display overview of unmerged project branches" helpSummary = "Display overview of unmerged project branches"
helpUsage = """ helpUsage = """
+3 -4
View File
@@ -15,17 +15,16 @@
import itertools import itertools
import color import color
from command import DEFAULT_LOCAL_JOBS import command
from command import PagedCommand
class Prune(PagedCommand): class Prune(command.PagedCommand):
COMMON = True COMMON = True
helpSummary = "Prune (delete) already merged topics" helpSummary = "Prune (delete) already merged topics"
helpUsage = """ helpUsage = """
%prog [<project>...] %prog [<project>...]
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
@classmethod @classmethod
def _ExecuteOne(cls, project_idx): def _ExecuteOne(cls, project_idx):
+2 -2
View File
@@ -15,7 +15,7 @@
import sys import sys
import color import color
from command import Command import command
import error import error
import git_command import git_command
from project import Project from project import Project
@@ -43,7 +43,7 @@ def _ResolveOntoManifest(project: Project) -> str:
return project.revisionExpr return project.revisionExpr
class Rebase(Command): class Rebase(command.Command):
COMMON = True COMMON = True
helpSummary = "Rebase local branches on upstream branch" helpSummary = "Rebase local branches on upstream branch"
helpUsage = """ helpUsage = """
+5 -7
View File
@@ -14,12 +14,10 @@
import optparse import optparse
from command import Command import command
from command import MirrorSafeCommand
import error import error
import repo_logging import repo_logging
from subcmds.sync import _PostRepoFetch from subcmds import sync
from subcmds.sync import _PostRepoUpgrade
logger = repo_logging.RepoLogger(__file__) logger = repo_logging.RepoLogger(__file__)
@@ -29,7 +27,7 @@ class SelfupdateError(error.RepoExitError):
"""Exit error for failed selfupdate command.""" """Exit error for failed selfupdate command."""
class Selfupdate(Command, MirrorSafeCommand): class Selfupdate(command.Command, command.MirrorSafeCommand):
COMMON = False COMMON = False
helpSummary = "Update repo to the latest version" helpSummary = "Update repo to the latest version"
helpUsage = """ helpUsage = """
@@ -63,7 +61,7 @@ need to be performed by an end-user.
rp.PreSync() rp.PreSync()
if opt.repo_upgraded: if opt.repo_upgraded:
_PostRepoUpgrade(self.manifest) sync._PostRepoUpgrade(self.manifest)
else: else:
result = rp.Sync_NetworkHalf() result = rp.Sync_NetworkHalf()
@@ -72,4 +70,4 @@ need to be performed by an end-user.
raise SelfupdateError(aggregate_errors=[result.error]) raise SelfupdateError(aggregate_errors=[result.error])
rp.bare_git.gc("--auto") rp.bare_git.gc("--auto")
_PostRepoFetch(rp, repo_verify=opt.repo_verify, verbose=True) sync._PostRepoFetch(rp, repo_verify=opt.repo_verify, verbose=True)
+4 -4
View File
@@ -12,10 +12,10 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from subcmds.sync import Sync from subcmds import sync
class Smartsync(Sync): class Smartsync(sync.Sync):
COMMON = True COMMON = True
helpSummary = "Update working tree to the latest known good revision" helpSummary = "Update working tree to the latest known good revision"
helpUsage = """ helpUsage = """
@@ -26,8 +26,8 @@ The '%prog' command is a shortcut for sync -s.
""" """
def _Options(self, p): def _Options(self, p):
Sync._Options(self, p, show_smart=False) sync.Sync._Options(self, p, show_smart=False)
def Execute(self, opt, args): def Execute(self, opt, args):
opt.smart_sync = True opt.smart_sync = True
Sync.Execute(self, opt, args) sync.Sync.Execute(self, opt, args)
+2 -2
View File
@@ -15,7 +15,7 @@
import sys import sys
import color import color
from command import InteractiveCommand import command
import git_command import git_command
import repo_logging import repo_logging
@@ -31,7 +31,7 @@ class _ProjectList(color.Coloring):
self.help = self.printer("help", fg="red", attr="bold") self.help = self.printer("help", fg="red", attr="bold")
class Stage(InteractiveCommand): class Stage(command.InteractiveCommand):
COMMON = True COMMON = True
helpSummary = "Stage file(s) for commit" helpSummary = "Stage file(s) for commit"
helpUsage = """ helpUsage = """
+5 -6
View File
@@ -13,10 +13,9 @@
# limitations under the License. # limitations under the License.
import functools import functools
from typing import NamedTuple from typing import NamedTuple, Optional
from command import Command import command
from command import DEFAULT_LOCAL_JOBS
import error import error
import git_command import git_command
import git_config import git_config
@@ -29,14 +28,14 @@ logger = repo_logging.RepoLogger(__file__)
class ExecuteOneResult(NamedTuple): class ExecuteOneResult(NamedTuple):
project_idx: int project_idx: int
error: Exception error: Optional[Exception]
class StartError(error.RepoExitError): class StartError(error.RepoExitError):
"""Exit error for failed start command.""" """Exit error for failed start command."""
class Start(Command): class Start(command.Command):
COMMON = True COMMON = True
helpSummary = "Start a new branch for development" helpSummary = "Start a new branch for development"
helpUsage = """ helpUsage = """
@@ -46,7 +45,7 @@ class Start(Command):
'%prog' begins a new branch of development, starting from the '%prog' begins a new branch of development, starting from the
revision specified in the manifest. revision specified in the manifest.
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
def _Options(self, p): def _Options(self, p):
p.add_option( p.add_option(
+3 -4
View File
@@ -18,12 +18,11 @@ import io
import os import os
import color import color
from command import DEFAULT_LOCAL_JOBS import command
from command import PagedCommand
import platform_utils import platform_utils
class Status(PagedCommand): class Status(command.PagedCommand):
COMMON = True COMMON = True
helpSummary = "Show the working tree status" helpSummary = "Show the working tree status"
helpUsage = """ helpUsage = """
@@ -82,7 +81,7 @@ the following meanings:
d: deleted ( in index, not in work tree ) d: deleted ( in index, not in work tree )
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
def _Options(self, p): def _Options(self, p):
p.add_option( p.add_option(
+431 -155
View File
@@ -28,7 +28,7 @@ import subprocess
import sys import sys
import tempfile import tempfile
import time import time
from typing import Dict, List, NamedTuple, Optional, Set, Tuple, Union from typing import Dict, Iterable, List, NamedTuple, Optional, Set, Tuple, Union
import urllib.error import urllib.error
import urllib.parse import urllib.parse
import urllib.request import urllib.request
@@ -39,6 +39,8 @@ import xmlrpc.client
try: try:
import threading as _threading import threading as _threading
except ImportError: except ImportError:
# This module was removed in Python 3.9, but we still support Python 3.6.
# pyrefly: ignore
import dummy_threading as _threading import dummy_threading as _threading
try: try:
@@ -53,17 +55,14 @@ except ImportError:
return (256, 256) return (256, 256)
from command import Command import command
from command import DEFAULT_LOCAL_JOBS
from command import MirrorSafeCommand
from command import WORKER_BATCH_SIZE
import error import error
import event_log import event_log
import git_command import git_command
import git_config import git_config
import git_refs import git_refs
import git_superproject import git_superproject
from hooks import RepoHook import hooks
import manifest_xml import manifest_xml
import platform_utils import platform_utils
import progress import progress
@@ -292,9 +291,58 @@ def _WithoutProjects(
return [p for p in projects if p not in dropped] return [p for p in projects if p not in dropped]
def _FlattenDeleteWorktreeError(err: DeleteWorktreeError) -> List[Exception]:
"""Extract individual exceptions from a `DeleteWorktreeError`."""
flat: List[Exception] = []
if err.args or not err.aggregate_errors:
flat.append(err)
flat.extend(err.aggregate_errors)
return flat
def _ResolveDotGit(dotgit: str) -> Optional[str]:
"""Return the normalized resolved gitdir path for a worktree `.git`."""
if os.path.isfile(dotgit) and not platform_utils.islink(dotgit):
try:
with open(dotgit, encoding="utf-8", errors="replace") as fp:
line = fp.readline()
except OSError:
return None
if not line.startswith("gitdir:"):
return None
target = line.split(":", 1)[1].strip()
if not target:
return None
return os.path.normcase(
os.path.realpath(os.path.join(os.path.dirname(dotgit), target))
)
if os.path.exists(dotgit):
return os.path.normcase(os.path.realpath(dotgit))
return None
def _UnfetchedSubmoduleGitdirPrefixes(
projects: List[Project], norm_obsolete: List[str]
) -> Tuple[str, ...]:
"""Return submodule gitdir prefixes for `sync-s` parents not yet fetched."""
prefixes: List[str] = []
for p in projects:
if not (p.relpath and p.sync_s and p.Exists):
continue
norm_relpath = p.relpath.replace("\\", "/")
prefix = "" if norm_relpath == "." else f"{norm_relpath}/"
if (
not prefix or any(c.startswith(prefix) for c in norm_obsolete)
) and p.GetSubmoduleRevisions() is None:
parent_gitdir = os.path.normcase(os.path.realpath(p.gitdir))
for sub in ("modules", "subprojects"):
prefixes.append(os.path.join(parent_gitdir, sub) + os.sep)
return tuple(prefixes)
def _chunksize(projects: int, jobs: int) -> int: def _chunksize(projects: int, jobs: int) -> int:
"""Calculate chunk size for the given number of projects and jobs.""" """Calculate chunk size for the given number of projects and jobs."""
return min(max(1, projects // jobs), WORKER_BATCH_SIZE) return min(max(1, projects // jobs), command.WORKER_BATCH_SIZE)
class _FetchOneResult(NamedTuple): class _FetchOneResult(NamedTuple):
@@ -447,7 +495,7 @@ class TeeStringIO(io.StringIO):
return ret return ret
class Sync(Command, MirrorSafeCommand): class Sync(command.Command, command.MirrorSafeCommand):
COMMON = True COMMON = True
RESPECT_SMART_SYNC_OVERRIDE = False RESPECT_SMART_SYNC_OVERRIDE = False
MULTI_MANIFEST_SUPPORT = True MULTI_MANIFEST_SUPPORT = True
@@ -592,7 +640,9 @@ later is required to fix a server side protocol bug.
"--jobs or 1). Ignored unless --no-interleaved is set", "--jobs or 1). Ignored unless --no-interleaved is set",
) )
jobs_checkout_default = self._GetHelpForCpuJobCount(DEFAULT_LOCAL_JOBS) jobs_checkout_default = self._GetHelpForCpuJobCount(
command.DEFAULT_LOCAL_JOBS
)
p.add_option( p.add_option(
"--jobs-checkout", "--jobs-checkout",
default=None, default=None,
@@ -605,6 +655,12 @@ later is required to fix a server side protocol bug.
), ),
) )
p.add_option(
"--force-jobs",
action="store_true",
help="force jobs to be the value of -j even if the manifest "
"caps it",
)
p.add_option( p.add_option(
"-f", "-f",
"--force-broken", "--force-broken",
@@ -852,19 +908,7 @@ later is required to fix a server side protocol bug.
action="store_true", action="store_true",
help=optparse.SUPPRESS_HELP, help=optparse.SUPPRESS_HELP,
) )
RepoHook.AddOptionGroup(p, "post-sync") hooks.RepoHook.AddOptionGroup(p, "post-sync")
def _GetBranch(self, manifest_project):
"""Returns the branch name for getting the approved smartsync manifest.
Args:
manifest_project: The manifestProject to query.
"""
b = manifest_project.GetBranch(manifest_project.CurrentBranch)
branch = b.merge
if branch.startswith(git_refs.R_HEADS):
branch = branch[len(git_refs.R_HEADS) :]
return branch
@classmethod @classmethod
def _GetCurrentBranchOnly(cls, opt, manifest): def _GetCurrentBranchOnly(cls, opt, manifest):
@@ -1151,7 +1195,21 @@ later is required to fix a server side protocol bug.
# job at the same time. Then the connection may fail if too many # job at the same time. Then the connection may fail if too many
# connections are pending and execeeded the socket listening backlog, # connections are pending and execeeded the socket listening backlog,
# especially on MacOS. # especially on MacOS.
len(cls.get_parallel_context()["sync_dict"]) context = cls.get_parallel_context()
_ = len(context["sync_dict"])
init_template_cache = context.get("init_template_cache")
if init_template_cache is not None:
Project.SeedInitTemplateCache(init_template_cache)
def _WarmInitTemplateCache(
self, objdirs: Iterable[str], force_sync: Optional[bool]
) -> None:
"""Populate parallel_context with init.templateDir state if needed."""
context = self.get_parallel_context()
if "init_template_cache" in context:
return
if force_sync is True or any(not os.path.exists(d) for d in objdirs):
context["init_template_cache"] = Project.WarmInitTemplateCache()
def _Fetch(self, projects, opt, err_event, ssh_proxy, errors): def _Fetch(self, projects, opt, err_event, ssh_proxy, errors):
ret = True ret = True
@@ -1178,7 +1236,8 @@ later is required to fix a server side protocol bug.
project = projects[result.project_idx] project = projects[result.project_idx]
start = result.start start = result.start
finish = result.finish finish = result.finish
self._fetch_times.Set(project, finish - start) if success:
self._fetch_times.Set(project, finish - start)
self._local_sync_state.SetFetchTime(project) self._local_sync_state.SetFetchTime(project)
self.event_log.AddSync( self.event_log.AddSync(
project, project,
@@ -1222,6 +1281,8 @@ later is required to fix a server side protocol bug.
# pass it as an argument to _FetchProjectList below as # pass it as an argument to _FetchProjectList below as
# multiprocessing is unable to pickle those. # multiprocessing is unable to pickle those.
self.get_parallel_context()["ssh_proxy"] = ssh_proxy self.get_parallel_context()["ssh_proxy"] = ssh_proxy
if jobs > 1:
self._WarmInitTemplateCache(objdir_project_map, opt.force_sync)
sync_progress_thread.start() sync_progress_thread.start()
if not opt.quiet: if not opt.quiet:
@@ -1454,6 +1515,7 @@ later is required to fix a server side protocol bug.
# Check for any errors before running any more tasks. # Check for any errors before running any more tasks.
# ...we'll let existing jobs finish, though. # ...we'll let existing jobs finish, though.
if success: if success:
self._checkout_times.Set(project, finish - start)
self._local_sync_state.SetCheckoutTime(project) self._local_sync_state.SetCheckoutTime(project)
else: else:
ret = False ret = False
@@ -1467,31 +1529,35 @@ later is required to fix a server side protocol bug.
pm.update(msg=project.name) pm.update(msg=project.name)
return ret return ret
for projects in _SafeCheckoutOrder(all_projects): proc_res = True
with self.ParallelContext(): try:
self.get_parallel_context()["projects"] = projects for projects in _SafeCheckoutOrder(all_projects):
proc_res = self.ExecuteInParallel( projects.sort(key=self._checkout_times.Get, reverse=True)
opt.jobs_checkout, with self.ParallelContext():
functools.partial( self.get_parallel_context()["projects"] = projects
self._CheckoutOne, proc_res = self.ExecuteInParallel(
opt.detach_head, opt.jobs_checkout,
opt.force_sync, functools.partial(
opt.force_checkout, self._CheckoutOne,
opt.rebase, opt.detach_head,
opt.verbose, opt.force_sync,
), opt.force_checkout,
range(len(projects)), opt.rebase,
callback=_ProcessResults, opt.verbose,
output=progress.Progress( ),
"Checking out", len(all_projects), quiet=opt.quiet range(len(projects)),
), callback=_ProcessResults,
# Use chunksize=1 to avoid the chance that some workers are output=progress.Progress(
# idle while other workers still have more than one job in "Checking out", len(all_projects), quiet=opt.quiet
# their chunk queue. ),
chunksize=1, # Use chunksize=1 to avoid the chance that some workers
) # are idle while other workers still have more than one
# job in their chunk queue.
self._local_sync_state.Save() chunksize=1,
)
finally:
self._checkout_times.Save()
self._local_sync_state.Save()
return proc_res and not err_results return proc_res and not err_results
def _PrintManifestNotices(self, opt): def _PrintManifestNotices(self, opt):
@@ -1525,22 +1591,20 @@ later is required to fix a server side protocol bug.
if len(projects) == 1: if len(projects) == 1:
return False return False
if len(projects) > 1: if len(projects) > 1:
# Objects are potentially shared with another project. if project.UseAlternates:
return False
# Objects are shared when another project uses the same objdir.
# See the logic in Project.Sync_NetworkHalf regarding UseAlternates. # See the logic in Project.Sync_NetworkHalf regarding UseAlternates.
# - When False, shared projects share (via symlink) # - When False and projects share .repo/project-objects/{NAME}.git
# .repo/project-objects/{PROJECT_NAME}.git as the one-and-only # as the one-and-only objects directory, all objects are precious
# objects directory. All objects are precious, since there is no # since no project has a complete set of refs. Projects with
# project with a complete set of refs. # distinct local .git directories (--use-local-gitdirs) do not
# share objects even when their manifest names match.
# - When True, shared projects share (via info/alternates) # - When True, shared projects share (via info/alternates)
# .repo/project-objects/{PROJECT_NAME}.git as an alternate object # .repo/project-objects/{PROJECT_NAME}.git as an alternate object
# store, which is written only on the first clone of the project, # store, which is written only on the first clone of the project,
# and is not written subsequently. (When Sync_NetworkHalf sees # and is not written subsequently.
# that it exists, it makes sure that the alternates file points return sum(1 for p in projects if p.objdir == project.objdir) > 1
# there, and uses a project-local .git/objects directory for all
# syncs going forward.
# We do not support switching between the options. The environment
# variable is present for testing and migration only.
return not project.UseAlternates
return False return False
@@ -1611,6 +1675,18 @@ later is required to fix a server side protocol bug.
if not opt.auto_gc: if not opt.auto_gc:
# Just repair preciousObjects state, and return. # Just repair preciousObjects state, and return.
for project in projects: for project in projects:
if (
project.manifest.UseLocalGitDirs is True
and not project.parent
and getattr(project, "gitdir", None) == project.objdir
and len(
project.manifest.GetProjectsWithName(
project.name, all_manifests=True
)
)
== 1
):
continue
cls._SetPreciousObjectsState(project, opt) cls._SetPreciousObjectsState(project, opt)
return return
@@ -1874,7 +1950,12 @@ later is required to fix a server side protocol bug.
else: else:
manifest.Unload() manifest.Unload()
def UpdateProjectList(self, opt, manifest): def UpdateProjectList(
self,
opt: optparse.Values,
manifest: manifest_xml.XmlManifest,
skip_paths: Optional[Set[str]] = None,
) -> int:
"""Update the cached projects list for |manifest| """Update the cached projects list for |manifest|
In a multi-manifest checkout, each manifest has its own project.list. In a multi-manifest checkout, each manifest has its own project.list.
@@ -1882,32 +1963,74 @@ later is required to fix a server side protocol bug.
Args: Args:
opt: Program options returned from optparse. See _Options(). opt: Program options returned from optparse. See _Options().
manifest: The manifest to use. manifest: The manifest to use.
skip_paths: Obsolete relpaths that already failed removal in an
earlier pass of the same sync and should be retained without
retrying `DeleteWorktree`; newly failed paths are added to it.
Returns: Returns:
0: success 0 on success.
1: failure
Raises:
DeleteWorktreeError: If one or more obsolete worktrees failed to be
removed.
""" """
new_project_paths = [] projects = self.GetProjects(
for project in self.GetProjects(
None, missing_ok=True, manifest=manifest, all_manifests=False None, missing_ok=True, manifest=manifest, all_manifests=False
): )
if project.relpath: new_project_paths: Set[str] = {p.relpath for p in projects if p.relpath}
new_project_paths.append(project.relpath)
file_name = "project.list" file_name = "project.list"
file_path = os.path.join(manifest.subdir, file_name) file_path = os.path.join(manifest.subdir, file_name)
old_project_paths = [] delete_errors: List[DeleteWorktreeError] = []
if os.path.exists(file_path): if os.path.exists(file_path):
with open(file_path) as fd: with open(file_path) as fd:
old_project_paths = fd.read().split("\n") old_project_paths = fd.read().split("\n")
# In reversed order, so subfolders are deleted before parent folder. # In reversed order, so subfolders are deleted before parent folder.
for path in sorted(old_project_paths, reverse=True): obsolete_paths = sorted(
if not path: (
continue p
if path not in new_project_paths: for p in set(old_project_paths)
if p and p not in new_project_paths
),
key=lambda p: p.replace("\\", "/").split("/"),
reverse=True,
)
if obsolete_paths:
# Submodules are derived from their parent's revision, which
# may not be fetched yet (e.g. a new pinned SHA before
# interleaved sync). Keep existing submodule checkouts under
# such parents until a post-fetch update can tell whether they
# were removed.
norm_obsolete = [p.replace("\\", "/") for p in obsolete_paths]
unresolved_gitdir_prefixes = _UnfetchedSubmoduleGitdirPrefixes(
projects, norm_obsolete
)
retained_obsolete: List[str] = []
for path, norm_path in zip(obsolete_paths, norm_obsolete):
gitdir = os.path.join(manifest.topdir, path, ".git")
norm_prefix = "" if norm_path == "." else f"{norm_path}/"
is_retained_ancestor = any(
not norm_prefix or r.startswith(norm_prefix)
for r in retained_obsolete
)
is_unresolved_submodule = bool(
unresolved_gitdir_prefixes
and (_ResolveDotGit(gitdir) or "").startswith(
unresolved_gitdir_prefixes
)
)
if is_unresolved_submodule:
new_project_paths.add(path)
continue
if (
skip_paths and path in skip_paths
) or is_retained_ancestor:
new_project_paths.add(path)
retained_obsolete.append(norm_path)
continue
# If the path has already been deleted, we don't need to do # If the path has already been deleted, we don't need to do
# it. # it.
gitdir = os.path.join(manifest.topdir, path, ".git")
if os.path.exists(gitdir): if os.path.exists(gitdir):
project = Project( project = Project(
manifest=manifest, manifest=manifest,
@@ -1922,14 +2045,31 @@ later is required to fix a server side protocol bug.
revisionId=None, revisionId=None,
groups=None, groups=None,
) )
project.DeleteWorktree( try:
verbose=opt.verbose, force=opt.force_remove_dirty project.DeleteWorktree(
) verbose=opt.verbose,
force=opt.force_remove_dirty,
)
except DeleteWorktreeError as e:
new_project_paths.add(path)
retained_obsolete.append(norm_path)
if skip_paths is not None:
skip_paths.add(path)
delete_errors.append(e)
new_project_paths.sort()
with open(file_path, "w") as fd: with open(file_path, "w") as fd:
fd.write("\n".join(new_project_paths)) fd.write("\n".join(sorted(new_project_paths)))
fd.write("\n") fd.write("\n")
if len(delete_errors) == 1:
raise delete_errors[0]
if delete_errors:
raise DeleteWorktreeError(
aggregate_errors=[
err
for e in delete_errors
for err in _FlattenDeleteWorktreeError(e)
]
)
return 0 return 0
def UpdateCopyLinkfileList(self, manifest): def UpdateCopyLinkfileList(self, manifest):
@@ -2163,45 +2303,67 @@ later is required to fix a server side protocol bug.
"manifest" "manifest"
) )
# Changes in behavior should update docs/smart-sync.md accordingly.
branch = None
target = None
if opt.smart_sync:
# SYNC_BRANCH overrides the branch sent to the manifest server
# (e.g. to select a specific CI branch that builds the same
# manifest branch). Setting SYNC_BRANCH="" forces the raw
# manifest branch, bypassing repo.<branch>.smartsyncbranch.
if "SYNC_BRANCH" in os.environ:
branch = os.environ["SYNC_BRANCH"] or (
manifest.manifestProject.GetMergeBranch(heads_only=True)
)
else:
manifest_branch = manifest.manifestProject.GetMergeBranch(
heads_only=True
)
if manifest_branch:
cfg_branch = manifest.manifestProject.GetSmartSyncBranch(
manifest_branch
)
branch = cfg_branch or manifest_branch
# Fail before running the helper or contacting the server.
if not branch:
raise SmartSyncError(
"error: cannot smart sync: the manifest checkout is not "
"tracking a remote branch; run `repo init -b <branch>` "
"or set SYNC_BRANCH to fix"
)
if "SYNC_TARGET" in os.environ:
target = os.environ["SYNC_TARGET"]
elif (
"TARGET_PRODUCT" in os.environ
and "TARGET_BUILD_VARIANT" in os.environ
and "TARGET_RELEASE" in os.environ
):
target = (
f"{os.environ['TARGET_PRODUCT']}-"
f"{os.environ['TARGET_RELEASE']}-"
f"{os.environ['TARGET_BUILD_VARIANT']}"
)
elif (
"TARGET_PRODUCT" in os.environ
and "TARGET_BUILD_VARIANT" in os.environ
):
target = (
f"{os.environ['TARGET_PRODUCT']}-"
f"{os.environ['TARGET_BUILD_VARIANT']}"
)
if not opt.quiet: if not opt.quiet:
print("Using manifest server %s" % manifest.manifest_server) print(f"Using manifest server {manifest.manifest_server}")
server_url, transport = self._ResolveManifestServerTransport( server_url, transport = self._ResolveManifestServerTransport(
opt, manifest opt, manifest
) )
# Changes in behavior should update docs/smart-sync.md accordingly.
try: try:
server = xmlrpc.client.Server(server_url, transport=transport) server = xmlrpc.client.Server(server_url, transport=transport)
if opt.smart_sync: if opt.smart_sync:
# SYNC_BRANCH overrides the branch sent to the manifest server assert branch is not None
# (e.g. to select a specific CI branch that builds the same
# manifest branch). An empty value is treated as unset.
branch = os.environ.get("SYNC_BRANCH")
if not branch:
branch = self._GetBranch(manifest.manifestProject)
target = None
if "SYNC_TARGET" in os.environ:
target = os.environ["SYNC_TARGET"]
elif (
"TARGET_PRODUCT" in os.environ
and "TARGET_BUILD_VARIANT" in os.environ
and "TARGET_RELEASE" in os.environ
):
target = "%s-%s-%s" % (
os.environ["TARGET_PRODUCT"],
os.environ["TARGET_RELEASE"],
os.environ["TARGET_BUILD_VARIANT"],
)
elif (
"TARGET_PRODUCT" in os.environ
and "TARGET_BUILD_VARIANT" in os.environ
):
target = "%s-%s" % (
os.environ["TARGET_PRODUCT"],
os.environ["TARGET_BUILD_VARIANT"],
)
success, manifest_str = _GetApprovedManifest( success, manifest_str = _GetApprovedManifest(
server, branch, target, best_effort server, branch, target, best_effort
) )
@@ -2389,7 +2551,7 @@ later is required to fix a server side protocol bug.
if not opt.jobs_network: if not opt.jobs_network:
opt.jobs_network = 1 opt.jobs_network = 1
if not opt.jobs_checkout: if not opt.jobs_checkout:
opt.jobs_checkout = DEFAULT_LOCAL_JOBS opt.jobs_checkout = command.DEFAULT_LOCAL_JOBS
opt.jobs = os.cpu_count() opt.jobs = os.cpu_count()
# Try to stay under user rlimit settings. # Try to stay under user rlimit settings.
@@ -2404,7 +2566,10 @@ later is required to fix a server side protocol bug.
opt.jobs_network = min(opt.jobs_network, jobs_soft_limit) opt.jobs_network = min(opt.jobs_network, jobs_soft_limit)
opt.jobs_checkout = min(opt.jobs_checkout, jobs_soft_limit) opt.jobs_checkout = min(opt.jobs_checkout, jobs_soft_limit)
sync_j_max = mp.manifest.default.sync_j_max or None if not opt.force_jobs and mp.manifest.default.sync_j_max:
sync_j_max = mp.manifest.default.sync_j_max
else:
sync_j_max = None
# Check for shared options. # Check for shared options.
# Prioritize --jobs, then --jobs-network, then --jobs-checkout. # Prioritize --jobs, then --jobs-network, then --jobs-checkout.
@@ -2422,7 +2587,8 @@ later is required to fix a server side protocol bug.
if sync_j_max and value > sync_j_max: if sync_j_max and value > sync_j_max:
if not limit_warned: if not limit_warned:
logger.warning( logger.warning(
"warning: manifest limits %s to %d", "warning: manifest limits %s to %d; "
"use --force-jobs to override",
name, name,
sync_j_max, sync_j_max,
) )
@@ -2456,7 +2622,7 @@ later is required to fix a server side protocol bug.
def _RunPostSyncHook(self, opt, sync_duration_seconds=None): def _RunPostSyncHook(self, opt, sync_duration_seconds=None):
"""Run post-sync hook if configured in manifest <repo-hooks>.""" """Run post-sync hook if configured in manifest <repo-hooks>."""
hook = RepoHook.FromSubcmd( hook = hooks.RepoHook.FromSubcmd(
hook_type="post-sync", hook_type="post-sync",
manifest=self.manifest, manifest=self.manifest,
opt=opt, opt=opt,
@@ -2712,7 +2878,8 @@ later is required to fix a server side protocol bug.
"repo.newprojectcount", str(len(all_projects) - len(existing)) "repo.newprojectcount", str(len(all_projects) - len(existing))
) )
self._fetch_times = _FetchTimes(manifest) self._fetch_times = _SyncTimes(manifest, ".repo_fetchtimes.json")
self._checkout_times = _SyncTimes(manifest, ".repo_checkouttimes.json")
self._local_sync_state = LocalSyncState(manifest) self._local_sync_state = LocalSyncState(manifest)
self._bloated_projects = [] self._bloated_projects = []
@@ -2835,6 +3002,92 @@ later is required to fix a server side protocol bug.
return _threading.Thread(target=_monitor_loop, daemon=True) return _threading.Thread(target=_monitor_loop, daemon=True)
def _UpdateProjectLists(
self,
opt: optparse.Values,
err_event: multiprocessing.Event,
errors: List[Exception],
skip_by_manifest: Optional[Dict[str, Set[str]]] = None,
) -> bool:
"""Updates project lists for all manifests.
This removes the worktrees of projects no longer in the manifests.
Args:
opt: Program options from optparse.
err_event: An event to set if any error occurs.
errors: A list to append any encountered exceptions to.
skip_by_manifest: Optional mapping from `manifest.subdir` to
obsolete relpaths that already failed removal in an earlier
pass and should not be retried.
Returns:
True if an error occurred while updating project lists.
"""
if opt.network_only:
return False
err_update_projects = False
for m in self.ManifestList(opt):
if m.IsMirror or m.IsArchive:
continue
skip = (
skip_by_manifest.setdefault(m.subdir, set())
if skip_by_manifest is not None
else None
)
try:
self.UpdateProjectList(opt, m, skip_paths=skip)
except Exception as e:
err_event.set()
err_update_projects = True
if isinstance(e, DeleteWorktreeError):
errors.extend(_FlattenDeleteWorktreeError(e))
else:
errors.append(e)
if opt.fail_fast:
logger.error("error: Local checkouts *not* updated.")
raise SyncFailFastError(aggregate_errors=errors)
return err_update_projects
def _UpdateCopyLinkfileLists(
self,
opt: optparse.Values,
err_event: multiprocessing.Event,
errors: List[Exception],
) -> bool:
"""Updates copy/link files for all manifests.
Args:
opt: Program options from optparse.
err_event: An event to set if any error occurs.
errors: A list to append any encountered exceptions to.
Returns:
True if an error occurred while updating copy/link files.
"""
if opt.network_only:
return False
err_update_linkfiles = False
for m in self.ManifestList(opt):
if m.IsMirror or m.IsArchive:
continue
try:
self.UpdateCopyLinkfileList(m)
except Exception as e:
err_event.set()
err_update_linkfiles = True
errors.append(e)
if opt.fail_fast:
logger.error(
"error: Local update copyfile or linkfile failed."
)
raise SyncFailFastError(aggregate_errors=errors)
return err_update_linkfiles
def _UpdateManifestLists( def _UpdateManifestLists(
self, self,
opt: optparse.Values, opt: optparse.Values,
@@ -2852,35 +3105,10 @@ later is required to fix a server side protocol bug.
A tuple (err_update_projects, err_update_linkfiles) indicating A tuple (err_update_projects, err_update_linkfiles) indicating
an error for each task. an error for each task.
""" """
err_update_projects = False err_update_projects = self._UpdateProjectLists(opt, err_event, errors)
err_update_linkfiles = False err_update_linkfiles = self._UpdateCopyLinkfileLists(
for m in self.ManifestList(opt): opt, err_event, errors
if m.IsMirror or m.IsArchive: )
continue
try:
self.UpdateProjectList(opt, m)
except Exception as e:
err_event.set()
err_update_projects = True
errors.append(e)
if isinstance(e, DeleteWorktreeError):
errors.extend(e.aggregate_errors)
if opt.fail_fast:
logger.error("error: Local checkouts *not* updated.")
raise SyncFailFastError(aggregate_errors=errors)
try:
self.UpdateCopyLinkfileList(m)
except Exception as e:
err_event.set()
err_update_linkfiles = True
errors.append(e)
if opt.fail_fast:
logger.error(
"error: Local update copyfile or linkfile failed."
)
raise SyncFailFastError(aggregate_errors=errors)
return err_update_projects, err_update_linkfiles return err_update_projects, err_update_linkfiles
def _ReportErrors( def _ReportErrors(
@@ -3185,6 +3413,15 @@ later is required to fix a server side protocol bug.
return _InterleavedSyncResult(results=results) return _InterleavedSyncResult(results=results)
def _GetSyncTime(self, opt: optparse.Values, project: Project) -> float:
"""Estimate sync duration of |project| for the phases in |opt|."""
total = 0.0
if not opt.local_only:
total += self._fetch_times.Get(project)
if not opt.network_only and project.worktree:
total += self._checkout_times.Get(project)
return total
def _ProcessSyncInterleavedResults( def _ProcessSyncInterleavedResults(
self, self,
finished_relpaths: Set[str], finished_relpaths: Set[str],
@@ -3207,11 +3444,11 @@ later is required to fix a server side protocol bug.
if result.stderr_text and (opt.verbose or not success): if result.stderr_text and (opt.verbose or not success):
pm.display_message(result.stderr_text) pm.display_message(result.stderr_text)
if result.fetch_start: if result.fetch_start is not None:
self._fetch_times.Set( if result.fetch_success:
project, self._fetch_times.Set(
result.fetch_finish - result.fetch_start, project, result.fetch_finish - result.fetch_start
) )
self._local_sync_state.SetFetchTime(project) self._local_sync_state.SetFetchTime(project)
self.event_log.AddSync( self.event_log.AddSync(
project, project,
@@ -3220,8 +3457,12 @@ later is required to fix a server side protocol bug.
result.fetch_finish, result.fetch_finish,
result.fetch_success, result.fetch_success,
) )
if result.checkout_start: if result.checkout_start is not None:
if result.checkout_success: if result.checkout_success:
self._checkout_times.Set(
project,
result.checkout_finish - result.checkout_start,
)
self._local_sync_state.SetCheckoutTime(project) self._local_sync_state.SetCheckoutTime(project)
self.event_log.AddSync( self.event_log.AddSync(
project, project,
@@ -3272,11 +3513,16 @@ later is required to fix a server side protocol bug.
checkout operations back-to-back. These independent tasks are run in checkout operations back-to-back. These independent tasks are run in
parallel. parallel.
It respects two constraints for correctness: It respects three constraints for correctness:
1. Projects in nested directories (e.g. 'foo' and 'foo/bar') are 1. Projects in nested directories (e.g. 'foo' and 'foo/bar') are
processed in hierarchical order. processed in hierarchical order.
2. Projects that share git objects are processed serially to prevent 2. Projects that share git objects are processed serially to prevent
race conditions. race conditions.
3. Projects removed from the manifest are deleted before syncing
begins, as a new project may be nested inside an obsolete worktree
(e.g. 'foo/bar' replacing 'foo'). Project lists are updated again
after syncing to remove derived submodules dropped when reloading
the manifest.
""" """
# Temporary state for tracking errors in interleaved mode. # Temporary state for tracking errors in interleaved mode.
self._interleaved_err_network = False self._interleaved_err_network = False
@@ -3289,6 +3535,10 @@ later is required to fix a server side protocol bug.
_RelPath = lambda p: p.RelPath(local=opt.this_manifest_only) _RelPath = lambda p: p.RelPath(local=opt.this_manifest_only)
err_event = multiprocessing.Event() err_event = multiprocessing.Event()
skip_by_manifest: Dict[str, Set[str]] = {}
err_update_projects = self._UpdateProjectLists(
opt, err_event, errors, skip_by_manifest
)
finished_relpaths = set() finished_relpaths = set()
project_list = list(all_projects) project_list = list(all_projects)
pm = progress.Progress( pm = progress.Progress(
@@ -3375,6 +3625,13 @@ later is required to fix a server side protocol bug.
if not level_projects: if not level_projects:
continue continue
level_projects.sort(
key=functools.partial(
self._GetSyncTime, opt
),
reverse=True,
)
objdir_project_map = collections.defaultdict( objdir_project_map = collections.defaultdict(
list list
) )
@@ -3388,6 +3645,10 @@ later is required to fix a server side protocol bug.
continue continue
jobs = max(1, min(opt.jobs, len(work_items))) jobs = max(1, min(opt.jobs, len(work_items)))
if jobs > 1:
self._WarmInitTemplateCache(
objdir_project_map, opt.force_sync
)
callback = functools.partial( callback = functools.partial(
self._ProcessSyncInterleavedResults, self._ProcessSyncInterleavedResults,
finished_relpaths, finished_relpaths,
@@ -3428,11 +3689,20 @@ later is required to fix a server side protocol bug.
sync_progress_thread.join() sync_progress_thread.join()
finally: finally:
self._fetch_times.Save() self._fetch_times.Save()
self._checkout_times.Save()
self._local_sync_state.Save() self._local_sync_state.Save()
pm.end() pm.end()
err_update_projects, err_update_linkfiles = self._UpdateManifestLists( # Derived submodules discovered or dropped when the manifest was
# reloaded during syncing are only known now. Pass |skip_by_manifest|
# so obsolete worktrees that already failed removal pre-sync are not
# reported twice.
err_update_projects = (
self._UpdateProjectLists(opt, err_event, errors, skip_by_manifest)
or err_update_projects
)
err_update_linkfiles = self._UpdateCopyLinkfileLists(
opt, err_event, errors opt, err_event, errors
) )
if not self.outer_client.manifest.IsArchive: if not self.outer_client.manifest.IsArchive:
@@ -3505,11 +3775,17 @@ def _PostRepoFetch(rp, repo_verify=True, verbose=False):
) )
class _FetchTimes: class _SyncTimes:
"""Moving average of per-project durations for slowest-first ordering."""
_ALPHA = 0.5 _ALPHA = 0.5
def __init__(self, manifest): def __init__(
self._path = os.path.join(manifest.repodir, ".repo_fetchtimes.json") self,
manifest: manifest_xml.XmlManifest,
filename: str = ".repo_fetchtimes.json",
) -> None:
self._path = os.path.join(manifest.repodir, filename)
self._saved = None self._saved = None
self._seen = {} self._seen = {}
+8 -9
View File
@@ -19,16 +19,15 @@ import re
import sys import sys
from typing import List from typing import List
from command import DEFAULT_LOCAL_JOBS import command
from command import InteractiveCommand
import editor import editor
import error import error
import git_refs import git_refs
import git_superproject import git_superproject
from hooks import RepoHook import hooks
from project import ReviewableBranch from project import ReviewableBranch
import repo_logging import repo_logging
from subcmds.sync import LocalSyncState from subcmds import sync
_DEFAULT_UNUSUAL_COMMIT_THRESHOLD = 5 _DEFAULT_UNUSUAL_COMMIT_THRESHOLD = 5
@@ -104,7 +103,7 @@ def _SplitEmails(values):
return result return result
class Upload(InteractiveCommand): class Upload(command.InteractiveCommand):
COMMON = True COMMON = True
helpSummary = "Upload changes for code review" helpSummary = "Upload changes for code review"
helpUsage = """ helpUsage = """
@@ -211,7 +210,7 @@ threshold to a different value.
Gerrit Code Review: https://www.gerritcodereview.com/ Gerrit Code Review: https://www.gerritcodereview.com/
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
def _Options(self, p): def _Options(self, p):
p.add_option( p.add_option(
@@ -382,7 +381,7 @@ Gerrit Code Review: https://www.gerritcodereview.com/
default=True, default=True,
help="disable verifying ssl certs (unsafe)", help="disable verifying ssl certs (unsafe)",
) )
RepoHook.AddOptionGroup(p, "pre-upload", allow_fix=True) hooks.RepoHook.AddOptionGroup(p, "pre-upload", allow_fix=True)
def _SingleBranch(self, opt, branch, people): def _SingleBranch(self, opt, branch, people):
project = branch.project project = branch.project
@@ -828,7 +827,7 @@ Gerrit Code Review: https://www.gerritcodereview.com/
for (project, available) in pending for (project, available) in pending
if project.manifest.topdir == manifest.topdir if project.manifest.topdir == manifest.topdir
] ]
hook = RepoHook.FromSubcmd( hook = hooks.RepoHook.FromSubcmd(
hook_type="pre-upload", hook_type="pre-upload",
manifest=manifest, manifest=manifest,
opt=opt, opt=opt,
@@ -837,7 +836,7 @@ Gerrit Code Review: https://www.gerritcodereview.com/
if not hook.Run( if not hook.Run(
project_list=pending_proj_names, worktree_list=pending_worktrees project_list=pending_proj_names, worktree_list=pending_worktrees
): ):
if LocalSyncState(manifest).IsPartiallySynced(): if sync.LocalSyncState(manifest).IsPartiallySynced():
logger.info( logger.info(
"Tip: A partially synced tree was detected. " "Tip: A partially synced tree was detected. "
"If this failure involves cross-project dependencies, " "If this failure involves cross-project dependencies, "
+2 -3
View File
@@ -16,14 +16,13 @@ import platform
import sys import sys
from typing import Any, Tuple from typing import Any, Tuple
from command import Command import command
from command import MirrorSafeCommand
import git_command import git_command
import git_refs import git_refs
import wrapper import wrapper
class Version(Command, MirrorSafeCommand): class Version(command.Command, command.MirrorSafeCommand):
wrapper_version = None wrapper_version = None
wrapper_path = None wrapper_path = None
+3 -3
View File
@@ -16,7 +16,7 @@ import os
import sys import sys
from typing import List from typing import List
from command import Command import command
import error import error
import platform_utils import platform_utils
from project import DeleteWorktreeError from project import DeleteWorktreeError
@@ -26,7 +26,7 @@ class Error(error.RepoExitError):
"""Exit error when wipe command fails.""" """Exit error when wipe command fails."""
class Wipe(Command): class Wipe(command.Command):
"""Delete projects from the worktree and .repo""" """Delete projects from the worktree and .repo"""
COMMON = True COMMON = True
@@ -98,7 +98,7 @@ Examples:
) )
try: try:
if project.HasChanges(): if project.HasChanges:
uncommitted_projects.append(project.name) uncommitted_projects.append(project.name)
except error.GitError: except error.GitError:
uncommitted_projects.append(f"{project.name} (corrupted)") uncommitted_projects.append(f"{project.name} (corrupted)")
+6
View File
@@ -166,6 +166,12 @@ def clean_environ() -> Dict[str, str]:
"pythonwarnings", "pythonwarnings",
# LUCI's git wrapper replaces HOME with this. # LUCI's git wrapper replaces HOME with this.
"infra_git_wrapper_home", "infra_git_wrapper_home",
# Smart sync reads these (e.g. set by Android's envsetup/lunch).
"sync_branch",
"sync_target",
"target_product",
"target_release",
"target_build_variant",
): ):
continue continue
env[key] = value env[key] = value
+2
View File
@@ -28,6 +28,7 @@ def _http_request(
) -> Tuple[int, bytes]: ) -> Tuple[int, bytes]:
"""Send a request to `server` & return the response status & body.""" """Send a request to `server` & return the response status & body."""
url = urllib.parse.urlsplit(server.url) url = urllib.parse.urlsplit(server.url)
assert url.hostname is not None
conn = http.client.HTTPConnection(url.hostname, url.port, timeout=30) conn = http.client.HTTPConnection(url.hostname, url.port, timeout=30)
try: try:
conn.request(method, path) conn.request(method, path)
@@ -113,6 +114,7 @@ def test_http_post_body(
if expect_continue: if expect_continue:
headers["Expect"] = "100-continue" headers["Expect"] = "100-continue"
url = urllib.parse.urlsplit(git_server.url) url = urllib.parse.urlsplit(git_server.url)
assert url.hostname is not None
conn = http.client.HTTPConnection(url.hostname, url.port, timeout=30) conn = http.client.HTTPConnection(url.hostname, url.port, timeout=30)
try: try:
# http.client skips any interim `100 Continue` response. # http.client skips any interim `100 Continue` response.
+118 -1
View File
@@ -14,8 +14,11 @@
"""E2E tests for the core `repo init` & `repo sync` options.""" """E2E tests for the core `repo init` & `repo sync` options."""
import contextlib
from pathlib import Path from pathlib import Path
from typing import List, Set import threading
from typing import Any, Callable, Dict, Iterator, List, Set
import xmlrpc.server
from harness import GIT_DEFAULTS_PARAMS from harness import GIT_DEFAULTS_PARAMS
from harness import GitHost from harness import GitHost
@@ -353,3 +356,117 @@ def test_manifest_depth(host: GitHost, client: RepoClient) -> None:
assert client.is_shallow(manifests) assert client.is_shallow(manifests)
assert not client.is_shallow("platform/foo") assert not client.is_shallow("platform/foo")
assert client.git("platform/foo", "rev-list", "--count", "HEAD") == "2" assert client.git("platform/foo", "rev-list", "--count", "HEAD") == "2"
@contextlib.contextmanager
def _manifest_server(
handler: Callable[[Any], Dict[str, Any]],
) -> Iterator[str]:
"""Run a local XML-RPC manifest server exposing GetApprovedManifestV2."""
server = xmlrpc.server.SimpleXMLRPCServer(
("127.0.0.1", 0), logRequests=False
)
server.register_function(handler, "GetApprovedManifestV2")
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
addr, port = server.server_address[:2]
yield f"http://{addr}:{port}/"
finally:
server.shutdown()
server.server_close()
thread.join()
@pytest.mark.parametrize("git_defaults", GIT_DEFAULTS_PARAMS)
def test_smart_sync_branch(host: GitHost, client: RepoClient) -> None:
"""`init --smart-sync-branch` persists per-manifest-branch server branch."""
foo = host.create_repo("platform/foo", {"f": "v1"})
v1 = foo.rev_parse("main")
warmup = foo.commit({"f": "warmup"})
foo.commit({"f": "tot"})
r16 = foo.commit({"f": "r16"}, branch="release/16")
requests: List[Dict[str, Any]] = []
by_branch = {
"main": v1,
"git_main-dev-warmup-abfs": warmup,
"release/16": r16,
}
def handle_v2(req: Dict[str, Any]) -> Dict[str, Any]:
requests.append(req)
rev = by_branch[req["branch"]]
return {
"success": True,
"manifest_xml": manifest_xml(
f'<project name="platform/foo" path="foo" revision="{rev}" />',
fetch=host.url,
),
}
with _manifest_server(handle_v2) as server_url:
m_xml = manifest_xml(
f'<manifest-server url="{server_url}" />',
'<project name="platform/foo" path="foo" />',
)
manifest = host.create_manifest(m_xml)
manifest.commit({"default.xml": m_xml}, branch="release/16")
client.init(
host.manifest_url,
"-s",
"--smart-sync-branch=git_main-dev-warmup-abfs",
)
assert (
client.config(".repo/manifests", "repo.main.smartsyncbranch")
== "git_main-dev-warmup-abfs"
)
client.sync()
assert requests[-1]["branch"] == "git_main-dev-warmup-abfs"
assert client.head("foo") == warmup
# SYNC_BRANCH="" forces the raw manifest branch over checkout config.
saved_env = dict(client.env)
client.env["SYNC_BRANCH"] = ""
try:
client.sync()
finally:
client.env = saved_env
assert requests[-1]["branch"] == "main"
assert client.head("foo") == v1
# Switching manifest branches does not leak main's override.
client.init(host.manifest_url, "-b", "release/16")
client.sync()
assert requests[-1]["branch"] == "release/16"
assert client.head("foo") == r16
# Switching back to main restores main's configured override.
client.init(host.manifest_url, "-b", "main")
client.sync()
assert requests[-1]["branch"] == "git_main-dev-warmup-abfs"
assert client.head("foo") == warmup
# Passing an empty --smart-sync-branch clears the override.
client.init(host.manifest_url, "--smart-sync-branch=")
assert (
client.config(".repo/manifests", "repo.main.smartsyncbranch")
is None
)
client.sync()
assert requests[-1]["branch"] == "main"
assert client.head("foo") == v1
# Non-empty --smart-sync-branch requires tracking a remote branch.
manifest_sha = manifest.rev_parse("main")
res = client.init(
host.manifest_url,
"-b",
manifest_sha,
"--smart-sync-branch=git_main-dev-warmup-abfs",
check=False,
)
assert res.returncode != 0
assert "requires the manifest checkout to track a branch" in res.stderr
+132
View File
@@ -245,6 +245,93 @@ def test_nested_projects(host: GitHost, client: RepoClient) -> None:
assert (client.top / "foo/f").read_text() == "foo" assert (client.top / "foo/f").read_text() == "foo"
@pytest.mark.parametrize(
"sync_args", [[], ["--no-interleaved"]], ids=["interleaved", "phased"]
)
def test_project_replaced_by_nested_project(
host: GitHost, client: RepoClient, sync_args: List[str]
) -> None:
"""Replacing a parent project with a nested child project removes parent.
Regression test for b/574048485: in interleaved mode, checking out the new
child project `foo/bar` before deleting the obsolete parent worktree `foo`
made `foo` appear dirty (`git status` in `foo` saw `foo/bar`'s files as
modifications and untracked files under `bar/`).
"""
host.create_repo(
"platform/foo",
{
"root.txt": "parent",
"bar/shared/version.txt": "v1",
"bar/old-version/file.txt": "old",
},
)
host.create_repo(
"platform/foo_bar",
{
"shared/version.txt": "v2",
"new-version/file.txt": "new",
},
)
host.create_manifest(
manifest_xml('<project name="platform/foo" path="foo" />')
)
client.init(host.manifest_url)
client.sync(*sync_args)
assert (client.top / "foo/root.txt").read_text() == "parent"
assert (client.top / "foo/bar/shared/version.txt").read_text() == "v1"
host.update_manifest(
manifest_xml('<project name="platform/foo_bar" path="foo/bar" />')
)
client.sync(*sync_args)
assert client.list_projects() == {"foo/bar": "platform/foo_bar"}
assert (client.repodir / "project.list").read_text() == "foo/bar\n"
assert not (client.top / "foo/.git").exists()
assert not (client.top / "foo/root.txt").exists()
assert not (client.top / "foo/bar/old-version").exists()
assert (client.top / "foo/bar/shared/version.txt").read_text() == "v2"
assert (client.top / "foo/bar/new-version/file.txt").read_text() == "new"
assert client.git("foo/bar", "status", "--porcelain") == ""
@pytest.mark.parametrize(
"sync_args", [[], ["--no-interleaved"]], ids=["interleaved", "phased"]
)
def test_nested_project_merged_into_parent(
host: GitHost, client: RepoClient, sync_args: List[str]
) -> None:
"""A nested project whose files move to its parent is removed first.
Otherwise the parent's new files clash with the removed project's
checkout. The reverse of test_project_replaced_by_nested_project.
"""
foo = host.create_repo("platform/foo", {"f": "foo"})
host.create_repo("platform/bar", {"shared/version.txt": "v1"})
host.create_manifest(
manifest_xml(
'<project name="platform/foo" path="foo" />',
'<project name="platform/bar" path="foo/bar" />',
)
)
client.init(host.manifest_url)
client.sync(*sync_args)
latest = foo.commit({"bar/shared/version.txt": "v2"})
host.update_manifest(
manifest_xml('<project name="platform/foo" path="foo" />')
)
client.sync(*sync_args)
assert client.list_projects() == {"foo": "platform/foo"}
assert client.head("foo") == latest
assert not os.path.lexists(client.top / "foo/bar/.git")
assert (client.top / "foo/bar/shared/version.txt").read_text() == "v2"
assert client.git("foo", "status", "--porcelain") == ""
def test_copyfile_linkfile(host: GitHost, client: RepoClient) -> None: def test_copyfile_linkfile(host: GitHost, client: RepoClient) -> None:
"""copyfile and linkfile are created, updated and cleaned up. """copyfile and linkfile are created, updated and cleaned up.
@@ -330,6 +417,51 @@ def test_submodules(
assert (client.top / path / "sub/s").read_text() == "2" assert (client.top / path / "sub/s").read_text() == "2"
@pytest.mark.parametrize(
"sync_args", [[], ["--no-interleaved"]], ids=["interleaved", "phased"]
)
def test_pinned_parent_keeps_unfetched_submodule(
host: GitHost, client: RepoClient, sync_args: List[str]
) -> None:
"""A sync-s parent moving to an unfetched SHA keeps its submodule."""
sub = host.create_repo("platform/sub", {"s": "1"})
host.create_repo("platform/bar", {"b": "old"})
gitmodules = '[submodule "sub"]\n\tpath = sub\n\turl = ../sub\n'
foo = host.create_repo(
"platform/foo",
{".gitmodules": gitmodules, "sub": Gitlink(sub.rev_parse("main"))},
)
def manifest_for(rev: str, include_bar: bool = False) -> str:
extra = (
'\n <project name="platform/bar" path="foo/bar" />'
if include_bar
else ""
)
return manifest_xml(
f'<project name="platform/foo" path="foo" sync-s="true" '
f'revision="{rev}" />{extra}'
)
host.create_manifest(manifest_for(foo.rev_parse("main"), include_bar=True))
client.init(host.manifest_url)
client.sync(*sync_args)
(client.top / "foo/sub/s").write_text("dirty")
latest = foo.commit({"bar/b": "from-foo"})
host.update_manifest(manifest_for(latest))
client.sync("-v", *sync_args)
assert client.head("foo") == latest
assert (client.top / "foo/sub/s").read_text() == "dirty"
assert (client.top / "foo/bar/b").read_text() == "from-foo"
assert not (client.top / "foo/bar/.git").exists()
expected_sub = os.path.join("foo", "sub")
assert (
client.repodir / "project.list"
).read_text() == f"foo\n{expected_sub}\n"
# The default layout reads HEAD from the ref cache instead of GetHead(). # The default layout reads HEAD from the ref cache instead of GetHead().
@pytest.mark.parametrize("init_args", [[], ["--worktree"]]) @pytest.mark.parametrize("init_args", [[], ["--worktree"]])
def test_prune_detached_head( def test_prune_detached_head(
+111 -6
View File
@@ -275,7 +275,49 @@ def test_force_sync(host: GitHost, client: RepoClient) -> None:
assert client.git("foo", "status", "--porcelain") == "" assert client.git("foo", "status", "--porcelain") == ""
def test_force_remove_dirty(host: GitHost, client: RepoClient) -> None: @pytest.mark.parametrize("mode_args", SYNC_MODES, ids=["interleaved", "phased"])
def test_network_only_keeps_removed_project(
host: GitHost, client: RepoClient, mode_args: List[str]
) -> None:
"""`sync -n` leaves projects dropped from the manifest to `sync -l`.
Removing their checkouts and copyfiles changes the worktree, which `-n`
must not do.
"""
host.create_repo("platform/foo", {"f": "foo"})
host.create_repo("platform/bar", {"f": "bar"})
host.create_manifest(
manifest_xml(
'<project name="platform/foo" path="foo" />',
'<project name="platform/bar" path="bar">'
'<copyfile src="f" dest="bar-copy" />'
"</project>",
)
)
client.init(host.manifest_url)
client.sync(*mode_args)
host.update_manifest(
manifest_xml('<project name="platform/foo" path="foo" />')
)
assert client.sync("-n", *mode_args).stderr == ""
assert (client.top / "bar/f").read_text() == "bar"
assert (client.top / "bar-copy").read_text() == "bar"
with host.watch_requests() as requests, host.offline():
client.sync("-l", *mode_args)
assert requests == []
assert not (client.top / "bar").exists()
assert not (client.top / "bar-copy").exists()
assert client.list_projects() == {"foo": "platform/foo"}
@pytest.mark.parametrize("mode_args", SYNC_MODES, ids=["interleaved", "phased"])
def test_force_remove_dirty(
host: GitHost, client: RepoClient, mode_args: List[str]
) -> None:
"""Removed projects with uncommitted changes need `--force-remove-dirty`.""" """Removed projects with uncommitted changes need `--force-remove-dirty`."""
host.create_repo("platform/foo", {"f": "foo"}) host.create_repo("platform/foo", {"f": "foo"})
host.create_repo("platform/bar", {"f": "bar"}) host.create_repo("platform/bar", {"f": "bar"})
@@ -286,7 +328,7 @@ def test_force_remove_dirty(host: GitHost, client: RepoClient) -> None:
) )
) )
client.init(host.manifest_url) client.init(host.manifest_url)
client.sync() client.sync(*mode_args)
(client.top / "bar/f").write_text("dirty") (client.top / "bar/f").write_text("dirty")
host.update_manifest( host.update_manifest(
manifest_xml('<project name="platform/foo" path="foo" />') manifest_xml('<project name="platform/foo" path="foo" />')
@@ -295,15 +337,21 @@ def test_force_remove_dirty(host: GitHost, client: RepoClient) -> None:
# Repeat to check that the project stays tracked in project.list (rather # Repeat to check that the project stays tracked in project.list (rather
# than forgotten & left behind) after the first failure. # than forgotten & left behind) after the first failure.
for _ in range(2): for _ in range(2):
result = client.sync(check=False) result = client.sync(*mode_args, check=False)
assert result.returncode != 0 assert result.returncode != 0
# Logged once when removal fails and once in the final error summary:
# removal is not attempted (and reported) a second time in the same
# sync.
assert ( assert (
"bar: Cannot remove project: uncommitted changes are present" result.stderr.count(
in result.stderr "bar: Cannot remove project: uncommitted changes are present"
)
== 2
) )
assert result.stderr.count("Updating local project lists failed") == 1
assert (client.top / "bar/f").read_text() == "dirty" assert (client.top / "bar/f").read_text() == "dirty"
result = client.sync("--force-remove-dirty") result = client.sync("--force-remove-dirty", *mode_args)
assert "bar: Removing dirty project" in result.stderr assert "bar: Removing dirty project" in result.stderr
assert not (client.top / "bar").exists() assert not (client.top / "bar").exists()
assert client.list_projects() == {"foo": "platform/foo"} assert client.list_projects() == {"foo": "platform/foo"}
@@ -337,3 +385,60 @@ def test_detach(host: GitHost, client: RepoClient) -> None:
# A regular sync afterwards stays detached. # A regular sync afterwards stays detached.
client.sync() client.sync()
assert client.is_detached("foo") assert client.is_detached("foo")
@pytest.mark.parametrize("mode", ["detached", "commit", "tag"])
def test_smart_sync_without_tracked_branch(
host: GitHost, client: RepoClient, mode: str
) -> None:
"""Smart sync requires `.repo/manifests` to track a remote branch.
When `sync-smartsync="true"` is set in the manifest, plain `repo sync`
falls back to a regular sync without contacting the manifest server, while
an explicit `repo sync -s` fails early with an actionable error
(b/580004259).
"""
foo = host.create_repo("platform/foo", {"f": "foo"})
manifest = host.create_manifest(
manifest_xml(
f'<manifest-server url="{host.url}/manifestserver" />',
'<project name="platform/foo" path="foo" />',
default='remote="origin" revision="main" sync-smartsync="true"',
)
)
pinned = manifest.rev_parse("main")
manifest.tag("v1.0", pinned)
# Detached `.repo/manifests` falls back to `init.defaultBranch` (`master`
# under Git 2 defaults) when updating the manifest repo during `repo sync`.
manifest.set_ref("refs/heads/master", pinned)
if mode == "detached":
client.init(host.manifest_url)
client.git(".repo/manifests", "checkout", "-q", "--detach")
assert client.is_detached(".repo/manifests")
elif mode == "commit":
client.init(host.manifest_url, "-b", pinned)
assert (
client.config(".repo/manifests", "branch.default.merge") == pinned
)
else:
client.init(host.manifest_url, "-b", "refs/tags/v1.0")
assert (
client.config(".repo/manifests", "branch.default.merge")
== "refs/tags/v1.0"
)
with host.watch_requests() as requests:
explicit = client.sync("-s", check=False)
assert explicit.returncode != 0
assert "not tracking a remote branch" in explicit.stderr
assert "Using manifest server" not in explicit.output
assert requests == []
with host.watch_requests() as requests:
implicit = client.sync()
assert implicit.stderr == ""
assert "Using manifest server" not in implicit.output
assert not [r for r in requests if "manifestserver" in r.path]
assert client.head("foo") == foo.rev_parse("main")
assert (client.top / "foo/f").read_text() == "foo"
+5 -3
View File
@@ -14,6 +14,7 @@
"""Unittests for the color.py module.""" """Unittests for the color.py module."""
import os
from unittest import mock from unittest import mock
import pytest import pytest
@@ -21,6 +22,7 @@ import utils_for_test
import color import color
import git_config import git_config
import pager
@pytest.fixture @pytest.fixture
@@ -147,8 +149,8 @@ class TestColoringInit:
pager_active: bool, pager_active: bool,
expected: bool, expected: bool,
) -> None: ) -> None:
with mock.patch("os.isatty", return_value=isatty), mock.patch( with mock.patch.object(
"pager.active", pager_active os, "isatty", return_value=isatty
): ), mock.patch.object(pager, "active", pager_active):
c = _make_coloring(state) c = _make_coloring(state)
assert c.is_on is expected assert c.is_on is expected
+8 -8
View File
@@ -18,7 +18,7 @@ from typing import Iterable, List, Optional
import pytest import pytest
from command import Command import command
class FakeProject: class FakeProject:
@@ -136,7 +136,7 @@ def test_get_projects_keeps_derived_subprojects_for_repeated_repo():
sync_s=True, sync_s=True,
) )
manifest = FakeManifest([project_a, project_b]) manifest = FakeManifest([project_a, project_b])
cmd = Command(manifest=manifest) cmd = command.Command(manifest=manifest)
projects = cmd.GetProjects([]) projects = cmd.GetProjects([])
@@ -165,7 +165,7 @@ def test_get_projects_submodule_override(
derived_subprojects=[submodule], derived_subprojects=[submodule],
sync_s=sync_s, sync_s=sync_s,
) )
cmd = Command(manifest=FakeManifest([project])) cmd = command.Command(manifest=FakeManifest([project]))
projects = cmd.GetProjects([], submodules_ok=submodules_ok) projects = cmd.GetProjects([], submodules_ok=submodules_ok)
@@ -214,7 +214,7 @@ def test_get_projects_uses_groups_from_each_manifest_unless_overridden(
all_projects=[outer_project, *child_manifest.projects], all_projects=[outer_project, *child_manifest.projects],
effective_groups="outer-group", effective_groups="outer-group",
) )
cmd = Command(manifest=outer_manifest) cmd = command.Command(manifest=outer_manifest)
projects = cmd.GetProjects( projects = cmd.GetProjects(
[], [],
@@ -248,7 +248,7 @@ def test_get_projects_by_name_uses_groups_from_each_manifest() -> None:
all_projects=[outer_project, *child_manifest.projects], all_projects=[outer_project, *child_manifest.projects],
effective_groups="outer-group", effective_groups="outer-group",
) )
cmd = Command(manifest=outer_manifest) cmd = command.Command(manifest=outer_manifest)
projects = cmd.GetProjects( projects = cmd.GetProjects(
["shared"], ["shared"],
@@ -296,7 +296,7 @@ def test_find_projects_uses_groups_from_each_manifest() -> None:
) )
outer_manifest.outer_client = outer_manifest outer_manifest.outer_client = outer_manifest
cmd = Command(manifest=outer_manifest) cmd = command.Command(manifest=outer_manifest)
projects = cmd.FindProjects(["match"], all_manifests=True) projects = cmd.FindProjects(["match"], all_manifests=True)
@@ -322,7 +322,7 @@ def test_find_projects_uses_explicit_groups() -> None:
[default_project, override_project], [default_project, override_project],
effective_groups="default-group", effective_groups="default-group",
) )
cmd = Command(manifest=manifest) cmd = command.Command(manifest=manifest)
projects = cmd.FindProjects( projects = cmd.FindProjects(
["override"], ["override"],
@@ -339,7 +339,7 @@ def test_find_projects_allows_missing_projects() -> None:
"missing", "missing",
exists=False, exists=False,
) )
cmd = Command(manifest=FakeManifest([project])) cmd = command.Command(manifest=FakeManifest([project]))
projects = cmd.FindProjects( projects = cmd.FindProjects(
["missing"], ["missing"],
+3 -1
View File
@@ -14,13 +14,15 @@
"""Unittests for the editor.py module.""" """Unittests for the editor.py module."""
from typing import Iterator
import pytest import pytest
import editor import editor
@pytest.fixture(autouse=True) @pytest.fixture(autouse=True)
def reset_editor() -> None: def reset_editor() -> Iterator[None]:
"""Take care of resetting Editor state across tests.""" """Take care of resetting Editor state across tests."""
editor.Editor._editor = None editor.Editor._editor = None
yield yield
+2 -2
View File
@@ -25,10 +25,10 @@ import error
import fetch import fetch
import git_command import git_command
import project import project
from subcmds import all_modules import subcmds
_IMPORTS = all_modules + [ _IMPORTS = subcmds.all_modules + [
error, error,
project, project,
git_command, git_command,
+8 -61
View File
@@ -18,6 +18,7 @@ import io
import os import os
import re import re
import subprocess import subprocess
import sys
import unittest import unittest
from unittest import mock from unittest import mock
@@ -65,60 +66,6 @@ class GitCommandTest(unittest.TestCase):
) )
class GitCommandWaitTest(unittest.TestCase):
"""Tests the GitCommand class .Wait()"""
def setUp(self):
class MockPopen:
rc = 0
def __init__(self):
self.stdout = io.BufferedReader(io.BytesIO())
self.stderr = io.BufferedReader(io.BytesIO())
def communicate(
self, input: str = None, timeout: float = None
) -> [str, str]:
"""Mock communicate fn."""
return ["", ""]
def wait(self, timeout=None):
return self.rc
self.popen = popen = MockPopen()
def popen_mock(*args, **kwargs):
return popen
def realpath_mock(val):
return val
mock.patch.object(subprocess, "Popen", side_effect=popen_mock).start()
mock.patch.object(
os.path, "realpath", side_effect=realpath_mock
).start()
def tearDown(self):
mock.patch.stopall()
def test_raises_when_verify_non_zero_result(self):
self.popen.rc = 1
r = git_command.GitCommand(None, ["status"], verify_command=True)
with self.assertRaises(git_command.GitCommandError):
r.Wait()
def test_returns_when_no_verify_non_zero_result(self):
self.popen.rc = 1
r = git_command.GitCommand(None, ["status"], verify_command=False)
self.assertEqual(1, r.Wait())
def test_default_returns_non_zero_result(self):
self.popen.rc = 1
r = git_command.GitCommand(None, ["status"])
self.assertEqual(1, r.Wait())
class GitCommandStreamLogsTest(unittest.TestCase): class GitCommandStreamLogsTest(unittest.TestCase):
"""Tests the GitCommand class stderr log streaming cases.""" """Tests the GitCommand class stderr log streaming cases."""
@@ -130,7 +77,7 @@ class GitCommandStreamLogsTest(unittest.TestCase):
self.mock_popen = mock.MagicMock() self.mock_popen = mock.MagicMock()
self.mock_popen.return_value = self.mock_process self.mock_popen.return_value = self.mock_process
mock.patch("subprocess.Popen", self.mock_popen).start() mock.patch.object(subprocess, "Popen", self.mock_popen).start()
def tearDown(self): def tearDown(self):
mock.patch.stopall() mock.patch.stopall()
@@ -199,7 +146,7 @@ class GitCommandStreamLogsTest(unittest.TestCase):
self.mock_process.communicate.assert_called_once_with(input=None) self.mock_process.communicate.assert_called_once_with(input=None)
self.mock_process.stderr.read1.assert_not_called() self.mock_process.stderr.read1.assert_not_called()
@mock.patch("sys.stderr") @mock.patch.object(sys, "stderr")
def test_streams_stderr_when_no_stream_is_set(self, mock_stderr): def test_streams_stderr_when_no_stream_is_set(self, mock_stderr):
mock_stderr.isatty.return_value = False mock_stderr.isatty.return_value = False
logs = "\n".join( logs = "\n".join(
@@ -229,7 +176,7 @@ class GitCommandStreamLogsTest(unittest.TestCase):
mock_stderr.write.assert_called_once_with(logs) mock_stderr.write.assert_called_once_with(logs)
self.assertEqual(cmd.stderr, logs) self.assertEqual(cmd.stderr, logs)
@mock.patch("sys.stderr") @mock.patch.object(sys, "stderr")
def test_push_injects_progress_only_when_tty_and_not_suppressed( def test_push_injects_progress_only_when_tty_and_not_suppressed(
self, mock_stderr: mock.MagicMock self, mock_stderr: mock.MagicMock
) -> None: ) -> None:
@@ -240,7 +187,7 @@ class GitCommandStreamLogsTest(unittest.TestCase):
cmd = self.mock_popen.call_args[0][0] cmd = self.mock_popen.call_args[0][0]
self.assertEqual(cmd, ["git", "push", "--progress"]) self.assertEqual(cmd, ["git", "push", "--progress"])
@mock.patch("sys.stderr") @mock.patch.object(sys, "stderr")
def test_push_does_not_inject_progress_when_no_progress_set( def test_push_does_not_inject_progress_when_no_progress_set(
self, mock_stderr: mock.MagicMock self, mock_stderr: mock.MagicMock
) -> None: ) -> None:
@@ -263,7 +210,7 @@ class GitCommandCaptureBytesTest(unittest.TestCase):
self.mock_popen = mock.MagicMock() self.mock_popen = mock.MagicMock()
self.mock_popen.return_value = self.mock_process self.mock_popen.return_value = self.mock_process
mock.patch("subprocess.Popen", self.mock_popen).start() mock.patch.object(subprocess, "Popen", self.mock_popen).start()
def tearDown(self) -> None: def tearDown(self) -> None:
mock.patch.stopall() mock.patch.stopall()
@@ -497,13 +444,13 @@ class GitRequireTests(unittest.TestCase):
"""Test fatal require calls with old versions.""" """Test fatal require calls with old versions."""
with self.assertRaises(git_command.GitRequireError) as e: with self.assertRaises(git_command.GitRequireError) as e:
git_command.git_require((2,), fail=True) git_command.git_require((2,), fail=True)
self.assertNotEqual(0, e.code) self.assertNotEqual(0, e.exception.exit_code)
def test_older_fatal_msg(self): def test_older_fatal_msg(self):
"""Test fatal require calls with old versions and message.""" """Test fatal require calls with old versions and message."""
with self.assertRaises(git_command.GitRequireError) as e: with self.assertRaises(git_command.GitRequireError) as e:
git_command.git_require((2,), fail=True, msg="so sad") git_command.git_require((2,), fail=True, msg="so sad")
self.assertNotEqual(0, e.code) self.assertNotEqual(0, e.exception.exit_code)
class GitCommandErrorTest(unittest.TestCase): class GitCommandErrorTest(unittest.TestCase):
+232 -1
View File
@@ -17,7 +17,8 @@
import json import json
import os import os
from pathlib import Path from pathlib import Path
from typing import Any, Optional import types
from typing import Any, List, Optional, Tuple
import pytest import pytest
import utils_for_test import utils_for_test
@@ -214,6 +215,9 @@ def test_get_sync_analysis_state_data(rw_config_file: Path) -> None:
class Options: class Options:
"""Container for testing.""" """Container for testing."""
verbose: str
mp_update: str
options = Options() options = Options()
options.verbose = "true" options.verbose = "true"
options.mp_update = "false" options.mp_update = "false"
@@ -560,3 +564,230 @@ def test_set_string_after_external_edit_preserves_external_keys(
assert reader2.GetString("other.key") == "ext2" assert reader2.GetString("other.key") == "ext2"
assert reader2.GetString("user.name") == "Alice" assert reader2.GetString("user.name") == "Alice"
assert reader2.GetString("user.email") == "alice@example.com" assert reader2.GetString("user.email") == "alice@example.com"
def test_batch_writes_coalesces_new_keys_without_git_replace(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Test BatchWrites writes new sections atomically with one JSON save."""
cfg_path = tmp_path / "config"
cfg_path.write_text("[core]\n\trepositoryformatversion = 0\n")
config = git_config.GitConfig(str(cfg_path))
orig_do = config._do
seen_do = []
save_count = 0
orig_save = config._SaveJson
def spy_do(*args: str) -> str:
seen_do.append(args)
return orig_do(*args)
def spy_save(cache: Any) -> None:
nonlocal save_count
save_count += 1
orig_save(cache)
monkeypatch.setattr(config, "_do", spy_do)
monkeypatch.setattr(config, "_SaveJson", spy_save)
with config.BatchWrites():
config.SetString("filter.lfs.smudge", "git-lfs smudge --skip -- %f")
config.SetString("filter.lfs.process", "git-lfs filter-process --skip")
config.SetString("remote.goog.url", "https://example.com/platform/art")
config.SetString(
"remote.goog.fetch",
[
"+refs/heads/*:refs/remotes/goog/*",
"+refs/tags/*:refs/tags/*",
],
)
config.SetBoolean("maintenance.auto", False)
assert seen_do == [("--null", "--list")]
assert save_count == 1
assert not (tmp_path / "config.lock").exists()
reloaded = git_config.GitConfig(str(cfg_path))
fresh = reloaded._ReadGit()
assert fresh["filter.lfs.smudge"] == ["git-lfs smudge --skip -- %f"]
assert fresh["filter.lfs.process"] == ["git-lfs filter-process --skip"]
assert fresh["remote.goog.url"] == ["https://example.com/platform/art"]
assert fresh["remote.goog.fetch"] == [
"+refs/heads/*:refs/remotes/goog/*",
"+refs/tags/*:refs/tags/*",
]
assert fresh["maintenance.auto"] == ["false"]
def test_batch_writes_falls_back_on_lock_or_external_edit(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Test BatchWrites preserves concurrent edits and respects config.lock."""
# 1. External edit between cache load and flush falls back to git config CLI
# so neither the external key nor the batched key is lost.
cfg_path = tmp_path / "config"
cfg_path.write_text("[core]\n\tfilemode = true\n")
config = git_config.GitConfig(str(cfg_path))
with config.BatchWrites():
config.SetString("remote.origin.url", "https://example.com/a")
with open(cfg_path, "a", encoding="utf-8") as fp:
fp.write("[external]\n\tadded = yes\n")
reloaded = git_config.GitConfig(str(cfg_path))
fresh = reloaded._ReadGit()
assert fresh["external.added"] == ["yes"]
assert fresh["remote.origin.url"] == ["https://example.com/a"]
assert config.GetString("external.added") == "yes"
# 2. Pre-existing config.lock is never clobbered; BatchWrites delegates to
# _do instead.
locked_path = tmp_path / "locked_config"
locked_path.write_text("[core]\n\tfilemode = true\n")
lock_file = tmp_path / "locked_config.lock"
lock_file.write_text("held-by-other-process")
locked_cfg = git_config.GitConfig(str(locked_path))
_ = locked_cfg.GetBoolean("core.filemode")
do_calls = []
monkeypatch.setattr(
locked_cfg,
"_do",
lambda *args: do_calls.append(args) or "core.filemode\ntrue\0",
)
with locked_cfg.BatchWrites():
locked_cfg.SetString("remote.origin.url", "https://example.com/b")
assert lock_file.read_text() == "held-by-other-process"
assert ("--replace-all", "remote.origin.url", "https://example.com/b") in (
do_calls
)
def test_batch_writes_direct_do_refreshes_fingerprint_and_branch_save_batches(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Direct _do in BatchWrites updates fingerprint; Save uses BatchWrites."""
cfg_path = tmp_path / "config"
cfg_path.write_text("[core]\n\tfilemode = true\n")
config = git_config.GitConfig(str(cfg_path))
_ = config.GetBoolean("core.filemode")
orig_do = config._do
seen_do = []
def spy_do(*args: str, **kwargs: Any) -> str:
seen_do.append(args)
return orig_do(*args, **kwargs)
monkeypatch.setattr(config, "_do", spy_do)
# Updating an existing key runs _do() directly inside BatchWrites; the next
# SetString must not see a stale _cache_fingerprint or re-run --list.
with config.BatchWrites():
config.SetString("core.filemode", "false")
config.SetString("new.section.key", "val")
assert seen_do == [("--replace-all", "core.filemode", "false")]
assert config.GetString("core.filemode") == "false"
assert config.GetString("new.section.key") == "val"
# Remote.Save() and Branch.Save() coalesce new keys via BatchWrites without
# spawning git config --replace-all subprocesses.
seen_do.clear()
remote = config.GetRemote("origin")
remote.url = "https://example.com/repo"
remote.ResetFetch()
remote.Save()
branch = config.GetBranch("main")
branch.remote = remote
branch.merge = "refs/heads/main"
branch.Save()
assert seen_do == []
# When include.path is present, _CanQueueInBatch returns False so writes go
# through git config immediately instead of queueing for lockfile append.
inc_file = tmp_path / "inc.gitconfig"
inc_file.write_text("[extra]\n\tflag = 1\n")
config.SetString("include.path", str(inc_file))
_ = config.GetString("extra.flag")
seen_do.clear()
with config.BatchWrites():
config.SetString("after.include", "yes")
assert ("--replace-all", "after.include", "yes") in seen_do
def test_batch_writes_failure_paths(tmp_path: Path) -> None:
"""BatchWrites clears state on failure and preserves body errors."""
# 1. Failed fallback flush clears poisoned in-memory cache and dirty flag.
fail_path = tmp_path / "fail_config"
fail_path.write_text("[core]\n\tfilemode = true\n")
lock_path = tmp_path / "fail_config.lock"
lock_path.write_text("locked")
fail_cfg = git_config.GitConfig(str(fail_path))
assert fail_cfg.GetBoolean("core.filemode") is True
with pytest.raises(error.GitError):
with fail_cfg.BatchWrites():
fail_cfg.SetString("remote.origin.url", "https://bad.example.com")
lock_path.unlink()
assert fail_cfg.GetString("remote.origin.url") is None
with fail_cfg.BatchWrites():
pass
assert (
git_config.GitConfig(str(fail_path)).GetString("remote.origin.url")
is None
)
# 2. If body raises and flush also fails, original body error propagates.
body_path = tmp_path / "body_config"
body_path.write_text("[core]\n\tfilemode = true\n")
body_lock = tmp_path / "body_config.lock"
body_lock.write_text("locked")
body_cfg = git_config.GitConfig(str(body_path))
assert body_cfg.GetBoolean("core.filemode") is True
with pytest.raises(RuntimeError, match="body boom"):
with body_cfg.BatchWrites():
body_cfg.SetString("remote.origin.url", "https://bad.example.com")
raise RuntimeError("body boom")
body_lock.unlink()
assert body_cfg.GetString("remote.origin.url") is None
def test_batch_writes_sync_analysis_state(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""UpdateSyncAnalysisState coalesces new syncstate keys without CLI."""
state_cfg_path = tmp_path / "state_config"
state_cfg_path.write_text(
'[remote "origin"]\n\turl = https://example.com/repo\n',
encoding="utf-8",
)
state_cfg = git_config.GitConfig(str(state_cfg_path))
_ = state_cfg.GetString("remote.origin.url")
seen_do: List[Tuple[str, ...]] = []
orig_do = state_cfg._do
monkeypatch.setattr(
state_cfg,
"_do",
lambda *args, **kw: seen_do.append(args) or orig_do(*args, **kw),
)
opts = types.SimpleNamespace(jobs=8, quiet=True)
state_cfg.UpdateSyncAnalysisState(opts, {"superproject": False})
assert seen_do == []
fresh_state = git_config.GitConfig(str(state_cfg_path))._ReadGit()
assert fresh_state["repo.syncstate.main.version"] == ["1"]
assert fresh_state["repo.syncstate.options.jobs"] == ["8"]
assert fresh_state["repo.syncstate.options.quiet"] == ["true"]
assert fresh_state["repo.syncstate.remote.origin.url"] == [
"https://example.com/repo"
]
+338
View File
@@ -16,7 +16,9 @@
import os import os
from pathlib import Path from pathlib import Path
import stat
import subprocess import subprocess
import time
from typing import Any, List from typing import Any, List
from unittest import mock from unittest import mock
@@ -25,6 +27,7 @@ import utils_for_test
import git_command import git_command
import git_refs import git_refs
import platform_utils
def _run(repo: str, *args: str) -> str: def _run(repo: str, *args: str) -> str:
@@ -138,6 +141,7 @@ def test_root_ref_snapshot_falls_back_for_unborn_head(
def read_head(name: str) -> None: def read_head(name: str) -> None:
assert name == "HEAD" assert name == "HEAD"
assert refs._symref is not None
refs._symref[name] = "refs/heads/main" refs._symref[name] = "refs/heads/main"
monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head) monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head)
@@ -168,6 +172,7 @@ def test_old_git_keeps_separate_head_fallback(
def read_head(name: str) -> None: def read_head(name: str) -> None:
assert name == "HEAD" assert name == "HEAD"
assert refs._symref is not None
refs._symref[name] = "refs/heads/main" refs._symref[name] = "refs/heads/main"
monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head) monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head)
@@ -201,6 +206,7 @@ def test_for_each_ref_failure_falls_back_to_symbolic_ref(
def read_head(name: str) -> None: def read_head(name: str) -> None:
assert name == "HEAD" assert name == "HEAD"
assert refs._symref is not None
refs._symref[name] = "refs/heads/main" refs._symref[name] = "refs/heads/main"
monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head) monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head)
@@ -406,3 +412,335 @@ def test_update_symref_propagates_chained_symrefs(tmp_path: Path) -> None:
assert refs.get("refs/remotes/m/main") == c2 assert refs.get("refs/remotes/m/main") == c2
assert refs.get("refs/remotes/m/alias") == c2 assert refs.get("refs/remotes/m/alias") == c2
assert refs.get("HEAD") == c2 assert refs.get("HEAD") == c2
def test_files_backend_reads_loose_and_packed_without_git(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Files backend reads packed-refs, loose refs, and symrefs in Python."""
repo = _init_repo(tmp_path, reftable=False)
branch = _run(repo, "symbolic-ref", "--short", "HEAD")
c1 = _run(repo, "rev-parse", "HEAD")
_run(repo, "tag", "-a", "-m", "v1", "v1.0")
tag_oid = _run(repo, "rev-parse", "refs/tags/v1.0")
_run(repo, "branch", "packed-only", c1)
_run(repo, "branch", "overridden", c1)
_run(repo, "pack-refs", "--all")
Path(os.path.join(repo, "a")).write_text("2", encoding="utf-8")
_run(repo, "add", "a")
_run(repo, "commit", "-q", "-m", "second")
c2 = _run(repo, "rev-parse", "HEAD")
_run(repo, "update-ref", "refs/heads/overridden", c2)
_run(repo, "symbolic-ref", "refs/remotes/m/main", "refs/heads/overridden")
monkeypatch.delenv("GIT_DIR", raising=False)
monkeypatch.delenv("GIT_COMMON_DIR", raising=False)
seen_cmds: List[List[str]] = []
class SpyGitCommand(git_command.GitCommand):
def __init__(
self,
proj: Any,
cmdv: List[str],
*args: Any,
**kwargs: Any,
) -> None:
seen_cmds.append(list(cmdv))
super().__init__(proj, cmdv, *args, **kwargs)
monkeypatch.setattr(git_command, "GitCommand", SpyGitCommand)
refs = git_refs.GitRefs(os.path.join(repo, ".git"))
assert refs.get("HEAD") == c2
assert refs.is_files_authoritative
with mock.patch.object(
refs,
"_LoadAll",
side_effect=AssertionError("expected warm GitRefs snapshot"),
):
assert refs.get(f"refs/heads/{branch}") == c2
assert refs.get("refs/heads/packed-only") == c1
assert refs.get("refs/heads/overridden") == c2
assert refs.get("refs/remotes/m/main") == c2
assert refs.get("refs/tags/v1.0") == tag_oid
assert seen_cmds == []
# A concurrent .lock file triggers clean fallback to git for-each-ref.
lock_file = Path(repo, ".git", "refs", "heads", "packed-only.lock")
lock_file.write_text("", encoding="utf-8")
refs2 = git_refs.GitRefs(os.path.join(repo, ".git"))
assert refs2.get("refs/heads/packed-only") == c1
assert not refs2.is_files_authoritative
assert any(cmd and cmd[0] == "for-each-ref" for cmd in seen_cmds)
def test_try_resolve_ref_loose_packed_symref_and_missing(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""TryResolveRef resolves loose/packed/symref refs without full scan."""
repo = _init_repo(tmp_path, reftable=False)
c1 = _run(repo, "rev-parse", "HEAD")
_run(repo, "branch", "packed-branch", c1)
_run(repo, "pack-refs", "--all")
Path(os.path.join(repo, "a")).write_text("2", encoding="utf-8")
_run(repo, "add", "a")
_run(repo, "commit", "-q", "-m", "second")
c2 = _run(repo, "rev-parse", "HEAD")
_run(repo, "update-ref", "refs/remotes/origin/main", c2)
_run(
repo,
"symbolic-ref",
"refs/remotes/m/main",
"refs/remotes/origin/main",
)
monkeypatch.delenv("GIT_DIR", raising=False)
monkeypatch.delenv("GIT_COMMON_DIR", raising=False)
gitdir = os.path.join(repo, ".git")
refs = git_refs.GitRefs(gitdir)
assert refs.TryResolveRef("refs/remotes/origin/main") == c2
assert refs.TryResolveRef("refs/heads/packed-branch") == c1
assert refs.TryResolveRef("refs/remotes/m/main") == c2
assert refs.TryResolveRef("refs/remotes/origin/missing") == ""
assert not refs.is_loaded
# A .lock file forces None so callers fall back to git.
lock_file = Path(gitdir, "refs", "remotes", "origin", "main.lock")
lock_file.write_text("", encoding="utf-8")
assert refs.TryResolveRef("refs/remotes/origin/main") is None
def test_detects_new_packed_refs_and_same_mtime_replace(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""GitRefs reloads on newly created packed-refs and same-mtime replaces."""
monkeypatch.delenv("GIT_DIR", raising=False)
monkeypatch.delenv("GIT_COMMON_DIR", raising=False)
repo = os.path.join(tmp_path, "fresh")
utils_for_test.init_git_tree(repo, ref_format="files")
gitdir = os.path.join(repo, ".git")
refs = git_refs.GitRefs(gitdir)
assert refs.all == {}
c1 = "1" * 40
c2 = "2" * 40
Path(gitdir, "packed-refs").write_text(
f"# pack-refs with: peeled fully-peeled sorted \n"
f"{c1} refs/remotes/origin/main\n",
encoding="utf-8",
)
assert refs.get("refs/remotes/origin/main") == c1
loose = Path(gitdir, "refs", "remotes", "origin", "topic")
loose.parent.mkdir(parents=True, exist_ok=True)
loose.write_text(c1 + "\n", encoding="utf-8")
assert refs.get("refs/remotes/origin/topic") == c1
# Replace the ref the way git does, by renaming a new file over it, then
# put the mtimes back so that only the new inode shows.
file_st = os.stat(loose)
dir_st = os.stat(loose.parent)
replacement = Path(gitdir, "topic.new")
replacement.write_text(c2 + "\n", encoding="utf-8")
os.replace(replacement, loose)
os.utime(loose, ns=(file_st.st_atime_ns, file_st.st_mtime_ns))
os.utime(loose.parent, ns=(dir_st.st_atime_ns, dir_st.st_mtime_ns))
assert refs.get("refs/remotes/origin/topic") == c2
@pytest.mark.skipif(
platform_utils.isWindows(),
reason="st_ctime is the creation time on Windows",
)
def test_ctime_only_change_keeps_snapshot_warm(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A ref file whose ctime alone moves does not invalidate the snapshot."""
monkeypatch.delenv("GIT_DIR", raising=False)
monkeypatch.delenv("GIT_COMMON_DIR", raising=False)
repo = _init_repo(tmp_path, reftable=False)
branch = _run(repo, "symbolic-ref", "--short", "HEAD")
head = _run(repo, "rev-parse", "HEAD")
gitdir = os.path.join(repo, ".git")
refs = git_refs.GitRefs(gitdir)
assert refs.get(f"refs/heads/{branch}") == head
# Backup agents and file system crawlers tag files with xattrs, which moves
# ctime and nothing else. Flipping a mode bit and putting it back does the
# same; repeat it until the timestamp granularity lets the change show.
loose = os.path.join(gitdir, "refs", "heads", branch)
before = os.lstat(loose)
mode = stat.S_IMODE(before.st_mode)
for _ in range(1000):
os.chmod(loose, mode ^ stat.S_IXUSR)
os.chmod(loose, mode)
if os.lstat(loose).st_ctime_ns != before.st_ctime_ns:
break
time.sleep(0.001)
else:
pytest.fail("chmod did not move ctime")
with mock.patch.object(
refs,
"_LoadAll",
side_effect=AssertionError("expected warm GitRefs snapshot"),
):
assert refs.get(f"refs/heads/{branch}") == head
@pytest.mark.parametrize("reftable", [False, True])
def test_update_ref_keeps_snapshot_warm(
tmp_path: Path, reftable: bool, monkeypatch: pytest.MonkeyPatch
) -> None:
"""UpdateRef stays warm on files backend and reloads on reftable."""
if reftable and not utils_for_test.supports_reftable():
pytest.skip("reftable not supported")
monkeypatch.delenv("GIT_DIR", raising=False)
monkeypatch.delenv("GIT_COMMON_DIR", raising=False)
repo = _init_repo(tmp_path, reftable=reftable)
branch = _run(repo, "symbolic-ref", "--short", "HEAD")
c1 = _run(repo, "rev-parse", "HEAD")
_run(repo, "symbolic-ref", "refs/remotes/m/main", f"refs/heads/{branch}")
Path(os.path.join(repo, "a")).write_text("2", encoding="utf-8")
_run(repo, "add", "a")
_run(repo, "commit", "-q", "-m", "second")
c2 = _run(repo, "rev-parse", "HEAD")
refs = git_refs.GitRefs(os.path.join(repo, ".git"))
assert refs.symref("refs/remotes/m/main") == f"refs/heads/{branch}"
assert refs.get("refs/remotes/m/main") == c2
_run(repo, "update-ref", "--no-deref", "refs/remotes/m/main", c1)
refs.UpdateRef("refs/remotes/m/main")
if reftable:
assert not refs.is_loaded
assert refs.symref("refs/remotes/m/main") == ""
assert refs.get("refs/remotes/m/main") == c1
else:
with mock.patch.object(
refs,
"_LoadAll",
side_effect=AssertionError("expected warm GitRefs snapshot"),
):
assert refs.symref("refs/remotes/m/main") == ""
assert refs.get("refs/remotes/m/main") == c1
# Setting GIT_COMMON_DIR invalidates instead of reading self._gitdir.
monkeypatch.setenv("GIT_COMMON_DIR", os.path.join(repo, ".git"))
assert refs.TryResolveRef("refs/remotes/m/main") is None
refs.UpdateRef("refs/remotes/m/main")
assert not refs.is_loaded
def test_symlinked_ref_fallback_reloads_and_breaks_cycles(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Symlink fallback reloads on target changes and breaks dir cycles."""
monkeypatch.delenv("GIT_DIR", raising=False)
monkeypatch.delenv("GIT_COMMON_DIR", raising=False)
repo = _init_repo(tmp_path, reftable=False)
c1 = _run(repo, "rev-parse", "HEAD")
Path(os.path.join(repo, "a")).write_text("2", encoding="utf-8")
_run(repo, "add", "a")
_run(repo, "commit", "-q", "-m", "second")
c2 = _run(repo, "rev-parse", "HEAD")
gitdir = os.path.join(repo, ".git")
ext_dir = tmp_path / "ext_refs"
ext_dir.mkdir()
target_ref = ext_dir / "linked"
target_ref.write_text(c1 + "\n", encoding="utf-8")
link_ref = os.path.join(gitdir, "refs", "heads", "linked")
loop_dir = os.path.join(gitdir, "refs", "heads", "loop")
refs_root = os.path.join(gitdir, "refs")
used_junction = False
try:
os.symlink(str(target_ref), link_ref)
os.symlink(refs_root, loop_dir, target_is_directory=True)
tracked_ref = "refs/heads/linked"
except OSError:
if not platform_utils.isWindows():
raise
used_junction = True
ext_link_dir = os.path.join(gitdir, "refs", "remotes", "ext")
os.makedirs(os.path.dirname(ext_link_dir), exist_ok=True)
subprocess.run(
["cmd", "/c", "mklink", "/J", ext_link_dir, str(ext_dir)],
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
subprocess.run(
["cmd", "/c", "mklink", "/J", loop_dir, refs_root],
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
tracked_ref = "refs/remotes/ext/linked"
try:
# Remove loop_dir before running git for-each-ref, then verify cycle
# handling directly on _TrackTreeMtimes so git itself does not walk
# deep cyclic paths.
if used_junction:
os.rmdir(loop_dir)
else:
os.unlink(loop_dir)
refs = git_refs.GitRefs(gitdir)
assert refs.get(tracked_ref) == c1
assert not refs.is_files_authoritative
replacement = ext_dir / "linked.new"
replacement.write_text(c2 + "\n", encoding="utf-8")
st = os.stat(target_ref)
os.replace(replacement, target_ref)
os.utime(target_ref, ns=(st.st_atime_ns, st.st_mtime_ns))
assert refs.get(tracked_ref) == c2
if used_junction:
subprocess.run(
["cmd", "/c", "mklink", "/J", loop_dir, refs_root],
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
else:
os.symlink(refs_root, loop_dir, target_is_directory=True)
refs._TrackTreeMtimes("refs")
assert os.path.join("refs", "heads", "loop") in refs._mtime
finally:
if used_junction:
for junc in (
loop_dir,
os.path.join(gitdir, "refs", "remotes", "ext"),
):
if os.path.lexists(junc):
os.rmdir(junc)
# Also exercise S_ISLNK target and dangling-link tracking directly so
# unprivileged Windows runs cover POSIX symlink stat transitions too.
lnk = stat.S_IFLNK | 0o777
reg = stat.S_IFREG | 0o644
fake_link = os.stat_result((lnk, 10, 1, 1, 0, 0, 8, 1, 1, 1))
fake_t1 = os.stat_result((reg, 20, 1, 1, 0, 0, 41, 1, 2, 1))
fake_t2 = os.stat_result((reg, 21, 1, 1, 0, 0, 41, 1, 2, 1))
with mock.patch("os.lstat", return_value=fake_link):
with mock.patch("os.stat", return_value=fake_t1):
s_t1 = git_refs.GitRefs._StatPath("sym")
with mock.patch("os.stat", return_value=fake_t2):
s_t2 = git_refs.GitRefs._StatPath("sym")
with mock.patch("os.stat", side_effect=FileNotFoundError):
s_dangling = git_refs.GitRefs._StatPath("sym")
assert s_t1 != s_t2
assert s_t1 != s_dangling
assert s_dangling is not None and not stat.S_ISDIR(s_dangling[0])
+13 -12
View File
@@ -24,6 +24,7 @@ from unittest import mock
import pytest import pytest
import test_manifest_xml import test_manifest_xml
import git_command
import git_superproject import git_superproject
import git_trace2_event_log import git_trace2_event_log
import manifest_xml import manifest_xml
@@ -464,8 +465,8 @@ class SuperprojectTestCase(unittest.TestCase):
def test_Init_success(self): def test_Init_success(self):
"""Test _Init succeeds and creates the work git dir.""" """Test _Init succeeds and creates the work git dir."""
self.assertFalse(os.path.exists(self._superproject._work_git)) self.assertFalse(os.path.exists(self._superproject._work_git))
with mock.patch( with mock.patch.object(
"git_command.GitCommand", autospec=True git_command, "GitCommand", autospec=True
) as mock_git_command: ) as mock_git_command:
instance = mock_git_command.return_value instance = mock_git_command.return_value
instance.Wait.return_value = 0 instance.Wait.return_value = 0
@@ -491,16 +492,16 @@ class SuperprojectTestCase(unittest.TestCase):
"""Test _Init returns early if the work git dir already exists.""" """Test _Init returns early if the work git dir already exists."""
os.mkdir(self._superproject._superproject_path) os.mkdir(self._superproject._superproject_path)
os.mkdir(self._superproject._work_git) os.mkdir(self._superproject._work_git)
with mock.patch( with mock.patch.object(
"git_command.GitCommand", autospec=True git_command, "GitCommand", autospec=True
) as mock_git_command: ) as mock_git_command:
self.assertTrue(self._superproject._Init()) self.assertTrue(self._superproject._Init())
mock_git_command.assert_not_called() mock_git_command.assert_not_called()
def test_Init_failure_cleans_up(self): def test_Init_failure_cleans_up(self):
"""Test _Init cleans up the temporary directory if 'git init' fails.""" """Test _Init cleans up the temporary directory if 'git init' fails."""
with mock.patch( with mock.patch.object(
"git_command.GitCommand", autospec=True git_command, "GitCommand", autospec=True
) as mock_git_command: ) as mock_git_command:
instance = mock_git_command.return_value instance = mock_git_command.return_value
instance.Wait.return_value = 1 instance.Wait.return_value = 1
@@ -539,8 +540,8 @@ class SuperprojectTestCase(unittest.TestCase):
os.mkdir(self._superproject._superproject_path) os.mkdir(self._superproject._superproject_path)
os.mkdir(self._superproject._work_git) os.mkdir(self._superproject._work_git)
with mock.patch.object(self._superproject, "_Init", return_value=True): with mock.patch.object(self._superproject, "_Init", return_value=True):
with mock.patch( with mock.patch.object(
"git_command.GitCommand", autospec=True git_command, "GitCommand", autospec=True
) as mock_git_command: ) as mock_git_command:
with mock.patch.object( with mock.patch.object(
self._superproject, "_GetRef" self._superproject, "_GetRef"
@@ -599,12 +600,12 @@ class SuperprojectTestCase(unittest.TestCase):
def test_GetRef_resolves_only_the_requested_ref(self) -> None: def test_GetRef_resolves_only_the_requested_ref(self) -> None:
command = mock.MagicMock(stdout="1234\n") command = mock.MagicMock(stdout="1234\n")
command.Wait.return_value = 0 command.Wait.return_value = 0
with mock.patch( with mock.patch.object(
"git_command.GitCommand", return_value=command git_command, "GitCommand", return_value=command
) as git_command: ) as mock_git_command:
self.assertEqual("1234", self._superproject._GetRef("HEAD")) self.assertEqual("1234", self._superproject._GetRef("HEAD"))
git_command.assert_called_once_with( mock_git_command.assert_called_once_with(
None, None,
["rev-parse", "--verify", "--quiet", "HEAD"], ["rev-parse", "--verify", "--quiet", "HEAD"],
gitdir=self._superproject._work_git, gitdir=self._superproject._work_git,
+10 -7
View File
@@ -437,7 +437,9 @@ class TestEventLogVerbose:
event_log.verbose = False event_log.verbose = False
with contextlib.redirect_stderr( with contextlib.redirect_stderr(
io.StringIO() io.StringIO()
) as mock_stderr, mock.patch("socket.socket", side_effect=OSError): ) as mock_stderr, mock.patch.object(
socket, "socket", side_effect=OSError
):
event_log.Write(path="af_unix:stream:/tmp/test_sock") event_log.Write(path="af_unix:stream:/tmp/test_sock")
assert mock_stderr.getvalue() == "" assert mock_stderr.getvalue() == ""
@@ -447,8 +449,8 @@ class TestEventLogVerbose:
event_log.verbose = True event_log.verbose = True
with contextlib.redirect_stderr( with contextlib.redirect_stderr(
io.StringIO() io.StringIO()
) as mock_stderr, mock.patch( ) as mock_stderr, mock.patch.object(
"socket.socket", side_effect=OSError("Mock error") socket, "socket", side_effect=OSError("Mock error")
): ):
event_log.Write(path="af_unix:stream:/tmp/test_sock") event_log.Write(path="af_unix:stream:/tmp/test_sock")
assert ( assert (
@@ -462,8 +464,8 @@ class TestEventLogVerbose:
event_log.verbose = False event_log.verbose = False
with contextlib.redirect_stderr( with contextlib.redirect_stderr(
io.StringIO() io.StringIO()
) as mock_stderr, mock.patch( ) as mock_stderr, mock.patch.object(
"tempfile.NamedTemporaryFile", side_effect=FileExistsError tempfile, "NamedTemporaryFile", side_effect=FileExistsError
): ):
event_log.Write(path="/tmp") event_log.Write(path="/tmp")
assert mock_stderr.getvalue() == "" assert mock_stderr.getvalue() == ""
@@ -474,8 +476,9 @@ class TestEventLogVerbose:
event_log.verbose = True event_log.verbose = True
with contextlib.redirect_stderr( with contextlib.redirect_stderr(
io.StringIO() io.StringIO()
) as mock_stderr, mock.patch( ) as mock_stderr, mock.patch.object(
"tempfile.NamedTemporaryFile", tempfile,
"NamedTemporaryFile",
side_effect=FileExistsError("Mock error"), side_effect=FileExistsError("Mock error"),
): ):
event_log.Write(path="/tmp") event_log.Write(path="/tmp")
+15 -12
View File
@@ -14,18 +14,21 @@
"""Tests for the main repo script and subcommand routing.""" """Tests for the main repo script and subcommand routing."""
import builtins
from pathlib import Path from pathlib import Path
import time
from unittest import mock from unittest import mock
import pytest import pytest
import event_log import event_log
from main import _Repo import main
import manifest_xml
@pytest.fixture(name="repo") @pytest.fixture(name="repo")
def fixture_repo(): def fixture_repo():
repo = _Repo("repodir") repo = main._Repo("repodir")
# Overriding the command list here ensures that we are only testing # Overriding the command list here ensures that we are only testing
# against a fixed set of commands, reducing fragility to new # against a fixed set of commands, reducing fragility to new
# subcommands being added to the main repo tool. # subcommands being added to the main repo tool.
@@ -38,7 +41,7 @@ def fixture_mock_config():
return mock.MagicMock() return mock.MagicMock()
@mock.patch("time.sleep") @mock.patch.object(time, "sleep")
def test_autocorrect_delay(mock_sleep, repo, mock_config): def test_autocorrect_delay(mock_sleep, repo, mock_config):
"""Test autocorrect with positive delay.""" """Test autocorrect with positive delay."""
mock_config.GetString.return_value = "10" mock_config.GetString.return_value = "10"
@@ -50,7 +53,7 @@ def test_autocorrect_delay(mock_sleep, repo, mock_config):
assert res == "start" assert res == "start"
@mock.patch("time.sleep") @mock.patch.object(time, "sleep")
def test_autocorrect_delay_one(mock_sleep, repo, mock_config): def test_autocorrect_delay_one(mock_sleep, repo, mock_config):
"""Test autocorrect with '1' (0.1s delay, not immediate).""" """Test autocorrect with '1' (0.1s delay, not immediate)."""
mock_config.GetString.return_value = "1" mock_config.GetString.return_value = "1"
@@ -61,7 +64,7 @@ def test_autocorrect_delay_one(mock_sleep, repo, mock_config):
assert res == "start" assert res == "start"
@mock.patch("time.sleep", side_effect=KeyboardInterrupt()) @mock.patch.object(time, "sleep", side_effect=KeyboardInterrupt())
def test_autocorrect_delay_interrupt(mock_sleep, repo, mock_config): def test_autocorrect_delay_interrupt(mock_sleep, repo, mock_config):
"""Test autocorrect handles KeyboardInterrupt during delay.""" """Test autocorrect handles KeyboardInterrupt during delay."""
mock_config.GetString.return_value = "10" mock_config.GetString.return_value = "10"
@@ -72,7 +75,7 @@ def test_autocorrect_delay_interrupt(mock_sleep, repo, mock_config):
assert res is None assert res is None
@mock.patch("time.sleep") @mock.patch.object(time, "sleep")
def test_autocorrect_immediate(mock_sleep, repo, mock_config): def test_autocorrect_immediate(mock_sleep, repo, mock_config):
"""Test autocorrect with immediate/negative delay.""" """Test autocorrect with immediate/negative delay."""
# Test numeric negative. # Test numeric negative.
@@ -125,7 +128,7 @@ def test_autocorrect_never(repo, mock_config):
assert res is None assert res is None
@mock.patch("builtins.input", return_value="y") @mock.patch.object(builtins, "input", return_value="y")
def test_autocorrect_prompt_yes(mock_input, repo, mock_config): def test_autocorrect_prompt_yes(mock_input, repo, mock_config):
"""Test autocorrect with prompt and user answers yes.""" """Test autocorrect with prompt and user answers yes."""
mock_config.GetString.return_value = "prompt" mock_config.GetString.return_value = "prompt"
@@ -135,7 +138,7 @@ def test_autocorrect_prompt_yes(mock_input, repo, mock_config):
assert res == "start" assert res == "start"
@mock.patch("builtins.input", return_value="n") @mock.patch.object(builtins, "input", return_value="n")
def test_autocorrect_prompt_no(mock_input, repo, mock_config): def test_autocorrect_prompt_no(mock_input, repo, mock_config):
"""Test autocorrect with prompt and user answers no.""" """Test autocorrect with prompt and user answers no."""
mock_config.GetString.return_value = "prompt" mock_config.GetString.return_value = "prompt"
@@ -145,7 +148,7 @@ def test_autocorrect_prompt_no(mock_input, repo, mock_config):
assert res is None assert res is None
@mock.patch("builtins.input", return_value="y") @mock.patch.object(builtins, "input", return_value="y")
def test_autocorrect_multiple_candidates(mock_input, repo, mock_config): def test_autocorrect_multiple_candidates(mock_input, repo, mock_config):
"""Test autocorrect with multiple matches forces a prompt.""" """Test autocorrect with multiple matches forces a prompt."""
mock_config.GetString.return_value = "10" # Normally just delay mock_config.GetString.return_value = "10" # Normally just delay
@@ -158,7 +161,7 @@ def test_autocorrect_multiple_candidates(mock_input, repo, mock_config):
assert res == "start" assert res == "start"
@mock.patch("builtins.input", side_effect=KeyboardInterrupt()) @mock.patch.object(builtins, "input", side_effect=KeyboardInterrupt())
def test_autocorrect_prompt_interrupt(mock_input, repo, mock_config): def test_autocorrect_prompt_interrupt(mock_input, repo, mock_config):
"""Test autocorrect with prompt and user interrupts.""" """Test autocorrect with prompt and user interrupts."""
mock_config.GetString.return_value = "prompt" mock_config.GetString.return_value = "prompt"
@@ -181,7 +184,7 @@ def test_event_log_disabled_skips_add_and_git_hash() -> None:
project.GetCommitRevisionId.assert_not_called() project.GetCommitRevisionId.assert_not_called()
@mock.patch("manifest_xml.RepoClient") @mock.patch.object(manifest_xml, "RepoClient")
def test_run_long_gates_event_log_on_flag( def test_run_long_gates_event_log_on_flag(
mock_client_cls: mock.MagicMock, tmp_path: Path mock_client_cls: mock.MagicMock, tmp_path: Path
) -> None: ) -> None:
@@ -189,7 +192,7 @@ def test_run_long_gates_event_log_on_flag(
client = mock_client_cls.return_value client = mock_client_cls.return_value
client.manifest.IsMirror = False client.manifest.IsMirror = False
client.manifest.manifestProject.config.DumpConfigDict.return_value = {} client.manifest.manifestProject.config.DumpConfigDict.return_value = {}
repo = _Repo("repodir") repo = main._Repo("repodir")
for log_path, expected in ( for log_path, expected in (
(None, False), (None, False),
+3 -1
View File
@@ -180,7 +180,9 @@ class TestValue:
"""Check utility parsing code.""" """Check utility parsing code."""
def _get_node(self, text: str) -> xml.dom.minidom.Element: def _get_node(self, text: str) -> xml.dom.minidom.Element:
return xml.dom.minidom.parseString(text).firstChild node = xml.dom.minidom.parseString(text).firstChild
assert isinstance(node, xml.dom.minidom.Element)
return node
def test_bool_default(self) -> None: def test_bool_default(self) -> None:
"""Check XmlBool default handling.""" """Check XmlBool default handling."""
+480 -86
View File
@@ -20,6 +20,7 @@ import os
from pathlib import Path from pathlib import Path
import shutil import shutil
import subprocess import subprocess
import sys
import tempfile import tempfile
from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Type from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Type
import unittest import unittest
@@ -37,6 +38,7 @@ import git_trace2_event_log
import manifest_xml import manifest_xml
import platform_utils import platform_utils
import project import project
import repo_trace
class FakeProject: class FakeProject:
@@ -274,8 +276,10 @@ class ProjectTests(unittest.TestCase):
"""Check UploadForReview emits repo.uploadstate trace2 data events.""" """Check UploadForReview emits repo.uploadstate trace2 data events."""
proj, _ = self._create_project_for_upload_test() proj, _ = self._create_project_for_upload_test()
with mock.patch("git_command.GitCommand") as mock_git_cmd, mock.patch( with mock.patch.object(
"project.ReviewableBranch" git_command, "GitCommand"
) as mock_git_cmd, mock.patch.object(
project, "ReviewableBranch"
) as mock_rb_cls: ) as mock_rb_cls:
mock_cmd = mock.MagicMock() mock_cmd = mock.MagicMock()
mock_cmd.Wait.return_value = 0 mock_cmd.Wait.return_value = 0
@@ -314,8 +318,10 @@ class ProjectTests(unittest.TestCase):
"""Check event emission when stderr contains no review URLs.""" """Check event emission when stderr contains no review URLs."""
proj, _ = self._create_project_for_upload_test() proj, _ = self._create_project_for_upload_test()
with mock.patch("git_command.GitCommand") as mock_git_cmd, mock.patch( with mock.patch.object(
"project.ReviewableBranch" git_command, "GitCommand"
) as mock_git_cmd, mock.patch.object(
project, "ReviewableBranch"
) as mock_rb_cls: ) as mock_rb_cls:
mock_cmd = mock.MagicMock() mock_cmd = mock.MagicMock()
mock_cmd.Wait.return_value = 0 mock_cmd.Wait.return_value = 0
@@ -348,8 +354,10 @@ class ProjectTests(unittest.TestCase):
"""Check UploadForReview succeeds when git_event_log is None.""" """Check UploadForReview succeeds when git_event_log is None."""
proj, _ = self._create_project_for_upload_test() proj, _ = self._create_project_for_upload_test()
with mock.patch("git_command.GitCommand") as mock_git_cmd, mock.patch( with mock.patch.object(
"project.ReviewableBranch" git_command, "GitCommand"
) as mock_git_cmd, mock.patch.object(
project, "ReviewableBranch"
) as mock_rb_cls: ) as mock_rb_cls:
mock_cmd = mock.MagicMock() mock_cmd = mock.MagicMock()
mock_cmd.Wait.return_value = 0 mock_cmd.Wait.return_value = 0
@@ -371,9 +379,13 @@ class ProjectTests(unittest.TestCase):
"""Check exceptions during tracing are caught and do not fail upload.""" """Check exceptions during tracing are caught and do not fail upload."""
proj, _ = self._create_project_for_upload_test() proj, _ = self._create_project_for_upload_test()
with mock.patch("git_command.GitCommand") as mock_git_cmd, mock.patch( with mock.patch.object(
"project.ReviewableBranch" git_command, "GitCommand"
) as mock_rb_cls, mock.patch("project.logger") as mock_logger: ) as mock_git_cmd, mock.patch.object(
project, "ReviewableBranch"
) as mock_rb_cls, mock.patch.object(
project, "logger"
) as mock_logger:
mock_cmd = mock.MagicMock() mock_cmd = mock.MagicMock()
mock_cmd.Wait.return_value = 0 mock_cmd.Wait.return_value = 0
mock_cmd.stderr = "remote: https://example.com/c/test/+/1\n" mock_cmd.stderr = "remote: https://example.com/c/test/+/1\n"
@@ -396,8 +408,10 @@ class ProjectTests(unittest.TestCase):
"""Check integration with real git_trace2_event_log.EventLog.""" """Check integration with real git_trace2_event_log.EventLog."""
proj, _ = self._create_project_for_upload_test() proj, _ = self._create_project_for_upload_test()
with mock.patch("git_command.GitCommand") as mock_git_cmd, mock.patch( with mock.patch.object(
"project.ReviewableBranch" git_command, "GitCommand"
) as mock_git_cmd, mock.patch.object(
project, "ReviewableBranch"
) as mock_rb_cls: ) as mock_rb_cls:
mock_cmd = mock.MagicMock() mock_cmd = mock.MagicMock()
mock_cmd.Wait.return_value = 0 mock_cmd.Wait.return_value = 0
@@ -439,9 +453,11 @@ class ProjectTests(unittest.TestCase):
for quiet in (False, True): for quiet in (False, True):
with self.subTest(quiet=quiet): with self.subTest(quiet=quiet):
proj, _ = self._create_project_for_upload_test() proj, _ = self._create_project_for_upload_test()
with mock.patch( with mock.patch.object(
"git_command.GitCommand" git_command, "GitCommand"
) as mock_git_cmd, mock.patch("project.ReviewableBranch"): ) as mock_git_cmd, mock.patch.object(
project, "ReviewableBranch"
):
mock_cmd = mock.MagicMock() mock_cmd = mock.MagicMock()
mock_cmd.Wait.return_value = 0 mock_cmd.Wait.return_value = 0
mock_cmd.stderr = "" mock_cmd.stderr = ""
@@ -823,7 +839,7 @@ class ProjectTests(unittest.TestCase):
proj._GetStatusSnapshot = mock.MagicMock(return_value=status) proj._GetStatusSnapshot = mock.MagicMock(return_value=status)
proj.IsRebaseInProgress = mock.MagicMock(return_value=True) proj.IsRebaseInProgress = mock.MagicMock(return_value=True)
self.assertTrue(proj.HasChanges()) self.assertTrue(proj.HasChanges)
self.assertEqual( self.assertEqual(
["rebase in progress"], proj.UncommittedFiles(get_all=False) ["rebase in progress"], proj.UncommittedFiles(get_all=False)
) )
@@ -1195,7 +1211,7 @@ class ProjectTests(unittest.TestCase):
with open(head_file, "w", encoding="utf-8", newline="") as fp: with open(head_file, "w", encoding="utf-8", newline="") as fp:
fp.write("ref:\t refs/heads/fallback-branch\r\n") fp.write("ref:\t refs/heads/fallback-branch\r\n")
with mock.patch("platform_utils.islink", return_value=True): with mock.patch.object(platform_utils, "islink", return_value=True):
with mock.patch.object( with mock.patch.object(
fakeproj.work_git, fakeproj.work_git,
"symbolic_ref", "symbolic_ref",
@@ -1215,7 +1231,7 @@ class ProjectTests(unittest.TestCase):
with open(head_file, "w", encoding="utf-8", newline="") as fp: with open(head_file, "w", encoding="utf-8", newline="") as fp:
fp.write(f"{sha_upper}\r\n") fp.write(f"{sha_upper}\r\n")
with mock.patch("platform_utils.islink", return_value=True): with mock.patch.object(platform_utils, "islink", return_value=True):
with mock.patch.object( with mock.patch.object(
fakeproj.work_git, fakeproj.work_git,
"symbolic_ref", "symbolic_ref",
@@ -1233,7 +1249,7 @@ class ProjectTests(unittest.TestCase):
with open(head_file, "w", encoding="utf-8", newline="") as fp: with open(head_file, "w", encoding="utf-8", newline="") as fp:
fp.write("ref: refs/heads/.invalid\r\n") fp.write("ref: refs/heads/.invalid\r\n")
with mock.patch("platform_utils.islink", return_value=True): with mock.patch.object(platform_utils, "islink", return_value=True):
with mock.patch.object( with mock.patch.object(
fakeproj.work_git, fakeproj.work_git,
"symbolic_ref", "symbolic_ref",
@@ -1476,7 +1492,7 @@ class ProjectTests(unittest.TestCase):
hex41 = "a" * 41 hex41 = "a" * 41
proj.SetRevision(hex41) proj.SetRevision(hex41)
with mock.patch("git_command.GitCommand") as mock_git: with mock.patch.object(git_command, "GitCommand") as mock_git:
mock_cmd = mock.MagicMock() mock_cmd = mock.MagicMock()
mock_cmd.Wait.return_value = 0 mock_cmd.Wait.return_value = 0
mock_git.return_value = mock_cmd mock_git.return_value = mock_cmd
@@ -1984,6 +2000,64 @@ class ManifestPropertiesFetchedCorrectly(unittest.TestCase):
fakeproj.config.SetString("manifest.platform", "auto") fakeproj.config.SetString("manifest.platform", "auto")
self.assertEqual(fakeproj.manifest_platform, "auto") self.assertEqual(fakeproj.manifest_platform, "auto")
def test_smart_sync_branch_config(self) -> None:
"""Test branch-scoped smart sync branch getters and setters."""
with utils_for_test.TempGitTree() as tempdir:
fakeproj = self.setUpManifest(tempdir)
for branch, target in (
("main", "git_main-dev-warmup-abfs"),
("Main", "git_Main-warmup"),
("release/16", "git_r16-warmup"),
("16-release", "git_16-warmup"),
):
fakeproj.SetSmartSyncBranch(branch, target)
self.assertEqual(
fakeproj.GetSmartSyncBranch("main"),
"git_main-dev-warmup-abfs",
)
self.assertEqual(
fakeproj.GetSmartSyncBranch("Main"), "git_Main-warmup"
)
self.assertEqual(
fakeproj.GetSmartSyncBranch("release/16"), "git_r16-warmup"
)
self.assertEqual(
fakeproj.GetSmartSyncBranch("16-release"), "git_16-warmup"
)
self.assertIsNone(fakeproj.GetSmartSyncBranch("other"))
self.assertEqual(
fakeproj.config.GetString("repo.release/16.smartsyncbranch"),
"git_r16-warmup",
)
# Default lookup resolves the tracked manifest branch.
fakeproj.work_git.checkout("-q", "-b", "default")
b = fakeproj.GetBranch("default")
b.merge = "refs/heads/release/16"
b.Save()
self.assertEqual(
fakeproj.GetMergeBranch(heads_only=True), "release/16"
)
self.assertEqual(fakeproj.GetSmartSyncBranch(), "git_r16-warmup")
# Empty string or None clears only that branch's setting.
fakeproj.SetSmartSyncBranch("release/16", "")
self.assertIsNone(fakeproj.GetSmartSyncBranch("release/16"))
self.assertIsNone(fakeproj.GetSmartSyncBranch())
self.assertEqual(
fakeproj.GetSmartSyncBranch("main"),
"git_main-dev-warmup-abfs",
)
fakeproj.SetSmartSyncBranch("main", None)
self.assertIsNone(fakeproj.GetSmartSyncBranch("main"))
# Non-branch refs (e.g. tags) are ignored by default lookup.
b.merge = "refs/tags/v1.0"
b.Save()
self.assertIsNone(fakeproj.GetSmartSyncBranch())
def test_check_immutable_revision_metaproject_skips_manifest_load(self): def test_check_immutable_revision_metaproject_skips_manifest_load(self):
"""MetaProjects must not parse manifest.xml during immutable check. """MetaProjects must not parse manifest.xml during immutable check.
@@ -2490,7 +2564,7 @@ class StatelessSyncTests(unittest.TestCase):
proj.work_git = mock.MagicMock() proj.work_git = mock.MagicMock()
syncbuf = project.SyncBuffer(proj.config) syncbuf = project.SyncBuffer(proj.config)
with mock.patch("git_command.GitCommand") as mock_git_cmd: with mock.patch.object(git_command, "GitCommand") as mock_git_cmd:
mock_cmd_instance = mock.MagicMock() mock_cmd_instance = mock.MagicMock()
mock_cmd_instance.Wait.return_value = 0 mock_cmd_instance.Wait.return_value = 0
mock_git_cmd.return_value = mock_cmd_instance mock_git_cmd.return_value = mock_cmd_instance
@@ -2839,7 +2913,7 @@ class SyncOptimizationTests(unittest.TestCase):
if os.path.exists(shallow_path): if os.path.exists(shallow_path):
os.unlink(shallow_path) os.unlink(shallow_path)
with mock.patch("git_command.GitCommand") as mock_git_cmd: with mock.patch.object(git_command, "GitCommand") as mock_git_cmd:
mock_cmd_instance = mock.MagicMock() mock_cmd_instance = mock.MagicMock()
mock_cmd_instance.Wait.return_value = 0 mock_cmd_instance.Wait.return_value = 0
mock_git_cmd.return_value = mock_cmd_instance mock_git_cmd.return_value = mock_cmd_instance
@@ -2862,7 +2936,7 @@ class SyncOptimizationTests(unittest.TestCase):
with open(os.path.join(proj.gitdir, "shallow"), "w") as f: with open(os.path.join(proj.gitdir, "shallow"), "w") as f:
f.write("") f.write("")
with mock.patch("git_command.GitCommand") as mock_git_cmd: with mock.patch.object(git_command, "GitCommand") as mock_git_cmd:
res = proj._RemoteFetch( res = proj._RemoteFetch(
current_branch_only=True, current_branch_only=True,
depth=1, depth=1,
@@ -2885,7 +2959,7 @@ class SyncOptimizationTests(unittest.TestCase):
with open(os.path.join(other.gitdir, "shallow"), "w") as f: with open(os.path.join(other.gitdir, "shallow"), "w") as f:
f.write("") f.write("")
with mock.patch("git_command.GitCommand") as mock_git_cmd: with mock.patch.object(git_command, "GitCommand") as mock_git_cmd:
mock_cmd_instance = mock.MagicMock() mock_cmd_instance = mock.MagicMock()
mock_cmd_instance.Wait.return_value = 0 mock_cmd_instance.Wait.return_value = 0
mock_git_cmd.return_value = mock_cmd_instance mock_git_cmd.return_value = mock_cmd_instance
@@ -2914,7 +2988,7 @@ class SyncOptimizationTests(unittest.TestCase):
with open(os.path.join(other.gitdir, "shallow"), "w") as f: with open(os.path.join(other.gitdir, "shallow"), "w") as f:
f.write("") f.write("")
with mock.patch("git_command.GitCommand") as mock_git_cmd: with mock.patch.object(git_command, "GitCommand") as mock_git_cmd:
res = proj._RemoteFetch( res = proj._RemoteFetch(
current_branch_only=True, current_branch_only=True,
depth=1, depth=1,
@@ -3108,7 +3182,7 @@ class SyncOptimizationTests(unittest.TestCase):
mock_remote.PreConnectFetch.return_value = True mock_remote.PreConnectFetch.return_value = True
proj.GetRemote = mock.MagicMock(return_value=mock_remote) proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("git_command.GitCommand") as mock_git_cmd: with mock.patch.object(git_command, "GitCommand") as mock_git_cmd:
mock_cmd_instance = mock.MagicMock() mock_cmd_instance = mock.MagicMock()
mock_cmd_instance.Wait.return_value = 0 mock_cmd_instance.Wait.return_value = 0
mock_git_cmd.return_value = mock_cmd_instance mock_git_cmd.return_value = mock_cmd_instance
@@ -3147,7 +3221,7 @@ class SyncOptimizationTests(unittest.TestCase):
mock_remote.PreConnectFetch.return_value = True mock_remote.PreConnectFetch.return_value = True
proj.GetRemote = mock.MagicMock(return_value=mock_remote) proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("git_command.GitCommand") as mock_git_cmd: with mock.patch.object(git_command, "GitCommand") as mock_git_cmd:
mock_cmd_instance = mock.MagicMock() mock_cmd_instance = mock.MagicMock()
mock_cmd_instance.Wait.return_value = 0 mock_cmd_instance.Wait.return_value = 0
mock_git_cmd.return_value = mock_cmd_instance mock_git_cmd.return_value = mock_cmd_instance
@@ -3189,7 +3263,7 @@ class SyncOptimizationTests(unittest.TestCase):
mock_remote.PreConnectFetch.return_value = True mock_remote.PreConnectFetch.return_value = True
proj.GetRemote = mock.MagicMock(return_value=mock_remote) proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("git_command.GitCommand") as mock_git_cmd: with mock.patch.object(git_command, "GitCommand") as mock_git_cmd:
mock_cmd_instance = mock.MagicMock() mock_cmd_instance = mock.MagicMock()
mock_cmd_instance.Wait.return_value = 0 mock_cmd_instance.Wait.return_value = 0
mock_git_cmd.return_value = mock_cmd_instance mock_git_cmd.return_value = mock_cmd_instance
@@ -3237,7 +3311,7 @@ class SyncOptimizationTests(unittest.TestCase):
mock_remote.PreConnectFetch.return_value = True mock_remote.PreConnectFetch.return_value = True
proj.GetRemote = mock.MagicMock(return_value=mock_remote) proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("git_command.GitCommand") as mock_git_cmd: with mock.patch.object(git_command, "GitCommand") as mock_git_cmd:
mock_cmd_instance = mock.MagicMock() mock_cmd_instance = mock.MagicMock()
mock_cmd_instance.Wait.return_value = 0 mock_cmd_instance.Wait.return_value = 0
mock_git_cmd.return_value = mock_cmd_instance mock_git_cmd.return_value = mock_cmd_instance
@@ -3338,8 +3412,8 @@ class SyncOptimizationTests(unittest.TestCase):
inst.stdout = "" inst.stdout = ""
return inst return inst
with mock.patch( with mock.patch.object(
"git_command.GitCommand", side_effect=_fake_git_command git_command, "GitCommand", side_effect=_fake_git_command
): ):
res = proj._RemoteFetch(current_branch_only=True) res = proj._RemoteFetch(current_branch_only=True)
@@ -3477,6 +3551,11 @@ class FetchCmdTests(unittest.TestCase):
proj.GetRevisionId = mock.MagicMock(return_value="1234abcd") proj.GetRevisionId = mock.MagicMock(return_value="1234abcd")
# Every ID names a commit unless a test says otherwise. # Every ID names a commit unless a test says otherwise.
proj.bare_git.ResolveCommit.side_effect = lambda rev: rev proj.bare_git.ResolveCommit.side_effect = lambda rev: rev
# These tests supply ref values through bare_git, so give them a ref
# snapshot that knows nothing and does not claim to be complete.
proj.bare_ref = mock.MagicMock()
proj.bare_ref.get.return_value = ""
proj.bare_ref.is_files_authoritative = False
return proj return proj
def test_fetch_cmd_execution_unpinned_branch(self) -> None: def test_fetch_cmd_execution_unpinned_branch(self) -> None:
@@ -3490,7 +3569,7 @@ class FetchCmdTests(unittest.TestCase):
mock_remote.ToLocal.return_value = "refs/remotes/origin/main" mock_remote.ToLocal.return_value = "refs/remotes/origin/main"
proj.GetRemote = mock.MagicMock(return_value=mock_remote) proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("subprocess.run") as mock_run: with mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="") mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
res = proj._CustomFetch() res = proj._CustomFetch()
@@ -3523,7 +3602,7 @@ class FetchCmdTests(unittest.TestCase):
mock_remote.ToLocal.return_value = "refs/remotes/origin/main" mock_remote.ToLocal.return_value = "refs/remotes/origin/main"
proj.GetRemote = mock.MagicMock(return_value=mock_remote) proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("subprocess.run") as mock_run: with mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="") mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
res = proj._CustomFetch() res = proj._CustomFetch()
@@ -3543,7 +3622,7 @@ class FetchCmdTests(unittest.TestCase):
proj.bare_git.ResolveRef.return_value = sha proj.bare_git.ResolveRef.return_value = sha
proj.GetRemote = mock.MagicMock() proj.GetRemote = mock.MagicMock()
with mock.patch("subprocess.run") as mock_run: with mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="") mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
res = proj._CustomFetch() res = proj._CustomFetch()
@@ -3571,7 +3650,7 @@ class FetchCmdTests(unittest.TestCase):
mock_remote.ToLocal.return_value = "refs/remotes/origin/main" mock_remote.ToLocal.return_value = "refs/remotes/origin/main"
proj.GetRemote = mock.MagicMock(return_value=mock_remote) proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("subprocess.run") as mock_run: with mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="") mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
self.assertFalse(proj._CustomFetch()) self.assertFalse(proj._CustomFetch())
@@ -3588,7 +3667,7 @@ class FetchCmdTests(unittest.TestCase):
mock_remote.ToLocal.return_value = "refs/remotes/origin/main" mock_remote.ToLocal.return_value = "refs/remotes/origin/main"
proj.GetRemote = mock.MagicMock(return_value=mock_remote) proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("subprocess.run") as mock_run: with mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="") mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
self.assertFalse(proj._CustomFetch()) self.assertFalse(proj._CustomFetch())
@@ -3603,7 +3682,7 @@ class FetchCmdTests(unittest.TestCase):
mock_remote.ToLocal.return_value = "refs/remotes/origin/main" mock_remote.ToLocal.return_value = "refs/remotes/origin/main"
proj.GetRemote = mock.MagicMock(return_value=mock_remote) proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("subprocess.run") as mock_run: with mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="") mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
self.assertFalse(proj._CustomFetch()) self.assertFalse(proj._CustomFetch())
@@ -3618,7 +3697,7 @@ class FetchCmdTests(unittest.TestCase):
proj.revisionExpr = sha proj.revisionExpr = sha
proj.bare_git.ResolveRef.side_effect = error.GitError("missing") proj.bare_git.ResolveRef.side_effect = error.GitError("missing")
with mock.patch("subprocess.run") as mock_run: with mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="") mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
self.assertFalse(proj._CustomFetch()) self.assertFalse(proj._CustomFetch())
@@ -3639,7 +3718,7 @@ class FetchCmdTests(unittest.TestCase):
mock_remote.ToLocal.return_value = "refs/tags/v1.0" mock_remote.ToLocal.return_value = "refs/tags/v1.0"
proj.GetRemote = mock.MagicMock(return_value=mock_remote) proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("subprocess.run") as mock_run: with mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="") mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
self.assertTrue(proj._CustomFetch()) self.assertTrue(proj._CustomFetch())
@@ -3667,7 +3746,7 @@ class FetchCmdTests(unittest.TestCase):
mock_remote.ToLocal.return_value = "refs/tags/v1.0" mock_remote.ToLocal.return_value = "refs/tags/v1.0"
proj.GetRemote = mock.MagicMock(return_value=mock_remote) proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("subprocess.run") as mock_run: with mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="") mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
self.assertFalse(proj._CustomFetch()) self.assertFalse(proj._CustomFetch())
@@ -3707,7 +3786,7 @@ class FetchCmdTests(unittest.TestCase):
) )
return mock.MagicMock(returncode=0, stderr="") return mock.MagicMock(returncode=0, stderr="")
with mock.patch("subprocess.run", side_effect=_edit_config): with mock.patch.object(subprocess, "run", side_effect=_edit_config):
self.assertTrue(proj._CustomFetch()) self.assertTrue(proj._CustomFetch())
self.assertTrue(proj.config.GetBoolean("custom.initialized")) self.assertTrue(proj.config.GetBoolean("custom.initialized"))
@@ -3730,7 +3809,9 @@ class FetchCmdTests(unittest.TestCase):
fp.write("[custom]\n\tfailedrun = true\n") fp.write("[custom]\n\tfailedrun = true\n")
return mock.MagicMock(returncode=2, stderr="boom") return mock.MagicMock(returncode=2, stderr="boom")
with mock.patch("subprocess.run", side_effect=_fail_after_edit): with mock.patch.object(
subprocess, "run", side_effect=_fail_after_edit
):
self.assertFalse(proj._CustomFetch()) self.assertFalse(proj._CustomFetch())
self.assertTrue(proj.config.GetBoolean("custom.failedrun")) self.assertTrue(proj.config.GetBoolean("custom.failedrun"))
@@ -3790,9 +3871,9 @@ class FetchCmdTests(unittest.TestCase):
mock_remote.ToLocal.return_value = "refs/remotes/origin/main" mock_remote.ToLocal.return_value = "refs/remotes/origin/main"
proj.GetRemote = mock.MagicMock(return_value=mock_remote) proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("repo_trace.Trace") as mock_trace, mock.patch( with mock.patch.object(
"subprocess.run" repo_trace, "Trace"
) as mock_run: ) as mock_trace, mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="") mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
self.assertTrue(proj._CustomFetch()) self.assertTrue(proj._CustomFetch())
@@ -3805,9 +3886,9 @@ class FetchCmdTests(unittest.TestCase):
) )
proj.revisionId = sha proj.revisionId = sha
with mock.patch("repo_trace.Trace") as mock_trace, mock.patch( with mock.patch.object(
"subprocess.run" repo_trace, "Trace"
) as mock_run: ) as mock_trace, mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="") mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
self.assertTrue(proj._CustomFetch()) self.assertTrue(proj._CustomFetch())
@@ -3819,9 +3900,9 @@ class FetchCmdTests(unittest.TestCase):
sha, sha,
) )
with mock.patch( with mock.patch.object(
"repo_trace.Trace", side_effect=OSError("trace disk full") repo_trace, "Trace", side_effect=OSError("trace disk full")
), mock.patch("subprocess.run") as mock_run: ), mock.patch.object(subprocess, "run") as mock_run:
with self.assertRaises(OSError): with self.assertRaises(OSError):
proj._CustomFetch() proj._CustomFetch()
mock_run.assert_not_called() mock_run.assert_not_called()
@@ -3975,6 +4056,112 @@ class FetchCmdGitTests(unittest.TestCase):
self.assertFalse(proj._CustomFetch()) self.assertFalse(proj._CustomFetch())
def test_custom_fetch_cold_and_warm_packed_zero_git_subprocesses(
self,
) -> None:
"""Cold and warm packed-refs _CustomFetch run zero git subprocesses."""
with tempfile.TemporaryDirectory(
prefix="repo-tests"
) as topdir, mock.patch.dict(
os.environ, {"GIT_DEFAULT_REF_FORMAT": "files"}, clear=False
):
os.environ.pop("GIT_DIR", None)
os.environ.pop("GIT_COMMON_DIR", None)
c1 = "1111abcd" * 5
c2 = "2222ef01" * 5
hook = os.path.join(topdir, "fake_fetch.py")
state = Path(topdir, "state.txt")
lrev_out = Path(topdir, "lrev.txt")
Path(hook).write_text(
"import os, pathlib\n"
"gitdir = pathlib.Path(os.environ['REPO_PATH'], '.git')\n"
"pathlib.Path('lrev.txt').write_text(\n"
" os.environ.get('REPO_LREV', ''), encoding='utf-8'\n"
")\n"
"mode, oid = pathlib.Path('state.txt').read_text().split(':')\n"
"packed = gitdir / 'packed-refs'\n"
"st = packed.stat() if packed.exists() else None\n"
"if mode != 'skip_ref':\n"
" hdr = '# pack-refs with: peeled sorted \\n'\n"
" ref = f'{oid} refs/remotes/origin/main\\n'\n"
" packed.write_bytes((hdr + ref).encode())\n"
" if st is not None:\n"
" ns = (st.st_atime_ns, st.st_mtime_ns)\n"
" os.utime(packed, ns=ns)\n"
"(gitdir / 'FETCH_HEAD').write_bytes(\n"
" f'{oid}\\tnot-for-merge\\tcommit\\n'.encode()\n"
")\n",
encoding="utf-8",
)
state.write_text(f"write:{c1}", encoding="utf-8")
proj = self._make_project(
topdir,
f'"{sys.executable}" fake_fetch.py',
revision_expr="main",
revision_id=None,
)
proj.config.SetString("remote.origin.url", "http://example.com/r")
proj.config.SetString(
"remote.origin.fetch", "+refs/heads/*:refs/remotes/origin/*"
)
seen_cmds: List[List[str]] = []
class SpyGitCommand(git_command.GitCommand):
def __init__(
self,
proj_arg: Any,
cmdv: List[str],
*args: Any,
**kwargs: Any,
) -> None:
seen_cmds.append(list(cmdv))
super().__init__(proj_arg, cmdv, *args, **kwargs)
with mock.patch.object(git_command, "GitCommand", SpyGitCommand):
# Cold unpinned fetch: tracking ref does not exist yet.
self.assertTrue(proj._CustomFetch())
self.assertEqual(seen_cmds, [])
self.assertEqual(lrev_out.read_text(encoding="utf-8"), "")
self.assertTrue(proj.bare_ref.is_loaded)
self.assertEqual(proj.GetRevisionId(), c1)
# Cold TryResolveRef hit resolves packed-refs without scanning.
proj.bare_ref.Invalidate()
self.assertEqual(proj.GetRevisionId(), c1)
self.assertFalse(proj.bare_ref.is_loaded)
self.assertEqual(seen_cmds, [])
# Warm unpinned fetch: bare_ref is pre-loaded AND packed-refs
# keeps identical size and mtime, requiring Invalidate().
self.assertEqual(
proj.bare_ref.get("refs/remotes/origin/main"), c1
)
state.write_text(f"write:{c2}", encoding="utf-8")
self.assertTrue(proj._CustomFetch())
self.assertEqual(seen_cmds, [])
self.assertEqual(lrev_out.read_text(encoding="utf-8"), c1)
self.assertEqual(
proj.bare_ref.get("refs/remotes/origin/main"), c2
)
self.assertEqual(proj.GetRevisionId(), c2)
# Authoritative missing tracking ref fails with zero git spawns.
proj.revisionExpr = "missing"
with self.assertRaises(error.ManifestInvalidRevisionError):
proj.GetRevisionId()
state.write_text(f"skip_ref:{c2}", encoding="utf-8")
self.assertFalse(proj._CustomFetch())
self.assertEqual(seen_cmds, [])
# Rev-parse syntax falls back to git instead of fast-failing.
proj.revisionExpr = "main~1"
with self.assertRaises(error.ManifestInvalidRevisionError):
proj.GetRevisionId()
self.assertTrue(
any(cmd and cmd[0] == "rev-parse" for cmd in seen_cmds)
)
class ReprojectCmdTests(unittest.TestCase): class ReprojectCmdTests(unittest.TestCase):
"""Tests for the repo.reprojectcmd feature.""" """Tests for the repo.reprojectcmd feature."""
@@ -4039,9 +4226,9 @@ class ReprojectCmdTests(unittest.TestCase):
self, proj: project.Project, **outputs: Any self, proj: project.Project, **outputs: Any
) -> mock.MagicMock: ) -> mock.MagicMock:
"""Run _Reproject against mocked Git; return the subprocess mock.""" """Run _Reproject against mocked Git; return the subprocess mock."""
with mock.patch( with mock.patch.object(
"git_command.GitCommand", side_effect=self._git_command(**outputs) git_command, "GitCommand", side_effect=self._git_command(**outputs)
), mock.patch("subprocess.run") as mock_run: ), mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stdout="") mock_run.return_value = mock.MagicMock(returncode=0, stdout="")
proj._Reproject(self.REVID) proj._Reproject(self.REVID)
return mock_run return mock_run
@@ -4050,9 +4237,9 @@ class ReprojectCmdTests(unittest.TestCase):
"""Test the command runs from the client root with REPO_TREV set.""" """Test the command runs from the client root with REPO_TREV set."""
with utils_for_test.TempGitTree() as tempdir: with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir) proj = self._get_project(tempdir)
with mock.patch( with mock.patch.object(
"git_command.GitCommand", side_effect=self._git_command() git_command, "GitCommand", side_effect=self._git_command()
), mock.patch("subprocess.run") as mock_run: ), mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stdout="") mock_run.return_value = mock.MagicMock(returncode=0, stdout="")
proj._Reproject(self.REVID) proj._Reproject(self.REVID)
@@ -4110,7 +4297,7 @@ class ReprojectCmdTests(unittest.TestCase):
with utils_for_test.TempGitTree() as tempdir: with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir) proj = self._get_project(tempdir)
proj.IsRebaseInProgress.return_value = True proj.IsRebaseInProgress.return_value = True
with mock.patch("subprocess.run") as mock_run: with mock.patch.object(subprocess, "run") as mock_run:
with self.assertRaises(project.LocalSyncFail) as e: with self.assertRaises(project.LocalSyncFail) as e:
proj._Reproject(self.REVID) proj._Reproject(self.REVID)
self.assertIn("rebase in progress", str(e.exception)) self.assertIn("rebase in progress", str(e.exception))
@@ -4120,9 +4307,9 @@ class ReprojectCmdTests(unittest.TestCase):
"""Test a non-zero exit fails the sync with the command's output.""" """Test a non-zero exit fails the sync with the command's output."""
with utils_for_test.TempGitTree() as tempdir: with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir) proj = self._get_project(tempdir)
with mock.patch( with mock.patch.object(
"git_command.GitCommand", side_effect=self._git_command() git_command, "GitCommand", side_effect=self._git_command()
), mock.patch("subprocess.run") as mock_run: ), mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock( mock_run.return_value = mock.MagicMock(
returncode=3, stdout="disk on fire\n" returncode=3, stdout="disk on fire\n"
) )
@@ -4136,9 +4323,9 @@ class ReprojectCmdTests(unittest.TestCase):
with utils_for_test.TempGitTree() as tempdir: with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir) proj = self._get_project(tempdir)
proj.work_git.GetHead.side_effect = [self.HEAD_ID, self.REVID] proj.work_git.GetHead.side_effect = [self.HEAD_ID, self.REVID]
with mock.patch( with mock.patch.object(
"git_command.GitCommand", side_effect=self._git_command() git_command, "GitCommand", side_effect=self._git_command()
), mock.patch("subprocess.run") as mock_run: ), mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stdout="") mock_run.return_value = mock.MagicMock(returncode=0, stdout="")
with self.assertRaises(project.LocalSyncFail) as e: with self.assertRaises(project.LocalSyncFail) as e:
proj._Reproject(self.REVID) proj._Reproject(self.REVID)
@@ -4154,9 +4341,9 @@ class ReprojectCmdTests(unittest.TestCase):
self.HEAD_ID, # HEAD before command self.HEAD_ID, # HEAD before command
self.REVID, # HEAD after command self.REVID, # HEAD after command
] ]
with mock.patch( with mock.patch.object(
"git_command.GitCommand", side_effect=self._git_command() git_command, "GitCommand", side_effect=self._git_command()
), mock.patch("subprocess.run") as mock_run: ), mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stdout="") mock_run.return_value = mock.MagicMock(returncode=0, stdout="")
with self.assertRaises(project.LocalSyncFail) as e: with self.assertRaises(project.LocalSyncFail) as e:
proj._Reproject(self.REVID) proj._Reproject(self.REVID)
@@ -4166,10 +4353,11 @@ class ReprojectCmdTests(unittest.TestCase):
"""Test _Reproject does not diff the index against REPO_TREV^{tree}.""" """Test _Reproject does not diff the index against REPO_TREV^{tree}."""
with utils_for_test.TempGitTree() as tempdir: with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir) proj = self._get_project(tempdir)
with mock.patch( with mock.patch.object(
"git_command.GitCommand", git_command,
"GitCommand",
side_effect=self._git_command(), side_effect=self._git_command(),
) as mock_git_cmd, mock.patch("subprocess.run") as mock_run: ) as mock_git_cmd, mock.patch.object(subprocess, "run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stdout="") mock_run.return_value = mock.MagicMock(returncode=0, stdout="")
proj._Reproject(self.REVID) proj._Reproject(self.REVID)
@@ -4517,7 +4705,7 @@ class ReprojectCmdTests(unittest.TestCase):
"""Test _Reproject traces execution and propagates trace OSError.""" """Test _Reproject traces execution and propagates trace OSError."""
with utils_for_test.TempGitTree() as tempdir: with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir) proj = self._get_project(tempdir)
with mock.patch("repo_trace.Trace") as mock_trace: with mock.patch.object(repo_trace, "Trace") as mock_trace:
self._reproject(proj) self._reproject(proj)
mock_trace.assert_called_once_with( mock_trace.assert_called_once_with(
@@ -4528,12 +4716,12 @@ class ReprojectCmdTests(unittest.TestCase):
self.REVID, self.REVID,
) )
with mock.patch( with mock.patch.object(
"git_command.GitCommand", side_effect=self._git_command() git_command, "GitCommand", side_effect=self._git_command()
), mock.patch( ), mock.patch.object(
"repo_trace.Trace", side_effect=OSError("trace disk full") repo_trace, "Trace", side_effect=OSError("trace disk full")
), mock.patch( ), mock.patch.object(
"subprocess.run" subprocess, "run"
) as mock_run: ) as mock_run:
with self.assertRaises(OSError): with self.assertRaises(OSError):
proj._Reproject(self.REVID) proj._Reproject(self.REVID)
@@ -4553,9 +4741,11 @@ class ReprojectCmdTests(unittest.TestCase):
fp.write("[custom]\n\treprojected = true\n") fp.write("[custom]\n\treprojected = true\n")
return mock.MagicMock(returncode=1, stdout="failed\n") return mock.MagicMock(returncode=1, stdout="failed\n")
with mock.patch( with mock.patch.object(
"git_command.GitCommand", side_effect=self._git_command() git_command, "GitCommand", side_effect=self._git_command()
), mock.patch("subprocess.run", side_effect=_fail_after_edit): ), mock.patch.object(
subprocess, "run", side_effect=_fail_after_edit
):
with self.assertRaises(project.LocalSyncFail): with self.assertRaises(project.LocalSyncFail):
proj._Reproject(self.REVID) proj._Reproject(self.REVID)
@@ -5034,10 +5224,10 @@ class DownloadPatchSetTests(unittest.TestCase):
def test_download_patch_set_with_no_filter_and_no_tags(self) -> None: def test_download_patch_set_with_no_filter_and_no_tags(self) -> None:
"""Test DownloadPatchSet includes --no-filter and --no-tags.""" """Test DownloadPatchSet includes --no-filter and --no-tags."""
with mock.patch( with mock.patch.object(
"git_command.git_require", return_value=True git_command, "git_require", return_value=True
) as mock_git_require: ) as mock_git_require:
with mock.patch("git_command.GitCommand") as mock_git_cmd: with mock.patch.object(git_command, "GitCommand") as mock_git_cmd:
mock_cmd_instance = mock.MagicMock() mock_cmd_instance = mock.MagicMock()
mock_cmd_instance.Wait.return_value = 0 mock_cmd_instance.Wait.return_value = 0
mock_git_cmd.return_value = mock_cmd_instance mock_git_cmd.return_value = mock_cmd_instance
@@ -5063,8 +5253,8 @@ class DownloadPatchSetTests(unittest.TestCase):
def test_download_patch_set_legacy_git_omits_no_filter(self) -> None: def test_download_patch_set_legacy_git_omits_no_filter(self) -> None:
"""Test DownloadPatchSet omits --no-filter on git < 2.17.""" """Test DownloadPatchSet omits --no-filter on git < 2.17."""
with mock.patch("git_command.git_require", return_value=False): with mock.patch.object(git_command, "git_require", return_value=False):
with mock.patch("git_command.GitCommand") as mock_git_cmd: with mock.patch.object(git_command, "GitCommand") as mock_git_cmd:
mock_cmd_instance = mock.MagicMock() mock_cmd_instance = mock.MagicMock()
mock_cmd_instance.Wait.return_value = 0 mock_cmd_instance.Wait.return_value = 0
mock_git_cmd.return_value = mock_cmd_instance mock_git_cmd.return_value = mock_cmd_instance
@@ -5149,13 +5339,24 @@ class InitGitDirConfigTests(unittest.TestCase):
super().__init__(proj_arg, cmdv, *args, **kwargs) super().__init__(proj_arg, cmdv, *args, **kwargs)
with mock.patch.object(git_command, "GitCommand", SpyGitCommand): with mock.patch.object(git_command, "GitCommand", SpyGitCommand):
proj._InitGitDir(quiet=True) with proj.config.BatchWrites():
proj._InitRemote() proj._InitGitDir(quiet=True)
proj._InitRemote()
init_cmds = [cmd for cmd in seen_cmds if cmd and cmd[0] == "init"] init_cmds = [cmd for cmd in seen_cmds if cmd and cmd[0] == "init"]
self.assertEqual([["init", "-q", "--template="]], init_cmds) self.assertEqual([["init", "-q", "--template="]], init_cmds)
self.assertFalse(
any("--replace-all" in cmd for cmd in seen_cmds),
f"Expected BatchWrites to avoid --replace-all, got {seen_cmds}",
)
self.assertFalse(
any("core.bare" in cmd for cmd in seen_cmds),
f"Expected local init to avoid core.bare config spawns: "
f"{seen_cmds}",
)
reloaded = git_config.GitConfig.ForRepository(gitdir=proj.gitdir) reloaded = git_config.GitConfig.ForRepository(gitdir=proj.gitdir)
fresh = reloaded._ReadGit() fresh = reloaded._ReadGit()
self.assertEqual(["false"], fresh.get("core.bare"))
self.assertEqual( self.assertEqual(
["git-lfs smudge --skip -- %f"], fresh.get("filter.lfs.smudge") ["git-lfs smudge --skip -- %f"], fresh.get("filter.lfs.smudge")
) )
@@ -5238,6 +5439,70 @@ class InitGitDirConfigTests(unittest.TestCase):
os.path.exists(os.path.join(hooks_dir, "post-checkout")) os.path.exists(os.path.join(hooks_dir, "post-checkout"))
) )
def test_warm_and_seed_init_template_cache(self) -> None:
"""Warming in parent and seeding in worker avoids re-querying git."""
with tempfile.TemporaryDirectory() as topdir, mock.patch.dict(
project._INIT_TEMPLATE_CACHE, {}, clear=True
), mock.patch.dict(
os.environ,
{
"GIT_CONFIG_NOSYSTEM": "1",
"GIT_CONFIG_GLOBAL": os.path.join(topdir, "empty_gitconfig"),
},
clear=False,
):
os.environ.pop("GIT_TEMPLATE_DIR", None)
os.environ.pop("GIT_CONFIG_PARAMETERS", None)
os.environ.pop("GIT_CONFIG_COUNT", None)
seen_cmds: List[List[str]] = []
class SpyGitCommand(git_command.GitCommand):
def __init__(
self,
proj_arg: Any,
cmdv: List[str],
*args: Any,
**kwargs: Any,
) -> None:
seen_cmds.append(list(cmdv))
super().__init__(proj_arg, cmdv, *args, **kwargs)
with mock.patch.object(git_command, "GitCommand", SpyGitCommand):
entry = project.Project.WarmInitTemplateCache()
self.assertFalse(entry[1])
self.assertEqual(
[["config", "--get", "init.templatedir"]], seen_cmds
)
# Simulate a fresh spawn/forkserver worker seeded via InitWorker.
seen_cmds.clear()
project._INIT_TEMPLATE_CACHE.clear()
project.Project.SeedInitTemplateCache(entry)
proj = self._make_uninitialized_project(topdir, clone_depth=1)
with mock.patch.object(git_command, "GitCommand", SpyGitCommand):
proj._InitGitDir(quiet=True)
tpl_cmds = [cmd for cmd in seen_cmds if "init.templatedir" in cmd]
init_cmds = [cmd for cmd in seen_cmds if cmd and cmd[0] == "init"]
self.assertEqual([], tpl_cmds)
self.assertEqual([["init", "-q", "--template="]], init_cmds)
# Seeding True omits --template= without querying git config.
seen_cmds.clear()
project._INIT_TEMPLATE_CACHE.clear()
project.Project.SeedInitTemplateCache((entry[0], True))
proj2 = self._make_uninitialized_project(
os.path.join(topdir, "sub"), clone_depth=1
)
with mock.patch.object(git_command, "GitCommand", SpyGitCommand):
proj2._InitGitDir(quiet=True)
tpl_cmds2 = [cmd for cmd in seen_cmds if "init.templatedir" in cmd]
init_cmds2 = [cmd for cmd in seen_cmds if cmd and cmd[0] == "init"]
self.assertEqual([], tpl_cmds2)
self.assertEqual([["init", "-q"]], init_cmds2)
class SparseCheckoutTests(unittest.TestCase): class SparseCheckoutTests(unittest.TestCase):
"""Check sparse-checkout handling.""" """Check sparse-checkout handling."""
@@ -5807,3 +6072,132 @@ class SparseCheckoutFetchFilterTests(unittest.TestCase):
self.assertTrue( self.assertTrue(
self._bundle_applied(proj, partial_clone_exclude={proj.name}) self._bundle_applied(proj, partial_clone_exclude={proj.name})
) )
@pytest.mark.parametrize(
"current_branch, merge, expected_all, expected_heads_only",
[
("default", "refs/heads/main", "main", "main"),
("default", "refs/heads/android/main", "android/main", "android/main"),
# `repo init -b` with a tag, change, or commit SHA keeps the ref for
# `m/<branch>`, but is not tracking a remote branch under `refs/heads/`.
("default", "refs/tags/v1.0", "refs/tags/v1.0", None),
(
"default",
"refs/changes/23/655123/1",
"refs/changes/23/655123/1",
None,
),
(
"default",
"1234567890abcdef1234567890abcdef12345678",
"1234567890abcdef1234567890abcdef12345678",
None,
),
("default", "main", "main", None),
# A local branch without an upstream.
("default", None, None, None),
("default", "", None, None),
("default", "refs/heads/", None, None),
# A detached HEAD or empty branch name.
(None, "refs/heads/main", None, None),
(None, None, None, None),
("", "refs/heads/main", None, None),
],
)
def test_manifest_project_get_merge_branch(
current_branch: Optional[str],
merge: Optional[str],
expected_all: Optional[str],
expected_heads_only: Optional[str],
) -> None:
"""Check ManifestProject.GetMergeBranch with and without heads_only."""
mp = mock.MagicMock(CurrentBranch=current_branch)
mp.GetBranch.return_value.merge = merge
assert (
project.ManifestProject.GetMergeBranch(mp, heads_only=False)
== expected_all
)
assert (
project.ManifestProject.GetMergeBranch(mp, heads_only=True)
== expected_heads_only
)
if current_branch:
assert mp.GetBranch.call_args_list == [
mock.call(current_branch),
mock.call(current_branch),
]
else:
mp.GetBranch.assert_not_called()
@pytest.mark.parametrize(
"git_cmds, expected_current, expected_all, expected_heads_only",
[
(
[["config", "branch.main.merge", "refs/heads/android-main"]],
"main",
"android-main",
"android-main",
),
(
[["config", "branch.main.merge", "refs/tags/v1.0"]],
"main",
"refs/tags/v1.0",
None,
),
(
[
[
"config",
"branch.main.merge",
"1234567890abcdef1234567890abcdef12345678",
]
],
"main",
"1234567890abcdef1234567890abcdef12345678",
None,
),
([], "main", None, None),
(
[
["config", "branch.main.merge", "refs/heads/android-main"],
["checkout", "-q", "--detach"],
],
None,
None,
None,
),
],
)
def test_manifest_project_get_merge_branch_checkout(
tmp_path: Path,
git_cmds: List[List[str]],
expected_current: Optional[str],
expected_all: Optional[str],
expected_heads_only: Optional[str],
) -> None:
"""Check GetMergeBranch and XmlManifest.branch on a real checkout."""
repodir = tmp_path / ".repo"
manifests = repodir / "manifests"
manifests_git = repodir / "manifests.git"
manifests.mkdir(parents=True)
utils_for_test.init_git_tree(manifests)
(manifests / "default.xml").write_text("<manifest />\n")
shutil.move(str(manifests / ".git"), str(manifests_git))
(manifests / ".git").write_text("gitdir: ../manifests.git\n")
for args in [
["add", "default.xml"],
["commit", "-q", "-m", "init"],
] + git_cmds:
subprocess.run(["git", "-C", str(manifests), *args], check=True)
manifest_file = repodir / manifest_xml.MANIFEST_FILE_NAME
manifest = manifest_xml.XmlManifest(str(repodir), str(manifest_file))
manifest.Link("default.xml")
mp = manifest.manifestProject
assert mp.CurrentBranch == expected_current
assert mp.GetMergeBranch() == expected_all
assert mp.GetMergeBranch(heads_only=True) == expected_heads_only
assert manifest.branch == expected_all
+1
View File
@@ -27,6 +27,7 @@ def test_trace_max_size_enforced(monkeypatch: pytest.MonkeyPatch) -> None:
# Enable trace for the test, in case users have it disabled. # Enable trace for the test, in case users have it disabled.
monkeypatch.setattr(repo_trace, "_TRACE", True) monkeypatch.setattr(repo_trace, "_TRACE", True)
assert repo_trace._TRACE_FILE is not None
with repo_trace.Trace(content, first_trace=True): with repo_trace.Trace(content, first_trace=True):
pass pass
+12 -6
View File
@@ -53,7 +53,9 @@ def test_parse_ssh_version(input_str: str, expected: Tuple[int, ...]) -> None:
def test_version() -> None: def test_version() -> None:
"""Check version() handling.""" """Check version() handling."""
with mock.patch("ssh._run_ssh_version", return_value="OpenSSH_1.2\n"): with mock.patch.object(
ssh, "_run_ssh_version", return_value="OpenSSH_1.2\n"
):
assert ssh.version() == (1, 2) assert ssh.version() == (1, 2)
@@ -67,7 +69,7 @@ def test_context_manager_empty() -> None:
def test_context_manager_child_cleanup() -> None: def test_context_manager_child_cleanup() -> None:
"""Verify orphaned clients & masters get cleaned up.""" """Verify orphaned clients & masters get cleaned up."""
with multiprocessing.Manager() as manager: with multiprocessing.Manager() as manager:
with mock.patch("ssh.version", return_value=(1, 2)): with mock.patch.object(ssh, "version", return_value=(1, 2)):
with ssh.ProxyManager(manager) as ssh_proxy: with ssh.ProxyManager(manager) as ssh_proxy:
client = subprocess.Popen(["sleep", "964853320"]) client = subprocess.Popen(["sleep", "964853320"])
ssh_proxy.add_client(client) ssh_proxy.add_client(client)
@@ -87,13 +89,17 @@ def test_ssh_sock(monkeypatch: pytest.MonkeyPatch) -> None:
) )
# Old ssh version uses port. # Old ssh version uses port.
with mock.patch("ssh.version", return_value=(6, 6)): with mock.patch.object(ssh, "version", return_value=(6, 6)):
with proxy as ssh_proxy: with proxy as ssh_proxy:
assert ssh_proxy.sock().endswith("%p") sock = ssh_proxy.sock()
assert sock is not None
assert sock.endswith("%p")
proxy._sock_path = None proxy._sock_path = None
# New ssh version uses hash. # New ssh version uses hash.
with mock.patch("ssh.version", return_value=(6, 7)): with mock.patch.object(ssh, "version", return_value=(6, 7)):
with proxy as ssh_proxy: with proxy as ssh_proxy:
assert ssh_proxy.sock().endswith("%C") sock = ssh_proxy.sock()
assert sock is not None
assert sock.endswith("%C")
proxy._sock_path = None proxy._sock_path = None
+5 -4
View File
@@ -19,7 +19,7 @@ from typing import Type
import pytest import pytest
from command import Command import command
import subcmds import subcmds
from subcmds import sync from subcmds import sync
@@ -49,7 +49,7 @@ def test_naming(name: str) -> None:
@pytest.mark.parametrize("name, cls", subcmds.all_commands.items()) @pytest.mark.parametrize("name, cls", subcmds.all_commands.items())
def test_help_desc_style(name: str, cls: Type[Command]) -> None: def test_help_desc_style(name: str, cls: Type[command.Command]) -> None:
"""Force some consistency in option descriptions. """Force some consistency in option descriptions.
Python's optparse & argparse has a few default options like --help. Python's optparse & argparse has a few default options like --help.
@@ -79,7 +79,7 @@ def test_help_desc_style(name: str, cls: Type[Command]) -> None:
@pytest.mark.parametrize("name, cls", subcmds.all_commands.items()) @pytest.mark.parametrize("name, cls", subcmds.all_commands.items())
def test_cli_option_style(name: str, cls: Type[Command]) -> None: def test_cli_option_style(name: str, cls: Type[command.Command]) -> None:
"""Force some consistency in option flags.""" """Force some consistency in option flags."""
cmd = cls() cmd = cls()
parser = cmd.OptionParser parser = cmd.OptionParser
@@ -112,6 +112,7 @@ def test_cli_option_dest() -> None:
bad_opts.append((str(opt), opt.dest)) bad_opts.append((str(opt), opt.dest))
# Hook the option check list. # Hook the option check list.
assert optparse.Option.CHECK_METHODS is not None
optparse.Option.CHECK_METHODS.insert(0, _check_dest) optparse.Option.CHECK_METHODS.insert(0, _check_dest)
try: try:
@@ -142,7 +143,7 @@ def test_cli_option_dest() -> None:
@pytest.mark.parametrize("name, cls", subcmds.all_commands.items()) @pytest.mark.parametrize("name, cls", subcmds.all_commands.items())
def test_common_validate_options(name: str, cls: Type[Command]) -> None: def test_common_validate_options(name: str, cls: Type[command.Command]) -> None:
"""Verify CommonValidateOptions sets up expected fields.""" """Verify CommonValidateOptions sets up expected fields."""
cmd = cls() cmd = cls()
opts, args = cmd.OptionParser.parse_args([]) opts, args = cmd.OptionParser.parse_args([])
+23 -1
View File
@@ -14,7 +14,7 @@
"""Unittests for the subcmds/init.py module.""" """Unittests for the subcmds/init.py module."""
from typing import List from typing import List, Optional
import pytest import pytest
@@ -29,6 +29,8 @@ from subcmds import init
["--smart-sync"], ["--smart-sync"],
["--no-smart-sync"], ["--no-smart-sync"],
["--head"], ["--head"],
["--smart-sync-branch=git_main-warmup"],
["--smart-sync-branch="],
), ),
) )
def test_cli_parser_good(argv: List[str]) -> None: def test_cli_parser_good(argv: List[str]) -> None:
@@ -38,6 +40,24 @@ def test_cli_parser_good(argv: List[str]) -> None:
cmd.ValidateOptions(opts, args) cmd.ValidateOptions(opts, args)
@pytest.mark.parametrize(
"argv,expected",
(
([], None),
(["--smart-sync-branch=git_main-warmup"], "git_main-warmup"),
(["--smart-sync-branch="], ""),
),
)
def test_cli_smart_sync_branch_value(
argv: List[str], expected: Optional[str]
) -> None:
"""Check --smart-sync-branch parsed values."""
cmd = init.Init()
opts, args = cmd.OptionParser.parse_args(argv)
cmd.ValidateOptions(opts, args)
assert opts.smart_sync_branch == expected
@pytest.mark.parametrize( @pytest.mark.parametrize(
"argv", "argv",
( (
@@ -45,6 +65,8 @@ def test_cli_parser_good(argv: List[str]) -> None:
["url", "asdf"], ["url", "asdf"],
# Conflicting options. # Conflicting options.
["--mirror", "--archive"], ["--mirror", "--archive"],
["--standalone-manifest", "--smart-sync-branch=git_main-warmup"],
["--standalone-manifest", "--smart-sync-branch="],
), ),
) )
def test_cli_parser_bad(argv: List[str]) -> None: def test_cli_parser_bad(argv: List[str]) -> None:
+1065 -93
View File
File diff suppressed because it is too large. Load diff
+23 -11
View File
@@ -14,12 +14,14 @@
"""Unittests for the subcmds/upload.py module.""" """Unittests for the subcmds/upload.py module."""
from typing import List, Optional from typing import Iterator, List, Optional
from unittest import mock from unittest import mock
import pytest import pytest
import editor
import error import error
import git_superproject
from subcmds import upload from subcmds import upload
@@ -32,7 +34,7 @@ _STUB_PEOPLE = ([], [])
@pytest.fixture @pytest.fixture
def cmd() -> upload.Upload: def cmd() -> Iterator[upload.Upload]:
"""Fixture to provide an Upload command instance with mocked methods.""" """Fixture to provide an Upload command instance with mocked methods."""
cmd = upload.Upload() cmd = upload.Upload()
with mock.patch.object( with mock.patch.object(
@@ -183,8 +185,10 @@ def test_MultipleBranches_yes_with_current_branch_flag_bypasses_editor(
branch2 = _create_mock_branch("b2", project_relpath="p2") branch2 = _create_mock_branch("b2", project_relpath="p2")
pending = [(branch1.project, [branch1]), (branch2.project, [branch2])] pending = [(branch1.project, [branch1]), (branch2.project, [branch2])]
with mock.patch.object(cmd, "_UploadAndReport") as mock_upload, mock.patch( with mock.patch.object(
"editor.Editor.EditString" cmd, "_UploadAndReport"
) as mock_upload, mock.patch.object(
editor.Editor, "EditString"
) as mock_edit: ) as mock_edit:
cmd._MultipleBranches(opt, pending, _STUB_PEOPLE) cmd._MultipleBranches(opt, pending, _STUB_PEOPLE)
mock_edit.assert_not_called() mock_edit.assert_not_called()
@@ -202,8 +206,10 @@ def test_MultipleBranches_yes_with_cbr_flag_bypasses_editor(
branch2 = _create_mock_branch("b2", project_relpath="p2") branch2 = _create_mock_branch("b2", project_relpath="p2")
pending = [(branch1.project, [branch1]), (branch2.project, [branch2])] pending = [(branch1.project, [branch1]), (branch2.project, [branch2])]
with mock.patch.object(cmd, "_UploadAndReport") as mock_upload, mock.patch( with mock.patch.object(
"editor.Editor.EditString" cmd, "_UploadAndReport"
) as mock_upload, mock.patch.object(
editor.Editor, "EditString"
) as mock_edit: ) as mock_edit:
cmd._MultipleBranches(opt, pending, _STUB_PEOPLE) cmd._MultipleBranches(opt, pending, _STUB_PEOPLE)
mock_edit.assert_not_called() mock_edit.assert_not_called()
@@ -221,8 +227,10 @@ def test_MultipleBranches_yes_with_branch_flag_bypasses_editor(
branch2 = _create_mock_branch("feature", project_relpath="p2") branch2 = _create_mock_branch("feature", project_relpath="p2")
pending = [(branch1.project, [branch1]), (branch2.project, [branch2])] pending = [(branch1.project, [branch1]), (branch2.project, [branch2])]
with mock.patch.object(cmd, "_UploadAndReport") as mock_upload, mock.patch( with mock.patch.object(
"editor.Editor.EditString" cmd, "_UploadAndReport"
) as mock_upload, mock.patch.object(
editor.Editor, "EditString"
) as mock_edit: ) as mock_edit:
cmd._MultipleBranches(opt, pending, _STUB_PEOPLE) cmd._MultipleBranches(opt, pending, _STUB_PEOPLE)
mock_edit.assert_not_called() mock_edit.assert_not_called()
@@ -265,8 +273,10 @@ def test_MultipleBranches_yes_without_branch_or_cbr_uses_editor(
" branch b1 ( 1 commit, 2026-08-26) to remote branch " " branch b1 ( 1 commit, 2026-08-26) to remote branch "
"refs/heads/main:\n" "refs/heads/main:\n"
) )
with mock.patch.object(cmd, "_UploadAndReport") as mock_upload, mock.patch( with mock.patch.object(
"editor.Editor.EditString", return_value=edited_script cmd, "_UploadAndReport"
) as mock_upload, mock.patch.object(
editor.Editor, "EditString", return_value=edited_script
) as mock_edit: ) as mock_edit:
cmd._MultipleBranches(opt, pending, _STUB_PEOPLE) cmd._MultipleBranches(opt, pending, _STUB_PEOPLE)
mock_edit.assert_called_once() mock_edit.assert_called_once()
@@ -324,7 +334,9 @@ def test_UploadBranch_forwards_quiet(
cmd.CommonValidateOptions(opt, []) cmd.CommonValidateOptions(opt, [])
branch = mock.MagicMock() branch = mock.MagicMock()
branch.project.dest_branch = None branch.project.dest_branch = None
with mock.patch("git_superproject.UseSuperproject", return_value=False): with mock.patch.object(
git_superproject, "UseSuperproject", return_value=False
):
cmd._UploadBranch(opt, branch, _STUB_PEOPLE) cmd._UploadBranch(opt, branch, _STUB_PEOPLE)
branch.UploadForReview.assert_called_once() branch.UploadForReview.assert_called_once()
+4 -3
View File
@@ -35,7 +35,10 @@ def _create_mock_project(tempdir, name, objdir_path=None, has_changes=False):
os.makedirs(gitdir, exist_ok=True) os.makedirs(gitdir, exist_ok=True)
os.makedirs(objdir, exist_ok=True) os.makedirs(objdir, exist_ok=True)
proj = project.Project( class MockProject(project.Project):
HasChanges = mock.PropertyMock(return_value=has_changes)
proj = MockProject(
manifest=mock.MagicMock(), manifest=mock.MagicMock(),
name=name, name=name,
remote=mock.MagicMock(), remote=mock.MagicMock(),
@@ -47,8 +50,6 @@ def _create_mock_project(tempdir, name, objdir_path=None, has_changes=False):
revisionId="abcd", revisionId="abcd",
) )
proj.HasChanges = mock.MagicMock(return_value=has_changes)
def side_effect_delete_worktree(force=False, verbose=False): def side_effect_delete_worktree(force=False, verbose=False):
if os.path.exists(proj.worktree): if os.path.exists(proj.worktree):
shutil.rmtree(proj.worktree) shutil.rmtree(proj.worktree)
+5 -3
View File
@@ -83,9 +83,11 @@ class TestRepoWrapper:
def test_version(self, repo_wrapper: wrapper.Wrapper) -> None: def test_version(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Make sure _Version works.""" """Make sure _Version works."""
with pytest.raises(SystemExit) as e: with pytest.raises(SystemExit) as e:
with mock.patch("sys.stdout", new_callable=io.StringIO) as stdout: with mock.patch.object(
with mock.patch( sys, "stdout", new_callable=io.StringIO
"sys.stderr", new_callable=io.StringIO ) as stdout:
with mock.patch.object(
sys, "stderr", new_callable=io.StringIO
) as stderr: ) as stderr:
repo_wrapper._Version() repo_wrapper._Version()
assert e.value.code == 0 assert e.value.code == 0
+1 -1
View File
@@ -50,7 +50,7 @@ def init_git_tree(
else: else:
# Use template dir for init. # Use template dir for init.
templatedir = stack.enter_context( templatedir = stack.enter_context(
tempfile.mkdtemp(prefix="git-template") tempfile.TemporaryDirectory(prefix="git-template")
) )
(Path(templatedir) / "HEAD").write_text("ref: refs/heads/main\n") (Path(templatedir) / "HEAD").write_text("ref: refs/heads/main\n")
cmd += ["--template", templatedir] cmd += ["--template", templatedir]
+1
View File
@@ -31,6 +31,7 @@ def Wrapper():
modname = "wrapper" modname = "wrapper"
loader = importlib.machinery.SourceFileLoader(modname, WrapperPath()) loader = importlib.machinery.SourceFileLoader(modname, WrapperPath())
spec = importlib.util.spec_from_loader(modname, loader) spec = importlib.util.spec_from_loader(modname, loader)
assert spec is not None
module = importlib.util.module_from_spec(spec) module = importlib.util.module_from_spec(spec)
loader.exec_module(module) loader.exec_module(module)
return module return module