Compare commits

...
60 Commits
Author SHA1 Message Date
Gavin Mak a816b88d23 repo: bump launcher version to 2.74
https://gerrit-review.git.corp.google.com/c/git-repo/+/655063 bumped
VERSION to 2.72, but the next release tag is v2.74.

Change-Id: Ic2c0bcb6c1ccc4e553ba7d3f55eb557c47ebb71c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/657024
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-13 14:18:57 -08:00
Gavin Mak 032f2341ec init, sync: support per-branch smart sync branch config
Add `repo init --smart-sync-branch=BRANCH` to persist a server-side
Smart Sync branch override in `repo.<manifest-branch>.smartsyncbranch`
inside .repo/manifests.git/config (with an empty string clearing it).

During Smart Sync, non-empty SYNC_BRANCH still takes top precedence,
while SYNC_BRANCH="" forces the raw manifest branch over checkout
config.

Bug: 543101585
Bug: 585378823
Change-Id: I933a48d03ed5ebd83aa5aba9ddde761dfac23243
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655063
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-11-13 13:55:47 -08:00
Gavin Mak af53524261 sync: sort interleaved sync slowest-first
Phased sync orders each fetch batch slowest-first using
`.repo_fetchtimes.json`, whereas interleaved sync dispatched each
hierarchical level in path order, letting slow projects near the end of
a level start late and extend the tail of the sync.

Combining fetch and checkout times into `.repo_fetchtimes.json` would
mix units with phased mode and older repo versions, and cause `-n` or
`-l` runs to either skew full-sync estimates or sort by the wrong
phase.

Keep fetch and checkout estimates separate instead:

- Parameterize `_SyncTimes` by filename and track fetch durations in
  `.repo_fetchtimes.json` and checkout durations in
  `.repo_checkouttimes.json`, recording a phase only when it succeeds.
- Sort each interleaved level slowest-first by the phases active in
  `opt` (fetch when not `-l`, checkout when not `-n` and the project
  has a worktree).
- Sort each `_SafeCheckoutOrder` level in phased `_Checkout` by
  `_checkout_times` and save in `finally` so phased and interleaved
  runs maintain and use the same per-phase estimates.

Bug: 543101585
Change-Id: Ie8b15a95ab2d8019b058d5f839a93501d0d85979
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/657085
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-11-13 13:44:18 -08:00
Gavin Mak 2259e50864 sync: add --force-jobs flag to override manifest sync-j-max
Checkouts backed by a local daemon via `repo.fetchcmd` and
`repo.reprojectcmd` do not hit remote Git servers during `repo sync` and
benefit from higher `-j` concurrency than a manifest's `sync-j-max` cap
allows.

Add `--force-jobs` to `repo sync` (and `repo smartsync`) to bypass the
manifest `sync-j-max` limit and mention `--force-jobs` in the clamp
warning message.

Bug: 585378822
Change-Id: Id0310f85e5a169207f7acfbe004744e77df2780f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/657543
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-13 13:18:29 -08:00
Gavin Mak 1702d7fb55 project: avoid git spawns in custom fetch ref checks
During _CustomFetch, GetEnvVars resolves REPO_LREV via GetRevisionId
before fetchcmd runs, and postcondition checks afterward run ResolveRef
on the tracking ref and FETCH_HEAD. On cold clones, missing branches, or
packed-refs tracking refs, GetRevisionId fell back to ResolveCommit,
spawning `git rev-parse <rev>^{commit}` and peeling objects from lazily
populated object stores such as ABFS.

Resolve non-tag refs/* in GetRevisionId via GitRefs.TryResolveRef and
authoritative files snapshots so loose, packed, and missing refs resolve
without spawning git or reading .git/objects. In _CustomFetch,
invalidate bare_ref after fetchcmd runs, resolve tracking refs through
bare_ref to warm the snapshot for _InitMRef and Sync_LocalHalf, and read
FETCH_HEAD's leading object ID directly when unambiguous.

Bug: 543101585
Change-Id: Ifb5c0f182e539a2836672a47da348fbeb41e5068
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655964
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-11-12 16:10:31 -08:00
Gavin Mak 7cbd9d1762 project: only unset core.bare when it is true
When creating a gitdir, _InitGitDir spawned `git config --unset-all
core.bare` for every non-mirror project. That is needed when `git init`
wrote `core.bare = true`, as it does for a gitdir under
.repo/project-objects. For a local worktree gitdir (<worktree>/.git, as
with --use-local-gitdirs) `git init` already writes `core.bare = false`,
so the spawn only removed a setting that was already correct. Checked
on git 1.9.1 (the oldest version repo supports), 2.7.4, 2.20.1 and
2.56.

Unset core.bare on non-mirror repositories only when it is currently
true. Local worktree gitdirs now keep the `core.bare = false` that
`git init` wrote.

Bug: 543101585
Change-Id: I662ffce61dc68ebca921941c1a5e5ebce2d22338
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655965
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-12 16:07:48 -08:00
Hung-Wei Chen f01eb8774f git_config, project: coalesce SetString writes in BatchWrites
This change attempts to skip sequential git config calls in order to
further speed up git-repo's performance for VFS integration. On
git-main we currently call git config 26K+ times, causing roughly 5~6
seconds of latency (see b/543101585#comment23's TRACE_FILE for data).
Given the expected time of VFS integration is targeting 30s, 6 seconds
makes a huge difference.

During an initial `repo sync`, initializing each project (`_InitGitDir`
and `_InitRemote`) invokes `SetString` multiple times per project, and
`SyncAnalysisState` writes `repo.syncstate.*` keys once per sync.
Because the `git config` CLI has no batch-write mode to set multiple
keys in a single invocation, each `SetString` call previously spawned a
separate `git config` subprocess and rewrote `.repo_config.json` per key
(see b/543101585#comment23 and b/543101585#comment25 for trace profiles
and benchmark data).

Bug: 543101585
Bug: 564817989
Change-Id: Ia1d4613266fb6b7742e8db637039e72ed26a13a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/657083
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-12 15:37:33 -08:00
Gavin Mak 2aca835f1f sync: warm init.templateDir cache for sync workers
When initializing missing project object directories in parallel, each
sync worker cold-misses the init.templateDir lookup cache and spawns
`git config --get init.templatedir`. Relying on parent process memory
alone also misses workers started via `spawn` or `forkserver`.

Resolve the init.templateDir lookup once in the parent process when
parallel sync workers may initialize an object directory (either
because an objdir is missing or `--force-sync` is active), pass the
cached entry through `parallel_context`, and seed it in `InitWorker`.

Bug: 543101585
Change-Id: I6bac2352a9fd4b62d25ed2a89b670ac21aa70f4e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/657084
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-11-12 15:33:07 -08:00
Gavin Mak c9cfb541aa sync: trim preciousObjects work
Only set preciousObjects when objdir differs from gitdir, check actual
objdir sharing in _GetPreciousObjectsState, and skip post-sync
preciousObjects scans on unshared --use-local-gitdirs projects.

Bug: 543101585
Change-Id: I5077aed128f22cf19f520c7866cfd62f0c2724be
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/652266
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-12 14:51:41 -08:00
Mike Frysinger c85e5ff8a5 pyrefly: initial support
This suppresses a lot of findings in most files so we can run pyrefly
by default and prevent new findings from showing up.  This will need
time to slowly chip away file by file.

Concretely, it's enabled for ~60% of files, and disabled for the other
~40% files.  Not the worst start.

I've opted for disabling files completely at this point rather than
using a baseline because the generated baseline is >200KiB in size.
I think we need to get through a good chunk of that before we switch.

We're going with pyrefly because it's what Google has adopted, and it
evaluated alternatives including mypy, pyright, and its own pytype.

Change-Id: Ib73752dd240a9a84ce0f56059a82292141b6af66
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656364
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-12 11:33:10 -08:00
Mike Frysinger ba7bc573ad typing: fix various errors caught by pyrefly
These should be fairly straightforward and not require much thought.
I've punted the more complicated typing changes.

Change-Id: I79167f07e0ec259586520104f07e4cee096a382e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656663
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-11-12 11:23:08 -08:00
Mike Frysinger f74f219954 project: fix invalid logger call
This used to be a print() call, but the migration was incomplete.

Change-Id: If17e5108aa3d2a2948c4834e9c6ed895d3809f91
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656724
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-12 09:17:39 -08:00
Mike Frysinger f9a1895400 project: fix Project.HasChanges API mismatches
The base Project class defined HasChanges as a function, but the
MetaProject & RepoProject subclasses defined it as a property.
Obviously this is bad & confusing, so standardize it.

Change-Id: Ia3ac9d0592307366296b8ef991dc5d7bb2204490
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656723
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-12 09:17:27 -08:00
Brian Gan 13fe1b9d84 pytest: only keep temp dirs of failed tests
By default, pytest keeps the temp dirs of every test for the last 3
sessions.  Each e2e test clones & checks out a snapshot of the tree
under its `tmp_path`, so these add up, and /tmp is often a tmpfs.

Set `tmp_path_retention_policy` to `failed` so a test's `tmp_path` is
removed as soon as the test passes, and the session's temp dir is
removed when all tests pass.  The `tmp_path` of a failed test is still
kept for debugging.  Pass `-o tmp_path_retention_policy=all` to keep
the dirs of passing tests too.

Test: vpython3 run_tests
Change-Id: I2cf464b063d62f155ffdbfed24a12ea9774086d6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656093
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
2026-11-12 08:43:05 -08:00
Mike Frysinger b1a1cc603a tests: switch to mock.patch.object
Python style reviewers often point people to use mock.patch.object
since it can be statically checked better.  Migrate code, and update
the SKILL file to cover this too.

Change-Id: If7aa8e3ddcdb3d739187b12e3034de1de990a7df
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656246
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-11-11 18:19:24 -08:00
Mike Frysinger 2b7969d1cf run_tests: turn into multicall program
Add shortcuts to easily run individual tools while re-using the embedded
vpython/UV script settings.

Change-Id: I4c63e7fd402666628a18cc77187c958d93910f80
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656245
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-11-11 15:34:09 -08:00
Gavin Mak bb9b4c2b60 git_refs, project: read files-backend refs in Python
GitRefs._LoadAll spawns `git for-each-ref` to load HEAD and refs/*, and
Project._InitAnyMRef leaves warm snapshots stale after direct UpdateRef
writes. On virtualized checkouts with thousands of repositories, these
spawns add avoidable process overhead.

Read HEAD, packed-refs, and loose refs directly in GitRefs when the
repository uses a plain files ref backend, recording (mode, mtime, size,
inode) stat tuples during the single directory pass and rechecking
packed-refs afterward to guard against concurrent pack-refs updates.
The tuples leave out ctime, which backup agents and file system crawlers
move without changing a ref. Fall back to `git for-each-ref` when
GIT_DIR, GIT_COMMON_DIR, reftable, commondir, custom refStorage config,
symlinks, or lockfiles are present.

Also add GitRefs.TryResolveRef for single-ref lookups without scanning
all of refs/, GitRefs.Invalidate, and GitRefs.UpdateRef to keep cached
snapshots warm after _InitAnyMRef writes a direct ref.

Bug: 543101585
Change-Id: I804ac45712208e9f9e6437012661bc1de58b9825
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655963
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-11-11 14:56:42 -08:00
Brian Gan 33aaf275be sync: handle smart sync when the manifest checkout has no branch
If the manifest checkout in `.repo/manifests` is on a detached HEAD, or
on a local branch without an upstream, `Sync._GetBranch` read a missing
`merge` config and crashed with `'NoneType' object has no attribute
'startswith'`. When smart sync is enabled by default, this broke plain
`repo sync` for such checkouts because `AttributeError` bypassed the
fallback to a regular ToT sync in `_ExecuteHelper`, which only catches
`SmartSyncError` and `ManifestParseError`. Pinning the manifest via
`repo init -b <sha>` or `repo init -b refs/tags/<tag>` also sent the
raw revision to the manifest server as a branch name.

Add `ManifestProject.GetMergeBranch(heads_only=...)` (shared with
`XmlManifest._Load`) to resolve the upstream ref of the current
manifest branch, returning `None` under `heads_only=True` unless it
tracks `refs/heads/*`, and raise `SmartSyncError` in `_SmartSyncSetup`
before running the remote helper or contacting the manifest server.
Implicit smart sync now falls back to a regular ToT sync, and explicit
smart sync fails early with an actionable error message.

Bug: 580004259
Test: vpython3 run_tests
Change-Id: Ice1f9bfe7ea8c923a283bcbd0ac7576cf33e42f6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655123
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-11-11 12:02:47 -08:00
Mike Frysinger f6523426b9 style: use from subcmds import sync everywhere
Align with Google style guides and `from subcmds import sync` as a
module rather than import symbols out of it.

Change-Id: Iac4ae43fc8f43304b6ffe381cf290378423cd133
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655847
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-11-11 09:45:00 -08:00
Mike Frysinger d519ee40a9 style: use import hooks everywhere
Align with Google style guides and `import hooks` as a module
rather than import symbols out of it.

Change-Id: I85a071ee73f8904ac33eae437d7853785762ba71
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655846
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-11 09:40:37 -08:00
Mike Frysinger 9b774577cb pytest: enable xdist usage
The testsuite has grown in size quite a bit.  It takes ~3 min on
my cloudtop with ~48 cores.  Running it in parallel makes it take
~20 seconds.

I've capped it to 32 jobs because it seems to complete in the same
amount of time as 128 jobs, and there's startup cost to spinning
up each node.

Change-Id: I6daa456a92e9db213118c9d16c9dee646db448e0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655887
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-11-11 09:14:21 -08:00
Mike Frysinger 0714b7e434 run_tests: regen uv.lock
Since vpython no longer does this automatically, I upgraded flake8
incorrectly by only updating the script file.  Add a note to remind
people they need to regen the lock file themselves.

Change-Id: I7884714a110bbf843c76aca6e3a2ff14b932b943
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655845
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-11-11 05:56:58 -08:00
Mike Frysinger 96363629bd flake8: fix warnings with newer versions
The flake8 version hasn't updated properly due the uv.lock file
being out of date.  Fix newer warnings before unwedging that.

We have a lot of B042 errors that probably should be fixed by reworking
the code, but for now suppress them since we've been doing this for a
long time in repo.
error.py:25:5: B042 Exception class with `__init__` should pass all args to `super().__init__()` to work in edge cases of `pickle` and `copy.copy()`. It should also not take any kwargs.

The only other warning is fairly simple:
subcmds/sync.py:1154:9: B018 Found useless Call expression. Consider either assigning it to a variable or removing it.

Change-Id: I532000a66a5d8493a490fa30080b137aff5f706e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655844
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-11-10 17:59:34 -08:00
Mike Frysinger 11bf826a10 git_command: move _GetBasicEnv to module level
RepoSourceVersion() and _build_env() are module-level functions that
called GitCommand._GetBasicEnv(), while GitCommand itself never called
it directly.  When tests mock git_command.GitCommand before
RepoSourceVersion() has cached its result (such as running
test_upload_for_review_real_event_log in isolation), calling
GitCommand._GetBasicEnv() returned a MagicMock instead of an env dict,
causing subprocess.run to fail with ValueError.

Change-Id: Ib8b866b012602c549ce2d59105526d847b0d7d51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656383
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-10 17:58:35 -08:00
Brian Gan 1e88f21cb1 sync: remove obsolete projects before interleaved checkouts
In `_SyncInterleaved`, `_UpdateManifestLists` only ran after fetching
and checking out all projects. When a manifest replaced a parent project
`foo` with a nested child project `foo/bar`, `foo/bar` was checked out
inside the still-existing obsolete `foo` worktree before
`UpdateProjectList` deleted `foo`. Because `bar/` was tracked by `foo`,
`git status` in `foo` saw `foo/bar`'s newly checked-out files as
modified and untracked files belonging to `foo`, causing
`DeleteWorktree` to fail with `DeleteDirtyWorktreeError` (and similarly,
when `foo/bar` was merged back into `foo`, `foo`'s checkout collided
with the not-yet-deleted `foo/bar` worktree):

  error: foo: Cannot remove project: uncommitted changes are present.
  error: Unable to fully sync the tree
  error: Updating local project lists failed.

Split `_UpdateManifestLists` into `_UpdateProjectLists` and
`_UpdateCopyLinkfileLists`, and run `_UpdateProjectLists` before the
sync loop in `_SyncInterleaved` so obsolete manifest projects are
deleted before new project checkouts begin (and `--fail-fast` aborts
before fetching if an obsolete project cannot be removed). Run
`_UpdateProjectLists` again after the sync loop (passing the set of
paths that already failed removal pre-sync so they are not retried or
reported twice) together with `_UpdateCopyLinkfileLists` so derived
submodules discovered or dropped when `_ReloadManifest` runs during the
loop are updated in `project.list` and copyfiles/linkfiles are updated.

Because the pre-sync `_UpdateProjectLists` pass in `_SyncInterleaved`
runs before projects are fetched (unlike `_SyncPhased`, which runs it
after `_FetchMain`), a `sync-s="true"` parent project whose manifest
revision moved to an unfetched commit cannot derive its submodules yet
(`GetSubmoduleRevisions() is None`). Retain existing submodule checkouts
under such parents until a post-fetch pass can determine whether their
submodules were removed. Also continue past individual
`DeleteWorktreeError` failures in `UpdateProjectList` (skipping obsolete
ancestors of any retained child worktree) so remaining clean obsolete
projects are still removed while failed paths stay tracked in
`project.list`, and skip project list and copy/linkfile updates when
`--network-only` is set so `-n` does not mutate local worktrees.

Bug: 574048485
Test: vpython3 run_tests
Change-Id: I530337977f063173a9a8a48df3250627cd4bc57f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656283
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
2026-11-10 15:25:06 -08:00
Mike Frysinger 4bb901eab0 style: use import command everywhere
Align with Google style guides and `import command` as a module
rather than import symbols out of it.

The help subcmd needed a minor rewrite as it had a "command"
variable already.

Change-Id: I9b465c85246670d63827df0744b00e59033e87b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656243
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-10 12:59:38 -08:00
Mike Frysinger b2e30cef91 tests: delete buggy git_command tests
The Popen mocking is incomplete & buggy:
* It uses a nested class which hinders pickling.
* Nested classes should rarely be used in the first place, stylewise.
* It does not mock the full Popen API causing failures.

For example, this fails:
$ pytest -vvv tests/test_git_command.py::GitCommandWaitTest
...
FAILED tests/test_git_command.py::GitCommandWaitTest::test_default_returns_non_zero_result - AttributeError: 'MockPopen' object has no attribute 'returncode'
FAILED tests/test_git_command.py::GitCommandWaitTest::test_raises_when_verify_non_zero_result - AttributeError: 'MockPopen' object has no attribute 'returncode'
FAILED tests/test_git_command.py::GitCommandWaitTest::test_returns_when_no_verify_non_zero_result - AttributeError: 'MockPopen' object has no attribute 'returncode'
...

Delete the tests since they don't seem to add much value currently.

Bug: b/293344017
Change-Id: I3d613a84657e30cbc7eac0b363ebfcb46b95eb49
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655886
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-10 11:17:32 -08:00
Mike Frysinger 27402ddb25 style: use import main everywhere
Align with Google style guides and `import main` as a module
rather than import symbols out of it.

Change-Id: Iecfb3c89a14d01b24ebfb0aa4584d8d6a94f70fd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654821
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-10 11:08:42 -08:00
Mike Frysinger 5b04e6e7c5 style: use import subcmds everywhere
Align with Google style guides and `import subcmds` as a module
rather than import symbols out of it.

Change-Id: If19503b977798f8a30e5d54cea9cd63a5c7b81d1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654820
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-10 11:01:44 -08:00
Mike Frysinger eb668be30d style: use import command in unittests
Align with Google style guides and `import command` as a module
rather than import symbols out of it in the tests tree.

Change-Id: I5c151d22c9ed8f8c7ea762823c8b2e25225a2ac4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654819
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-10 10:45:58 -08:00
Mike Frysinger d7c431b996 style: use import progress everywhere
Align with Google style guides and `import progress` as a module
rather than import symbols out of it.

Change-Id: I3ef9109af8c6883ac0df7fa19bdb88bc6cdfd5e9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654818
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-11-10 09:45:00 -08:00
Mike Frysinger 020767b9d3 style: use import wrapper everywhere
Align with Google style guides and `import wrapper` as a module
rather than import symbols out of it.

Change-Id: Ifc6165974a242a1ce38bd09b1b4226dc0ba943d5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654817
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-10 07:24:10 -08:00
Mike Frysinger 0f83e19bbb style: use import git_trace2_event_log_base everywhere
Align with Google style guides and `import git_trace2_event_log_base` as
a module rather than import symbols out of it.

Change-Id: I98c68a9ddca06325a8e0a620346c408e1ff3b661
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654816
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-10 07:13:59 -08:00
Mike Frysinger 221a31cd0a style: use import git_trace2_event_log everywhere
Align with Google style guides and `import git_trace2_event_log` as a
module rather than import symbols out of it.

Change-Id: I4fdb0692ccbcf38ecdd756a4417c68f0cfaebc0c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654815
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-11-09 19:52:17 -08:00
Mike Frysinger ccddf51633 style: use import git_superproject everywhere
Align with Google style guides and `import git_superproject` as a module
rather than import symbols out of it.

Change-Id: I939f1123d5c6f79b56c244a5ffdf5bcfa136b2db
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654814
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-09 19:43:03 -08:00
Mike Frysinger 72e3c8c277 style: use import event_log everywhere
Align with Google style guides and `import event_log` as a module
rather than import symbols out of it.

Change-Id: I2c558f1161490b354f0b956ac0e86e658942c3a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654813
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-09 19:06:10 -08:00
Mike Frysinger 4cef373d98 style: use import manifest_xml everywhere
Align with Google style guides and `import manifest_xml` as a module
rather than import symbols out of it.

Change-Id: Ib600a5ccaf53b178e27e6e9d56847751d266bfe4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654812
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-09 17:34:52 -08:00
Gavin Mak 95d7c8c0df sync: don't read target tree after reprojectcmd
After repo.reprojectcmd exits 0, _Reproject ran `git diff-index --cached
--name-only <revid>^{tree}` to verify that the index matches the target
commit's tree.

Peeling <revid>^{tree} and comparing every entry against the index
forces Git to read the target commit and walk every tree object in the
target revision from .git/objects.

reprojectcmd exists for virtualized filesystems such as ABFS whose
checkout command materializes the worktree and writes .git/index from
its own tree metadata without reading .git/objects/pack/*-trees-*.pack.
Running diff-index against <revid>^{tree} afterward forces Git to fault
in the tree packfiles over the network (260 FetchTrees pack downloads /
1.64 GiB across a 3,296-project Android checkout).

Trust reprojectcmd's exit status for index materialization instead of
re-walking the target tree in Git, while keeping the HEAD postcondition
check that verifies the command did not modify refs.

Bug: 543101585
Change-Id: Ie5feea41a4191245232c6763a489d01596370c55

Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643004
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-09 15:15:49 -08:00
Gavin Mak 616396317c sync: verify fetchcmd postconditions without reading objects
After repo.fetchcmd exits, _CustomFetch verified the tracking ref and
FETCH_HEAD with ResolveCommit, i.e. `git rev-parse --verify
<ref>^{commit}`. Peeling to a commit makes Git look the object up: it
opens every pack index in objects/pack/ until it finds the commit, then
reads the commit from its packfile.

fetchcmd exists for object stores that are populated lazily, such as
ABFS, whose `abfs git fetch` registers packfiles as virtual files and
returns at once. There the peel is the first read of the packs the
command registered, and it blocks until they are downloaded. A project
whose revision did not change needs no objects to sync, yet paid that
download whenever fetchcmd registered new packs for it.

Compare the object IDs that the refs record instead (ResolveRef, which
runs the same rev-parse without the peel). Only when they differ peel
both sides to commits, so an annotated tag still matches the commit it
points to. A commit missing from the object store now fails the checkout
that reads it instead of the network half.

On a first sync the checkout reads the commit anyway, so there the
download only moves to the next command that needs the object.

Bug: 543101585
Change-Id: I51248ff1e9674b16e3ff8cee69572dbc8c7a8e99
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643003
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-09 15:15:43 -08:00
Gavin Mak 4978203b07 sync: skip shallow bloat check when using fetchcmd
Skip projects fetched via repo.fetchcmd in _CheckForBloatedProjects,
and expose Project.UseFetchCmd alongside Project.UseReprojectCmd.

The bloat check exists because Project._RemoteFetch passes --no-auto-gc
for clone-depth="1" repositories. Projects using repo.fetchcmd bypass
_RemoteFetch and manage object storage externally, so scanning their
worktrees for bloat is unnecessary and recommending git repack can
materialize virtualized objects locally.

Bug: 543101585
Change-Id: I16510ad3b5b39c3a5e2a1a75df706439252a088e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655083
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
2026-11-09 15:03:09 -08:00
Mike Frysinger cc3f52cd6c style: use import git_config everywhere
Align with Google style guides and `import git_config` as a module
rather than import symbols out of it.

Change-Id: I467f1f8c0868c53b71917ca0a9610606e1ce4d83
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654811
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-09 13:35:49 -08:00
John Cai e045aa19d8 Adding sparse checkout functionality
Sparse checkout allows Git to check out only a portion of the
repository. This is particularly helpful for monorepository workflows
where CI jobs only need a portion of the repository to run tests.

This speeds up the CI jobs, and puts less load on the Git server as it
relieves it from having to pack up all the objects.

In the Git project, though it is still labeled experimental, the feature
has been stable for a number of years and is unlikely to change at this
point.

This change allows configuring Git to use sparse checkout and provide a
list of directories for the sync command to retrieve.

Projects with sparse paths default to a blob:none partial clone so the
excluded paths are never fetched, unless a filter is already configured
or the project is excluded from partial clones.  Both the plain and the
git-worktree checkout flows configure sparse-checkout before populating
the working tree.

Change-Id: I772c7db5209b345306be378771c8f2040832bca1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/535061
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: John Cai <johncai86@gmail.com>
Commit-Queue: John Cai <johncai86@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-11-09 11:14:00 -08:00
Mike Frysinger c382d3807a style: use import color everywhere
Align with Google style guides and `import color` as a module
rather than import symbols out of it.

Change-Id: Id3f1f8ea324097d6e1ea5ed33c67b4b024abce7f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654810
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-09 09:07:27 -08:00
Mike Frysinger 9d01d216e0 style: use import test_manifest_xml everywhere
Align with Google style guides and `import test_manifest_xml` as a module
rather than import symbols out of it.

Change-Id: Ib85716bea2eb09028e05b86d445336405e1df870
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654809
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-11-09 08:52:28 -08:00
Mike Frysinger 52f01a18da style: use import pager everywhere
Align with Google style guides and `import pager` as a module
rather than import symbols out of it.

Change-Id: I20ec5024ece35ae27bf46364e97c59400aae0cf0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654808
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-09 07:55:40 -08:00
Mike Frysinger 3a3d1fc7cd style: use import editor everywhere
Align with Google style guides and `import editor` as a module
rather than import symbols out of it.

Change-Id: Ie941ff12f63c2b168bcec6510751e155156f26e7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654807
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-11-09 07:09:25 -08:00
Mike Frysinger 2029997aef style: use import io everywhere
Align with Google style guides and `import io` as a module
rather than import symbols out of it.

Change-Id: Ic3e78aa81ffc24687f61c7ce744e596fbdb13415
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654806
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-06 11:58:10 -08:00
Mike Frysinger caa4cd6564 style: use import collections everywhere
Align with Google style guides and `import collections` as a module
rather than import symbols out of it.

Change-Id: I165028d5b70a78492e361e6b6f9e3eea19d014b6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654805
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-06 11:57:56 -08:00
Mike Frysinger e15b3e0298 style: use import urllib.xxx everywhere
Align with Google style guides and `import urllib.xxx` as a module
rather than import symbols out of it.

Change-Id: Ia177f4fd5ee2a3f38f8a8860e9a2ba8beac8078f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654804
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-11-06 11:56:05 -08:00
Mike Frysinger a4b44c5026 style: use import types everywhere
Align with Google style guides and `import types` as a module
rather than import symbols out of it.

Change-Id: I20f8058eb1b5dafc1d2964c276e28b95b188bfff
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654803
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-06 10:29:39 -08:00
Mike Frysinger 8f2c0f1483 flake8: enable flake8-import-restrictions
Disable the warning across existing modules for now.  We'll need to
clean those up, but we can at least stop new bad modules from being
added to the tree.

Change-Id: I39be6450ecde86ce30553b6e277f870e0d3865c1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654623
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-11-05 11:03:12 -08:00
Mike Frysinger 7ed9700a95 flake8: fix nested import checks
Import modules at the top of the module in most cases.  Many of these
weren't actually saving any time:
* editor: shlex is already imported by the main module
* platform_utils: On Windows, the win32 module is almost always needed
* tests: There's no point in optimizing these.

The trace import in the main module is about the only one worth allowing
since that module itself is rarely used, and only when in a specific
debug mode.

Change-Id: I5d3fcecbe51e0ce78d4e856d7033565150ef4526
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654103
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-11-05 10:50:14 -08:00
hungweichen d621aa8ee4 event_log: only record events when --event-log is specified
Add an enabled flag to EventLog (defaulting to False on
Command.event_log and enabled by _Repo._RunLong when --event-log is
passed) and return None early from EventLog.Add when disabled.

Because EventLog.AddSync only resolves project.GetCommitRevisionId()
when EventLog.Add returns an event, disabling Command.event_log by
default eliminates 3,300+ synchronous git subprocess calls in
MainProcess result callbacks during normal repo sync runs without
touching subcmds/sync.py.

Bug: 564817989
Change-Id: I79dd66d1c392ba5ee562f104759e36d1dd6eefea
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649606
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-11-05 08:19:19 -08:00
hungweichen b16e5f49d6 git_refs, project: skip redundant git calls on unborn HEAD and warm refs
Read plain files-backend unborn HEAD symrefs directly before spawning
git symbolic-ref, update already-loaded bare_ref snapshots in
_InitAnyMRef when no external ref changes occurred, and reuse warm
bare_ref snapshots in GetRevisionId for non-tag refs.

Also skip redundant rev-parse and empty-index ls-files subprocesses in
_Reproject when HEAD is guaranteed to be an unborn local ref.

Bug: 564817989
Change-Id: I0da10cefb14753ce6e1e22e192d652790d59a5b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649605
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-11-05 07:52:57 -08:00
hungweichen 6596cb6499 git_config, project: keep config cache coherent and skip init templates
Update .repo_config.json atomically with file metadata and keep cached
sections, remotes, and branches coherent across SetString, Branch.Save,
ClearCache, and Reload. Also reload Project.config in workers when
fetchcmd or reprojectcmd modifies .git/config externally so subsequent
reads hit the JSON cache instead of re-spawning git config --list in
MainProcess.

Also pass --template= to git init when no custom template directory is
configured so Git does not copy sample hooks only to delete them.

Bug: 564817989
Change-Id: I5ff655b0ab0b34a4f77f13617bbe5a1fff488c8c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649604
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-11-05 07:43:32 -08:00
hungweichen ecb91f465e project: trace fetchcmd and reprojectcmd executions
Wrap repo.fetchcmd and repo.reprojectcmd subprocess invocations in
Trace blocks so TRACE_FILE records START/END spans for custom fetch and
reproject commands alongside standard Git commands.

Bug: 564817989
Change-Id: I68a4d863370e7c46d144e35e5826d01b386b9b07
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649603
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Haamed Gheibi <haamed@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-11-04 13:10:24 -08:00
Romaric Jodin 3656290156 upload: add --ignore-remote-commits
When uploading a branch that merges another remote-tracking branch (such
as an upstream branch into the destination branch), ReviewableBranch
only excludes the destination tracking branch (^base). As a result, git
rev-list walks the merged parent and counts all merged commits even
though they already exist on a remote branch, triggering the unusual
commit count warning in _VerifyPendingCommits.

Add --ignore-remote-commits to `repo upload` to pass --not --remotes to
git rev-list in ReviewableBranch so commits already reachable from
refs/remotes/* are excluded from the pending commit list.

Test: ./run_tests tests/test_project.py tests/test_subcmds_upload.py
Change-Id: Ib1fb864982729e20a621144580fe95daa478eaee
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/651284
Tested-by: Romaric Jodin <rjodin@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Romaric Jodin <rjodin@google.com>
2026-11-03 00:10:19 -08:00
Brian Gan b90c7cd8f9 docs: add Repology links for distro package versions
The Ubuntu and Debian release wiki pages added in
https://gerrit-review.googlesource.com/c/git-repo/+/649983 only list
distro release and EOL dates, not which versions of Python, Git, or
OpenSSH shipped in each distro release.

Add Repology package version links alongside the distro release pages
in python-support.md and requirements.json so maintainers can check
which versions shipped across historical Ubuntu and Debian releases.

Bug: 575551885
Change-Id: Ib256fd9b364074b52d3e15d185af519d8376cb28
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/650868
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
2026-11-02 13:20:14 -08:00
Brian Gan c5340585b8 docs: release: update release process to match current practice
The documented process has drifted from how releases are actually done:
* Releases are now cut weekly by an automated pipeline, so the manual
  steps are only a fallback, and for rollbacks.
* Releases are tested (both updating an existing checkout and a fresh
  init) before pushing anything.
* The tag is pushed & announced first, and only promoted to stable after
  a day.
* Direct pushes to the stable branch are rejected, so it has to be
  deleted & recreated via the Gerrit REST API instead.

Bug: 575551885
Change-Id: Ib36df07811c225558aba1a925b9703f3b2017e6f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649984
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
2026-10-27 12:29:45 -07:00
Brian Gan 86a7bec83d docs: release: remove stale Project References table
The table of git/python/ssh/distro versions hasn't been kept up to date
and duplicates information maintained upstream by the distros and
projects themselves.  Drop it, and point the places that referenced it
(python-support.md & requirements.json) at the Ubuntu & Debian release
pages instead.

Bug: 575551885
Change-Id: Id1cb2225b3852273ca717fc7a443449fb2fd006e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649983
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
2026-10-27 10:39:40 -07:00
107 changed files with 7802 additions and 1582 deletions

No files matched your search

+32 -1
View File
@@ -1,9 +1,32 @@
# https://flake8.pycqa.org/en/latest/user/configuration.html
[flake8] [flake8]
max-line-length = 80 max-line-length = 80
per-file-ignores = per-file-ignores =
# E501: line too long # E501: line too long
tests/test_git_superproject.py: E501 tests/test_git_superproject.py: E501,
# IMR241: When using the from syntax, only submodules are imported, not module elements.
# NB: Do not add any more here.
command.py: IMR241,
git_status.py: IMR241,
main.py: IMR241,
manifest_xml.py: IMR241,
subcmds/gc.py: IMR241,
subcmds/rebase.py: IMR241,
subcmds/sync.py: IMR241,
subcmds/upload.py: IMR241,
subcmds/wipe.py: IMR241,
tests/e2e/test_failures.py: IMR241,
tests/e2e/test_groups.py: IMR241,
tests/e2e/test_init_sync.py: IMR241,
tests/e2e/test_manifest_regressions.py: IMR241,
tests/e2e/test_partial_clone.py: IMR241,
tests/e2e/test_sync_modes.py: IMR241,
tests/e2e/test_workflows.py: IMR241,
tests/test_subcmds_sync.py: IMR241,
extend-ignore = extend-ignore =
# B042: Exception class with __init__ should pass all args to super().__init__()
# We should fix this by reworking our exception classes.
B042,
# E203: Whitespace before ':' # E203: Whitespace before ':'
# See https://github.com/PyCQA/pycodestyle/issues/373 # See https://github.com/PyCQA/pycodestyle/issues/373
E203, E203,
@@ -15,3 +38,11 @@ exclude =
.venv, .venv,
venv, venv,
.tox, .tox,
# IMR241: When using the from syntax, only submodules are imported, not module elements.
# NB: Do not add more modules here without strong discussion. This is aligned
# with Google's Python style guide.
imr241_exclude =
collections.abc,
pathlib,
typing,
typing_extensions,
+51 -8
View File
@@ -3471,6 +3471,12 @@ standard streams and filesystem paths.
: : : : asserted directly : : : : : asserted directly :
: : : : in the test : : : : : in the test :
: : : : docstring. : : : : : docstring. :
| **T5-29** | Object-Referenced Mocking via | High | Passing a dotted |
: : `mock.patch.object` : : string target to :
: : : : `mock.patch` :
: : : : instead of using :
: : : : `mock.patch.object` :
: : : : on the object. :
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
@@ -3903,13 +3909,13 @@ file where it was locally imported.**
**Don't:** **Don't:**
```python ```python
@mock.patch("main.time.sleep") @mock.patch.object(main.time, "sleep")
``` ```
**Do:** **Do:**
```python ```python
@mock.patch("time.sleep") @mock.patch.object(time, "sleep")
``` ```
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
@@ -3939,7 +3945,7 @@ LRU cache intact across tests.**
```python ```python
class SshTests(unittest.TestCase): class SshTests(unittest.TestCase):
def test_version(self): def test_version(self):
with mock.patch("ssh._run_ssh_version", return_value="OpenSSH_1.2\n"): with mock.patch.object(ssh, "_run_ssh_version", return_value="OpenSSH_1.2\n"):
self.assertEqual(ssh.version(), (1, 2)) self.assertEqual(ssh.version(), (1, 2))
``` ```
@@ -3951,7 +3957,7 @@ class SshTests(unittest.TestCase):
ssh.version.cache_clear() ssh.version.cache_clear()
def test_version(self): def test_version(self):
with mock.patch("ssh._run_ssh_version", return_value="OpenSSH_1.2\n"): with mock.patch.object(ssh, "_run_ssh_version", return_value="OpenSSH_1.2\n"):
self.assertEqual(ssh.version(), (1, 2)) self.assertEqual(ssh.version(), (1, 2))
``` ```
@@ -4015,15 +4021,15 @@ with contextlib.redirect_stderr(f):
**Don't:** **Don't:**
```python ```python
with mock.patch("sys.stderr") as mock_stderr: with mock.patch.object(time, "sleep") as mock_sleep:
with mock.patch("socket.socket") as ms: with mock.patch.object(socket, "socket") as ms:
run_test() run_test()
``` ```
**Do:** **Do:**
```python ```python
with mock.patch("sys.stderr") as mock_stderr, mock.patch("socket.socket") as ms: with mock.patch.object(time, "sleep") as mock_sleep, mock.patch.object(socket, "socket") as ms:
run_test() run_test()
``` ```
@@ -4075,7 +4081,7 @@ mock_socket.connect.side_effect = OSError("Mock error")
**Do:** **Do:**
```python ```python
mock.patch("socket.socket", side_effect=OSError) mock.patch.object(socket, "socket", side_effect=OSError)
``` ```
**Exceptions:** Tests explicitly verifying exact formatting serialization (e.g., **Exceptions:** Tests explicitly verifying exact formatting serialization (e.g.,
@@ -4613,6 +4619,43 @@ def test_Color_Parse_first_parameter_none(self):
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
#### T5-29: Object-Referenced Mocking via `mock.patch.object`
> **Rule:** Never use `mock.patch`; always use `mock.patch.object`.
>
> **What:** All attribute and function patching in unit tests must use
> `mock.patch.object(target, "attribute", ...)` against an explicitly imported
> module or class object rather than passing dotted string paths to
> `mock.patch("module.attribute", ...)`.
>
> **Applies To:** All unit tests utilizing `unittest.mock`.
>
> **Why:** String-based `mock.patch` paths bypass static analysis and module
> import validation, making tests brittle during refactoring and prone to silent
> resolution errors or import ordering bugs. Referencing the imported object
> directly via `mock.patch.object` ensures the target module is explicitly
> imported and validated at load time. Failing to adhere to this typically
> results in **Brittle Patch Routing / Unvalidated String Targets**.
**Trap 1: Passing a dotted string target to `mock.patch` instead of using
`mock.patch.object` on the object.**
**Don't:**
```python
with mock.patch("ssh._run_ssh_version", return_value="OpenSSH_1.2\n"):
...
```
**Do:**
```python
with mock.patch.object(ssh, "_run_ssh_version", return_value="OpenSSH_1.2\n"):
...
```
--------------------------------------------------------------------------------
### Cross-Domain Dependencies ### Cross-Domain Dependencies
* **Upstream:** T2 | Filesystem Atomicity & Worktree Layout - *Managing safe * **Upstream:** T2 | Filesystem Atomicity & Worktree Layout - *Managing safe
Symlink
+1
View File
@@ -0,0 +1 @@
run_tests
+1 -1
View File
@@ -82,7 +82,7 @@ def _Color(fg=None, bg=None, attr=None):
return code return code
DEFAULT = None DEFAULT: Optional[str] = None
class _CheckConsoleSentinel: class _CheckConsoleSentinel:
+9 -6
View File
@@ -20,7 +20,7 @@ import re
from typing import List, Optional, TYPE_CHECKING from typing import List, Optional, TYPE_CHECKING
import error import error
from event_log import EventLog import event_log
import progress import progress
@@ -43,7 +43,7 @@ WORKER_BATCH_SIZE = 32
# How many jobs to run in parallel by default? This assumes the jobs are # How many jobs to run in parallel by default? This assumes the jobs are
# largely I/O bound and do not hit the network. # largely I/O bound and do not hit the network.
DEFAULT_LOCAL_JOBS = min(os.cpu_count(), 8) DEFAULT_LOCAL_JOBS = min(os.cpu_count() or 1, 8)
class UsageError(error.RepoExitError): class UsageError(error.RepoExitError):
@@ -57,7 +57,7 @@ class Command:
# provide event summary to callers. Only used by sync subcommand currently. # provide event summary to callers. Only used by sync subcommand currently.
# #
# NB: This is being replaced by git trace2 events. See git_trace2_event_log. # NB: This is being replaced by git trace2 events. See git_trace2_event_log.
event_log = EventLog() event_log = event_log.EventLog(enabled=False)
# Whether this command is a "common" one, i.e. whether the user would # Whether this command is a "common" one, i.e. whether the user would
# commonly use it or it's a more uncommon command. This is used by the help # commonly use it or it's a more uncommon command. This is used by the help
@@ -105,7 +105,8 @@ class Command:
# Cache for the OptionParser property. # Cache for the OptionParser property.
self._optparse = None self._optparse = None
def WantPager(self, _opt): def WantPager(self, opt) -> bool:
del opt # Unused.
return False return False
@staticmethod @staticmethod
@@ -623,7 +624,8 @@ class InteractiveCommand(Command):
within a pager, even if the user asks to. within a pager, even if the user asks to.
""" """
def WantPager(self, _opt): def WantPager(self, opt) -> bool:
del opt # Unused.
return False return False
@@ -632,7 +634,8 @@ class PagedCommand(Command):
larger than one screen full. larger than one screen full.
""" """
def WantPager(self, _opt): def WantPager(self, opt) -> bool:
del opt # Unused.
return True return True
+3
View File
@@ -249,6 +249,7 @@ _repo() {
'(-s --smart-sync)'{-s,--smart-sync}'[Default to smart sync]' \ '(-s --smart-sync)'{-s,--smart-sync}'[Default to smart sync]' \
'--no-smart-sync[Do not default to smart sync]' \ '--no-smart-sync[Do not default to smart sync]' \
'--head[Do not default to smart sync]' \ '--head[Do not default to smart sync]' \
'--smart-sync-branch=[Smart sync branch]:branch:' \
'--clone-bundle[Use clone bundle]' \ '--clone-bundle[Use clone bundle]' \
'--no-clone-bundle[Do not use clone bundle]' \ '--no-clone-bundle[Do not use clone bundle]' \
'--git-lfs[Use git lfs]' \ '--git-lfs[Use git lfs]' \
@@ -315,6 +316,7 @@ _repo() {
${common_opts} \ ${common_opts} \
'--jobs-network=[Number of network jobs]:jobs:' \ '--jobs-network=[Number of network jobs]:jobs:' \
'--jobs-checkout=[Number of checkout jobs]:jobs:' \ '--jobs-checkout=[Number of checkout jobs]:jobs:' \
'--force-jobs[Force jobs even if manifest caps it]' \
'(-f --force-broken)'{-f,--force-broken}'[Continue sync even if a project fails]' \ '(-f --force-broken)'{-f,--force-broken}'[Continue sync even if a project fails]' \
'--fail-fast[Fail fast]' \ '--fail-fast[Fail fast]' \
'--force-sync[Overwrite existing git directories]' \ '--force-sync[Overwrite existing git directories]' \
@@ -411,6 +413,7 @@ _repo() {
'(-y --yes)'{-y,--yes}'[Answer yes to all prompts]' \ '(-y --yes)'{-y,--yes}'[Answer yes to all prompts]' \
'--ignore-untracked-files[Ignore untracked files]' \ '--ignore-untracked-files[Ignore untracked files]' \
'--no-ignore-untracked-files[Do not ignore untracked files]' \ '--no-ignore-untracked-files[Do not ignore untracked files]' \
'--ignore-remote-commits[Ignore commits already in remote branches]' \
'--no-cert-checks[Do not check certificates]' \ '--no-cert-checks[Do not check certificates]' \
'--no-verify[Do not verify]' \ '--no-verify[Do not verify]' \
'--verify[Verify]' \ '--verify[Verify]' \
+14 -9
View File
@@ -42,13 +42,13 @@ postconditions to be met:
1. When the manifest revision names a branch or tag, the mapped local tracking 1. When the manifest revision names a branch or tag, the mapped local tracking
ref (e.g. `refs/remotes/REPO_REMOTE/<branch>` for a branch, or ref (e.g. `refs/remotes/REPO_REMOTE/<branch>` for a branch, or
`refs/tags/<tag>` for a tag) must resolve to a valid commit in the object `refs/tags/<tag>` for a tag) must name the target commit, and must equal
store (`git rev-parse --verify <ref>^{commit}`), and must equal `REPO_TREV` `REPO_TREV` if `REPO_TREV` was non-empty. A tag ref may name an annotated
if `REPO_TREV` was non-empty. tag of the target commit.
2. `FETCH_HEAD` must resolve to the same target commit in the object store 2. `FETCH_HEAD` must name the same target commit.
(`git rev-parse --verify FETCH_HEAD^{commit}`). 3. The target commit, and the commit graph from it far enough to compute
3. The commit graph from the target commit must be reachable far enough to merge bases with local branches, must be readable from the object store.
compute merge bases with local branches. They may be materialized lazily, when first read.
### Invariants ### Invariants
@@ -66,5 +66,10 @@ postconditions to be met:
* A non-zero exit status aborts the project's sync, and the command's stderr * A non-zero exit status aborts the project's sync, and the command's stderr
is surfaced to the user. is surfaced to the user.
* `repo` verifies the tracking ref and target reachability after exit 0. Any * After exit 0, `repo` checks postconditions 1 and 2 by comparing the object
mismatch is treated as a failure. IDs that the refs record (`git rev-parse --verify <ref>`), so it does not
read the object store during the network half of the sync. It peels the
IDs to commits only when they differ, as for an annotated tag. Any mismatch
is treated as a failure.
* A target commit missing from the object store fails the checkout that reads
it.
+6 -2
View File
@@ -42,8 +42,11 @@ For example, if you want to change the manifest branch, you can simply run
change the git URL/branch that this tracks, re-run `repo init` with the new change the git URL/branch that this tracks, re-run `repo init` with the new
settings. settings.
* `.repo_fetchtimes.json`: Used by `repo sync` to record fetch times when * `.repo_fetchtimes.json`: Used by `repo sync` to record per-project fetch
syncing the various projects. times so subsequent syncs schedule the slowest projects first.
* `.repo_checkouttimes.json`: Used by `repo sync` to record per-project
checkout times so subsequent syncs schedule the slowest projects first.
* `.repo_localsyncstate.json`: Used by `repo sync` to detect and warn on * `.repo_localsyncstate.json`: Used by `repo sync` to detect and warn on
on partial tree syncs. Partial syncs are allowed by `repo` itself, but are on partial tree syncs. Partial syncs are allowed by `repo` itself, but are
@@ -177,6 +180,7 @@ User controlled settings are initialized when running `repo init`.
| repo.partialcloneexclude | `--partial-clone-exclude` | Comma-delimited list of project names (not paths) to exclude while using [partial git clones] | | repo.partialcloneexclude | `--partial-clone-exclude` | Comma-delimited list of project names (not paths) to exclude while using [partial git clones] |
| repo.reference | `--reference` | Reference repo client checkout | | repo.reference | `--reference` | Reference repo client checkout |
| repo.smartsync | `--smart-sync` | Default to [Smart Sync](./smart-sync.md) when syncing | | repo.smartsync | `--smart-sync` | Default to [Smart Sync](./smart-sync.md) when syncing |
| repo.\<branch\>.smartsyncbranch | `--smart-sync-branch` | Server branch to query when [Smart Sync](./smart-sync.md)ing `<branch>` |
| repo.submodules | `--submodules` | Sync git submodules | | repo.submodules | `--submodules` | Sync git submodules |
| repo.superproject | `--use-superproject` | Sync [superproject] | | repo.superproject | `--use-superproject` | Sync [superproject] |
| repo.worktree | `--worktree` | Use [git worktree] for checkouts | | repo.worktree | `--worktree` | Use [git worktree] for checkouts |
+69 -6
View File
@@ -74,7 +74,8 @@ following DTD:
<!ELEMENT project (annotation*, <!ELEMENT project (annotation*,
project*, project*,
copyfile*, copyfile*,
linkfile*)> linkfile*,
sparse-path*)>
<!ATTLIST project name CDATA #REQUIRED> <!ATTLIST project name CDATA #REQUIRED>
<!ATTLIST project path CDATA #IMPLIED> <!ATTLIST project path CDATA #IMPLIED>
<!ATTLIST project remote IDREF #IMPLIED> <!ATTLIST project remote IDREF #IMPLIED>
@@ -102,9 +103,13 @@ following DTD:
<!ATTLIST linkfile src CDATA #REQUIRED> <!ATTLIST linkfile src CDATA #REQUIRED>
<!ATTLIST linkfile dest CDATA #REQUIRED> <!ATTLIST linkfile dest CDATA #REQUIRED>
<!ELEMENT sparse-path EMPTY>
<!ATTLIST sparse-path path CDATA #REQUIRED>
<!ELEMENT extend-project (annotation*, <!ELEMENT extend-project (annotation*,
copyfile*, copyfile*,
linkfile*)> linkfile*,
sparse-path*)>
<!ATTLIST extend-project name CDATA #REQUIRED> <!ATTLIST extend-project name CDATA #REQUIRED>
<!ATTLIST extend-project path CDATA #IMPLIED> <!ATTLIST extend-project path CDATA #IMPLIED>
<!ATTLIST extend-project dest-path CDATA #IMPLIED> <!ATTLIST extend-project dest-path CDATA #IMPLIED>
@@ -424,10 +429,11 @@ attributes of an existing project without completely replacing the
existing project definition. This makes the local manifest more robust existing project definition. This makes the local manifest more robust
against changes to the original manifest. against changes to the original manifest.
The `extend-project` element can also contain `annotation`, `copyfile`, and The `extend-project` element can also contain `annotation`, `copyfile`,
`linkfile` child elements. These are added to the project's definition. A `linkfile`, and `sparse-path` child elements. These are added to the project's
`copyfile` or `linkfile` with a `dest` that already exists in the project definition. A `copyfile` or `linkfile` with a `dest` that already exists in the
will overwrite the original. project will overwrite the original. There is no way to remove a `sparse-path`
that the original project definition declared.
Attribute `path`: If specified, limit the change to projects checked out Attribute `path`: If specified, limit the change to projects checked out
at the specified path, rather than all projects with the given name. at the specified path, rather than all projects with the given name.
@@ -471,6 +477,63 @@ attribute "keep" which accepts the case insensitive values "true"
(default) or "false". This attribute determines whether or not the (default) or "false". This attribute determines whether or not the
annotation will be kept when exported with the manifest subcommand. annotation will be kept when exported with the manifest subcommand.
### Element sparse-path
Zero or more sparse-path elements may be specified as children of a
project element or an extend-project element. Each element specifies a
path to include in the working directory.
Requires Git 2.25.0 or later, which is when `git sparse-checkout` and cone mode
were introduced. Declaring a sparse-path with an older Git fails the sync for
that project.
Declaring at least one sparse-path is what enables Git sparse-checkout for
the project: the listed paths are checked out, and other directories are left
out of the working directory. Cone mode also checks out the files sitting
directly in the project's top-level directory and in each leading directory of
a sparse-path, so `src/backend` brings along the top-level files and the files
directly under `src/` too. A project with no sparse-path elements is checked
out in full.
The checkout uses [cone mode](https://git-scm.com/docs/git-sparse-checkout#_internalscone_mode_handling)
for better performance, which means only directory paths are supported;
individual files and globs are not. Cone mode does no pattern matching, so `*`,
`?`, `[`, `]`, and `\` are not glob syntax in a `path`. In fact, Git 2.36.0 and
later reject them with `specify directories rather than patterns`, which
surfaces as a sync failure for the project. Earlier versions starting with
2.25.0 have no such check and are treated as literals.
Sparse-checkout is configured only when the project's working directory is
first created. Changes to `sparse-path` in the manifest (added, changed, or
removed) are not applied to a project that is already checked out. To change
the paths in an existing checkout, run `git sparse-checkout` in the project
directly, or remove the project's working directory and sync again.
Because sparse-checkout only prunes the working directory, a project with at
least one sparse-path is fetched with `--filter=blob:none` so that blobs which
will never be checked out are not downloaded. Without the filter Git still
downloads every blob in the project and sparse-checkout merely declines to
write most of them to disk, so the filter is what turns sparse-checkout into a
saving in transfer size and server load. The filter from
`repo init --partial-clone --clone-filter` takes precedence, and projects listed
in `repo init --partial-clone-exclude` are fetched in full as usual. Like
sparse-checkout itself, the implied filter only applies when the project is
first fetched (or is already a partial clone), so adding a sparse-path to an
existing full checkout does not convert it to a partial clone. Mirrors are never
filtered.
Because that filter is only useful if the objects it excludes are not fetched
by some other route, a project that takes the implied `--filter=blob:none` also
skips the `/clone.bundle` download: the bundle is an unfiltered pack of every
object, so fetching it would undo the filter. Projects that set the filter
explicitly, or that are listed in `repo init --partial-clone-exclude`, keep
using the clone bundle.
Attribute `path`: A path relative to the project root to include in the
sparse-checkout. The path is validated like `<copyfile>` and `<linkfile>`
paths: it must stay inside the project, so absolute paths, `.` or `..`
components, `~`, and `.git` or `.repo*` components are all rejected. A trailing
`/` is allowed.
### Element copyfile ### Element copyfile
Zero or more copyfile elements may be specified as children of a Zero or more copyfile elements may be specified as children of a
+7 -4
View File
@@ -45,10 +45,10 @@ that are critical to us.
That said, let's assume we need functionality that only exists in Python 3.7. That said, let's assume we need functionality that only exists in Python 3.7.
How do we decide when it's acceptable to drop Python 3.6? How do we decide when it's acceptable to drop Python 3.6?
1. Review the [Project References](./release-process.md#project-references) to 1. Review what major distros are using the previous version of Python (e.g.
see what major distros are using the previous version of Python, and when via [Repology Python versions]), and when they go EOL (e.g. via
they go EOL. Generally we care about Ubuntu LTS & current/previous Debian [Ubuntu releases] & [Debian releases]). Generally we care about Ubuntu LTS
stable versions. & current/previous Debian stable versions.
* If they're all EOL already, then go for it, drop support. * If they're all EOL already, then go for it, drop support.
* If they aren't EOL, start a thread on [repo-discuss] to see how the user * If they aren't EOL, start a thread on [repo-discuss] to see how the user
base feels about the proposal. base feels about the proposal.
@@ -88,5 +88,8 @@ $ repo-2.32 init --repo-rev=repo-1 ...
[repo-discuss]: https://groups.google.com/forum/#!forum/repo-discuss [repo-discuss]: https://groups.google.com/forum/#!forum/repo-discuss
[Debian releases]: https://wiki.debian.org/DebianReleases#Versions_of_Debian_stable
[Repology Python versions]: https://repology.org/project/python/versions
[repo hooks]: ./repo-hooks.md [repo hooks]: ./repo-hooks.md
[repo launcher]: ../repo [repo launcher]: ../repo
[Ubuntu releases]: https://wiki.ubuntu.com/Releases
+54 -200
View File
@@ -5,11 +5,19 @@ related topics and flows.
[TOC] [TOC]
## Schedule ## Schedule {#schedule}
Releases are automated.
Every Tuesday morning [US PT], a pipeline checks if any new commits have landed
since last release.
If so, it tags & tests the latest `main`, pushes & announces the tag, and then
promotes it to `stable` a day later.
The release is skipped if there's a holiday or [production freeze](#freeze)
that week.
Maintainers can also trigger it by hand.
The [manual steps](#manual) below are only needed as a fallback, or to roll
back a release.
There is no specific schedule for when releases are made.
Usually it's more along the lines of "enough minor changes have been merged",
or "there's a known issue the maintainers know should get fixed".
If you find a fix has been merged for an issue important to you, but hasn't been If you find a fix has been merged for an issue important to you, but hasn't been
released after a week or so, feel free to [contact] us to request a new release. released after a week or so, feel free to [contact] us to request a new release.
@@ -48,6 +56,11 @@ Whenever major changes are made to the launcher script, you should increment the
At runtime, repo will check this to see if it needs to be updated (and notify At runtime, repo will check this to see if it needs to be updated (and notify
the user automatically). the user automatically).
When `VERSION` changes, the release pipeline publishes the new launcher to the
`git-repo-downloads` Google Cloud Storage bucket.
To publish it by hand, run `./release/sign-launcher.py`, which signs the
launcher and prints the commands to upload it.
## Key management ## Key management
Every release has a git tag that is signed with a key that repo recognizes. Every release has a git tag that is signed with a key that repo recognizes.
@@ -96,8 +109,8 @@ If that tag is valid, then repo will warn and use that commit instead.
If that tag cannot be verified, it gives up and forces the user to resolve. If that tag cannot be verified, it gives up and forces the user to resolve.
If env variable `REPO_SKIP_SELF_UPDATE` is defined, this will If the env variable `REPO_SKIP_SELF_UPDATE` is set to anything other than `0`,
bypass the self update algorithm. `repo sync` will bypass the self update algorithm.
### Force an update ### Force an update
@@ -110,29 +123,20 @@ All development happens on the `main` branch and should generally be stable.
Since the repo launcher defaults to tracking the `stable` branch, it is not Since the repo launcher defaults to tracking the `stable` branch, it is not
normally updated until a new release is available. normally updated until a new release is available.
If something goes wrong with a new release, an older release can be force pushed If something goes wrong with a new release, `stable` can be pointed back at an
and clients will automatically downgrade. older release and clients will automatically downgrade.
The `maint` branch is used to track the previous major release of repo.
It is not normally meant to be used by people as `stable` should be good enough.
Once a new major release is pushed to the `stable` branch, then the previous
major release can be pushed to `maint`.
For example, when `stable` moves from `v1.10.x` to `v1.11.x`, then the `maint`
branch will be updated from `v1.9.x` to `v1.10.x`.
We don't have parallel release branches/series. We don't have parallel release branches/series.
Typically all tags are made against the `main` branch and then pushed to the Typically all tags are made against the `main` branch and then promoted to the
`stable` branch to make it available to the rest of the world. `stable` branch to make it available to the rest of the world.
Since repo doesn't typically see a lot of changes, this tends to be OK. Since repo doesn't typically see a lot of changes, this tends to be OK.
## Creating a new release ## Creating a new release {#manual}
When you want to create a new release, you'll need to select a good version and Releases are normally created automatically (see [Schedule](#schedule)).
If you need to create one by hand, you'll need to select a good version and
create a signed tag using a key registered in repo itself. create a signed tag using a key registered in repo itself.
Typically we just tag the latest version of the `main` branch. Typically we just tag the latest version of the `main` branch.
The tag could be pushed now, but it won't be used by clients normally (since the
default `repo-rev` setting is `stable`).
This would allow some early testing on systems who explicitly select `main`.
### Creating a signed tag ### Creating a signed tag
@@ -147,10 +151,10 @@ internal processes for accessing the restricted keys.
```sh ```sh
# Pick the new version. # Pick the new version.
$ t=v2.30 $ t=v2.XX
# Create a new signed tag with the current HEAD. # Create a new signed tag of the latest main branch.
$ ./release/sign-tag.py $t $ ./release/sign-tag.py $t origin/main
# Verify the signed tag. # Verify the signed tag.
$ git show $t $ git show $t
@@ -158,199 +162,49 @@ $ git show $t
### Push the new release ### Push the new release
Once you're ready to make the release available to everyone, push it to the First push the tag, [announce](#announce) it, and wait a day to get feedback
`stable` branch. from people testing new releases early.
Make sure you never push the tag itself to the stable branch!
Only push the commit -- note the use of `^0` below.
```sh ```sh
$ git push https://gerrit-review.googlesource.com/git-repo $t $ git push https://gerrit-review.googlesource.com/git-repo $t
$ git push https://gerrit-review.googlesource.com/git-repo $t^0:stable
``` ```
If something goes horribly wrong, you can force push the previous version to the Then make the release available to everyone by updating the `stable` branch.
`stable` branch and people should automatically recover. Direct pushes to it are rejected, so it has to be deleted & recreated via the
Again, make sure you never push the tag itself! authenticated [Gerrit REST API] (see [go/repo-release] for maintainers).
Make sure you never point the stable branch at the tag itself!
Only use the commit -- note the use of `^0` below.
The commit is resolved before deleting anything so that a bad `$t` can't leave
`stable` missing.
```sh ```sh
$ oldrev="whatever-old-commit" $ api=https://gerrit-review.googlesource.com/a/projects/git-repo/branches/stable
$ git push https://gerrit-review.googlesource.com/git-repo $oldrev:stable --force $ rev=$(git rev-parse --verify "$t^0") && \
curl --fail -X DELETE "$api" && \
curl --fail -X PUT -H 'Content-Type: application/json' \
-d "{\"revision\":\"$rev\"}" "$api"
``` ```
### Announce the release If something goes horribly wrong, run the same commands with the previous
release (e.g. `t=$(git describe --abbrev=0 $t^0~)`) and people should
automatically recover.
Once you do push a new release to `stable`, make sure to announce it on the ### Announce the release {#announce}
[repo-discuss@googlegroups.com] group.
Once you push a new release, make sure to announce it on the
[repo-discuss@googlegroups.com] group, and follow up once it's on `stable`.
Here is an [example announcement]. Here is an [example announcement].
You can create a short changelog using the command: You can create a short changelog since the previous release using the command:
```sh ```sh
# If you haven't pushed to the stable branch yet, you can use origin/stable. $ git log --format="%h (%aN) %s" --no-merges \
# If you have pushed, change origin/stable to the previous release tag. $(git describe --abbrev=0 $t^0~)..$t
# This assumes "main" is the current tagged release. If it's newer, change it
# to the current release tag too.
$ git log --format="%h (%aN) %s" --no-merges origin/stable..main
``` ```
## Project References
Here's a table showing the relationship of major tools, their EOL dates, and
their status in Ubuntu & Debian.
Those distros tend to be good indicators of how long we need to support things.
Things in bold indicate stuff to take note of, but does not guarantee that we
still support them.
Things in italics are things we used to care about but probably don't anymore.
| Date | EOL | [Git][rel-g] | [Python][rel-p] | [SSH][rel-o] | [Ubuntu][rel-u] / [Debian][rel-d] | Git | Python | SSH |
|:--------:|:------------:|:------------:|:---------------:|:------------:|-----------------------------------|:---:|:------:|:---:|
| Apr 2008 | | | | 5.0 |
| Jun 2008 | | | | 5.1 |
| Oct 2008 | *Oct 2013* | | 2.6.0 | | *10.04 Lucid* - 10.10 Maverick / *Squeeze* |
| Dec 2008 | *Feb 2009* | | 3.0.0 |
| Feb 2009 | | | | 5.2 |
| Feb 2009 | *Mar 2012* | | | | Debian 5 Lenny | 1.5.6.5 | 2.5.2 |
| Jun 2009 | *Jun 2016* | | 3.1.0 | | *10.04 Lucid* - 10.10 Maverick / *Squeeze* |
| Sep 2009 | | | | 5.3 | *10.04 Lucid* |
| Feb 2010 | *Oct 2012* | 1.7.0 | | | *10.04 Lucid* - *12.04 Precise* - 12.10 Quantal |
| Mar 2010 | | | | 5.4 |
| Apr 2010 | | | | 5.5 | 10.10 Maverick |
| Apr 2010 | *Apr 2015* | | | | *10.04 Lucid* | 1.7.0.4 | 2.6.5 3.1.2 | 5.3 |
| Jul 2010 | *Dec 2019* | | *2.7.0* | | 11.04 Natty - *<current>* |
| Aug 2010 | | | | 5.6 |
| Oct 2010 | | | | | 10.10 Maverick | 1.7.1 | 2.6.6 3.1.3 | 5.5 |
| Jan 2011 | | | | 5.7 |
| Feb 2011 | | | | 5.8 | 11.04 Natty |
| Feb 2011 | *Feb 2016* | | | | Debian 6 Squeeze | 1.7.2.5 | 2.6.6 3.1.3 |
| Apr 2011 | | | | | 11.04 Natty | 1.7.4 | 2.7.1 3.2.0 | 5.8 |
| Sep 2011 | | | | 5.9 | *12.04 Precise* |
| Oct 2011 | *Feb 2016* | | 3.2.0 | | 11.04 Natty - 12.10 Quantal |
| Oct 2011 | | | | | 11.10 Ocelot | 1.7.5.4 | 2.7.2 3.2.2 | 5.8 |
| Apr 2012 | | | | 6.0 | 12.10 Quantal |
| Apr 2012 | *Apr 2019* | | | | *12.04 Precise* | 1.7.9.5 | 2.7.3 3.2.3 | 5.9 |
| Aug 2012 | | | | 6.1 | 13.04 Raring |
| Sep 2012 | *Sep 2017* | | 3.3.0 | | 13.04 Raring - 13.10 Saucy |
| Oct 2012 | *Dec 2014* | 1.8.0 | | | 13.04 Raring - 13.10 Saucy |
| Oct 2012 | | | | | 12.10 Quantal | 1.7.10.4 | 2.7.3 3.2.3 | 6.0 |
| Mar 2013 | | | | 6.2 | 13.10 Saucy |
| Apr 2013 | | | | | 13.04 Raring | 1.8.1.2 | 2.7.4 3.3.1 | 6.1 |
| May 2013 | *May 2018* | | | | Debian 7 Wheezy | 1.7.10.4 | 2.7.3 3.2.3 |
| Sep 2013 | | | | 6.3 |
| Oct 2013 | | | | | 13.10 Saucy | 1.8.3.2 | 2.7.5 3.3.2 | 6.2 |
| Nov 2013 | | | | 6.4 |
| Jan 2014 | | | | 6.5 |
| Feb 2014 | *Dec 2014* | **1.9.0** | | | *14.04 Trusty* |
| Mar 2014 | *Mar 2019* | | *3.4.0* | | *14.04 Trusty* - 15.10 Wily / *Jessie* |
| Mar 2014 | | | | 6.6 | *14.04 Trusty* - 14.10 Utopic |
| Apr 2014 | *Apr 2024* | | | | *14.04 Trusty* | 1.9.1 | 2.7.5 3.4.0 | 6.6 |
| May 2014 | *Dec 2014* | 2.0.0 |
| Aug 2014 | *Dec 2014* | *2.1.0* | | | 14.10 Utopic - 15.04 Vivid / *Jessie* |
| Oct 2014 | | | | 6.7 | 15.04 Vivid |
| Oct 2014 | | | | | 14.10 Utopic | 2.1.0 | 2.7.8 3.4.2 | 6.6 |
| Nov 2014 | *Sep 2015* | 2.2.0 |
| Feb 2015 | *Sep 2015* | 2.3.0 |
| Mar 2015 | | | | 6.8 |
| Apr 2015 | *May 2017* | 2.4.0 |
| Apr 2015 | *Jun 2020* | | | | *Debian 8 Jessie* | 2.1.4 | 2.7.9 3.4.2 |
| Apr 2015 | | | | | 15.04 Vivid | 2.1.4 | 2.7.9 3.4.3 | 6.7 |
| Jul 2015 | *May 2017* | 2.5.0 | | | 15.10 Wily |
| Jul 2015 | | | | 6.9 | 15.10 Wily |
| Aug 2015 | | | | 7.0 |
| Aug 2015 | | | | 7.1 |
| Sep 2015 | *May 2017* | 2.6.0 |
| Sep 2015 | *Sep 2020* | | *3.5.0* | | *16.04 Xenial* - 17.04 Zesty / *Stretch* |
| Oct 2015 | | | | | 15.10 Wily | 2.5.0 | 2.7.9 3.4.3 | 6.9 |
| Jan 2016 | *Jul 2017* | *2.7.0* | | | *16.04 Xenial* |
| Feb 2016 | | | | 7.2 | *16.04 Xenial* |
| Mar 2016 | *Jul 2017* | 2.8.0 |
| Apr 2016 | *Apr 2026* | | | | *16.04 Xenial* | 2.7.4 | 2.7.11 3.5.1 | 7.2 |
| Jun 2016 | *Jul 2017* | 2.9.0 | | | 16.10 Yakkety |
| Jul 2016 | | | | 7.3 | 16.10 Yakkety |
| Sep 2016 | *Sep 2017* | 2.10.0 |
| Oct 2016 | | | | | 16.10 Yakkety | 2.9.3 | 2.7.11 3.5.1 | 7.3 |
| Nov 2016 | *Sep 2017* | *2.11.0* | | | 17.04 Zesty / *Stretch* |
| Dec 2016 | **Dec 2021** | | **3.6.0** | | 17.10 Artful - **18.04 Bionic** - 18.10 Cosmic |
| Dec 2016 | | | | 7.4 | 17.04 Zesty / *Debian 9 Stretch* |
| Feb 2017 | *Sep 2017* | 2.12.0 |
| Mar 2017 | | | | 7.5 | 17.10 Artful |
| Apr 2017 | | | | | 17.04 Zesty | 2.11.0 | 2.7.13 3.5.3 | 7.4 |
| May 2017 | *May 2018* | 2.13.0 |
| Jun 2017 | *Jun 2022* | | | | *Debian 9 Stretch* | 2.11.0 | 2.7.13 3.5.3 | 7.4 |
| Aug 2017 | *Dec 2019* | 2.14.0 | | | 17.10 Artful |
| Oct 2017 | *Dec 2019* | 2.15.0 |
| Oct 2017 | | | | 7.6 | **18.04 Bionic** |
| Oct 2017 | | | | | 17.10 Artful | 2.14.1 | 2.7.14 3.6.3 | 7.5 |
| Jan 2018 | *Dec 2019* | 2.16.0 |
| Apr 2018 | *Mar 2021* | **2.17.0** | | | **18.04 Bionic** |
| Apr 2018 | | | | 7.7 | 18.10 Cosmic |
| Apr 2018 | **Apr 2028** | | | | **18.04 Bionic** | 2.17.0 | 2.7.15 3.6.5 | 7.6 |
| Jun 2018 | *Mar 2021* | 2.18.0 |
| Jun 2018 | **Jun 2023** | | 3.7.0 | | 19.04 Disco - **Buster** |
| Aug 2018 | | | | 7.8 |
| Sep 2018 | *Mar 2021* | 2.19.0 | | | 18.10 Cosmic |
| Oct 2018 | | | | 7.9 | 19.04 Disco / **Buster** |
| Oct 2018 | | | | | 18.10 Cosmic | 2.19.1 | 2.7.15 3.6.6 | 7.7 |
| Dec 2018 | *Mar 2021* | **2.20.0** | | | 19.04 Disco - 19.10 Eoan / **Buster** |
| Feb 2019 | *Mar 2021* | 2.21.0 |
| Apr 2019 | | | | 8.0 | 19.10 Eoan |
| Apr 2019 | | | | | 19.04 Disco | 2.20.1 | 2.7.16 3.7.3 | 7.9 |
| Jun 2019 | | 2.22.0 |
| Jul 2019 | **Jul 2024** | | | | **Debian 10 Buster** | 2.20.1 | 2.7.16 3.7.3 | 7.9 |
| Aug 2019 | *Mar 2021* | 2.23.0 |
| Oct 2019 | **Oct 2024** | | 3.8.0 | | **20.04 Focal** - 20.10 Groovy |
| Oct 2019 | | | | 8.1 |
| Oct 2019 | | | | | 19.10 Eoan | 2.20.1 | 2.7.17 3.7.5 | 8.0 |
| Nov 2019 | *Mar 2021* | 2.24.0 |
| Jan 2020 | *Mar 2021* | 2.25.0 | | | **20.04 Focal** |
| Feb 2020 | | | | 8.2 | **20.04 Focal** |
| Mar 2020 | *Mar 2021* | 2.26.0 |
| Apr 2020 | **Apr 2030** | | | | **20.04 Focal** | 2.25.1 | 2.7.17 3.8.2 | 8.2 |
| May 2020 | *Mar 2021* | 2.27.0 | | | 20.10 Groovy |
| May 2020 | | | | 8.3 |
| Jul 2020 | *Mar 2021* | 2.28.0 |
| Sep 2020 | | | | 8.4 | 21.04 Hirsute / **Bullseye** |
| Oct 2020 | *Mar 2021* | 2.29.0 |
| Oct 2020 | | | | | 20.10 Groovy | 2.27.0 | 2.7.18 3.8.6 | 8.3 |
| Oct 2020 | **Oct 2025** | | 3.9.0 | | 21.04 Hirsute / **Bullseye** |
| Dec 2020 | *Mar 2021* | 2.30.0 | | | 21.04 Hirsute / **Bullseye** |
| Mar 2021 | | 2.31.0 | | 8.5 |
| Apr 2021 | | | | 8.6 |
| Apr 2021 | *Jan 2022* | | | | 21.04 Hirsute | 2.30.2 | 2.7.18 3.9.4 | 8.4 |
| Jun 2021 | | 2.32.0 |
| Aug 2021 | | 2.33.0 | | 8.7 |
| Aug 2021 | **Aug 2026** | | | | **Debian 11 Bullseye** | 2.30.2 | 2.7.18 3.9.2 | 8.4 |
| Sep 2021 | | | | 8.8 |
| Oct 2021 | | 2.34.0 | 3.10.0 | | **22.04 Jammy** |
| Jan 2022 | | 2.35.0 |
| Feb 2022 | | | | 8.9 | **22.04 Jammy** |
| Apr 2022 | | 2.36.0 | | 9.0 |
| Apr 2022 | **Apr 2032** | | | | **22.04 Jammy** | 2.34.1 | 2.7.18 3.10.6 | 8.9 |
| Jun 2022 | | 2.37.0 |
| Oct 2022 | | 2.38.0 | | 9.1 |
| Oct 2022 | | | 3.11.0 | | **Bookworm** |
| Dec 2022 | | 2.39.0 | | | **Bookworm** |
| Feb 2023 | | | | 9.2 | **Bookworm** |
| Mar 2023 | | 2.40.0 | | 9.3 |
| Jun 2023 | | 2.41.0 |
| Jun 2023 | **Jun 2028** | | | | **Debian 12 Bookworm** | 2.39.2 | 3.11.2 | 9.2 |
| Aug 2023 | | 2.42.0 | | 9.4 |
| Oct 2023 | | | 3.12.0 | 9.5 |
| Nov 2022 | | 2.43.0 |
| Dec 2023 | | | | 9.6 |
| Feb 2024 | | 2.44.0 |
| Mar 2024 | | | | 9.7 |
| Oct 2024 | | | 3.13.0 |
| **Date** | **EOL** | **[Git][rel-g]** | **[Python][rel-p]** | **[SSH][rel-o]** | **[Ubuntu][rel-u] / [Debian][rel-d]** | **Git** | **Python** | **SSH** |
[contact]: ../README.md#contact [contact]: ../README.md#contact
[rel-d]: https://en.wikipedia.org/wiki/Debian_version_history
[rel-g]: https://en.wikipedia.org/wiki/Git#Releases
[rel-o]: https://www.openssh.com/releasenotes.html
[rel-p]: https://en.wikipedia.org/wiki/History_of_Python#Table_of_versions
[rel-u]: https://wiki.ubuntu.com/Releases
[example announcement]: https://groups.google.com/d/topic/repo-discuss/UGBNismWo1M/discussion [example announcement]: https://groups.google.com/d/topic/repo-discuss/UGBNismWo1M/discussion
[repo-discuss@googlegroups.com]: https://groups.google.com/forum/#!forum/repo-discuss [repo-discuss@googlegroups.com]: https://groups.google.com/forum/#!forum/repo-discuss
[go/repo-release]: https://goto.google.com/repo-release [go/repo-release]: https://goto.google.com/repo-release
[Gerrit REST API]: https://gerrit-review.googlesource.com/Documentation/rest-api-projects.html#branch-endpoints
+3 -2
View File
@@ -91,8 +91,9 @@ Detecting collisions with untracked files or unstaged working-tree modifications
After the command exits with status 0, `repo` expects the following After the command exits with status 0, `repo` expects the following
postconditions to be met: postconditions to be met:
1. `git diff-index --quiet --cached REPO_TREV^{tree}` exits 0 (the index 1. The index matches `REPO_TREV`'s tree (`repo` trusts the command's exit
matches the target tree). status and does not run `git diff-index` against `REPO_TREV^{tree}`, which
would read every tree object from `.git/objects`).
2. `HEAD` still names what it did before the command, and its resolved commit 2. `HEAD` still names what it did before the command, and its resolved commit
object ID has not changed. object ID has not changed.
+50 -20
View File
@@ -200,9 +200,27 @@ following precedence order (highest to lowest):
Running `repo init --no-smart-sync` (or `repo init --head`) disables default Running `repo init --no-smart-sync` (or `repo init --head`) disables default
Smart Sync for the checkout even when enabled by the user setting or manifest. Smart Sync for the checkout even when enabled by the user setting or manifest.
As with the [manifest setting](#sync-smartsync-manifest-attribute), a As with the [manifest setting](#sync-smartsync-manifest-attribute), a
checkout-enabled default Smart Sync falls back to a regular ToT sync if the checkout-enabled default Smart Sync falls back to a regular ToT sync if Smart
manifest does not define a `<manifest-server>` or the manifest server cannot Sync fails (e.g. the manifest checkout is not tracking a remote branch,
provide a usable manifest. `<manifest-server>` is not defined, or the server cannot provide a usable
manifest).
### `repo init --smart-sync-branch=BRANCH`
Users can configure a server-side Smart Sync branch override for the initialized
manifest branch by passing `--smart-sync-branch=BRANCH` to `repo init`:
```bash
repo init -u <url> -b main -s --smart-sync-branch=git_main-warmup
```
This persists `repo.<manifest-branch>.smartsyncbranch` in
`.repo/manifests.git/config`. Because the setting is scoped to the manifest
branch, switching branches with `repo init -b <other>` will not send the
override branch when syncing `<other>`. Passing `--smart-sync-branch=""` clears
the override for the target manifest branch. Setting a non-empty override
requires the manifest checkout to track a remote branch (not a tag or commit
SHA).
### sync-smartsync (Manifest Attribute) ### sync-smartsync (Manifest Attribute)
@@ -215,16 +233,17 @@ by setting `sync-smartsync="true"` on the `<default>` element:
When enabled in the manifest, `repo sync` will automatically perform a Smart When enabled in the manifest, `repo sync` will automatically perform a Smart
Sync unless explicitly overridden on the command line. Since the user did not Sync unless explicitly overridden on the command line. Since the user did not
ask for a Smart Sync, any failure to obtain a manifest from the manifest server ask for a Smart Sync, any failure to smart sync (e.g. the manifest checkout is
(e.g. the server is unreachable, or it returns a manifest that cannot be parsed) not tracking a remote branch, the server is unreachable, or it returns a
is silently ignored and `repo sync` falls back to a regular ToT sync; pass manifest that cannot be parsed) is silently ignored and `repo sync` falls back
`--verbose` to see the reason. Repo marks these requests as `best_effort` (see to a regular ToT sync; pass `--verbose` to see the reason. Repo marks these
[GetApprovedManifestV2](#GetApprovedManifestV2)), so the server may also decline requests as `best_effort` (see [GetApprovedManifestV2](#GetApprovedManifestV2)),
them, e.g. during a staged rollout. This is only a fallback for the so the server may also decline them, e.g. during a staged rollout. This is only
manifest-driven default: an explicit `-s`/`--smart-sync` or `-t`/`--smart-tag` a fallback for the manifest-driven default: an explicit `-s`/`--smart-sync` or
still fails the sync, as before. Options that specify an explicit target or `-t`/`--smart-tag` still fails the sync, as before. Options that specify an
manifest source (such as `-t`/`--smart-tag`, `-m`/`--manifest-name` or explicit target or manifest source (such as `-t`/`--smart-tag`,
`--superproject-revision`) will also disable the default Smart Sync behavior. `-m`/`--manifest-name` or `--superproject-revision`) will also disable the
default Smart Sync behavior.
Users can override this manifest default on their workstation with the Users can override this manifest default on their workstation with the
`sync.smartsync` user setting; see `sync.smartsync` user setting; see
@@ -236,11 +255,22 @@ Explicitly enables Smart Sync. Repo will call
`GetApprovedManifestV2({branch[, target]})`, or `GetApprovedManifestV2({branch[, target]})`, or
`GetApprovedManifest(branch[, target])` if the server does not implement it. `GetApprovedManifest(branch[, target])` if the server does not implement it.
The `branch` is determined by the current manifest branch as specified by The `branch` sent to the manifest server is resolved in the following order:
`--manifest-branch=BRANCH` when running `repo init`. If `${SYNC_BRANCH}` is
defined and non-empty, its value is sent instead. This allows selecting a 1. `${SYNC_BRANCH}`: If defined and non-empty, its value is sent directly.
branch other than the one the checkout tracks, e.g. a server-side (CI) branch If defined as an empty string (`SYNC_BRANCH=""`), repo sends the checkout's
name that builds from the same manifest branch. raw manifest branch, bypassing `repo.<manifest-branch>.smartsyncbranch`.
2. `repo.<manifest-branch>.smartsyncbranch`: If configured via
`repo init --smart-sync-branch=BRANCH`, its value is sent for that manifest
branch.
3. The current manifest branch (as selected by `--manifest-branch=BRANCH` when
running `repo init`).
Overriding the branch allows selecting a server-side (CI) branch name that
builds or warms caches from the same manifest branch. If `${SYNC_BRANCH}` is
unset or empty and the manifest checkout is not tracking a remote branch (e.g.
it is on a detached HEAD, or was initialized with `-b <sha>` or
`-b refs/tags/<tag>`), Smart Sync fails without contacting the manifest server.
The `target` is defined by environment variables in the order below. If none The `target` is defined by environment variables in the order below. If none
of them match, then `target` is omitted. These variables were decided as they of them match, then `target` is omitted. These variables were decided as they
@@ -266,8 +296,8 @@ Smart Sync by default by setting the boolean `sync.smartsync` key in their
Command line flags (`-s`, `--no-smart-sync`, `-t`) override the user setting, Command line flags (`-s`, `--no-smart-sync`, `-t`) override the user setting,
which in turn overrides the manifest's `sync-smartsync` default. A Smart Sync which in turn overrides the manifest's `sync-smartsync` default. A Smart Sync
enabled this way is treated as an explicit request, so it is subject to the enabled this way is treated as an explicit request, so it is subject to the
same restrictions as `-s` and fails the sync if no manifest can be obtained same restrictions as `-s` and fails the sync if Smart Sync fails instead of
from the server instead of silently falling back to ToT. silently falling back to ToT.
### --smart-tag=TAG / -t ### --smart-tag=TAG / -t
+1 -2
View File
@@ -14,6 +14,7 @@
import os import os
import re import re
import shlex
import subprocess import subprocess
import sys import sys
import tempfile import tempfile
@@ -89,8 +90,6 @@ least one of these before using this command.""", # noqa: E501
if platform_utils.isWindows(): if platform_utils.isWindows():
# Split on spaces, respecting quoted strings # Split on spaces, respecting quoted strings
import shlex
args = shlex.split(editor) args = shlex.split(editor)
shell = False shell = False
elif re.compile("^.*[$ \t'].*$").match(editor): elif re.compile("^.*[$ \t'].*$").match(editor):
+3 -3
View File
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from typing import List from typing import Optional, Sequence
class BaseRepoError(Exception): class BaseRepoError(Exception):
@@ -22,7 +22,7 @@ class BaseRepoError(Exception):
class RepoError(BaseRepoError): class RepoError(BaseRepoError):
"""Exceptions thrown inside repo that can be handled.""" """Exceptions thrown inside repo that can be handled."""
def __init__(self, *args, project: str = None) -> None: def __init__(self, *args, project: Optional[str] = None) -> None:
super().__init__(*args) super().__init__(*args)
self.project = project self.project = project
@@ -36,7 +36,7 @@ class RepoExitError(BaseRepoError):
self, self,
*args, *args,
exit_code: int = 1, exit_code: int = 1,
aggregate_errors: List[Exception] = None, aggregate_errors: Optional[Sequence[Exception]] = None,
**kwargs, **kwargs,
) -> None: ) -> None:
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
+11 -3
View File
@@ -47,10 +47,15 @@ class EventLog:
Specific tasks may include additional informational properties. Specific tasks may include additional informational properties.
""" """
def __init__(self): def __init__(self, enabled: bool = True) -> None:
"""Initializes the event log.""" """Initializes the event log.
Args:
enabled: Whether event logging is enabled.
"""
self._log = [] self._log = []
self._parent = None self._parent = None
self.enabled = enabled
def Add( def Add(
self, self,
@@ -74,8 +79,11 @@ class EventLog:
kind: The kind of the object for the unique identifier. kind: The kind of the object for the unique identifier.
Returns: Returns:
A dictionary of the event added to the log. A dictionary of the event added to the log, or None if disabled.
""" """
if not self.enabled:
return None
event = { event = {
"id": (kind, _NextEventId()), "id": (kind, _NextEventId()),
"name": name, "name": name,
+4 -4
View File
@@ -16,8 +16,8 @@
import subprocess import subprocess
import sys import sys
from urllib.parse import urlparse import urllib.parse
from urllib.request import urlopen import urllib.request
import error import error
@@ -32,7 +32,7 @@ def fetch_file(url, verbose=False):
Returns: Returns:
The contents of the file as bytes. The contents of the file as bytes.
""" """
scheme = urlparse(url).scheme scheme = urllib.parse.urlparse(url).scheme
if scheme == "gs": if scheme == "gs":
cmd = ["gcloud", "storage", "cat", url] cmd = ["gcloud", "storage", "cat", url]
errors = [] errors = []
@@ -54,5 +54,5 @@ def fetch_file(url, verbose=False):
file=sys.stderr, file=sys.stderr,
) )
raise FetchFileError(aggregate_errors=errors) raise FetchFileError(aggregate_errors=errors)
with urlopen(url) as f: with urllib.request.urlopen(url) as f:
return f.read() return f.read()
Symlink
+1
View File
@@ -0,0 +1 @@
run_tests
+26 -26
View File
@@ -18,14 +18,14 @@ import os
import re import re
import subprocess import subprocess
import sys import sys
from typing import Any, Optional from typing import Any, Dict, Optional
import error import error
from git_trace2_event_log_base import BaseEventLog import git_trace2_event_log_base
import platform_utils import platform_utils
import repo_logging import repo_logging
import repo_trace import repo_trace
from wrapper import Wrapper import wrapper
GIT = "git" GIT = "git"
@@ -46,7 +46,7 @@ logger = repo_logging.RepoLogger(__file__)
class _GitCall: class _GitCall:
@functools.lru_cache(maxsize=None) # noqa: B019 @functools.lru_cache(maxsize=None) # noqa: B019
def version_tuple(self): def version_tuple(self):
ret = Wrapper().ParseGitVersion() ret = wrapper.Wrapper().ParseGitVersion()
if ret is None: if ret is None:
msg = "fatal: unable to detect git version" msg = "fatal: unable to detect git version"
logger.error(msg) logger.error(msg)
@@ -87,7 +87,7 @@ def RepoSourceVersion():
# We avoid GitCommand so we don't run into circular deps -- GitCommand needs # We avoid GitCommand so we don't run into circular deps -- GitCommand needs
# to initialize version info we provide. # to initialize version info we provide.
if ver is None: if ver is None:
env = GitCommand._GetBasicEnv() env = _get_basic_env()
proj = os.path.dirname(os.path.abspath(__file__)) proj = os.path.dirname(os.path.abspath(__file__))
env[GIT_DIR] = os.path.join(proj, ".git") env[GIT_DIR] = os.path.join(proj, ".git")
@@ -220,6 +220,25 @@ def git_require(min_version, fail=False, msg=""):
return False return False
def _get_basic_env() -> Dict[str, str]:
"""Return a basic env for running git under.
This is guaranteed to be side-effect free.
"""
env = os.environ.copy()
for key in (
repo_trace.REPO_TRACE,
GIT_DIR,
"GIT_ALTERNATE_OBJECT_DIRECTORIES",
"GIT_OBJECT_DIRECTORY",
"GIT_WORK_TREE",
"GIT_GRAFT_FILE",
"GIT_INDEX_FILE",
):
env.pop(key, None)
return env
def _build_env( def _build_env(
_kwargs_only=(), _kwargs_only=(),
bare: Optional[bool] = False, bare: Optional[bool] = False,
@@ -232,7 +251,7 @@ def _build_env(
assert _kwargs_only == (), "_build_env only accepts keyword arguments." assert _kwargs_only == (), "_build_env only accepts keyword arguments."
env = GitCommand._GetBasicEnv() env = _get_basic_env()
if disable_editor: if disable_editor:
env["GIT_EDITOR"] = ":" env["GIT_EDITOR"] = ":"
@@ -347,7 +366,7 @@ class GitCommand:
command.extend(cmdv[1:]) command.extend(cmdv[1:])
event_log = ( event_log = (
BaseEventLog(env=env, add_init_count=True) git_trace2_event_log_base.BaseEventLog(env=env, add_init_count=True)
if add_event_log if add_event_log
else None else None
) )
@@ -542,25 +561,6 @@ class GitCommand:
return buffer return buffer
@staticmethod
def _GetBasicEnv():
"""Return a basic env for running git under.
This is guaranteed to be side-effect free.
"""
env = os.environ.copy()
for key in (
repo_trace.REPO_TRACE,
GIT_DIR,
"GIT_ALTERNATE_OBJECT_DIRECTORIES",
"GIT_OBJECT_DIRECTORY",
"GIT_WORK_TREE",
"GIT_GRAFT_FILE",
"GIT_INDEX_FILE",
):
env.pop(key, None)
return env
def VerifyCommand(self) -> None: def VerifyCommand(self) -> None:
if self.rc == 0: if self.rc == 0:
return None return None
+470 -59
View File
@@ -15,6 +15,7 @@
import contextlib import contextlib
import datetime import datetime
import errno import errno
import hashlib
import http.client import http.client
import json import json
import os import os
@@ -22,7 +23,8 @@ import re
import ssl import ssl
import subprocess import subprocess
import sys import sys
from typing import Union import tempfile
from typing import Any, Dict, Iterator, List, Optional, Tuple, Union
import urllib.error import urllib.error
import urllib.request import urllib.request
@@ -38,9 +40,93 @@ import repo_trace
SYNC_STATE_PREFIX = "repo.syncstate." SYNC_STATE_PREFIX = "repo.syncstate."
ID_RE = re.compile(r"^(?:[0-9a-f]{40}|[0-9a-f]{64})$") ID_RE = re.compile(r"^(?:[0-9a-f]{40}|[0-9a-f]{64})$")
_INI_SECTION_RE = re.compile(r"^[A-Za-z0-9-]+$")
_INI_KEY_RE = re.compile(r"^[A-Za-z][A-Za-z0-9-]*$")
REVIEW_CACHE = {} REVIEW_CACHE = {}
# Metadata key stored in .repo_config.json to detect same-tick config edits.
_JSON_META_KEY = "__repo_meta__"
def _SplitIniName(name: str) -> Optional[Tuple[str, str, str]]:
"""Split a config key into INI components when plain-append safe.
Args:
name: Dotted git config key name (e.g. "remote.origin.url").
Returns:
A (section, subsection, var) tuple if the key can be written as a
plain INI entry, or None if it requires git config CLI handling.
"""
parts = name.split(".")
if len(parts) < 2:
return None
section = parts[0]
var = parts[-1]
subsection = ".".join(parts[1:-1]) if len(parts) > 2 else ""
if not _INI_SECTION_RE.match(section) or not _INI_KEY_RE.match(var):
return None
if any(
ord(ch) < 0x20 or ord(ch) > 0x7E or ch in ('"', "\\")
for ch in subsection
):
return None
return (section, subsection, var)
def _IsPlainIniValue(val: Any) -> bool:
"""Check whether a config value needs no INI quoting or escaping.
Args:
val: Candidate configuration value passed to SetString.
Returns:
True if val is a string that can be written verbatim without quotes
or backslash escapes; False otherwise.
"""
if not isinstance(val, str):
return False
if val and (val[0] in (" ", "\t") or val[-1] in (" ", "\t")):
return False
return not any(
ord(ch) < 0x20 or ord(ch) > 0x7E or ch in ('"', "\\", ";", "#")
for ch in val
)
def _FormatIniBatch(ops: List[Tuple[str, List[str]]], existing: bytes) -> bytes:
"""Format queued key/value pairs as UTF-8 INI bytes to append.
Args:
ops: Ordered list of (name, values) pairs to append.
existing: Existing raw bytes of the target git config file.
Returns:
UTF-8 encoded INI stanza bytes ready to append after existing.
"""
lines: List[str] = []
if existing and not existing.endswith(b"\n"):
lines.append("\n")
last_header = None
for name, vals in ops:
parsed = _SplitIniName(name)
assert parsed is not None
section, subsection, var = parsed
header_key = (section.lower(), subsection)
if header_key != last_header:
if subsection:
lines.append(f'[{section} "{subsection}"]\n')
else:
lines.append(f"[{section}]\n")
last_header = header_key
for val in vals:
if val == "":
lines.append(f"\t{var} =\n")
else:
lines.append(f"\t{var} = {val}\n")
return "".join(lines).encode("utf-8")
def IsChange(rev): def IsChange(rev):
return rev.startswith(git_refs.R_CHANGES) return rev.startswith(git_refs.R_CHANGES)
@@ -161,7 +247,11 @@ class GitConfig:
self.file = str(configfile) self.file = str(configfile)
self.defaults = defaults self.defaults = defaults
self._cache_dict = None self._cache_dict = None
self._cache_fingerprint = None
self._section_dict = None self._section_dict = None
self._batch_depth = 0
self._batch_ops: Dict[str, Tuple[str, List[str]]] = {}
self._batch_json_dirty = False
self._remotes = {} self._remotes = {}
self._branches = {} self._branches = {}
@@ -175,6 +265,47 @@ class GitConfig:
def ClearCache(self): def ClearCache(self):
"""Clear the in-memory cache of config.""" """Clear the in-memory cache of config."""
self._cache_dict = None self._cache_dict = None
self._cache_fingerprint = None
self._section_dict = None
self._remotes = {}
self._branches = {}
def _RemoveJson(self) -> None:
with contextlib.suppress(OSError):
platform_utils.remove(self._json, missing_ok=True)
@contextlib.contextmanager
def BatchWrites(self) -> Iterator[None]:
"""Coalesce compatible SetString writes under Git's config.lock.
Yields:
None while SetString writes are queued for batch flushing on exit.
"""
self._batch_depth += 1
try:
yield
except BaseException:
self._batch_depth -= 1
if self._batch_depth == 0:
try:
self._FlushBatch(save_json=True)
except Exception:
self._batch_ops.clear()
self._batch_json_dirty = False
self.ClearCache()
self._RemoveJson()
raise
else:
self._batch_depth -= 1
if self._batch_depth == 0:
self._FlushBatch(save_json=True)
def Reload(self) -> None:
"""Clear cached state and reload configuration from disk."""
self.ClearCache()
self._batch_json_dirty = False
self._RemoveJson()
_ = self._cache
def Has(self, name, include_defaults=True): def Has(self, name, include_defaults=True):
"""Return true if this configuration file has the key.""" """Return true if this configuration file has the key."""
@@ -292,6 +423,14 @@ class GitConfig:
""" """
key = _key(name) key = _key(name)
if (
self._cache_dict is not None
and not self._batch_ops
and self._ReadFileFingerprint() != self._cache_fingerprint
):
self.ClearCache()
self._RemoveJson()
try: try:
old = self._cache[key] old = self._cache[key]
except KeyError: except KeyError:
@@ -306,8 +445,17 @@ class GitConfig:
if value is None: if value is None:
if old: if old:
if key in self._batch_ops:
del self._batch_ops[key]
else:
if self._batch_ops and (
key == "include.path" or key.startswith("includeif.")
):
self._FlushBatch(save_json=False)
self._do("--unset-all", name, allow_missing=True)
self._cache_fingerprint = self._ReadFileFingerprint()
del self._cache[key] del self._cache[key]
self._do("--unset-all", name) self._SyncCacheAfterSet(key)
elif isinstance(value, list): elif isinstance(value, list):
if len(value) == 0: if len(value) == 0:
@@ -317,14 +465,196 @@ class GitConfig:
self.SetString(name, value[0]) self.SetString(name, value[0])
elif old != value: elif old != value:
self._cache[key] = list(value) vals = list(value)
self._do("--replace-all", name, value[0]) if self._CanQueueInBatch(key, name, vals, old):
for i in range(1, len(value)): self._cache[key] = vals
self._do("--add", name, value[i]) self._batch_ops[key] = (name, vals)
else:
if self._batch_ops:
self._batch_ops.pop(key, None)
self._FlushBatch(save_json=False)
self._do("--replace-all", name, vals[0])
try:
for i in range(1, len(vals)):
self._do("--add", name, vals[i])
except BaseException:
self.ClearCache()
self._RemoveJson()
raise
self._cache[key] = vals
self._cache_fingerprint = self._ReadFileFingerprint()
self._SyncCacheAfterSet(key)
elif len(old) != 1 or old[0] != value: elif len(old) != 1 or old[0] != value:
self._cache[key] = [value] vals = [value]
self._do("--replace-all", name, value) if self._CanQueueInBatch(key, name, vals, old):
self._cache[key] = vals
self._batch_ops[key] = (name, vals)
else:
if self._batch_ops:
self._batch_ops.pop(key, None)
self._FlushBatch(save_json=False)
self._do("--replace-all", name, value)
self._cache[key] = vals
self._cache_fingerprint = self._ReadFileFingerprint()
self._SyncCacheAfterSet(key)
def _HasIncludes(self) -> bool:
"""Return True if cached config contains include or includeIf keys."""
return self._cache_dict is not None and (
"include.path" in self._cache_dict
or any(k.startswith("includeif.") for k in self._cache_dict)
)
def _CanQueueInBatch(
self,
key: str,
name: str,
vals: List[Any],
old: List[Optional[str]],
) -> bool:
"""Check whether a SetString write can be queued for lockfile append.
Args:
key: Lowercased canonical key used in self._cache.
name: Original dotted key name preserving subsection case.
vals: New value list requested by SetString.
old: Existing cached values for key before this write.
Returns:
True if the write can be appended at batch flush time; False if it
must be flushed immediately through git config.
"""
if self._batch_depth <= 0 or self.file == self._SYSTEM_CONFIG:
return False
if (
key == "include.path"
or key.startswith("includeif.")
or self._HasIncludes()
):
return False
if old and key not in self._batch_ops:
return False
return _SplitIniName(name) is not None and all(
_IsPlainIniValue(v) for v in vals
)
def _FlushBatch(self, save_json: bool = True) -> None:
"""Flush queued SetString writes via config.lock or git config CLI.
Args:
save_json: Whether to persist a dirty .repo_config.json cache
after flushing queued config updates.
"""
ops = list(self._batch_ops.values())
self._batch_ops.clear()
if ops and not self._TryWriteBatchLocked(ops):
self._ApplyBatchViaGit(ops)
if save_json and self._batch_json_dirty:
self._batch_json_dirty = False
if self._cache_dict is not None:
self._SaveJson(self._cache_dict)
def _TryWriteBatchLocked(self, ops: List[Tuple[str, List[str]]]) -> bool:
"""Atomically append queued entries under <file>.lock if unmodified.
Args:
ops: Ordered list of (name, values) pairs to append.
Returns:
True if the lockfile was written and renamed into place; False if
the lock was busy, the file changed externally, or an I/O error
occurred.
"""
if self.file == self._SYSTEM_CONFIG or os.path.islink(self.file):
return False
lock_path = f"{self.file}.lock"
try:
fd = os.open(lock_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o666)
except OSError:
return False
replaced = False
try:
existing = b""
curr_fp = None
if os.path.exists(self.file):
with open(self.file, "rb") as src_fp:
existing = src_fp.read()
st = os.fstat(src_fp.fileno())
curr_fp = (
st.st_mtime_ns,
st.st_ctime_ns,
st.st_size,
st.st_ino,
hashlib.sha1(existing).hexdigest(),
)
if hasattr(os, "fchmod") and not platform_utils.isWindows():
os.fchmod(fd, st.st_mode & 0o777)
if curr_fp != self._cache_fingerprint:
return False
payload = _FormatIniBatch(ops, existing)
with os.fdopen(fd, "wb") as lock_fp:
fd = -1
if existing:
lock_fp.write(existing)
lock_fp.write(payload)
os.replace(lock_path, self.file)
replaced = True
self._cache_fingerprint = self._ReadFileFingerprint()
return True
except OSError:
return False
finally:
if fd >= 0:
os.close(fd)
if not replaced:
platform_utils.remove(lock_path, missing_ok=True)
def _ApplyBatchViaGit(self, ops: List[Tuple[str, List[str]]]) -> None:
"""Apply queued batch operations via the git config CLI fallback.
Args:
ops: Ordered list of (name, values) pairs to write via git config.
"""
try:
for name, vals in ops:
self._do("--replace-all", name, vals[0])
for extra in vals[1:]:
self._do("--add", name, extra)
except BaseException:
self._batch_json_dirty = False
self.ClearCache()
self._RemoveJson()
raise
self.Reload()
def _SyncCacheAfterSet(self, key: str) -> None:
"""Keep the in-memory section index and JSON cache coherent.
Args:
key: Canonicalized configuration key that was modified.
"""
self._section_dict = None
if key.startswith("remote."):
self._remotes = {}
self._branches = {}
elif key.startswith("branch."):
self._branches = {}
if (
key == "include.path"
or key.startswith("includeif.")
or self._HasIncludes()
):
self.ClearCache()
self._RemoveJson()
self._batch_json_dirty = False
elif self._batch_depth > 0:
self._batch_json_dirty = True
else:
self._SaveJson(self._cache)
def GetRemote(self, name): def GetRemote(self, name):
"""Get the remote.$name.* configuration values as an object.""" """Get the remote.$name.* configuration values as an object."""
@@ -414,30 +744,115 @@ class GitConfig:
d = self._ReadJson() d = self._ReadJson()
if d is None: if d is None:
d = self._ReadGit() d = self._ReadGit()
self._SaveJson(d) if self._batch_depth > 0:
self._batch_json_dirty = True
else:
self._SaveJson(d)
return d return d
def _ReadFileFingerprint(self) -> Optional[Tuple[int, int, int, int, str]]:
"""Return (mtime_ns, ctime_ns, size, ino, sha1_hex) for self.file."""
try:
with open(self.file, "rb") as fd:
raw_cfg = fd.read()
st = os.fstat(fd.fileno())
except OSError:
return None
return (
st.st_mtime_ns,
st.st_ctime_ns,
st.st_size,
st.st_ino,
hashlib.sha1(raw_cfg).hexdigest(),
)
def _ReadJson(self): def _ReadJson(self):
try: try:
if os.path.getmtime(self._json) <= os.path.getmtime(self.file): cfg_stat = os.stat(self.file)
platform_utils.remove(self._json) json_stat = os.stat(self._json)
return None
except OSError: except OSError:
return None return None
if json_stat.st_mtime_ns < cfg_stat.st_mtime_ns:
self._RemoveJson()
return None
try: try:
with repo_trace.Trace(": parsing %s", self.file): with repo_trace.Trace(": parsing %s", self.file):
with open(self._json) as fd: with open(self._json, encoding="utf-8") as fd:
return json.load(fd) data = json.load(fd)
except (OSError, ValueError): except (OSError, ValueError):
platform_utils.remove(self._json, missing_ok=True) self._RemoveJson()
return None
if not isinstance(data, dict):
self._RemoveJson()
return None return None
meta = data.pop(_JSON_META_KEY, None)
if (
isinstance(meta, list)
and len(meta) == 2
and isinstance(meta[1], str)
):
if (
json_stat.st_mtime_ns == cfg_stat.st_mtime_ns
or cfg_stat.st_size != meta[0]
):
if cfg_stat.st_size != meta[0]:
self._RemoveJson()
return None
fp = self._ReadFileFingerprint()
if fp is None:
return None
if fp[4] != meta[1]:
self._RemoveJson()
return None
self._cache_fingerprint = fp
else:
self._cache_fingerprint = (
cfg_stat.st_mtime_ns,
cfg_stat.st_ctime_ns,
cfg_stat.st_size,
cfg_stat.st_ino,
meta[1],
)
elif json_stat.st_mtime_ns <= cfg_stat.st_mtime_ns:
self._RemoveJson()
return None
else:
fp = self._ReadFileFingerprint()
if fp is None:
return None
self._cache_fingerprint = fp
return data
def _SaveJson(self, cache): def _SaveJson(self, cache):
fp = self._ReadFileFingerprint()
if fp is None:
self._cache_fingerprint = None
self._RemoveJson()
return
self._cache_fingerprint = fp
payload = dict(cache)
payload[_JSON_META_KEY] = [fp[2], fp[4]]
json_dir = os.path.dirname(self._json) or "."
tmp_path = None
try: try:
with open(self._json, "w") as fd: with tempfile.NamedTemporaryFile(
json.dump(cache, fd, indent=2) "w",
dir=json_dir,
prefix=f".tmp{os.path.basename(self._json)}.",
delete=False,
encoding="utf-8",
) as fd:
tmp_path = fd.name
json.dump(payload, fd, indent=2)
os.replace(tmp_path, self._json)
except (OSError, TypeError): except (OSError, TypeError):
platform_utils.remove(self._json, missing_ok=True) if tmp_path is not None:
with contextlib.suppress(OSError):
platform_utils.remove(tmp_path, missing_ok=True)
self._RemoveJson()
def _ReadGit(self): def _ReadGit(self):
""" """
@@ -448,6 +863,7 @@ class GitConfig:
""" """
c = {} c = {}
if not os.path.exists(self.file): if not os.path.exists(self.file):
self._cache_fingerprint = None
return c return c
d = self._do("--null", "--list") d = self._do("--null", "--list")
@@ -463,9 +879,10 @@ class GitConfig:
else: else:
c[key] = [val] c[key] = [val]
self._cache_fingerprint = self._ReadFileFingerprint()
return c return c
def _do(self, *args): def _do(self, *args, allow_missing: bool = False) -> str:
if self.file == self._SYSTEM_CONFIG: if self.file == self._SYSTEM_CONFIG:
command = ["config", "--system", "--includes"] command = ["config", "--system", "--includes"]
else: else:
@@ -475,10 +892,10 @@ class GitConfig:
p = git_command.GitCommand( p = git_command.GitCommand(
None, command, capture_stdout=True, capture_stderr=True None, command, capture_stdout=True, capture_stderr=True
) )
if p.Wait() == 0: rc = p.Wait()
if rc == 0 or (allow_missing and rc == 5):
return p.stdout return p.stdout
else: raise error.GitError(f"git {' '.join(command)}: {p.stderr}")
raise error.GitError(f"git {' '.join(command)}: {p.stderr}")
class RepoConfig(GitConfig): class RepoConfig(GitConfig):
@@ -770,17 +1187,18 @@ class Remote:
def Save(self): def Save(self):
"""Save this remote to the configuration.""" """Save this remote to the configuration."""
self._Set("url", self.url) with self._config.BatchWrites():
# projectname is initialized for projects listed in the manifest, but self._Set("url", self.url)
# not for others (e.g. the manifest project). This class is used for # projectname is initialized for projects listed in the manifest,
# all of them. # but not for others (e.g. the manifest project). This class is
if self.pushUrl is not None and self.projectname is not None: # used for all of them.
self._Set("pushurl", self.pushUrl + "/" + self.projectname) if self.pushUrl is not None and self.projectname is not None:
else: self._Set("pushurl", self.pushUrl + "/" + self.projectname)
self._Set("pushurl", self.pushUrl) else:
self._Set("review", self.review) self._Set("pushurl", self.pushUrl)
self._Set("projectname", self.projectname) self._Set("review", self.review)
self._Set("fetch", list(map(str, self.fetch))) self._Set("projectname", self.projectname)
self._Set("fetch", list(map(str, self.fetch)))
def _Set(self, key, value): def _Set(self, key, value):
key = f"remote.{self.name}.{key}" key = f"remote.{self.name}.{key}"
@@ -814,21 +1232,13 @@ class Branch:
def Save(self): def Save(self):
"""Save this branch back into the configuration.""" """Save this branch back into the configuration."""
if self._config.HasSection("branch", self.name): with self._config.BatchWrites():
if self.remote: if self.remote:
self._Set("remote", self.remote.name) self._Set("remote", self.remote.name)
else: else:
self._Set("remote", None) self._Set("remote", None)
self._Set("merge", self.merge) self._Set("merge", self.merge)
else:
with open(self._config.file, "a") as fd:
fd.write('[branch "%s"]\n' % self.name)
if self.remote:
fd.write("\tremote = %s\n" % self.remote.name)
if self.merge:
fd.write("\tmerge = %s\n" % self.merge)
def _Set(self, key, value): def _Set(self, key, value):
key = f"branch.{self.name}.{key}" key = f"branch.{self.name}.{key}"
return self._config.SetString(key, value) return self._config.SetString(key, value)
@@ -864,23 +1274,24 @@ class SyncAnalysisState:
""" """
self._config = config self._config = config
now = datetime.datetime.now(datetime.timezone.utc) now = datetime.datetime.now(datetime.timezone.utc)
self._Set("main.synctime", now.isoformat(timespec="microseconds")) with self._config.BatchWrites():
self._Set("main.version", "1") self._Set("main.synctime", now.isoformat(timespec="microseconds"))
self._Set("sys.argv", sys.argv) self._Set("main.version", "1")
for key, value in superproject_logging_data.items(): self._Set("sys.argv", sys.argv)
self._Set(f"superproject.{key}", value) for key, value in superproject_logging_data.items():
for key, value in options.__dict__.items(): self._Set(f"superproject.{key}", value)
self._Set(f"options.{key}", value) for key, value in options.__dict__.items():
config_items = config.DumpConfigDict().items() self._Set(f"options.{key}", value)
EXTRACT_NAMESPACES = {"repo", "branch", "remote"} config_items = config.DumpConfigDict().items()
self._SetDictionary( EXTRACT_NAMESPACES = {"repo", "branch", "remote"}
{ self._SetDictionary(
k: v {
for k, v in config_items k: v
if not k.startswith(SYNC_STATE_PREFIX) for k, v in config_items
and k.split(".", 1)[0] in EXTRACT_NAMESPACES if not k.startswith(SYNC_STATE_PREFIX)
} and k.split(".", 1)[0] in EXTRACT_NAMESPACES
) }
)
def _SetDictionary(self, data): def _SetDictionary(self, data):
"""Save all key/value pairs of |data| dictionary. """Save all key/value pairs of |data| dictionary.
+527 -35
View File
@@ -13,6 +13,9 @@
# limitations under the License. # limitations under the License.
import os import os
import re
import stat
from typing import Dict, Optional, Set, Tuple
import git_command import git_command
import platform_utils import platform_utils
@@ -27,6 +30,28 @@ R_PUB = "refs/published/"
R_WORKTREE = "refs/worktree/" R_WORKTREE = "refs/worktree/"
R_WORKTREE_M = R_WORKTREE + "m/" R_WORKTREE_M = R_WORKTREE + "m/"
R_M = "refs/remotes/m/" R_M = "refs/remotes/m/"
# Maximum symbolic-ref indirection depth (matches Git SYMREF_MAXDEPTH).
_SYMREF_MAXDEPTH = 5
_HEX_OID_RE = re.compile(r"^(?:[0-9a-f]{40}|[0-9a-f]{64})$")
_BAD_REF_CHARS = frozenset(" ~^:?*[\\")
_StatKey = Tuple[int, int, int, int]
def IsValidRefName(name: str) -> bool:
"""Validate a Git refname per git-check-ref-format rules."""
if not name or name.endswith(".") or ".." in name or "@{" in name:
return False
if any(
ch in _BAD_REF_CHARS or ord(ch) < 0x20 or ord(ch) == 0x7F for ch in name
):
return False
parts = name.split("/")
if len(parts) < 2:
return False
for part in parts:
if not part or part.startswith(".") or part.endswith(".lock"):
return False
return True
class GitRefs: class GitRefs:
@@ -34,7 +59,8 @@ class GitRefs:
self._gitdir = gitdir self._gitdir = gitdir
self._phyref = None self._phyref = None
self._symref = None self._symref = None
self._mtime = {} self._mtime: Dict[str, Optional[_StatKey]] = {}
self._files_authoritative = False
@property @property
def all(self): def all(self):
@@ -52,6 +78,18 @@ class GitRefs:
"""Whether a ref snapshot has already been loaded.""" """Whether a ref snapshot has already been loaded."""
return self._phyref is not None return self._phyref is not None
@property
def is_files_authoritative(self) -> bool:
"""Whether the loaded snapshot came from a complete files scan."""
return self._phyref is not None and self._files_authoritative
def Invalidate(self) -> None:
"""Discard any cached ref snapshot so the next access reloads."""
self._phyref = None
self._symref = None
self._mtime = {}
self._files_authoritative = False
def get(self, name): def get(self, name):
try: try:
return self.all[name] return self.all[name]
@@ -68,6 +106,9 @@ class GitRefs:
if name in self._mtime: if name in self._mtime:
del self._mtime[name] del self._mtime[name]
norm_os = os.path.join(*name.strip("/").split("/"))
if norm_os in self._mtime:
del self._mtime[norm_os]
def symref(self, name): def symref(self, name):
try: try:
@@ -76,24 +117,233 @@ class GitRefs:
except KeyError: except KeyError:
return "" return ""
def TryResolveRef(self, name: str) -> Optional[str]:
"""Resolve a single ref directly on the files backend without git.
Args:
name: Reference name ("HEAD" or "refs/...").
Returns:
The resolved hex object ID if present, "" if the repository uses a
clean files backend and the ref is authoritatively absent or
unborn, or None if git must be invoked as a fallback.
"""
if name != HEAD and (
not name.startswith("refs/") or not IsValidRefName(name)
):
return None
if self._phyref is not None and not self._NeedUpdate():
if name in self._phyref:
return self._phyref[name]
if self._files_authoritative:
return ""
return None
if self._CheckFilesBackendConfig() is None:
return None
refs_dir = os.path.join(self._gitdir, "refs")
if platform_utils.islink(refs_dir) or not os.path.isdir(refs_dir):
return None
head_file = os.path.join(self._gitdir, HEAD)
if platform_utils.islink(head_file) or not os.path.isfile(head_file):
return None
cur = name
for _ in range(_SYMREF_MAXDEPTH + 1):
parts = cur.split("/")
for idx in range(1, len(parts) + 1):
if platform_utils.islink(
os.path.join(self._gitdir, *parts[:idx])
):
return None
ref_path = os.path.join(self._gitdir, *parts)
if os.path.lexists(ref_path + ".lock"):
return None
if os.path.lexists(ref_path):
if os.path.isdir(ref_path):
return None
parsed = self._ParseLooseRefFile(ref_path)
if parsed is None:
return None
is_sym, val, _ = parsed
if not is_sym:
return val
cur = val
continue
if cur == HEAD:
return None
packed_path = os.path.join(self._gitdir, "packed-refs")
if os.path.lexists(packed_path):
packed_res = self._ReadPackedRefs(packed_path)
if packed_res is None:
return None
packed_map, _ = packed_res
else:
packed_map = {}
if os.path.lexists(ref_path):
return None
return packed_map.get(cur, "")
return ""
def UpdateSymref(self, name: str, target: str) -> None:
"""Update a symbolic reference in an already-loaded ref snapshot.
Args:
name: Full symbolic reference name (e.g. "refs/remotes/m/main").
target: Target reference name that the symref points to.
"""
if self._phyref is None:
return
if self._HasExternalRefChanges(name):
self.Invalidate()
return
self._symref[name] = target
self._ResolveSymrefs()
parts = name.strip("/").split("/")
for idx in range(1, len(parts)):
anc = os.path.join(*parts[:idx])
self._TrackMtime(anc)
self._TrackMtime(os.path.join(*parts))
def UpdateRef(self, name: str) -> None:
"""Refresh a direct reference in an already-loaded ref snapshot.
Args:
name: Full reference name just updated on disk (e.g.
"refs/remotes/m/main").
"""
if self._phyref is None:
return
if self._HasExternalRefChanges(name):
self.Invalidate()
return
parts = name.strip("/").split("/")
ref_rel = os.path.join(*parts)
parsed = self._ParseLooseRefFile(os.path.join(self._gitdir, ref_rel))
if parsed is None or parsed[0]:
self.Invalidate()
return
_, oid, ref_stat = parsed
self._symref.pop(name, None)
self._phyref[name] = oid
self._ResolveSymrefs()
for idx in range(1, len(parts)):
self._TrackMtime(os.path.join(*parts[:idx]))
self._mtime[ref_rel] = ref_stat
def _HasExternalRefChanges(self, updated_ref: str) -> bool:
"""Check whether refs outside |updated_ref| changed on disk.
Args:
updated_ref: Reference path just updated on disk.
Returns:
True if files backend is unsafe or any unrelated ref path changed.
"""
if self._CheckFilesBackendConfig() is None:
return True
if "packed-refs" not in self._mtime and os.path.lexists(
os.path.join(self._gitdir, "packed-refs")
):
return True
parts = updated_ref.strip("/").split("/")
expected_child = {
"/".join(parts[:idx]): parts[idx] for idx in range(1, len(parts))
}
for tracked, prev_stat in self._mtime.items():
norm = tracked.replace(os.sep, "/")
if norm == updated_ref:
continue
full_path = os.path.join(self._gitdir, tracked)
curr_stat = self._StatPath(full_path)
if prev_stat is None:
if curr_stat is not None:
return True
continue
if curr_stat is None:
return True
if norm in expected_child:
if curr_stat != prev_stat:
try:
children = platform_utils.listdir(full_path)
except OSError:
return True
next_comp = expected_child[norm]
for child in children:
if child == next_comp:
continue
child_key = os.path.join(tracked, child)
if self._mtime.get(child_key) is None:
return True
elif curr_stat != prev_stat:
return True
for anc, next_comp in expected_child.items():
anc_os = anc.replace("/", os.sep)
if self._mtime.get(anc_os) is None:
anc_full = os.path.join(self._gitdir, anc_os)
try:
if platform_utils.listdir(anc_full) != [next_comp]:
return True
except OSError:
return True
return False
def _ResolveSymrefs(self) -> None:
"""Resolve all symbolic references into _phyref up to max depth."""
for name in self._symref:
self._phyref.pop(name, None)
scan = self._symref
attempts = 0
while scan and attempts < _SYMREF_MAXDEPTH:
scan_next = {}
for name, dest in scan.items():
if dest in self._phyref:
self._phyref[name] = self._phyref[dest]
else:
scan_next[name] = dest
scan = scan_next
attempts += 1
def _EnsureLoaded(self): def _EnsureLoaded(self):
if self._phyref is None or self._NeedUpdate(): if self._phyref is None or self._NeedUpdate():
self._LoadAll() self._LoadAll()
def _NeedUpdate(self): def _NeedUpdate(self) -> bool:
with repo_trace.Trace(": scan refs %s", self._gitdir): with repo_trace.Trace(": scan refs %s", self._gitdir):
for name, mtime in self._mtime.items(): if self._files_authoritative and (
try: "GIT_DIR" in os.environ or "GIT_COMMON_DIR" in os.environ
if mtime != os.path.getmtime( ):
os.path.join(self._gitdir, name) return True
): for name, prev_stat in self._mtime.items():
return True if (
except OSError: self._StatPath(os.path.join(self._gitdir, name))
!= prev_stat
):
return True return True
return False return False
def _LoadAll(self): def _LoadAll(self):
with repo_trace.Trace(": load refs %s", self._gitdir): with repo_trace.Trace(": load refs %s", self._gitdir):
if self._TryReadFilesBackend():
self._ResolveSymrefs()
return
self._files_authoritative = False
self._phyref = {} self._phyref = {}
self._symref = {} self._symref = {}
self._mtime = {} self._mtime = {}
@@ -105,28 +355,241 @@ class GitRefs:
# --include-root-refs does not report an unborn HEAD. # --include-root-refs does not report an unborn HEAD.
self._ReadSymbolicRef(HEAD) self._ReadSymbolicRef(HEAD)
scan = self._symref self._ResolveSymrefs()
attempts = 0 self._RefreshMtimes()
while scan and attempts < 5:
scan_next = {}
for name, dest in scan.items():
if dest in self._phyref:
self._phyref[name] = self._phyref[dest]
else:
scan_next[name] = dest
scan = scan_next
attempts += 1
self._TrackMtime(HEAD) def _CheckFilesBackendConfig(self) -> Optional[_StatKey]:
self._TrackMtime("config") """Verify standard files ref backend preconditions and stat config."""
self._TrackMtime("packed-refs") if "GIT_DIR" in os.environ or "GIT_COMMON_DIR" in os.environ:
self._TrackTreeMtimes("refs") return None
self._TrackTreeMtimes("reftable") if not os.path.isdir(self._gitdir):
return None
for special in ("reftable", "commondir"):
if os.path.lexists(os.path.join(self._gitdir, special)):
return None
cfg_path = os.path.join(self._gitdir, "config")
if platform_utils.islink(cfg_path):
return None
try:
with open(cfg_path, "rb") as cfg_fd:
if b"refstorage" in cfg_fd.read().lower():
return None
return self._FormatStat(os.fstat(cfg_fd.fileno()))
except OSError:
return None
@classmethod
def _ParseLooseRefFile(
cls, path: str
) -> Optional[Tuple[bool, str, _StatKey]]:
"""Parse a loose ref or HEAD file into (is_symref, val, stat)."""
if platform_utils.islink(path):
return None
try:
with open(path, "r", encoding="utf-8") as fd:
raw = fd.read(1024)
st = cls._FormatStat(os.fstat(fd.fileno()))
except (OSError, UnicodeDecodeError):
return None
line = raw.rstrip("\r\n")
if not line or "\n" in line or "\r" in line:
return None
if line.startswith("ref:"):
target = line[4:].strip()
if not target or target == R_HEADS + ".invalid":
return None
if target != HEAD and (
not target.startswith("refs/") or not IsValidRefName(target)
):
return None
return (True, target, st)
oid = line.strip()
if not _HEX_OID_RE.match(oid) or cls._IsNullRef(oid):
return None
return (False, oid, st)
@classmethod
def _ReadPackedRefs(
cls, packed_path: str
) -> Optional[Tuple[Dict[str, str], _StatKey]]:
"""Parse a packed-refs file into (ref_map, stat)."""
if platform_utils.islink(packed_path):
return None
packed: Dict[str, str] = {}
try:
with open(packed_path, "r", encoding="utf-8") as packed_fd:
for raw_line in packed_fd:
if raw_line.startswith("#"):
continue
line = raw_line.rstrip("\r\n")
if line.startswith("^"):
if not _HEX_OID_RE.match(line[1:]):
return None
continue
parts = line.split(" ", 1)
if len(parts) != 2:
return None
oid, ref_name = parts
if (
not _HEX_OID_RE.match(oid)
or cls._IsNullRef(oid)
or not ref_name.startswith("refs/")
or not IsValidRefName(ref_name)
):
return None
packed[ref_name] = oid
st = cls._FormatStat(os.fstat(packed_fd.fileno()))
except (OSError, UnicodeDecodeError):
return None
return (packed, st)
def _TryReadFilesBackend(self) -> bool:
"""Load HEAD, packed-refs, and loose refs directly when safe.
Returns:
True if the repository uses a standard files ref backend and all
refs and stat fingerprints were read cleanly without invoking git.
"""
cfg_stat = self._CheckFilesBackendConfig()
if cfg_stat is None:
return False
mtime: Dict[str, Optional[_StatKey]] = {
"config": cfg_stat,
"reftable": None,
}
phyref: Dict[str, str] = {}
symref: Dict[str, str] = {}
parsed_head = self._ParseLooseRefFile(os.path.join(self._gitdir, HEAD))
if parsed_head is None:
return False
head_is_sym, head_val, head_stat = parsed_head
mtime[HEAD] = head_stat
if head_is_sym:
symref[HEAD] = head_val
else:
phyref[HEAD] = head_val
packed_path = os.path.join(self._gitdir, "packed-refs")
packed_stat: Optional[_StatKey] = None
if os.path.lexists(packed_path):
packed_res = self._ReadPackedRefs(packed_path)
if packed_res is None:
return False
packed_map, packed_stat = packed_res
phyref.update(packed_map)
mtime["packed-refs"] = packed_stat
refs_dir = os.path.join(self._gitdir, "refs")
if platform_utils.islink(refs_dir):
return False
refs_stat = self._StatPath(refs_dir)
if refs_stat is None or not stat.S_ISDIR(refs_stat[0]):
return False
mtime["refs"] = refs_stat
try:
to_scan = [("refs", "refs", refs_dir)]
while to_scan:
ref_prefix, rel_os_prefix, abs_dir = to_scan.pop()
with os.scandir(abs_dir) as entries:
for entry in entries:
if entry.is_symlink():
return False
ref_name = f"{ref_prefix}/{entry.name}"
if not IsValidRefName(ref_name):
return False
rel_os = os.path.join(rel_os_prefix, entry.name)
if entry.is_dir(follow_symlinks=False):
if ref_name in phyref:
return False
# DirEntry.stat() leaves st_ino == 0 on Windows,
# whereas _StatPath() uses os.lstat() and populates
# st_ino. Stat via _StatPath() so _NeedUpdate() and
# _HasExternalRefChanges() compare matching keys.
d_stat = self._StatPath(entry.path)
if d_stat is None or not stat.S_ISDIR(d_stat[0]):
return False
mtime[rel_os] = d_stat
to_scan.append((ref_name, rel_os, entry.path))
elif entry.is_file(follow_symlinks=False):
parsed = self._ParseLooseRefFile(entry.path)
if parsed is None:
return False
is_sym, val, f_stat = parsed
mtime[rel_os] = f_stat
if is_sym:
phyref.pop(ref_name, None)
symref[ref_name] = val
else:
symref.pop(ref_name, None)
phyref[ref_name] = val
else:
return False
except OSError:
return False
# Guard against a concurrent `git pack-refs --prune` writing
# packed-refs and unlinking loose refs mid-scan.
if self._StatPath(packed_path) != packed_stat:
return False
self._phyref = phyref
self._symref = symref
self._mtime = mtime
self._files_authoritative = True
return True
def _RefreshMtimes(self) -> None:
"""Record current modification times for ref storage paths."""
self._mtime = {}
self._TrackMtime(HEAD, track_missing=True)
self._TrackMtime("config", track_missing=True)
self._TrackMtime("packed-refs", track_missing=True)
self._TrackMtime("refs", track_missing=True)
self._TrackMtime("reftable", track_missing=True)
self._TrackTreeMtimes("refs")
self._TrackTreeMtimes("reftable")
@staticmethod
def _FormatStat(st: os.stat_result) -> _StatKey:
# ctime is left out: tools that only touch metadata, such as backup
# agents and file system crawlers, move it without changing a ref.
return (st.st_mode, st.st_mtime_ns, st.st_size, st.st_ino)
@classmethod
def _StatPath(cls, path: str) -> Optional[_StatKey]:
try:
st = os.lstat(path)
except OSError:
return None
mount_tag = getattr(stat, "IO_REPARSE_TAG_MOUNT_POINT", -1)
is_reparse_dir = getattr(st, "st_reparse_tag", 0) == mount_tag
if stat.S_ISLNK(st.st_mode) or is_reparse_dir:
# Track the symlink target so mutations to symlinked refs or
# packed-refs invalidate the fallback snapshot, while keeping
# S_IFLNK in mode and negative size when dangling.
link_mode = stat.S_IFLNK | stat.S_IMODE(st.st_mode)
try:
target_st = os.stat(path)
except OSError:
return (link_mode, st.st_mtime_ns, -1 - st.st_size, st.st_ino)
return (
link_mode,
target_st.st_mtime_ns,
target_st.st_size,
target_st.st_ino,
)
return cls._FormatStat(st)
@staticmethod @staticmethod
def _IsNullRef(ref_id: str) -> bool: def _IsNullRef(ref_id: str) -> bool:
"""Check if a ref_id is a null object ID.""" """Check if a ref_id is a null object ID."""
return ref_id and all(ch == "0" for ch in ref_id) return bool(ref_id) and all(ch == "0" for ch in ref_id)
def _ReadRefs(self) -> bool: def _ReadRefs(self) -> bool:
"""Read all references using git for-each-ref. """Read all references using git for-each-ref.
@@ -165,6 +628,22 @@ class GitRefs:
def _ReadSymbolicRef(self, name: str) -> None: def _ReadSymbolicRef(self, name: str) -> None:
"""Read a symbolic reference.""" """Read a symbolic reference."""
if name == HEAD:
path = os.path.join(self._gitdir, HEAD)
try:
if not platform_utils.islink(path):
with open(
path, "r", encoding="utf-8", errors="replace"
) as fd:
line = fd.readline().strip()
if line.startswith("ref:"):
ref = line[4:].strip()
if ref and ref != R_HEADS + ".invalid":
self._symref[name] = ref
return
except OSError:
pass
p = git_command.GitCommand( p = git_command.GitCommand(
None, None,
["symbolic-ref", "-q", name], ["symbolic-ref", "-q", name],
@@ -192,32 +671,45 @@ class GitRefs:
if ref_id: if ref_id:
self._phyref[name] = ref_id self._phyref[name] = ref_id
def _TrackMtime(self, name: str) -> None: def _TrackMtime(self, name: str, track_missing: bool = False) -> None:
"""Track the modification time of a specific gitdir path.""" """Track the stat fingerprint of a specific gitdir path."""
path = os.path.join(self._gitdir, name) st = self._StatPath(os.path.join(self._gitdir, name))
try: if st is not None or track_missing:
self._mtime[name] = os.path.getmtime(path) self._mtime[name] = st
except OSError:
return
def _TrackTreeMtimes(self, root: str) -> None: def _TrackTreeMtimes(self, root: str) -> None:
"""Recursively track modification times for a directory tree.""" """Recursively track modification times for a directory tree."""
root_path = os.path.join(self._gitdir, root) root_path = os.path.join(self._gitdir, root)
try: try:
if not platform_utils.isdir(root_path): if not platform_utils.isdir(root_path):
self._TrackMtime(root)
return return
except OSError: except OSError:
return return
visited: Set[Tuple[int, int]] = set()
to_scan = [root] to_scan = [root]
while to_scan: while to_scan:
name = to_scan.pop() name = to_scan.pop()
self._TrackMtime(name) self._TrackMtime(name)
path = os.path.join(self._gitdir, name) path = os.path.join(self._gitdir, name)
if not platform_utils.isdir(path): try:
dir_st = os.stat(path)
except OSError:
continue continue
if not stat.S_ISDIR(dir_st.st_mode):
continue
if dir_st.st_ino:
dir_key = (dir_st.st_dev, dir_st.st_ino)
if dir_key in visited:
continue
visited.add(dir_key)
for child in platform_utils.listdir(path): try:
children = platform_utils.listdir(path)
except OSError:
continue
for child in children:
child_name = os.path.join(name, child) child_name = os.path.join(name, child)
child_path = os.path.join(self._gitdir, child_name) child_path = os.path.join(self._gitdir, child_name)
if platform_utils.isdir(child_path): if platform_utils.isdir(child_path):
+3 -3
View File
@@ -14,7 +14,7 @@
"""Read a worktree's state from one machine-readable git status snapshot.""" """Read a worktree's state from one machine-readable git status snapshot."""
from collections import OrderedDict import collections
import os import os
from typing import Iterator, List, Optional, TYPE_CHECKING from typing import Iterator, List, Optional, TYPE_CHECKING
@@ -45,8 +45,8 @@ class StatusSnapshot:
"""A consistent view of worktree, index, and branch state.""" """A consistent view of worktree, index, and branch state."""
def __init__(self) -> None: def __init__(self) -> None:
self.index_changes = OrderedDict() self.index_changes = collections.OrderedDict()
self.worktree_changes = OrderedDict() self.worktree_changes = collections.OrderedDict()
self.untracked = [] self.untracked = []
self.branch_oid = None self.branch_oid = None
self.branch_head = None self.branch_head = None
+8 -9
View File
@@ -29,12 +29,11 @@ import os
import sys import sys
import tempfile import tempfile
import time import time
from typing import NamedTuple from typing import NamedTuple, Optional
import urllib.parse import urllib.parse
import git_command import git_command
from git_config import IsId import git_config
from git_config import RepoConfig
import platform_utils import platform_utils
@@ -55,7 +54,7 @@ class CommitIdsResult(NamedTuple):
"""Return the commit ids and whether caller should exit.""" """Return the commit ids and whether caller should exit."""
# A dictionary with the projects/commit ids on success, otherwise None. # A dictionary with the projects/commit ids on success, otherwise None.
commit_ids: dict commit_ids: Optional[dict]
# Whether the caller should exit. # Whether the caller should exit.
fatal: bool fatal: bool
@@ -64,7 +63,7 @@ class UpdateProjectsResult(NamedTuple):
"""Return the overriding manifest file and whether caller should exit.""" """Return the overriding manifest file and whether caller should exit."""
# Path name of the overriding manifest file if successful, otherwise None. # Path name of the overriding manifest file if successful, otherwise None.
manifest_path: str manifest_path: Optional[str]
# Whether the caller should exit. # Whether the caller should exit.
fatal: bool fatal: bool
@@ -324,7 +323,7 @@ class Superproject:
# creating a local branch of the same name. # creating a local branch of the same name.
refspec = ( refspec = (
self.revision self.revision
if IsId(self.revision) if git_config.IsId(self.revision)
else f"{self.revision}:{self.revision}" else f"{self.revision}:{self.revision}"
) )
cmd.append(refspec) cmd.append(refspec)
@@ -429,7 +428,7 @@ class Superproject:
if not self._Init(): if not self._Init():
return SyncResult(False, should_exit) return SyncResult(False, should_exit)
if IsId(self.revision) and self.commit_id: if git_config.IsId(self.revision) and self.commit_id:
return SyncResult(True, False) return SyncResult(True, False)
if not self._Fetch(): if not self._Fetch():
return SyncResult(False, should_exit) return SyncResult(False, should_exit)
@@ -568,7 +567,7 @@ class Superproject:
@functools.lru_cache(maxsize=None) @functools.lru_cache(maxsize=None)
def _UseSuperprojectFromConfiguration(): def _UseSuperprojectFromConfiguration():
"""Returns the user choice of whether to use superproject.""" """Returns the user choice of whether to use superproject."""
user_cfg = RepoConfig.ForUser() user_cfg = git_config.RepoConfig.ForUser()
time_now = int(time.time()) time_now = int(time.time())
user_value = user_cfg.GetBoolean("repo.superprojectChoice") user_value = user_cfg.GetBoolean("repo.superprojectChoice")
@@ -602,7 +601,7 @@ def _UseSuperprojectFromConfiguration():
return user_value return user_value
# We don't have an unexpired choice, ask for one. # We don't have an unexpired choice, ask for one.
system_cfg = RepoConfig.ForSystem() system_cfg = git_config.RepoConfig.ForSystem()
system_value = system_cfg.GetBoolean("repo.superprojectChoice") system_value = system_cfg.GetBoolean("repo.superprojectChoice")
if system_value: if system_value:
# The system configuration is proposing that we should enable the # The system configuration is proposing that we should enable the
+2 -2
View File
@@ -15,10 +15,10 @@
"""Event logging in the git trace2 EVENT format.""" """Event logging in the git trace2 EVENT format."""
import git_command import git_command
from git_trace2_event_log_base import BaseEventLog import git_trace2_event_log_base
class EventLog(BaseEventLog): class EventLog(git_trace2_event_log_base.BaseEventLog):
"""Event log that records events that occurred during a repo invocation. """Event log that records events that occurred during a repo invocation.
Events are written to the log as a consecutive JSON entries, one per line. Events are written to the log as a consecutive JSON entries, one per line.
+2
View File
@@ -98,6 +98,8 @@ class BaseEventLog:
try: try:
env[KEY] = self._full_sid env[KEY] = self._full_sid
except UnicodeEncodeError: except UnicodeEncodeError:
# This is a specific fallback for Windows for string-vs-bytes.
# pyrefly: ignore
env[KEY] = self._full_sid.encode() env[KEY] = self._full_sid.encode()
if repo_source_version is not None: if repo_source_version is not None:
+3
View File
@@ -323,6 +323,7 @@ class RepoHook:
""" """
# Exec, storing global context in the context dict. We catch exceptions # Exec, storing global context in the context dict. We catch exceptions
# and convert to a HookError w/ just the failing traceback. # and convert to a HookError w/ just the failing traceback.
assert self._script_fullpath is not None
try: try:
exec(compile(data, self._script_fullpath, "exec"), context) exec(compile(data, self._script_fullpath, "exec"), context)
except Exception: except Exception:
@@ -367,6 +368,7 @@ class RepoHook:
# Always run hooks with CWD as topdir. # Always run hooks with CWD as topdir.
os.chdir(self._repo_topdir) os.chdir(self._repo_topdir)
assert self._script_fullpath is not None
# Put the hook dir as the first item of sys.path so hooks can do # Put the hook dir as the first item of sys.path so hooks can do
# relative imports. We want to replace the repo dir as [0] so # relative imports. We want to replace the repo dir as [0] so
# hooks can't import repo files. # hooks can't import repo files.
@@ -407,6 +409,7 @@ class RepoHook:
def _CheckHook(self): def _CheckHook(self):
# Bail with a nice error if we can't find the hook. # Bail with a nice error if we can't find the hook.
assert self._script_fullpath is not None
if not os.path.isfile(self._script_fullpath): if not os.path.isfile(self._script_fullpath):
raise error.HookError( raise error.HookError(
"Couldn't find repo hook: %s" % self._script_fullpath "Couldn't find repo hook: %s" % self._script_fullpath
Symlink
+1
View File
@@ -0,0 +1 @@
run_tests
+44 -38
View File
@@ -37,27 +37,26 @@ import repo_logging
try: try:
# Optional third-party module.
# pyrefly: ignore
import kerberos import kerberos
except ImportError: except ImportError:
kerberos = None kerberos = None
from color import SetDefaultColoring import color
from command import InteractiveCommand import command
from command import MirrorSafeCommand import editor
from editor import Editor
import error import error
import event_log import event_log
import git_command import git_command
from git_config import RepoConfig import git_config
from git_trace2_event_log import EventLog import git_trace2_event_log
from manifest_xml import RepoClient import manifest_xml
from pager import RunPager import pager
from pager import TerminatePager
import repo_trace import repo_trace
from subcmds import all_commands import subcmds
from subcmds.version import Version from subcmds.version import Version
from wrapper import Wrapper import wrapper
from wrapper import WrapperPath
logger = repo_logging.RepoLogger(__file__) logger = repo_logging.RepoLogger(__file__)
@@ -183,7 +182,7 @@ global_options.add_option(
class _Repo: class _Repo:
def __init__(self, repodir): def __init__(self, repodir):
self.repodir = repodir self.repodir = repodir
self.commands = all_commands self.commands = subcmds.all_commands
def _PrintHelp(self, short: bool = False, all_commands: bool = False): def _PrintHelp(self, short: bool = False, all_commands: bool = False):
"""Show --help screen.""" """Show --help screen."""
@@ -195,7 +194,7 @@ class _Repo:
help_commands = "".join(f"\n {x}" for x in wrapped_commands) help_commands = "".join(f"\n {x}" for x in wrapped_commands)
print(f"Available commands:{help_commands}") print(f"Available commands:{help_commands}")
print("\nRun `repo help <command>` for command-specific details.") print("\nRun `repo help <command>` for command-specific details.")
print("Bug reports:", Wrapper().BUG_URL) print("Bug reports:", wrapper.Wrapper().BUG_URL)
else: else:
cmd = self.commands["help"]() cmd = self.commands["help"]()
if all_commands: if all_commands:
@@ -230,9 +229,9 @@ class _Repo:
return name, [] return name, []
key = f"alias.{name}" key = f"alias.{name}"
alias = RepoConfig.ForRepository(self.repodir).GetString(key) alias = git_config.RepoConfig.ForRepository(self.repodir).GetString(key)
if alias is None: if alias is None:
alias = RepoConfig.ForUser().GetString(key) alias = git_config.RepoConfig.ForUser().GetString(key)
if alias is None: if alias is None:
return name, [] return name, []
@@ -263,18 +262,18 @@ class _Repo:
self._PrintHelp(short=True) self._PrintHelp(short=True)
return 1 return 1
git_trace2_event_log = EventLog() git_event_log = git_trace2_event_log.EventLog()
run = ( run = lambda: self._RunLong(name, gopts, argv, git_event_log) or 0
lambda: self._RunLong(name, gopts, argv, git_trace2_event_log) or 0
)
with repo_trace.Trace( with repo_trace.Trace(
"starting new command: %s [sid=%s]", "starting new command: %s [sid=%s]",
", ".join([name] + argv), ", ".join([name] + argv),
git_trace2_event_log.full_sid, git_event_log.full_sid,
first_trace=True, first_trace=True,
): ):
if gopts.trace_python: if gopts.trace_python:
import trace # This is a debug module that is not otherwise used, so don't
# import it all the time.
import trace # noqa: IMR200
tracer = trace.Trace( tracer = trace.Trace(
count=False, count=False,
@@ -288,7 +287,7 @@ class _Repo:
return result return result
def _autocorrect_command_name( def _autocorrect_command_name(
self, name: str, config: RepoConfig self, name: str, config: git_config.RepoConfig
) -> Optional[str]: ) -> Optional[str]:
"""Autocorrect command name based on user's git config.""" """Autocorrect command name based on user's git config."""
@@ -386,12 +385,12 @@ class _Repo:
def _RunLong(self, name, gopts, argv, git_trace2_event_log): def _RunLong(self, name, gopts, argv, git_trace2_event_log):
"""Execute the (longer running) requested subcommand.""" """Execute the (longer running) requested subcommand."""
result = 0 result = 0
SetDefaultColoring(gopts.color) color.SetDefaultColoring(gopts.color)
outer_client = RepoClient(self.repodir) outer_client = manifest_xml.RepoClient(self.repodir)
repo_client = outer_client repo_client = outer_client
if gopts.submanifest_path: if gopts.submanifest_path:
repo_client = RepoClient( repo_client = manifest_xml.RepoClient(
self.repodir, self.repodir,
submanifest_path=gopts.submanifest_path, submanifest_path=gopts.submanifest_path,
outer_client=outer_client, outer_client=outer_client,
@@ -413,10 +412,14 @@ class _Repo:
outer_manifest=outer_client.manifest, outer_manifest=outer_client.manifest,
git_event_log=git_trace2_event_log, git_event_log=git_trace2_event_log,
) )
cmd.event_log.enabled = bool(gopts.event_log)
Editor.globalConfig = cmd.client.globalConfig editor.Editor.globalConfig = cmd.client.globalConfig
if not isinstance(cmd, MirrorSafeCommand) and cmd.manifest.IsMirror: if (
not isinstance(cmd, command.MirrorSafeCommand)
and cmd.manifest.IsMirror
):
logger.error("fatal: '%s' requires a working directory", name) logger.error("fatal: '%s' requires a working directory", name)
return 1 return 1
@@ -433,7 +436,9 @@ class _Repo:
cmd.CommonValidateOptions(copts, cargs) cmd.CommonValidateOptions(copts, cargs)
git_trace2_event_log.verbose = copts.verbose git_trace2_event_log.verbose = copts.verbose
if gopts.pager is not False and not isinstance(cmd, InteractiveCommand): if gopts.pager is not False and not isinstance(
cmd, command.InteractiveCommand
):
config = cmd.client.globalConfig config = cmd.client.globalConfig
if gopts.pager: if gopts.pager:
use_pager = True use_pager = True
@@ -442,7 +447,7 @@ class _Repo:
if use_pager is None: if use_pager is None:
use_pager = cmd.WantPager(copts) use_pager = cmd.WantPager(copts)
if use_pager: if use_pager:
RunPager(config) pager.RunPager(config)
start = time.time() start = time.time()
cmd_event = cmd.event_log.Add(name, event_log.TASK_COMMAND, start) cmd_event = cmd.event_log.Add(name, event_log.TASK_COMMAND, start)
@@ -588,9 +593,10 @@ class _Repo:
file=sys.stderr, file=sys.stderr,
) )
cmd.event_log.FinishEvent( if cmd_event is not None:
cmd_event, finish, result is None or result == 0 cmd.event_log.FinishEvent(
) cmd_event, finish, result is None or result == 0
)
git_trace2_event_log.DefParamRepoEvents( git_trace2_event_log.DefParamRepoEvents(
cmd.manifest.manifestProject.config.DumpConfigDict() cmd.manifest.manifestProject.config.DumpConfigDict()
) )
@@ -627,7 +633,7 @@ def _CheckWrapperVersion(ver_str, repo_path):
sys.exit(1) sys.exit(1)
# Pull out the version of the repo launcher we know about to compare. # Pull out the version of the repo launcher we know about to compare.
exp = Wrapper().VERSION exp = wrapper.Wrapper().VERSION
ver = tuple(map(int, ver_str.split("."))) ver = tuple(map(int, ver_str.split(".")))
exp_str = ".".join(map(str, exp)) exp_str = ".".join(map(str, exp))
@@ -645,7 +651,7 @@ repo: error:
ver_str, ver_str,
min_str, min_str,
exp_str, exp_str,
WrapperPath(), wrapper.WrapperPath(),
repo_path, repo_path,
) )
sys.exit(1) sys.exit(1)
@@ -660,7 +666,7 @@ repo: error:
... You should upgrade soon: ... You should upgrade soon:
cp %s %s cp %s %s
""", """,
WrapperPath(), wrapper.WrapperPath(),
repo_path, repo_path,
) )
else: else:
@@ -671,7 +677,7 @@ repo: error:
!!! The launcher is not writable. Please talk to your sysadmin or distro !!! The launcher is not writable. Please talk to your sysadmin or distro
!!! to get an update installed. !!! to get an update installed.
""", """,
WrapperPath(), wrapper.WrapperPath(),
repo_path, repo_path,
) )
@@ -955,7 +961,7 @@ def _Main(argv):
print("fatal: %s" % e, file=sys.stderr) print("fatal: %s" % e, file=sys.stderr)
result = 128 result = 128
TerminatePager() pager.TerminatePager()
sys.exit(result) sys.exit(result)
+4 -1
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man. .\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "June 2026" "repo init" "Repo Manual" .TH REPO "1" "November 2026" "repo init" "Repo Manual"
.SH NAME .SH NAME
repo \- repo init - manual page for repo init repo \- repo init - manual page for repo init
.SH SYNOPSIS .SH SYNOPSIS
@@ -109,6 +109,9 @@ default to smart sync when syncing projects
\fB\-\-no\-smart\-sync\fR, \fB\-\-head\fR \fB\-\-no\-smart\-sync\fR, \fB\-\-head\fR
do not default to smart sync when syncing projects do not default to smart sync when syncing projects
.TP .TP
\fB\-\-smart\-sync\-branch\fR=\fI\,BRANCH\/\fR
server branch to query when smart syncing the manifest branch (empty string to clear)
.TP
\fB\-\-clone\-bundle\fR \fB\-\-clone\-bundle\fR
enable use of \fI\,/clone.bundle\/\fP on HTTP/HTTPS (default if not \fB\-\-partial\-clone\fR) enable use of \fI\,/clone.bundle\/\fP on HTTP/HTTPS (default if not \fB\-\-partial\-clone\fR)
.TP .TP
+68 -7
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man. .\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "August 2026" "repo manifest" "Repo Manual" .TH REPO "1" "October 2026" "repo manifest" "Repo Manual"
.SH NAME .SH NAME
repo \- repo manifest - manual page for repo manifest repo \- repo manifest - manual page for repo manifest
.SH SYNOPSIS .SH SYNOPSIS
@@ -149,7 +149,8 @@ include*)>
<!ELEMENT project (annotation*, <!ELEMENT project (annotation*,
project*, project*,
copyfile*, copyfile*,
linkfile*)> linkfile*,
sparse\-path*)>
.TP .TP
<!ATTLIST project name <!ATTLIST project name
CDATA #REQUIRED> CDATA #REQUIRED>
@@ -201,10 +202,14 @@ CDATA #IMPLIED>
<!ELEMENT linkfile EMPTY> <!ELEMENT linkfile EMPTY>
<!ATTLIST linkfile src CDATA #REQUIRED> <!ATTLIST linkfile src CDATA #REQUIRED>
<!ATTLIST linkfile dest CDATA #REQUIRED> <!ATTLIST linkfile dest CDATA #REQUIRED>
.IP
<!ELEMENT sparse\-path EMPTY>
<!ATTLIST sparse\-path path CDATA #REQUIRED>
.TP .TP
<!ELEMENT extend\-project (annotation*, <!ELEMENT extend\-project (annotation*,
copyfile*, copyfile*,
linkfile*)> linkfile*,
sparse\-path*)>
.TP .TP
<!ATTLIST extend\-project name <!ATTLIST extend\-project name
CDATA #REQUIRED> CDATA #REQUIRED>
@@ -514,10 +519,11 @@ of an existing project without completely replacing the existing project
definition. This makes the local manifest more robust against changes to the definition. This makes the local manifest more robust against changes to the
original manifest. original manifest.
.PP .PP
The `extend\-project` element can also contain `annotation`, `copyfile`, and The `extend\-project` element can also contain `annotation`, `copyfile`,
`linkfile` child elements. These are added to the project's definition. A `linkfile`, and `sparse\-path` child elements. These are added to the project's
`copyfile` or `linkfile` with a `dest` that already exists in the project will definition. A `copyfile` or `linkfile` with a `dest` that already exists in the
overwrite the original. project will overwrite the original. There is no way to remove a `sparse\-path`
that the original project definition declared.
.PP .PP
Attribute `path`: If specified, limit the change to projects checked out at the Attribute `path`: If specified, limit the change to projects checked out at the
specified path, rather than all projects with the given name. specified path, rather than all projects with the given name.
@@ -561,6 +567,61 @@ insensitive values "true" (default) or "false". This attribute determines
whether or not the annotation will be kept when exported with the manifest whether or not the annotation will be kept when exported with the manifest
subcommand. subcommand.
.PP .PP
Element sparse\-path
.PP
Zero or more sparse\-path elements may be specified as children of a project
element or an extend\-project element. Each element specifies a path to include
in the working directory.
.PP
Requires Git 2.25.0 or later, which is when `git sparse\-checkout` and cone mode
were introduced. Declaring a sparse\-path with an older Git fails the sync for
that project.
.PP
Declaring at least one sparse\-path is what enables Git sparse\-checkout for the
project: the listed paths are checked out, and other directories are left out of
the working directory. Cone mode also checks out the files sitting directly in
the project's top\-level directory and in each leading directory of a
sparse\-path, so `src/backend` brings along the top\-level files and the files
directly under `src/` too. A project with no sparse\-path elements is checked out
in full. The checkout uses [cone
mode](https://git\-scm.com/docs/git\-sparse\-checkout#_internalscone_mode_handling)
for better performance, which means only directory paths are supported;
individual files and globs are not. Cone mode does no pattern matching, so `*`,
`?`, `[`, `]`, and `\e` are not glob syntax in a `path`. In fact, Git 2.36.0 and
later reject them with `specify directories rather than patterns`, which
surfaces as a sync failure for the project. Earlier versions starting with
2.25.0 have no such check and are treated as literals.
.PP
Sparse\-checkout is configured only when the project's working directory is first
created. Changes to `sparse\-path` in the manifest (added, changed, or removed)
are not applied to a project that is already checked out. To change the paths in
an existing checkout, run `git sparse\-checkout` in the project directly, or
remove the project's working directory and sync again.
.PP
Because sparse\-checkout only prunes the working directory, a project with at
least one sparse\-path is fetched with `\-\-filter=blob:none` so that blobs which
will never be checked out are not downloaded. Without the filter Git still
downloads every blob in the project and sparse\-checkout merely declines to write
most of them to disk, so the filter is what turns sparse\-checkout into a saving
in transfer size and server load. The filter from `repo init \fB\-\-partial\-clone\fR
\fB\-\-clone\-filter\fR` takes precedence, and projects listed in `repo init
\fB\-\-partial\-clone\-exclude\fR` are fetched in full as usual. Like sparse\-checkout
itself, the implied filter only applies when the project is first fetched (or is
already a partial clone), so adding a sparse\-path to an existing full checkout
does not convert it to a partial clone. Mirrors are never filtered.
.PP
Because that filter is only useful if the objects it excludes are not fetched by
some other route, a project that takes the implied `\-\-filter=blob:none` also
skips the `/clone.bundle` download: the bundle is an unfiltered pack of every
object, so fetching it would undo the filter. Projects that set the filter
explicitly, or that are listed in `repo init \fB\-\-partial\-clone\-exclude\fR`, keep
using the clone bundle.
.PP
Attribute `path`: A path relative to the project root to include in the
sparse\-checkout. The path is validated like `<copyfile>` and `<linkfile>` paths:
it must stay inside the project, so absolute paths, `.` or `..` components, `~`,
and `.git` or `.repo*` components are all rejected. A trailing `/` is allowed.
.PP
Element copyfile Element copyfile
.PP .PP
Zero or more copyfile elements may be specified as children of a project Zero or more copyfile elements may be specified as children of a project
+4 -1
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man. .\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "September 2026" "repo smartsync" "Repo Manual" .TH REPO "1" "November 2026" "repo smartsync" "Repo Manual"
.SH NAME .SH NAME
repo \- repo smartsync - manual page for repo smartsync repo \- repo smartsync - manual page for repo smartsync
.SH SYNOPSIS .SH SYNOPSIS
@@ -23,6 +23,9 @@ number of network jobs to run in parallel (defaults to \fB\-\-jobs\fR or 1). Ign
\fB\-\-jobs\-checkout\fR=\fI\,JOBS\/\fR \fB\-\-jobs\-checkout\fR=\fI\,JOBS\/\fR
number of local checkout jobs to run in parallel (defaults to \fB\-\-jobs\fR or based on number of CPU cores). Ignored unless \fB\-\-no\-interleaved\fR is set number of local checkout jobs to run in parallel (defaults to \fB\-\-jobs\fR or based on number of CPU cores). Ignored unless \fB\-\-no\-interleaved\fR is set
.TP .TP
\fB\-\-force\-jobs\fR
force jobs to be the value of \fB\-j\fR even if the manifest caps it
.TP
\fB\-f\fR, \fB\-\-force\-broken\fR \fB\-f\fR, \fB\-\-force\-broken\fR
obsolete option (to be deleted in the future) obsolete option (to be deleted in the future)
.TP .TP
+4 -1
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man. .\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "October 2026" "repo sync" "Repo Manual" .TH REPO "1" "November 2026" "repo sync" "Repo Manual"
.SH NAME .SH NAME
repo \- repo sync - manual page for repo sync repo \- repo sync - manual page for repo sync
.SH SYNOPSIS .SH SYNOPSIS
@@ -23,6 +23,9 @@ number of network jobs to run in parallel (defaults to \fB\-\-jobs\fR or 1). Ign
\fB\-\-jobs\-checkout\fR=\fI\,JOBS\/\fR \fB\-\-jobs\-checkout\fR=\fI\,JOBS\/\fR
number of local checkout jobs to run in parallel (defaults to \fB\-\-jobs\fR or based on number of CPU cores). Ignored unless \fB\-\-no\-interleaved\fR is set number of local checkout jobs to run in parallel (defaults to \fB\-\-jobs\fR or based on number of CPU cores). Ignored unless \fB\-\-no\-interleaved\fR is set
.TP .TP
\fB\-\-force\-jobs\fR
force jobs to be the value of \fB\-j\fR even if the manifest caps it
.TP
\fB\-f\fR, \fB\-\-force\-broken\fR \fB\-f\fR, \fB\-\-force\-broken\fR
obsolete option (to be deleted in the future) obsolete option (to be deleted in the future)
.TP .TP
+4 -1
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man. .\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "August 2026" "repo upload" "Repo Manual" .TH REPO "1" "October 2026" "repo upload" "Repo Manual"
.SH NAME .SH NAME
repo \- repo upload - manual page for repo upload repo \- repo upload - manual page for repo upload
.SH SYNOPSIS .SH SYNOPSIS
@@ -80,6 +80,9 @@ ignore untracked files in the working copy
\fB\-\-no\-ignore\-untracked\-files\fR \fB\-\-no\-ignore\-untracked\-files\fR
always ask about untracked files in the working copy always ask about untracked files in the working copy
.TP .TP
\fB\-\-ignore\-remote\-commits\fR
ignore commits already in remote branches
.TP
\fB\-\-no\-cert\-checks\fR \fB\-\-no\-cert\-checks\fR
disable verifying ssl certs (unsafe) disable verifying ssl certs (unsafe)
.SS Logging options: .SS Logging options:
+41 -12
View File
@@ -22,16 +22,16 @@ import urllib.parse
import xml.dom.minidom import xml.dom.minidom
import error import error
from git_config import GitConfig import git_config
import git_refs import git_refs
from git_superproject import Superproject import git_superproject
import platform_utils import platform_utils
from project import Annotation from project import Annotation
from project import ManifestProject from project import ManifestProject
from project import Project from project import Project
from project import RemoteSpec from project import RemoteSpec
from project import RepoProject from project import RepoProject
from wrapper import Wrapper import wrapper
MANIFEST_FILE_NAME = "manifest.xml" MANIFEST_FILE_NAME = "manifest.xml"
@@ -414,7 +414,7 @@ class XmlManifest:
default_groups: a string, the default manifest groups to use. default_groups: a string, the default manifest groups to use.
""" """
# TODO(vapier): Move this out of this class. # TODO(vapier): Move this out of this class.
self.globalConfig = GitConfig.ForUser() self.globalConfig = git_config.GitConfig.ForUser()
self.repodir = os.path.abspath(repodir) self.repodir = os.path.abspath(repodir)
self._CheckLocalPath(submanifest_path) self._CheckLocalPath(submanifest_path)
@@ -754,6 +754,11 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
le.setAttribute("dest", lf.dest) le.setAttribute("dest", lf.dest)
e.appendChild(le) e.appendChild(le)
for sp in p.sparse_paths:
se = doc.createElement("sparse-path")
se.setAttribute("path", sp)
e.appendChild(se)
groups = p.groups - {"all", f"name:{p.name}", f"path:{p.relpath}"} groups = p.groups - {"all", f"name:{p.name}", f"path:{p.relpath}"}
if groups: if groups:
e.setAttribute("groups", ",".join(sorted(groups))) e.setAttribute("groups", ",".join(sorted(groups)))
@@ -815,7 +820,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
e.setAttribute("revision", self._superproject.revision) e.setAttribute("revision", self._superproject.revision)
root.appendChild(e) root.appendChild(e)
if self._contactinfo.bugurl != Wrapper().BUG_URL: if self._contactinfo.bugurl != wrapper.Wrapper().BUG_URL:
root.appendChild(doc.createTextNode("")) root.appendChild(doc.createTextNode(""))
e = doc.createElement("contactinfo") e = doc.createElement("contactinfo")
e.setAttribute("bugurl", self._contactinfo.bugurl) e.setAttribute("bugurl", self._contactinfo.bugurl)
@@ -846,6 +851,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
"annotation", "annotation",
"copyfile", "copyfile",
"linkfile", "linkfile",
"sparse-path",
} }
doc = self.ToXml(**kwargs) doc = self.ToXml(**kwargs)
@@ -1176,7 +1182,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
self._submanifests = {} self._submanifests = {}
self._repo_hooks_project = None self._repo_hooks_project = None
self._superproject = None self._superproject = None
self._contactinfo = ContactInfo(Wrapper().BUG_URL) self._contactinfo = ContactInfo(wrapper.Wrapper().BUG_URL)
self._notice = None self._notice = None
self.branch = None self.branch = None
self._manifest_server = None self._manifest_server = None
@@ -1205,11 +1211,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
self.manifestFile = override self.manifestFile = override
try: try:
m = self.manifestProject self.branch = self.manifestProject.GetMergeBranch()
b = m.GetBranch(m.CurrentBranch).merge
if b is not None and b.startswith(git_refs.R_HEADS):
b = b[len(git_refs.R_HEADS) :]
self.branch = b
parent_groups = self.parent_groups.copy() parent_groups = self.parent_groups.copy()
if self.path_prefix: if self.path_prefix:
@@ -1569,6 +1571,8 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
self._ParseLinkFile(p, n) self._ParseLinkFile(p, n)
elif n.nodeName == "annotation": elif n.nodeName == "annotation":
self._ParseAnnotation(p, n) self._ParseAnnotation(p, n)
elif n.nodeName == "sparse-path":
self._ParseSparsePath(p, n)
if node.nodeName == "repo-hooks": if node.nodeName == "repo-hooks":
# Only one project can be the hooks project # Only one project can be the hooks project
@@ -1608,7 +1612,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
"no revision for superproject %s within %s" "no revision for superproject %s within %s"
% (name, self.manifestFile) % (name, self.manifestFile)
) )
self._superproject = Superproject( self._superproject = git_superproject.Superproject(
self, self,
name=name, name=name,
remote=remote.ToRemoteSpec(name), remote=remote.ToRemoteSpec(name),
@@ -2033,6 +2037,8 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
self._ParseLinkFile(project, n) self._ParseLinkFile(project, n)
elif n.nodeName == "annotation": elif n.nodeName == "annotation":
self._ParseAnnotation(project, n) self._ParseAnnotation(project, n)
elif n.nodeName == "sparse-path":
self._ParseSparsePath(project, n)
elif n.nodeName == "project": elif n.nodeName == "project":
project.subprojects.append( project.subprojects.append(
self._ParseProject(n, parent=project) self._ParseProject(n, parent=project)
@@ -2307,6 +2313,29 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
) )
element.AddAnnotation(name, value, keep) element.AddAnnotation(name, value, keep)
def _ParseSparsePath(
self, project: Project, node: xml.dom.minidom.Element
) -> None:
"""Parses a <sparse-path> element and adds it to the project.
Args:
project: The Project object to add the sparse path to.
node: The XML node representing the <sparse-path> element.
Raises:
ManifestInvalidPathError: |path| is not usable as a project
relative path.
"""
path = self._reqatt(node, "path")
# Sparse paths name directories inside the project checkout, so they
# get the same scrutiny as <copyfile> & <linkfile> sources.
msg = self._CheckLocalPath(path, dir_ok=True)
if msg:
raise error.ManifestInvalidPathError(
f'<sparse-path> invalid "path": {path}: {msg}'
)
project.sparse_paths.append(path)
def _get_remote(self, node): def _get_remote(self, node):
name = node.getAttribute("remote") name = node.getAttribute("remote")
if not name: if not name:
+6 -7
View File
@@ -17,6 +17,7 @@ import os
import platform import platform
import shutil import shutil
import stat import stat
from typing import Union
def isWindows(): def isWindows():
@@ -28,6 +29,10 @@ def isWindows():
return platform.system() == "Windows" return platform.system() == "Windows"
if isWindows():
import platform_utils_win32
def symlink(source, link_name): def symlink(source, link_name):
"""Creates a symbolic link pointing to source named link_name. """Creates a symbolic link pointing to source named link_name.
@@ -35,8 +40,6 @@ def symlink(source, link_name):
to know whether to create a "File" or a "Directory" symbolic link. to know whether to create a "File" or a "Directory" symbolic link.
""" """
if isWindows(): if isWindows():
import platform_utils_win32
source = _validate_winpath(source) source = _validate_winpath(source)
link_name = _validate_winpath(link_name) link_name = _validate_winpath(link_name)
target = os.path.join(os.path.dirname(link_name), source) target = os.path.join(os.path.dirname(link_name), source)
@@ -222,7 +225,7 @@ def rmdir(path):
os.rmdir(_makelongpath(path)) os.rmdir(_makelongpath(path))
def removedirs(path: str) -> None: def removedirs(path: Union[str, os.PathLike]) -> None:
"""Remove a directory tree of symlinks and empty directories. """Remove a directory tree of symlinks and empty directories.
Walks |path| bottom-up without following symlinks. Removes symlinks Walks |path| bottom-up without following symlinks. Removes symlinks
@@ -273,8 +276,6 @@ def islink(path):
Availability: Windows, Unix. Availability: Windows, Unix.
""" """
if isWindows(): if isWindows():
import platform_utils_win32
return platform_utils_win32.islink(_makelongpath(path)) return platform_utils_win32.islink(_makelongpath(path))
else: else:
return os.path.islink(path) return os.path.islink(path)
@@ -289,8 +290,6 @@ def readlink(path):
Availability: Windows, Unix. Availability: Windows, Unix.
""" """
if isWindows(): if isWindows():
import platform_utils_win32
return platform_utils_win32.readlink(_makelongpath(path)) return platform_utils_win32.readlink(_makelongpath(path))
else: else:
return os.readlink(path) return os.readlink(path)
+12 -2
View File
@@ -17,6 +17,8 @@ from ctypes import wintypes
import errno import errno
# Windows-only ctypes APIs are not in typeshed when checking on non-Windows.
# pyrefly: ignore
kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
UCHAR = ctypes.c_ubyte UCHAR = ctypes.c_ubyte
@@ -164,7 +166,9 @@ def _create_symlink(source, link_name, dwFlags):
# "the unprivileged create flag is unsupported below Windows 10 (1703, # "the unprivileged create flag is unsupported below Windows 10 (1703,
# v10.0.14972). retry without it." # v10.0.14972). retry without it."
if not CreateSymbolicLinkW(link_name, source, dwFlags): if not CreateSymbolicLinkW(link_name, source, dwFlags):
# pyrefly: ignore
code = ctypes.get_last_error() code = ctypes.get_last_error()
# pyrefly: ignore
error_desc = ctypes.FormatError(code).strip() error_desc = ctypes.FormatError(code).strip()
if code == ERROR_PRIVILEGE_NOT_HELD: if code == ERROR_PRIVILEGE_NOT_HELD:
raise OSError(errno.EPERM, error_desc, link_name) raise OSError(errno.EPERM, error_desc, link_name)
@@ -190,7 +194,9 @@ def readlink(path):
) )
if reparse_point_handle == INVALID_HANDLE_VALUE: if reparse_point_handle == INVALID_HANDLE_VALUE:
_raise_winerror( _raise_winerror(
ctypes.get_last_error(), f'Error opening symbolic link "{path}"' # pyrefly: ignore
ctypes.get_last_error(),
f'Error opening symbolic link "{path}"',
) )
target_buffer = ctypes.c_buffer(MAXIMUM_REPARSE_DATA_BUFFER_SIZE) target_buffer = ctypes.c_buffer(MAXIMUM_REPARSE_DATA_BUFFER_SIZE)
n_bytes_returned = wintypes.DWORD() n_bytes_returned = wintypes.DWORD()
@@ -207,7 +213,9 @@ def readlink(path):
CloseHandle(reparse_point_handle) CloseHandle(reparse_point_handle)
if not io_result: if not io_result:
_raise_winerror( _raise_winerror(
ctypes.get_last_error(), f'Error reading symbolic link "{path}"' # pyrefly: ignore
ctypes.get_last_error(),
f'Error reading symbolic link "{path}"',
) )
rdb = REPARSE_DATA_BUFFER.from_buffer(target_buffer) rdb = REPARSE_DATA_BUFFER.from_buffer(target_buffer)
if rdb.ReparseTag == IO_REPARSE_TAG_SYMLINK: if rdb.ReparseTag == IO_REPARSE_TAG_SYMLINK:
@@ -221,6 +229,8 @@ def readlink(path):
def _raise_winerror(code, error_desc): def _raise_winerror(code, error_desc):
# pyrefly: ignore
win_error_desc = ctypes.FormatError(code).strip() win_error_desc = ctypes.FormatError(code).strip()
error_desc = f"{error_desc}: {win_error_desc}" error_desc = f"{error_desc}: {win_error_desc}"
# pyrefly: ignore
raise ctypes.WinError(code, error_desc) raise ctypes.WinError(code, error_desc)
+2
View File
@@ -20,6 +20,8 @@ import time
try: try:
import threading as _threading import threading as _threading
except ImportError: except ImportError:
# This module was removed in Python 3.9, but we still support Python 3.6.
# pyrefly: ignore
import dummy_threading as _threading import dummy_threading as _threading
import repo_trace import repo_trace
+767 -276
View File
File diff suppressed because it is too large. Load diff
+73
View File
@@ -41,7 +41,80 @@ extend_skip_glob = '*_pb2.py'
single_line_exclusions = ['abc', 'collections.abc', 'typing'] single_line_exclusions = ['abc', 'collections.abc', 'typing']
[tool.pytest.ini_options] [tool.pytest.ini_options]
addopts = [
"--durations=5",
"--durations-min=1.0",
"--strict-markers",
"--numprocesses=auto",
# Currently the test suite doesn't seem to get faster with more threads,
# so cap it here for now since there's a startup cost for every one.
"--maxprocesses=32",
]
markers = """ markers = """
skip_cq: Skip tests in the CQ. Should be rarely used! skip_cq: Skip tests in the CQ. Should be rarely used!
e2e: Hermetic end-to-end tests (tests/e2e/). Deselect with -m 'not e2e'. e2e: Hermetic end-to-end tests (tests/e2e/). Deselect with -m 'not e2e'.
""" """
# Only keep the temp dirs of failed tests. By default, pytest keeps the temp
# dirs of all tests for the last 3 runs, and each e2e test has its own checkout.
# Use `-o tmp_path_retention_policy=all` to inspect the dirs of passing tests.
tmp_path_retention_policy = "failed"
# https://pyrefly.org/en/docs/configuration/
[tool.pyrefly]
project-includes = [
# go/keep-sorted start
"**/*.py*",
"release/update-hooks",
"release/update-manpages",
"repo",
"run_tests",
# go/keep-sorted end
]
# We use relative module imports for our own modules.
enable-fallback-search-path = true
# NB: Do not add more modules here. We need to shrink this list.
project-excludes = [
"command.py",
"git_command.py",
"git_config.py",
"git_refs.py",
"git_status.py",
"git_superproject.py",
"main.py",
"manifest_xml.py",
"project.py",
"subcmds/branches.py",
"subcmds/cherry_pick.py",
"subcmds/config.py",
"subcmds/diffmanifests.py",
"subcmds/forall.py",
"subcmds/gc.py",
"subcmds/grep.py",
"subcmds/help.py",
"subcmds/info.py",
"subcmds/init.py",
"subcmds/manifest.py",
"subcmds/rebase.py",
"subcmds/selfupdate.py",
"subcmds/smartsync.py",
"subcmds/stage.py",
"subcmds/start.py",
"subcmds/status.py",
"subcmds/sync.py",
"subcmds/upload.py",
"subcmds/version.py",
"subcmds/wipe.py",
"tests/e2e/harness.py",
"tests/e2e/test_groups.py",
"tests/test_command.py",
"tests/test_git_command.py",
"tests/test_git_config.py",
"tests/test_git_superproject.py",
"tests/test_git_trace2_event_log.py",
"tests/test_manifest_xml.py",
"tests/test_project.py",
"tests/test_subcmds_info.py",
"tests/test_subcmds_sync.py",
"tests/test_subcmds_wipe.py",
"tests/test_wrapper.py",
]
Symlink
+1
View File
@@ -0,0 +1 @@
run_tests
Symlink
+1
View File
@@ -0,0 +1 @@
run_tests
+16 -10
View File
@@ -26,7 +26,7 @@ import platform
import shlex import shlex
import subprocess import subprocess
import sys import sys
from typing import NamedTuple, Tuple from typing import List, NamedTuple, Tuple
# These should never be newer than the main.py version since this needs to be a # These should never be newer than the main.py version since this needs to be a
@@ -119,17 +119,15 @@ if __name__ == "__main__":
# repo default configuration # repo default configuration
# #
REPO_URL = os.environ.get("REPO_URL", None) REPO_URL = (
if not REPO_URL: os.environ.get("REPO_URL") or "https://gerrit.googlesource.com/git-repo"
REPO_URL = "https://gerrit.googlesource.com/git-repo" )
REPO_REV = os.environ.get("REPO_REV") REPO_REV = os.environ.get("REPO_REV") or "stable"
if not REPO_REV:
REPO_REV = "stable"
# URL to file bug reports for repo tool issues. # URL to file bug reports for repo tool issues.
BUG_URL = "https://issues.gerritcodereview.com/issues/new?component=1370071" BUG_URL = "https://issues.gerritcodereview.com/issues/new?component=1370071"
# increment this whenever we make important changes to this script # increment this whenever we make important changes to this script
VERSION = (2, 71) VERSION = (2, 74)
# increment this if the MAINTAINER_KEYS block is modified # increment this if the MAINTAINER_KEYS block is modified
KEYRING_VERSION = (2, 3) KEYRING_VERSION = (2, 3)
@@ -251,7 +249,7 @@ def GetParser():
return parser return parser
def InitParser(parser): def InitParser(parser: optparse.OptionParser) -> optparse.OptionParser:
"""Setup the CLI parser.""" """Setup the CLI parser."""
# NB: Keep in sync with command.py:_CommonOptions(). # NB: Keep in sync with command.py:_CommonOptions().
@@ -457,6 +455,14 @@ def InitParser(parser):
dest="smart_sync", dest="smart_sync",
help="do not default to smart sync when syncing projects", help="do not default to smart sync when syncing projects",
) )
group.add_option(
"--smart-sync-branch",
action="store",
default=None,
metavar="BRANCH",
help="server branch to query when smart syncing the manifest branch "
"(empty string to clear)",
)
group.add_option( group.add_option(
"--clone-bundle", "--clone-bundle",
action="store_true", action="store_true",
@@ -902,7 +908,7 @@ def _GetRepoConfig(name):
def _InitHttp(): def _InitHttp():
handlers = [] handlers: List[urllib.request.BaseHandler] = []
mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm() mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
try: try:
+2 -2
View File
@@ -16,7 +16,7 @@
import logging import logging
from color import Coloring import color
import error import error
@@ -34,7 +34,7 @@ class _ConfigMock:
return self.default_values.get(x, None) return self.default_values.get(x, None)
class _LogColoring(Coloring): class _LogColoring(color.Coloring):
"""Coloring outstream for logging.""" """Coloring outstream for logging."""
def __init__(self, config): def __init__(self, config):
+5 -1
View File
@@ -25,6 +25,7 @@ import os
import sys import sys
import tempfile import tempfile
import time import time
from typing import Optional
import platform_utils import platform_utils
@@ -35,7 +36,7 @@ REPO_TRACE = "REPO_TRACE"
# Temporarily set tracing to always on unless user expicitly sets to 0. # Temporarily set tracing to always on unless user expicitly sets to 0.
_TRACE = os.environ.get(REPO_TRACE) != "0" _TRACE = os.environ.get(REPO_TRACE) != "0"
_TRACE_TO_STDERR = False _TRACE_TO_STDERR = False
_TRACE_FILE = None _TRACE_FILE: Optional[str] = None
_TRACE_FILE_NAME = "TRACE_FILE" _TRACE_FILE_NAME = "TRACE_FILE"
_MAX_SIZE = 70 # in MiB _MAX_SIZE = 70 # in MiB
_NEW_COMMAND_SEP = "+++++++++++++++NEW COMMAND+++++++++++++++++++" _NEW_COMMAND_SEP = "+++++++++++++++NEW COMMAND+++++++++++++++++++"
@@ -104,6 +105,7 @@ class Trace(contextlib.ContextDecorator):
f"PID: {os.getpid()} START: {self._time()} :{self._trace_msg}\n" f"PID: {os.getpid()} START: {self._time()} :{self._trace_msg}\n"
) )
assert _TRACE_FILE is not None
with open(_TRACE_FILE, "a") as f: with open(_TRACE_FILE, "a") as f:
print(print_msg, file=f) print(print_msg, file=f)
@@ -120,6 +122,7 @@ class Trace(contextlib.ContextDecorator):
f"PID: {os.getpid()} END: {self._time()} :{self._trace_msg}\n" f"PID: {os.getpid()} END: {self._time()} :{self._trace_msg}\n"
) )
assert _TRACE_FILE is not None
with open(_TRACE_FILE, "a") as f: with open(_TRACE_FILE, "a") as f:
print(print_msg, file=f) print(print_msg, file=f)
@@ -141,6 +144,7 @@ def _GetTraceFile(quiet):
def _ClearOldTraces(): def _ClearOldTraces():
"""Clear the oldest commands if trace file is too big.""" """Clear the oldest commands if trace file is too big."""
assert _TRACE_FILE is not None
try: try:
with open(_TRACE_FILE, errors="ignore") as f: with open(_TRACE_FILE, errors="ignore") as f:
if os.path.getsize(f.name) / (1024 * 1024) <= _MAX_SIZE: if os.path.getsize(f.name) / (1024 * 1024) <= _MAX_SIZE:
+8 -3
View File
@@ -19,9 +19,14 @@
# Hard versions are when we refuse to work the tool. Users will be shown an # Hard versions are when we refuse to work the tool. Users will be shown an
# error message before we abort entirely. # error message before we abort entirely.
# When deciding whether to upgrade a version requirement, check out the distro # When deciding whether to upgrade a version requirement, check out what
# lists to see who will be impacted: # versions major distros (e.g. Ubuntu LTS & Debian stable) ship and when they
# https://gerrit.googlesource.com/git-repo/+/HEAD/docs/release-process.md#Project-References # go EOL to see who will be impacted:
# https://wiki.ubuntu.com/Releases
# https://wiki.debian.org/DebianReleases#Versions_of_Debian_stable
# https://repology.org/project/python/versions
# https://repology.org/project/git/versions
# https://repology.org/project/openssh/versions
{ {
# The repo launcher itself. This allows us to force people to upgrade as some # The repo launcher itself. This allows us to force people to upgrade as some
+66 -7
View File
@@ -13,15 +13,20 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
# NB: Regen uv.lock whenever changing this script by running:
# $ vpython3 -vpython-tool upgrade run_tests
# /// script # /// script
# requires-python = '>=3.11,<3.12' # requires-python = '>=3.11,<3.12'
# dependencies = [ # dependencies = [
# 'pytest==8.3.4', # 'pytest==8.3.4',
# 'pytest-xdist==3.8.0',
# 'black==25.1.0', # 'black==25.1.0',
# 'flake8==6.0.0', # 'flake8==7.4.1',
# 'flake8-bugbear==24.12.12', # 'flake8-bugbear==26.9.30',
# 'flake8-comprehensions==3.16.0', # 'flake8-comprehensions==3.17.0',
# 'isort==5.10.1' # 'flake8-import-restrictions==2.1',
# 'isort==5.10.1',
# 'pyrefly==1.4.0',
# ] # ]
# /// # ///
@@ -30,10 +35,15 @@
import functools import functools
import os import os
import shlex import shlex
import shutil
import subprocess import subprocess
import sys import sys
import black
import flake8.main.cli
import isort.main
import pyrefly.__main__
import pytest
# NB: While tests/* support Python >=3.6 to match requirements.json for `repo`, # NB: While tests/* support Python >=3.6 to match requirements.json for `repo`,
# the higher level runner logic does not need to be held back. # the higher level runner logic does not need to be held back.
@@ -161,6 +171,30 @@ def run_isort():
).returncode ).returncode
def run_pyrefly() -> int:
"""Returns the exit code from pyrefly."""
argv = ["--version"]
log_cmd("pyrefly", argv)
subprocess.run(
[pyrefly.__main__.get_pyrefly_bin()] + argv,
check=True,
cwd=ROOT_DIR,
)
argv = [
"check",
"--verbose",
"--output-format=full-text",
"--progress-bar=simple",
]
log_cmd("pyrefly", argv)
return subprocess.run(
[pyrefly.__main__.get_pyrefly_bin()] + argv,
check=False,
cwd=ROOT_DIR,
).returncode
def run_check_metadata(): def run_check_metadata():
"""Returns the exit code from check-metadata.""" """Returns the exit code from check-metadata."""
argv = [] argv = []
@@ -198,8 +232,8 @@ def cipd_bootstrap() -> None:
return return
def main(argv): def main_run_tests(argv: list[str]) -> int:
"""The main entry.""" """The run_tests program."""
cipd_bootstrap() cipd_bootstrap()
checks = ( checks = (
functools.partial(run_pytest, argv), functools.partial(run_pytest, argv),
@@ -207,6 +241,7 @@ def main(argv):
run_black, run_black,
run_flake8, run_flake8,
run_isort, run_isort,
run_pyrefly,
run_check_metadata, run_check_metadata,
run_update_manpages, run_update_manpages,
) )
@@ -215,5 +250,29 @@ def main(argv):
return 1 if sum(c() for c in checks) else 0 return 1 if sum(c() for c in checks) else 0
def main_pyrefly(argv: list[str]) -> int:
"""The pyrefly program."""
result = subprocess.run(
[pyrefly.__main__.get_pyrefly_bin()] + argv, check=False
)
return result.returncode
main_black = black.main
main_flake8 = flake8.main.cli.main
main_isort = isort.main.main
main_pytest = pytest.main
def main(argv: list[str]) -> int:
"""The main dispatcher."""
# Figure out what script we're running as and dispatch to its main.
# This makes it easier for people to use the same env as run_tests.
argv0 = os.path.basename(sys.argv[0])
funcname = f"main_{argv0}"
func = globals().get(funcname, main_run_tests)
return func(argv)
if __name__ == "__main__": if __name__ == "__main__":
sys.exit(main(sys.argv[1:])) sys.exit(main(sys.argv[1:]))
+54 -67
View File
@@ -1,14 +1,12 @@
attrs==24.2.0 \ attrs==25.4.0 \
--hash=sha256:37c4653e51383dc1249dba15635d59011ca11714a8e61db8cd6dc73011e4b603 \ --hash=sha256:3e2be3ccc931d6c0cda7d37122b992d6d4b905968a437da7843cff016e2c41e8 \
--hash=sha256:81921eb96de3191c8258c199618104dd27ac608d9366f5e35d011eae1867ede2 --hash=sha256:adcf7e2a1fb3b36ac48d97835bb6d8ade15b8dcce26aba8bf1d14847b57a3373
# via flake8-bugbear # via flake8-bugbear
black==25.1.0 \ black==25.1.0 \
--hash=sha256:95e8176dae143ba9097f351d174fdaf0ccd29efb414b362ae3fd72bf0f710717 \ --hash=sha256:95e8176dae143ba9097f351d174fdaf0ccd29efb414b362ae3fd72bf0f710717 \
--hash=sha256:c9ed033e41998dd45d6f19bf9923e2444cb401185ccb26d1a2101a9049637c1f --hash=sha256:c9ed033e41998dd45d6f19bf9923e2444cb401185ccb26d1a2101a9049637c1f
click==8.0.3 \ click==8.4.1 \
--hash=sha256:353f466495adaeb40b6b5f592f9f91cb22372351c84caeb068132442a4518ef3 \ --hash=sha256:482be17c6991b8c19c5429a1e995d9b0efdbb63172824c41f99965dc0ade8ec2
--hash=sha256:353f466495adaeb40b6b5f592f9f91cb22372351c84caeb068132442a4518ef3 \
--hash=sha256:b0a0492d07c8e7e4a88ee153ca6553d357a0746792d6868bfca79dc1a344bb1b
# via black # via black
colorama==0.4.6 ; sys_platform == 'win32' \ colorama==0.4.6 ; sys_platform == 'win32' \
--hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 \ --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 \
@@ -16,19 +14,24 @@ colorama==0.4.6 ; sys_platform == 'win32' \
# via # via
# click # click
# pytest # pytest
flake8==6.0.0 \ execnet==2.1.1 \
--hash=sha256:9b5bf0ebd06aff1601e38dfc4e91f9b99ba3cdf446e74ee25a9cbbe01aef81ee \ --hash=sha256:26dee51f1b80cebd6d0ca8e74dd8745419761d3bef34163928cbebbdc4749fdc \
--hash=sha256:f34b5cb8bf908ff49e4673d3dbc557b2381a14132b637d53e9d3a59816f4b6ba --hash=sha256:58d6d58362e52bfab78f40ebfed9575a3334c41593bee4eb5fcda518290018dc
# via pytest-xdist
flake8==7.4.1 \
--hash=sha256:7ff3b14668bfec70da5ce3eeaf1756098330598a9a69f1ee312ed315d27c6508
# via # via
# flake8-bugbear # flake8-bugbear
# flake8-comprehensions # flake8-comprehensions
flake8-bugbear==24.12.12 \ flake8-bugbear==26.9.30 \
--hash=sha256:1b6967436f65ca22a42e5373aaa6f2d87966ade9aa38d4baf2a1be550767545e --hash=sha256:d0fec65b92eb79adde488515c925c33afabbf85d194bd881e845c71444960fc5
flake8-comprehensions==3.16.0 \ flake8-comprehensions==3.17.0 \
--hash=sha256:7c1eadc9d22e765f39857798febe7766b4d9c519793c6c149e3e13bf99693f70 --hash=sha256:3943a9c6f2593c3bc5cc64106c2f89d63c6ecd49c8343597f8257b8fcfc8b0a2
iniconfig==1.1.1 \ flake8-import-restrictions==2.1 \
--hash=sha256:011e24c64b7f47f6ebd835bb12a743f2fbe9a26d4cecaa7f53bc4f35ee9da8b3 \ --hash=sha256:77522e76b82a6ccb25820b1e195825a695094ceded23a18c1b5ce026c23e1071
--hash=sha256:9d81514370059186276e9a1799e5a9752e4c060233c254d6954b53fad2ff8a5e iniconfig==2.3.0 \
--hash=sha256:e6ae3625a739878eeb4baf25f17a0b4e9d985421e45b897d1332ed7478f822d9 \
--hash=sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12
# via pytest # via pytest
isort==5.10.1 \ isort==5.10.1 \
--hash=sha256:6f62d78e2f89b4500b080fe3a81690850cd254227f27f75c3a0c491a1f351ba7 \ --hash=sha256:6f62d78e2f89b4500b080fe3a81690850cd254227f27f75c3a0c491a1f351ba7 \
@@ -37,65 +40,49 @@ mccabe==0.7.0 \
--hash=sha256:6c2d30ab6be0e4a46919781807b4f0d834ebdd6c6e3dca0bda5a15f863427b6e \ --hash=sha256:6c2d30ab6be0e4a46919781807b4f0d834ebdd6c6e3dca0bda5a15f863427b6e \
--hash=sha256:b85abd0a321cdd357d0be34adbf49cf01453a857a23e94d44612ec6f159cb261 --hash=sha256:b85abd0a321cdd357d0be34adbf49cf01453a857a23e94d44612ec6f159cb261
# via flake8 # via flake8
mypy-extensions==0.4.3 \ mypy-extensions==1.1.0 \
--hash=sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d \ --hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \
--hash=sha256:38cbac35a502c4f13b9d9b1d06258fa51e01da4f5828b171b4d1200d00378e7c --hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \
--hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \
--hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \
--hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \
--hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \
--hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \
--hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \
--hash=sha256:4ec363d206d93f233330fa7f08eefe9a382dd1e326112a94244f95ec2f22f10e
# via black # via black
packaging==23.0 \ packaging==26.2 \
--hash=sha256:37cb5b4759d17ae6bad78ee3347f0ee1057452895fcd1bbff48fb12908ce00a1 \ --hash=sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2
# via # via
# black # black
# pytest # pytest
pathspec==0.9.0 \ pathspec==1.0.4 \
--hash=sha256:7d15c4ddb0b5c802d161efc417ec1a2558ea2653c2e8ad9c19098201dc1c993a \ --hash=sha256:eeaf9fc7cbc0a4492b666be276f38956baa67cd3faca745462db83d98f1cef87 \
--hash=sha256:aaf90f4ad38a16bd3ebe0fa64da7f6ee01819c881120db68cc6440941a9adc08 --hash=sha256:fb6ae2fd4e7c921a165808a552060e722767cfa526f99ca5156ed2ce45a5c723
# via black # via black
platformdirs==2.5.2 \ platformdirs==4.9.2 \
--hash=sha256:027d8e83a2d7de06bbac4e5ef7e023c02b863d7ea5d079477e722bb41ab25788 \ --hash=sha256:9170634f126f8efdae22fb58ae8a0eaa86f38365bc57897a6c4f781d1f5875bd \
--hash=sha256:e257a04b5d0528cb9836b766c8a16a85368e249cba7ae42888a403976ccd7d2d --hash=sha256:ef245e78eb7a2ba57599bdb39c4ff323cfae3de356a47207b58bb29e7f479f46
# via black # via black
pluggy==1.5.0 \ pluggy==1.6.0 \
--hash=sha256:40da5bf96b15e359e280fd760a80c11d7a8ea20c3c296c3f3e09f8be67cec10b \ --hash=sha256:88349c39e440567519b15b27a781bcd3430de6bae17c44c37e758795be96dc5a \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \ --hash=sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669
# via pytest # via pytest
py==1.11.0 \ pycodestyle==2.15.0 \
--hash=sha256:607c53218732647dff4acdfcd50cb62615cedf612e72d1724fb1a0cc6405b378 \ --hash=sha256:12fd2f73c7b8ee8845a0431111df8faf4c1a07d6e64e2ee7f0c74014dab14181
--hash=sha256:607c53218732647dff4acdfcd50cb62615cedf612e72d1724fb1a0cc6405b378
pycodestyle==2.10.0 \
--hash=sha256:8a4eaf0d0495c7395bdab3589ac2db602797d76207242c17d470186815706610 \
--hash=sha256:aacf8d28ecea10fad31b2d14943a42d85d007f4d2fee8f212fc05751c8460647
# via flake8 # via flake8
pyflakes==3.0.1 \ pyflakes==4.0.3 \
--hash=sha256:a968b8baa43926124b692c7e76279bc5fa8c6ef9a2281e4a2a475be69f954bdf \ --hash=sha256:330ba92b8c1db2eb0b8f4068f6c58674e2649a99e334769aa50e3e9c5b11c23a
--hash=sha256:ec55bf7fe21fff7f1ad2f7da62363d749e2a470500eab1b555334b67aa1ef8cf
# via flake8 # via flake8
pyparsing==3.0.7 \ pyrefly==1.4.0 \
--hash=sha256:943c220ca64e93940b74dd070152ed9ef1402bc3b28fd1f899843bee34e83708 \ --hash=sha256:460cec39bc8743915e9c2c8c10ba926ad0b29b099d89c1859fb97e0059f42fa7 \
--hash=sha256:a6c06a88f252e6c322f65faf8f418b16213b51bdfaece0524c1c1bc30c63c484 --hash=sha256:5a6370d686b6cbbac8869832f09a52012888371956fbc68d446a349bc85ccd43 \
--hash=sha256:7aff37c2b711c8240ed650da0c6148237b327bf7e5dfacee65884f1adea22305 \
--hash=sha256:b6951f51687304caebfedf13dd1a943aeafdbc95abcc4e24a683b97467a77007
pytest==8.3.4 \ pytest==8.3.4 \
--hash=sha256:00cba7845d2dccfe5403025a688d60ceae228b1ddaa866d2eb86b2edb6923e73 \ --hash=sha256:00cba7845d2dccfe5403025a688d60ceae228b1ddaa866d2eb86b2edb6923e73 \
--hash=sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6 --hash=sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6
toml==0.10.1 \ # via pytest-xdist
--hash=sha256:4099063d9b50219bf09bfa693ab8dcd96ce585cee9fdadce5a48a08ee23b0646 \ pytest-xdist==3.8.0 \
--hash=sha256:bda89d5935c2eac546d648028b9901107a595863cb36bae0c73ac804a9b4ce88 --hash=sha256:202ca578cfeb7370784a8c33d6d05bc6e13b4f25b5053c30a152269fd10f0b88 \
tomli==2.0.1 \ --hash=sha256:9e659fa2eacbed776070a08a075eff2d9c8e010bab4eea169640c22a1b079f2c
--hash=sha256:864f310e021d68415d6ba45b23e29808f99703b471d884267d6942eecdf8d6c5 \
--hash=sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc
typing-extensions==4.3.0 \
--hash=sha256:09126dab54422810a541e078c37e62024e7260a022d4f67c075a6707d9553649 \
--hash=sha256:25642c956049920a5aa49edcdd6ab1e06d7e5d467fc00e0506c44ac86fbfca02
+5
View File
@@ -65,3 +65,8 @@ wheel: <
name: "infra/python/wheels/exceptiongroup-py3" name: "infra/python/wheels/exceptiongroup-py3"
version: "version:1.1.2" version: "version:1.1.2"
> >
wheel: <
name: "infra/python/wheels/pytest-xdist-py2_py3"
version: "version:1.31.0"
>
+2
View File
@@ -17,6 +17,8 @@
import os import os
# Not installed in the test virtualenv.
# pyrefly: ignore
import setuptools import setuptools
+5 -6
View File
@@ -16,11 +16,10 @@ import collections
import functools import functools
import itertools import itertools
from command import Command import command
from command import DEFAULT_LOCAL_JOBS
import error import error
import git_command import git_command
from progress import Progress import progress
import repo_logging import repo_logging
@@ -31,7 +30,7 @@ class AbandonError(error.RepoExitError):
"""Exit error when abandon command fails.""" """Exit error when abandon command fails."""
class Abandon(Command): class Abandon(command.Command):
COMMON = True COMMON = True
helpSummary = "Permanently abandon a development branch" helpSummary = "Permanently abandon a development branch"
helpUsage = """ helpUsage = """
@@ -42,7 +41,7 @@ deleting it (and all its history) from your local repository.
It is equivalent to "git branch -D <branchname>". It is equivalent to "git branch -D <branchname>".
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
def _Options(self, p): def _Options(self, p):
p.add_option( p.add_option(
@@ -119,7 +118,7 @@ It is equivalent to "git branch -D <branchname>".
functools.partial(self._ExecuteOne, opt.all, nb), functools.partial(self._ExecuteOne, opt.all, nb),
range(len(all_projects)), range(len(all_projects)),
callback=_ProcessResults, callback=_ProcessResults,
output=Progress( output=progress.Progress(
f"Abandon {nb}", len(all_projects), quiet=opt.quiet f"Abandon {nb}", len(all_projects), quiet=opt.quiet
), ),
chunksize=1, chunksize=1,
+6 -7
View File
@@ -15,14 +15,13 @@
import itertools import itertools
import sys import sys
from color import Coloring import color
from command import Command import command
from command import DEFAULT_LOCAL_JOBS
class BranchColoring(Coloring): class BranchColoring(color.Coloring):
def __init__(self, config): def __init__(self, config):
Coloring.__init__(self, config, "branch") super().__init__(config, "branch")
self.current = self.printer("current", fg="green") self.current = self.printer("current", fg="green")
self.local = self.printer("local") self.local = self.printer("local")
self.notinproject = self.printer("notinproject", fg="red") self.notinproject = self.printer("notinproject", fg="red")
@@ -62,7 +61,7 @@ class BranchInfo:
return self.published_equal == len(self.projects) return self.published_equal == len(self.projects)
class Branches(Command): class Branches(command.Command):
COMMON = True COMMON = True
helpSummary = "View current topic branches" helpSummary = "View current topic branches"
helpUsage = """ helpUsage = """
@@ -96,7 +95,7 @@ the branch appears in, or does not appear in. If no project list
is shown, then the branch appears in all projects. is shown, then the branch appears in all projects.
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
@classmethod @classmethod
def _ExpandProjectToBranches(cls, project_idx): def _ExpandProjectToBranches(cls, project_idx):
+8 -9
View File
@@ -13,12 +13,11 @@
# limitations under the License. # limitations under the License.
import functools import functools
from typing import NamedTuple from typing import NamedTuple, Optional
from command import Command import command
from command import DEFAULT_LOCAL_JOBS
import error import error
from progress import Progress import progress
import repo_logging import repo_logging
@@ -27,9 +26,9 @@ logger = repo_logging.RepoLogger(__file__)
class CheckoutBranchResult(NamedTuple): class CheckoutBranchResult(NamedTuple):
# Whether the Project is on the branch (i.e. branch exists and no errors) # Whether the Project is on the branch (i.e. branch exists and no errors)
result: bool result: Optional[bool]
project_idx: int project_idx: int
error: Exception error: Optional[Exception]
class CheckoutCommandError(error.RepoExitError): class CheckoutCommandError(error.RepoExitError):
@@ -40,7 +39,7 @@ class MissingBranchError(error.RepoExitError):
"""Exception thrown when no project has specified branch.""" """Exception thrown when no project has specified branch."""
class Checkout(Command): class Checkout(command.Command):
COMMON = True COMMON = True
helpSummary = "Checkout a branch for development" helpSummary = "Checkout a branch for development"
helpUsage = """ helpUsage = """
@@ -54,7 +53,7 @@ The command is equivalent to:
repo forall [<project>...] -c git checkout <branchname> repo forall [<project>...] -c git checkout <branchname>
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
def ValidateOptions(self, opt, args): def ValidateOptions(self, opt, args):
if not args: if not args:
@@ -98,7 +97,7 @@ The command is equivalent to:
functools.partial(self._ExecuteOne, nb), functools.partial(self._ExecuteOne, nb),
range(len(all_projects)), range(len(all_projects)),
callback=_ProcessResults, callback=_ProcessResults,
output=Progress( output=progress.Progress(
f"Checkout {nb}", len(all_projects), quiet=opt.quiet f"Checkout {nb}", len(all_projects), quiet=opt.quiet
), ),
) )
+2 -2
View File
@@ -16,7 +16,7 @@ import re
import sys import sys
from typing import Tuple from typing import Tuple
from command import Command import command
import error import error
import git_command import git_command
import repo_logging import repo_logging
@@ -26,7 +26,7 @@ CHANGE_ID_RE = re.compile(r"^\s*Change-Id: I([0-9a-f]{40})\s*$")
logger = repo_logging.RepoLogger(__file__) logger = repo_logging.RepoLogger(__file__)
class CherryPick(Command): class CherryPick(command.Command):
COMMON = True COMMON = True
helpSummary = "Cherry-pick a change." helpSummary = "Cherry-pick a change."
helpUsage = """ helpUsage = """
+3 -4
View File
@@ -15,11 +15,10 @@
import functools import functools
import io import io
from command import DEFAULT_LOCAL_JOBS import command
from command import PagedCommand
class Diff(PagedCommand): class Diff(command.PagedCommand):
COMMON = True COMMON = True
helpSummary = "Show changes between commit and working tree" helpSummary = "Show changes between commit and working tree"
helpUsage = """ helpUsage = """
@@ -29,7 +28,7 @@ The -u option causes '%prog' to generate diff output with file paths
relative to the repository root, so the output can be applied relative to the repository root, so the output can be applied
to the Unix 'patch' command. to the Unix 'patch' command.
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
def _Options(self, p): def _Options(self, p):
p.add_option( p.add_option(
+8 -8
View File
@@ -12,17 +12,17 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from color import Coloring import color
from command import PagedCommand import command
from manifest_xml import RepoClient import manifest_xml
class _Coloring(Coloring): class _Coloring(color.Coloring):
def __init__(self, config): def __init__(self, config):
Coloring.__init__(self, config, "status") super().__init__(config, "status")
class Diffmanifests(PagedCommand): class Diffmanifests(command.PagedCommand):
"""A command to see logs in projects represented by manifests """A command to see logs in projects represented by manifests
This is used to see deeper differences between manifests. Where a simple This is used to see deeper differences between manifests. Where a simple
@@ -238,12 +238,12 @@ synced and their revisions won't be found.
self.printRevision self.printRevision
) = self.printText ) = self.printText
manifest1 = RepoClient(self.repodir) manifest1 = manifest_xml.RepoClient(self.repodir)
manifest1.Override(args[0], load_local_manifests=False) manifest1.Override(args[0], load_local_manifests=False)
if len(args) == 1: if len(args) == 1:
manifest2 = self.manifest manifest2 = self.manifest
else: else:
manifest2 = RepoClient(self.repodir) manifest2 = manifest_xml.RepoClient(self.repodir)
manifest2.Override(args[1], load_local_manifests=False) manifest2.Override(args[1], load_local_manifests=False)
diff = manifest1.projectsDiff(manifest2) diff = manifest1.projectsDiff(manifest2)
+2 -2
View File
@@ -15,7 +15,7 @@
import re import re
import sys import sys
from command import Command import command
import error import error
import repo_logging import repo_logging
@@ -28,7 +28,7 @@ class DownloadCommandError(error.RepoExitError):
"""Error raised when download command fails.""" """Error raised when download command fails."""
class Download(Command): class Download(command.Command):
COMMON = True COMMON = True
helpSummary = "Download and checkout a change" helpSummary = "Download and checkout a change"
helpUsage = """ helpUsage = """
+9 -11
View File
@@ -21,10 +21,8 @@ import signal
import subprocess import subprocess
import sys import sys
from color import Coloring import color
from command import Command import command
from command import DEFAULT_LOCAL_JOBS
from command import MirrorSafeCommand
import repo_logging import repo_logging
@@ -37,13 +35,13 @@ _CAN_COLOR = [
] ]
class ForallColoring(Coloring): class ForallColoring(color.Coloring):
def __init__(self, config): def __init__(self, config):
Coloring.__init__(self, config, "forall") super().__init__(config, "forall")
self.project = self.printer("project", attr="bold") self.project = self.printer("project", attr="bold")
class Forall(Command, MirrorSafeCommand): class Forall(command.Command, command.MirrorSafeCommand):
COMMON = False COMMON = False
helpSummary = "Run a shell command in each project" helpSummary = "Run a shell command in each project"
helpUsage = """ helpUsage = """
@@ -136,7 +134,7 @@ terminal and are not redirected.
If -e is used, when a command exits unsuccessfully, '%prog' will abort If -e is used, when a command exits unsuccessfully, '%prog' will abort
without iterating through the remaining projects. without iterating through the remaining projects.
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
@staticmethod @staticmethod
def _cmd_option(option, _opt_str, _value, parser): def _cmd_option(option, _opt_str, _value, parser):
@@ -197,7 +195,7 @@ without iterating through the remaining projects.
"--interactive", action="store_true", help="force interactive usage" "--interactive", action="store_true", help="force interactive usage"
) )
def WantPager(self, opt): def WantPager(self, opt) -> bool:
return opt.project_header and opt.jobs == 1 return opt.project_header and opt.jobs == 1
def ValidateOptions(self, opt, args): def ValidateOptions(self, opt, args):
@@ -234,9 +232,9 @@ without iterating through the remaining projects.
cn = None cn = None
if cn and cn in _CAN_COLOR: if cn and cn in _CAN_COLOR:
class ColorCmd(Coloring): class ColorCmd(color.Coloring):
def __init__(self, config, cmd): def __init__(self, config, cmd):
Coloring.__init__(self, config, cmd) super().__init__(config, cmd)
if ColorCmd(self.manifest.manifestProject.config, cn).is_on: if ColorCmd(self.manifest.manifestProject.config, cn).is_on:
cmd.insert(cmd.index(cn) + 1, "--color") cmd.insert(cmd.index(cn) + 1, "--color")
+5 -5
View File
@@ -15,14 +15,14 @@
import os import os
from typing import List, Set from typing import List, Set
from command import Command import command
import git_command import git_command
import platform_utils import platform_utils
from progress import Progress import progress
from project import Project from project import Project
class Gc(Command): class Gc(command.Command):
COMMON = True COMMON = True
helpSummary = "Cleaning up internal repo and Git state." helpSummary = "Cleaning up internal repo and Git state."
helpUsage = """ helpUsage = """
@@ -124,7 +124,7 @@ are still in the manifests.
if ask.lower() != "y": if ask.lower() != "y":
return 1 return 1
pm = Progress( pm = progress.Progress(
"Deleting", "Deleting",
len(to_delete), len(to_delete),
delay=False, delay=False,
@@ -180,7 +180,7 @@ are still in the manifests.
print(f"Would have repacked {len(repack_projects)} projects.") print(f"Would have repacked {len(repack_projects)} projects.")
return 0 return 0
pm = Progress( pm = progress.Progress(
"Repacking (this will take a while)", "Repacking (this will take a while)",
len(repack_projects), len(repack_projects),
delay=False, delay=False,
+8 -8
View File
@@ -16,9 +16,8 @@ import functools
import sys import sys
from typing import NamedTuple from typing import NamedTuple
from color import Coloring import color
from command import DEFAULT_LOCAL_JOBS import command
from command import PagedCommand
import error import error
import git_command import git_command
import repo_logging import repo_logging
@@ -27,9 +26,9 @@ import repo_logging
logger = repo_logging.RepoLogger(__file__) logger = repo_logging.RepoLogger(__file__)
class GrepColoring(Coloring): class GrepColoring(color.Coloring):
def __init__(self, config): def __init__(self, config):
Coloring.__init__(self, config, "grep") super().__init__(config, "grep")
self.project = self.printer("project", attr="bold") self.project = self.printer("project", attr="bold")
self.fail = self.printer("fail", fg="red") self.fail = self.printer("fail", fg="red")
@@ -50,7 +49,7 @@ class GrepCommandError(error.SilentRepoExitError):
aggregate errors exit silently.""" aggregate errors exit silently."""
class Grep(PagedCommand): class Grep(command.PagedCommand):
COMMON = True COMMON = True
helpSummary = "Print lines matching a pattern" helpSummary = "Print lines matching a pattern"
helpUsage = """ helpUsage = """
@@ -84,7 +83,7 @@ contain a line that matches both expressions:
repo grep --all-match -e NODE -e Unexpected repo grep --all-match -e NODE -e Unexpected
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
@staticmethod @staticmethod
def _carry_option(_option, opt_str, value, parser): def _carry_option(_option, opt_str, value, parser):
@@ -103,8 +102,9 @@ contain a line that matches both expressions:
if value is not None: if value is not None:
pt.append(value) pt.append(value)
def _CommonOptions(self, p): def _CommonOptions(self, p, opt_v=True):
"""Override common options slightly.""" """Override common options slightly."""
del opt_v # Unused.
super()._CommonOptions(p, opt_v=False) super()._CommonOptions(p, opt_v=False)
def _Options(self, p): def _Options(self, p):
+24 -24
View File
@@ -15,20 +15,20 @@
import re import re
import sys import sys
import textwrap import textwrap
from typing import Sequence
from color import Coloring import color
from command import MirrorSafeCommand import command
from command import PagedCommand
import error import error
from subcmds import all_commands import subcmds
from wrapper import Wrapper import wrapper
class InvalidHelpCommand(error.RepoExitError): class InvalidHelpCommand(error.RepoExitError):
"""Invalid command passed into help.""" """Invalid command passed into help."""
class Help(PagedCommand, MirrorSafeCommand): class Help(command.PagedCommand, command.MirrorSafeCommand):
COMMON = False COMMON = False
helpSummary = "Display detailed help on a command" helpSummary = "Display detailed help on a command"
helpUsage = """ helpUsage = """
@@ -38,17 +38,17 @@ class Help(PagedCommand, MirrorSafeCommand):
Displays detailed usage information about a command. Displays detailed usage information about a command.
""" """
def _PrintCommands(self, commandNames): def _PrintCommands(self, names: Sequence[str]) -> None:
"""Helper to display |commandNames| summaries.""" """Helper to display |names| summaries."""
maxlen = 0 maxlen = 0
for name in commandNames: for name in names:
maxlen = max(maxlen, len(name)) maxlen = max(maxlen, len(name))
fmt = " %%-%ds %%s" % maxlen fmt = " %%-%ds %%s" % maxlen
for name in commandNames: for name in names:
command = all_commands[name]() cmd = subcmds.all_commands[name]()
try: try:
summary = command.helpSummary.strip() summary = cmd.helpSummary.strip()
except AttributeError: except AttributeError:
summary = "" summary = ""
print(fmt % (name, summary)) print(fmt % (name, summary))
@@ -59,13 +59,13 @@ Displays detailed usage information about a command.
def PrintAllCommandsBody(self): def PrintAllCommandsBody(self):
print("The complete list of recognized repo commands is:") print("The complete list of recognized repo commands is:")
commandNames = sorted(all_commands) names = sorted(subcmds.all_commands)
self._PrintCommands(commandNames) self._PrintCommands(names)
print( print(
"See 'repo help <command>' for more information on a " "See 'repo help <command>' for more information on a "
"specific command." "specific command."
) )
print("Bug reports:", Wrapper().BUG_URL) print("Bug reports:", wrapper.Wrapper().BUG_URL)
def _PrintCommonCommands(self): def _PrintCommonCommands(self):
print("usage: repo COMMAND [ARGS]") print("usage: repo COMMAND [ARGS]")
@@ -74,22 +74,22 @@ Displays detailed usage information about a command.
def PrintCommonCommandsBody(self): def PrintCommonCommandsBody(self):
print("The most commonly used repo commands are:") print("The most commonly used repo commands are:")
commandNames = sorted( names = sorted(
name for name, command in all_commands.items() if command.COMMON name for name, cmd in subcmds.all_commands.items() if cmd.COMMON
) )
self._PrintCommands(commandNames) self._PrintCommands(names)
print( print(
"See 'repo help <command>' for more information on a specific " "See 'repo help <command>' for more information on a specific "
"command.\nSee 'repo help --all' for a complete list of recognized " "command.\nSee 'repo help --all' for a complete list of recognized "
"commands." "commands."
) )
print("Bug reports:", Wrapper().BUG_URL) print("Bug reports:", wrapper.Wrapper().BUG_URL)
def _PrintCommandHelp(self, cmd, header_prefix=""): def _PrintCommandHelp(self, cmd, header_prefix=""):
class _Out(Coloring): class _Out(color.Coloring):
def __init__(self, gc): def __init__(self, gc):
Coloring.__init__(self, gc, "help") super().__init__(gc, "help")
self.heading = self.printer("heading", attr="bold") self.heading = self.printer("heading", attr="bold")
self._first = True self._first = True
@@ -146,8 +146,8 @@ Displays detailed usage information about a command.
out._PrintSection("Description", "helpDescription") out._PrintSection("Description", "helpDescription")
def _PrintAllCommandHelp(self): def _PrintAllCommandHelp(self):
for name in sorted(all_commands): for name in sorted(subcmds.all_commands):
cmd = all_commands[name](manifest=self.manifest) cmd = subcmds.all_commands[name](manifest=self.manifest)
self._PrintCommandHelp(cmd, header_prefix=f"[{name}] ") self._PrintCommandHelp(cmd, header_prefix=f"[{name}] ")
def _Options(self, p): def _Options(self, p):
@@ -178,7 +178,7 @@ Displays detailed usage information about a command.
name = args[0] name = args[0]
try: try:
cmd = all_commands[name](manifest=self.manifest) cmd = subcmds.all_commands[name](manifest=self.manifest)
except KeyError: except KeyError:
print( print(
"repo: '%s' is not a repo command." % name, file=sys.stderr "repo: '%s' is not a repo command." % name, file=sys.stderr
+7 -8
View File
@@ -20,9 +20,8 @@ import optparse
import sys import sys
from typing import Any, Dict, List, NamedTuple, Optional, Tuple from typing import Any, Dict, List, NamedTuple, Optional, Tuple
from color import Coloring import color
from command import DEFAULT_LOCAL_JOBS import command
from command import PagedCommand
import git_refs import git_refs
@@ -46,14 +45,14 @@ class OutputFormat(enum.Enum):
JSON = enum.auto() JSON = enum.auto()
class _Coloring(Coloring): class _Coloring(color.Coloring):
def __init__(self, config): def __init__(self, config):
Coloring.__init__(self, config, "status") super().__init__(config, "status")
class Info(PagedCommand): class Info(command.PagedCommand):
COMMON = True COMMON = True
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
helpSummary = ( helpSummary = (
"Get info on the manifest branch, current branch or unmerged branches" "Get info on the manifest branch, current branch or unmerged branches"
) )
@@ -131,7 +130,7 @@ class Info(PagedCommand):
help=f"output format: {', '.join(formats)} (default: %default)", help=f"output format: {', '.join(formats)} (default: %default)",
) )
def WantPager(self, opt): def WantPager(self, opt) -> bool:
return OutputFormat[opt.format.upper()] == OutputFormat.TEXT return OutputFormat[opt.format.upper()] == OutputFormat.TEXT
def ValidateOptions(self, opt, args): def ValidateOptions(self, opt, args):
+21 -17
View File
@@ -12,17 +12,17 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import optparse
import os import os
import sys import sys
from typing import List
from color import Coloring import color
from command import InteractiveCommand import command
from command import MirrorSafeCommand
import error import error
import git_command import git_command
import repo_logging import repo_logging
from wrapper import Wrapper import wrapper
from wrapper import WrapperDir
logger = repo_logging.RepoLogger(__file__) logger = repo_logging.RepoLogger(__file__)
@@ -30,7 +30,7 @@ logger = repo_logging.RepoLogger(__file__)
_REPO_ALLOW_SHALLOW = os.environ.get("REPO_ALLOW_SHALLOW") _REPO_ALLOW_SHALLOW = os.environ.get("REPO_ALLOW_SHALLOW")
class Init(InteractiveCommand, MirrorSafeCommand): class Init(command.InteractiveCommand, command.MirrorSafeCommand):
COMMON = True COMMON = True
RESPECT_SMART_SYNC_OVERRIDE = False RESPECT_SMART_SYNC_OVERRIDE = False
MULTI_MANIFEST_SUPPORT = True MULTI_MANIFEST_SUPPORT = True
@@ -91,11 +91,11 @@ manifest, a subsequent `repo sync` (or `repo sync -d`) is necessary
to update the working directory files. to update the working directory files.
""" """
def _CommonOptions(self, p): def _CommonOptions(self, p, opt_v=True):
"""Disable due to re-use of Wrapper().""" """Disable due to re-use of Wrapper()."""
def _Options(self, p): def _Options(self, p):
Wrapper().InitParser(p) wrapper.Wrapper().InitParser(p)
m = p.add_option_group("Multi-manifest") m = p.add_option_group("Multi-manifest")
m.add_option( m.add_option(
"--outer-manifest", "--outer-manifest",
@@ -130,7 +130,7 @@ to update the working directory files.
"REPO_GIT_LFS": "git_lfs", "REPO_GIT_LFS": "git_lfs",
} }
def _SyncManifest(self, opt): def _SyncManifest(self, opt: optparse.Values) -> None:
"""Call manifestProject.Sync with arguments from opt. """Call manifestProject.Sync with arguments from opt.
Args: Args:
@@ -164,6 +164,7 @@ to update the working directory files.
git_lfs=opt.git_lfs, git_lfs=opt.git_lfs,
use_superproject=opt.use_superproject, use_superproject=opt.use_superproject,
smart_sync=opt.smart_sync, smart_sync=opt.smart_sync,
smart_sync_branch=opt.smart_sync_branch,
verbose=opt.verbose, verbose=opt.verbose,
current_branch_only=opt.current_branch_only, current_branch_only=opt.current_branch_only,
tags=opt.tags, tags=opt.tags,
@@ -244,9 +245,9 @@ to update the working directory files.
if self._HasColorSet(gc): if self._HasColorSet(gc):
return return
class _Test(Coloring): class _Test(color.Coloring):
def __init__(self): def __init__(self):
Coloring.__init__(self, gc, "test color display") super().__init__(gc, "test color display")
self._on = True self._on = True
out = _Test() out = _Test()
@@ -296,7 +297,7 @@ to update the working directory files.
print(" rm -r %s" % os.path.join(self.manifest.topdir, ".repo")) print(" rm -r %s" % os.path.join(self.manifest.topdir, ".repo"))
print("and try again.") print("and try again.")
def ValidateOptions(self, opt, args): def ValidateOptions(self, opt: optparse.Values, args: List[str]) -> None:
if opt.reference: if opt.reference:
opt.reference = os.path.expanduser(opt.reference) opt.reference = os.path.expanduser(opt.reference)
@@ -325,6 +326,11 @@ to update the working directory files.
" be used with --standalone-manifest." " be used with --standalone-manifest."
) )
if opt.standalone_manifest and opt.smart_sync_branch is not None:
self.OptionParser.error(
"--smart-sync-branch cannot be used with --standalone-manifest."
)
if opt.manifest_upstream_branch and opt.manifest_branch is None: if opt.manifest_upstream_branch and opt.manifest_branch is None:
self.OptionParser.error( self.OptionParser.error(
"--manifest-upstream-branch cannot be used without " "--manifest-upstream-branch cannot be used without "
@@ -344,9 +350,7 @@ to update the working directory files.
self.OptionParser.error("too many arguments to init") self.OptionParser.error("too many arguments to init")
def Execute(self, opt, args): def Execute(self, opt, args):
wrapper = Wrapper() reqs = wrapper.Wrapper().Requirements.from_dir(wrapper.WrapperDir())
reqs = wrapper.Requirements.from_dir(WrapperDir())
git_command.git_require(reqs.get_hard_ver("git"), fail=True) git_command.git_require(reqs.get_hard_ver("git"), fail=True)
min_git_version_soft = reqs.get_soft_ver("git") min_git_version_soft = reqs.get_soft_ver("git")
if not git_command.git_require(min_git_version_soft): if not git_command.git_require(min_git_version_soft):
@@ -368,13 +372,13 @@ to update the working directory files.
# Handle new --repo-rev requests. # Handle new --repo-rev requests.
if opt.repo_rev: if opt.repo_rev:
try: try:
remote_ref, rev = wrapper.check_repo_rev( remote_ref, rev = wrapper.Wrapper().check_repo_rev(
rp.worktree, rp.worktree,
opt.repo_rev, opt.repo_rev,
repo_verify=opt.repo_verify, repo_verify=opt.repo_verify,
quiet=opt.quiet, quiet=opt.quiet,
) )
except wrapper.CloneFailure as e: except wrapper.Wrapper().CloneFailure as e:
err_msg = "fatal: double check your --repo-rev setting." err_msg = "fatal: double check your --repo-rev setting."
logger.error(err_msg) logger.error(err_msg)
self.git_event_log.ErrorEvent(err_msg) self.git_event_log.ErrorEvent(err_msg)
+2 -3
View File
@@ -14,11 +14,10 @@
import os import os
from command import Command import command
from command import MirrorSafeCommand
class List(Command, MirrorSafeCommand): class List(command.Command, command.MirrorSafeCommand):
COMMON = True COMMON = True
helpSummary = "List projects and their associated directories" helpSummary = "List projects and their associated directories"
helpUsage = """ helpUsage = """
+2 -2
View File
@@ -18,7 +18,7 @@ import optparse
import os import os
import sys import sys
from command import PagedCommand import command
import repo_logging import repo_logging
@@ -35,7 +35,7 @@ class OutputFormat(enum.Enum):
JSON = enum.auto() JSON = enum.auto()
class Manifest(PagedCommand): class Manifest(command.PagedCommand):
COMMON = False COMMON = False
helpSummary = "Manifest inspection utility" helpSummary = "Manifest inspection utility"
helpUsage = """ helpUsage = """
+5 -5
View File
@@ -14,11 +14,11 @@
import optparse import optparse
from color import Coloring import color
from command import PagedCommand import command
class Overview(PagedCommand): class Overview(command.PagedCommand):
COMMON = True COMMON = True
helpSummary = "Display overview of unmerged project branches" helpSummary = "Display overview of unmerged project branches"
helpUsage = """ helpUsage = """
@@ -73,9 +73,9 @@ are displayed.
if not all_branches: if not all_branches:
return return
class Report(Coloring): class Report(color.Coloring):
def __init__(self, config): def __init__(self, config):
Coloring.__init__(self, config, "status") super().__init__(config, "status")
self.project = self.printer("header", attr="bold") self.project = self.printer("header", attr="bold")
self.text = self.printer("text") self.text = self.printer("text")
+6 -7
View File
@@ -14,18 +14,17 @@
import itertools import itertools
from color import Coloring import color
from command import DEFAULT_LOCAL_JOBS import command
from command import PagedCommand
class Prune(PagedCommand): class Prune(command.PagedCommand):
COMMON = True COMMON = True
helpSummary = "Prune (delete) already merged topics" helpSummary = "Prune (delete) already merged topics"
helpUsage = """ helpUsage = """
%prog [<project>...] %prog [<project>...]
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
@classmethod @classmethod
def _ExecuteOne(cls, project_idx): def _ExecuteOne(cls, project_idx):
@@ -56,9 +55,9 @@ class Prune(PagedCommand):
if not all_branches: if not all_branches:
return return
class Report(Coloring): class Report(color.Coloring):
def __init__(self, config): def __init__(self, config):
Coloring.__init__(self, config, "status") super().__init__(config, "status")
self.project = self.printer("header", attr="bold") self.project = self.printer("header", attr="bold")
out = Report(all_branches[0].project.config) out = Report(all_branches[0].project.config)
+5 -5
View File
@@ -14,8 +14,8 @@
import sys import sys
from color import Coloring import color
from command import Command import command
import error import error
import git_command import git_command
from project import Project from project import Project
@@ -25,9 +25,9 @@ import repo_logging
logger = repo_logging.RepoLogger(__file__) logger = repo_logging.RepoLogger(__file__)
class RebaseColoring(Coloring): class RebaseColoring(color.Coloring):
def __init__(self, config): def __init__(self, config):
Coloring.__init__(self, config, "rebase") super().__init__(config, "rebase")
self.project = self.printer("project", attr="bold") self.project = self.printer("project", attr="bold")
self.fail = self.printer("fail", fg="red") self.fail = self.printer("fail", fg="red")
@@ -43,7 +43,7 @@ def _ResolveOntoManifest(project: Project) -> str:
return project.revisionExpr return project.revisionExpr
class Rebase(Command): class Rebase(command.Command):
COMMON = True COMMON = True
helpSummary = "Rebase local branches on upstream branch" helpSummary = "Rebase local branches on upstream branch"
helpUsage = """ helpUsage = """
+5 -7
View File
@@ -14,12 +14,10 @@
import optparse import optparse
from command import Command import command
from command import MirrorSafeCommand
import error import error
import repo_logging import repo_logging
from subcmds.sync import _PostRepoFetch from subcmds import sync
from subcmds.sync import _PostRepoUpgrade
logger = repo_logging.RepoLogger(__file__) logger = repo_logging.RepoLogger(__file__)
@@ -29,7 +27,7 @@ class SelfupdateError(error.RepoExitError):
"""Exit error for failed selfupdate command.""" """Exit error for failed selfupdate command."""
class Selfupdate(Command, MirrorSafeCommand): class Selfupdate(command.Command, command.MirrorSafeCommand):
COMMON = False COMMON = False
helpSummary = "Update repo to the latest version" helpSummary = "Update repo to the latest version"
helpUsage = """ helpUsage = """
@@ -63,7 +61,7 @@ need to be performed by an end-user.
rp.PreSync() rp.PreSync()
if opt.repo_upgraded: if opt.repo_upgraded:
_PostRepoUpgrade(self.manifest) sync._PostRepoUpgrade(self.manifest)
else: else:
result = rp.Sync_NetworkHalf() result = rp.Sync_NetworkHalf()
@@ -72,4 +70,4 @@ need to be performed by an end-user.
raise SelfupdateError(aggregate_errors=[result.error]) raise SelfupdateError(aggregate_errors=[result.error])
rp.bare_git.gc("--auto") rp.bare_git.gc("--auto")
_PostRepoFetch(rp, repo_verify=opt.repo_verify, verbose=True) sync._PostRepoFetch(rp, repo_verify=opt.repo_verify, verbose=True)
+4 -4
View File
@@ -12,10 +12,10 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from subcmds.sync import Sync from subcmds import sync
class Smartsync(Sync): class Smartsync(sync.Sync):
COMMON = True COMMON = True
helpSummary = "Update working tree to the latest known good revision" helpSummary = "Update working tree to the latest known good revision"
helpUsage = """ helpUsage = """
@@ -26,8 +26,8 @@ The '%prog' command is a shortcut for sync -s.
""" """
def _Options(self, p): def _Options(self, p):
Sync._Options(self, p, show_smart=False) sync.Sync._Options(self, p, show_smart=False)
def Execute(self, opt, args): def Execute(self, opt, args):
opt.smart_sync = True opt.smart_sync = True
Sync.Execute(self, opt, args) sync.Sync.Execute(self, opt, args)
+5 -5
View File
@@ -14,8 +14,8 @@
import sys import sys
from color import Coloring import color
from command import InteractiveCommand import command
import git_command import git_command
import repo_logging import repo_logging
@@ -23,15 +23,15 @@ import repo_logging
logger = repo_logging.RepoLogger(__file__) logger = repo_logging.RepoLogger(__file__)
class _ProjectList(Coloring): class _ProjectList(color.Coloring):
def __init__(self, gc): def __init__(self, gc):
Coloring.__init__(self, gc, "interactive") super().__init__(gc, "interactive")
self.prompt = self.printer("prompt", fg="blue", attr="bold") self.prompt = self.printer("prompt", fg="blue", attr="bold")
self.header = self.printer("header", attr="bold") self.header = self.printer("header", attr="bold")
self.help = self.printer("help", fg="red", attr="bold") self.help = self.printer("help", fg="red", attr="bold")
class Stage(InteractiveCommand): class Stage(command.InteractiveCommand):
COMMON = True COMMON = True
helpSummary = "Stage file(s) for commit" helpSummary = "Stage file(s) for commit"
helpUsage = """ helpUsage = """
+9 -10
View File
@@ -13,14 +13,13 @@
# limitations under the License. # limitations under the License.
import functools import functools
from typing import NamedTuple from typing import NamedTuple, Optional
from command import Command import command
from command import DEFAULT_LOCAL_JOBS
import error import error
import git_command import git_command
from git_config import IsImmutable import git_config
from progress import Progress import progress
import repo_logging import repo_logging
@@ -29,14 +28,14 @@ logger = repo_logging.RepoLogger(__file__)
class ExecuteOneResult(NamedTuple): class ExecuteOneResult(NamedTuple):
project_idx: int project_idx: int
error: Exception error: Optional[Exception]
class StartError(error.RepoExitError): class StartError(error.RepoExitError):
"""Exit error for failed start command.""" """Exit error for failed start command."""
class Start(Command): class Start(command.Command):
COMMON = True COMMON = True
helpSummary = "Start a new branch for development" helpSummary = "Start a new branch for development"
helpUsage = """ helpUsage = """
@@ -46,7 +45,7 @@ class Start(Command):
'%prog' begins a new branch of development, starting from the '%prog' begins a new branch of development, starting from the
revision specified in the manifest. revision specified in the manifest.
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
def _Options(self, p): def _Options(self, p):
p.add_option( p.add_option(
@@ -87,7 +86,7 @@ revision specified in the manifest.
branch_merge = "" branch_merge = ""
err = None err = None
project = cls.get_parallel_context()["projects"][project_idx] project = cls.get_parallel_context()["projects"][project_idx]
if IsImmutable(project.revisionExpr): if git_config.IsImmutable(project.revisionExpr):
if project.dest_branch: if project.dest_branch:
branch_merge = project.dest_branch branch_merge = project.dest_branch
else: else:
@@ -137,7 +136,7 @@ revision specified in the manifest.
), ),
range(len(all_projects)), range(len(all_projects)),
callback=_ProcessResults, callback=_ProcessResults,
output=Progress( output=progress.Progress(
f"Starting {nb}", len(all_projects), quiet=opt.quiet f"Starting {nb}", len(all_projects), quiet=opt.quiet
), ),
chunksize=1, chunksize=1,
+6 -7
View File
@@ -17,13 +17,12 @@ import glob
import io import io
import os import os
from color import Coloring import color
from command import DEFAULT_LOCAL_JOBS import command
from command import PagedCommand
import platform_utils import platform_utils
class Status(PagedCommand): class Status(command.PagedCommand):
COMMON = True COMMON = True
helpSummary = "Show the working tree status" helpSummary = "Show the working tree status"
helpUsage = """ helpUsage = """
@@ -82,7 +81,7 @@ the following meanings:
d: deleted ( in index, not in work tree ) d: deleted ( in index, not in work tree )
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
def _Options(self, p): def _Options(self, p):
p.add_option( p.add_option(
@@ -180,9 +179,9 @@ the following meanings:
(head, _tail) = os.path.split(head) (head, _tail) = os.path.split(head)
proj_dirs.add(".repo") proj_dirs.add(".repo")
class StatusColoring(Coloring): class StatusColoring(color.Coloring):
def __init__(self, config): def __init__(self, config):
Coloring.__init__(self, config, "status") super().__init__(config, "status")
self.project = self.printer("header", attr="bold") self.project = self.printer("header", attr="bold")
self.untracked = self.printer("untracked", fg="red") self.untracked = self.printer("untracked", fg="red")
+461 -186
View File
File diff suppressed because it is too large. Load diff
+22 -13
View File
@@ -19,16 +19,15 @@ import re
import sys import sys
from typing import List from typing import List
from command import DEFAULT_LOCAL_JOBS import command
from command import InteractiveCommand import editor
from editor import Editor
import error import error
import git_refs import git_refs
import git_superproject import git_superproject
from hooks import RepoHook import hooks
from project import ReviewableBranch from project import ReviewableBranch
import repo_logging import repo_logging
from subcmds.sync import LocalSyncState from subcmds import sync
_DEFAULT_UNUSUAL_COMMIT_THRESHOLD = 5 _DEFAULT_UNUSUAL_COMMIT_THRESHOLD = 5
@@ -104,7 +103,7 @@ def _SplitEmails(values):
return result return result
class Upload(InteractiveCommand): class Upload(command.InteractiveCommand):
COMMON = True COMMON = True
helpSummary = "Upload changes for code review" helpSummary = "Upload changes for code review"
helpUsage = """ helpUsage = """
@@ -211,7 +210,7 @@ threshold to a different value.
Gerrit Code Review: https://www.gerritcodereview.com/ Gerrit Code Review: https://www.gerritcodereview.com/
""" """
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS PARALLEL_JOBS = command.DEFAULT_LOCAL_JOBS
def _Options(self, p): def _Options(self, p):
p.add_option( p.add_option(
@@ -369,6 +368,12 @@ Gerrit Code Review: https://www.gerritcodereview.com/
action="store_false", action="store_false",
help="always ask about untracked files in the working copy", help="always ask about untracked files in the working copy",
) )
p.add_option(
"--ignore-remote-commits",
action="store_true",
default=False,
help="ignore commits already in remote branches",
)
p.add_option( p.add_option(
"--no-cert-checks", "--no-cert-checks",
dest="validate_certs", dest="validate_certs",
@@ -376,7 +381,7 @@ Gerrit Code Review: https://www.gerritcodereview.com/
default=True, default=True,
help="disable verifying ssl certs (unsafe)", help="disable verifying ssl certs (unsafe)",
) )
RepoHook.AddOptionGroup(p, "pre-upload", allow_fix=True) hooks.RepoHook.AddOptionGroup(p, "pre-upload", allow_fix=True)
def _SingleBranch(self, opt, branch, people): def _SingleBranch(self, opt, branch, people):
project = branch.project project = branch.project
@@ -488,7 +493,7 @@ Gerrit Code Review: https://www.gerritcodereview.com/
branches[project_path] = b branches[project_path] = b
script.append("") script.append("")
script = Editor.EditString("\n".join(script)).split("\n") script = editor.Editor.EditString("\n".join(script)).split("\n")
project_re = re.compile(r"^#?\s*project\s*([^\s]+)/:$") project_re = re.compile(r"^#?\s*project\s*([^\s]+)/:$")
branch_re = re.compile(r"^\s*branch\s*([^\s(]+)\s*\(.*") branch_re = re.compile(r"^\s*branch\s*([^\s(]+)\s*\(.*")
@@ -738,10 +743,14 @@ Gerrit Code Review: https://www.gerritcodereview.com/
cbr = None cbr = None
if opt.current_branch: if opt.current_branch:
cbr = project.CurrentBranch cbr = project.CurrentBranch
up_branch = project.GetUploadableBranch(cbr) up_branch = project.GetUploadableBranch(
cbr, ignore_remote_commits=opt.ignore_remote_commits
)
avail = [up_branch] if up_branch else None avail = [up_branch] if up_branch else None
else: else:
avail = project.GetUploadableBranches(opt.branch) avail = project.GetUploadableBranches(
opt.branch, ignore_remote_commits=opt.ignore_remote_commits
)
return (project_idx, avail, cbr) return (project_idx, avail, cbr)
def ValidateOptions(self, opt: optparse.Values, args: List[str]) -> None: def ValidateOptions(self, opt: optparse.Values, args: List[str]) -> None:
@@ -818,7 +827,7 @@ Gerrit Code Review: https://www.gerritcodereview.com/
for (project, available) in pending for (project, available) in pending
if project.manifest.topdir == manifest.topdir if project.manifest.topdir == manifest.topdir
] ]
hook = RepoHook.FromSubcmd( hook = hooks.RepoHook.FromSubcmd(
hook_type="pre-upload", hook_type="pre-upload",
manifest=manifest, manifest=manifest,
opt=opt, opt=opt,
@@ -827,7 +836,7 @@ Gerrit Code Review: https://www.gerritcodereview.com/
if not hook.Run( if not hook.Run(
project_list=pending_proj_names, worktree_list=pending_worktrees project_list=pending_proj_names, worktree_list=pending_worktrees
): ):
if LocalSyncState(manifest).IsPartiallySynced(): if sync.LocalSyncState(manifest).IsPartiallySynced():
logger.info( logger.info(
"Tip: A partially synced tree was detected. " "Tip: A partially synced tree was detected. "
"If this failure involves cross-project dependencies, " "If this failure involves cross-project dependencies, "
+4 -5
View File
@@ -16,14 +16,13 @@ import platform
import sys import sys
from typing import Any, Tuple from typing import Any, Tuple
from command import Command import command
from command import MirrorSafeCommand
import git_command import git_command
import git_refs import git_refs
from wrapper import Wrapper import wrapper
class Version(Command, MirrorSafeCommand): class Version(command.Command, command.MirrorSafeCommand):
wrapper_version = None wrapper_version = None
wrapper_path = None wrapper_path = None
@@ -81,4 +80,4 @@ class Version(Command, MirrorSafeCommand):
print(f"OS {uname.system} {uname.release} ({uname.version})") print(f"OS {uname.system} {uname.release} ({uname.version})")
processor = uname.processor if uname.processor else "unknown" processor = uname.processor if uname.processor else "unknown"
print(f"CPU {uname.machine} ({processor})") print(f"CPU {uname.machine} ({processor})")
print("Bug reports:", Wrapper().BUG_URL) print("Bug reports:", wrapper.Wrapper().BUG_URL)
+3 -3
View File
@@ -16,7 +16,7 @@ import os
import sys import sys
from typing import List from typing import List
from command import Command import command
import error import error
import platform_utils import platform_utils
from project import DeleteWorktreeError from project import DeleteWorktreeError
@@ -26,7 +26,7 @@ class Error(error.RepoExitError):
"""Exit error when wipe command fails.""" """Exit error when wipe command fails."""
class Wipe(Command): class Wipe(command.Command):
"""Delete projects from the worktree and .repo""" """Delete projects from the worktree and .repo"""
COMMON = True COMMON = True
@@ -98,7 +98,7 @@ Examples:
) )
try: try:
if project.HasChanges(): if project.HasChanges:
uncommitted_projects.append(project.name) uncommitted_projects.append(project.name)
except error.GitError: except error.GitError:
uncommitted_projects.append(f"{project.name} (corrupted)") uncommitted_projects.append(f"{project.name} (corrupted)")
+6
View File
@@ -166,6 +166,12 @@ def clean_environ() -> Dict[str, str]:
"pythonwarnings", "pythonwarnings",
# LUCI's git wrapper replaces HOME with this. # LUCI's git wrapper replaces HOME with this.
"infra_git_wrapper_home", "infra_git_wrapper_home",
# Smart sync reads these (e.g. set by Android's envsetup/lunch).
"sync_branch",
"sync_target",
"target_product",
"target_release",
"target_build_variant",
): ):
continue continue
env[key] = value env[key] = value
+2
View File
@@ -28,6 +28,7 @@ def _http_request(
) -> Tuple[int, bytes]: ) -> Tuple[int, bytes]:
"""Send a request to `server` & return the response status & body.""" """Send a request to `server` & return the response status & body."""
url = urllib.parse.urlsplit(server.url) url = urllib.parse.urlsplit(server.url)
assert url.hostname is not None
conn = http.client.HTTPConnection(url.hostname, url.port, timeout=30) conn = http.client.HTTPConnection(url.hostname, url.port, timeout=30)
try: try:
conn.request(method, path) conn.request(method, path)
@@ -113,6 +114,7 @@ def test_http_post_body(
if expect_continue: if expect_continue:
headers["Expect"] = "100-continue" headers["Expect"] = "100-continue"
url = urllib.parse.urlsplit(git_server.url) url = urllib.parse.urlsplit(git_server.url)
assert url.hostname is not None
conn = http.client.HTTPConnection(url.hostname, url.port, timeout=30) conn = http.client.HTTPConnection(url.hostname, url.port, timeout=30)
try: try:
# http.client skips any interim `100 Continue` response. # http.client skips any interim `100 Continue` response.
+118 -1
View File
@@ -14,8 +14,11 @@
"""E2E tests for the core `repo init` & `repo sync` options.""" """E2E tests for the core `repo init` & `repo sync` options."""
import contextlib
from pathlib import Path from pathlib import Path
from typing import List, Set import threading
from typing import Any, Callable, Dict, Iterator, List, Set
import xmlrpc.server
from harness import GIT_DEFAULTS_PARAMS from harness import GIT_DEFAULTS_PARAMS
from harness import GitHost from harness import GitHost
@@ -353,3 +356,117 @@ def test_manifest_depth(host: GitHost, client: RepoClient) -> None:
assert client.is_shallow(manifests) assert client.is_shallow(manifests)
assert not client.is_shallow("platform/foo") assert not client.is_shallow("platform/foo")
assert client.git("platform/foo", "rev-list", "--count", "HEAD") == "2" assert client.git("platform/foo", "rev-list", "--count", "HEAD") == "2"
@contextlib.contextmanager
def _manifest_server(
handler: Callable[[Any], Dict[str, Any]],
) -> Iterator[str]:
"""Run a local XML-RPC manifest server exposing GetApprovedManifestV2."""
server = xmlrpc.server.SimpleXMLRPCServer(
("127.0.0.1", 0), logRequests=False
)
server.register_function(handler, "GetApprovedManifestV2")
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
addr, port = server.server_address[:2]
yield f"http://{addr}:{port}/"
finally:
server.shutdown()
server.server_close()
thread.join()
@pytest.mark.parametrize("git_defaults", GIT_DEFAULTS_PARAMS)
def test_smart_sync_branch(host: GitHost, client: RepoClient) -> None:
"""`init --smart-sync-branch` persists per-manifest-branch server branch."""
foo = host.create_repo("platform/foo", {"f": "v1"})
v1 = foo.rev_parse("main")
warmup = foo.commit({"f": "warmup"})
foo.commit({"f": "tot"})
r16 = foo.commit({"f": "r16"}, branch="release/16")
requests: List[Dict[str, Any]] = []
by_branch = {
"main": v1,
"git_main-dev-warmup-abfs": warmup,
"release/16": r16,
}
def handle_v2(req: Dict[str, Any]) -> Dict[str, Any]:
requests.append(req)
rev = by_branch[req["branch"]]
return {
"success": True,
"manifest_xml": manifest_xml(
f'<project name="platform/foo" path="foo" revision="{rev}" />',
fetch=host.url,
),
}
with _manifest_server(handle_v2) as server_url:
m_xml = manifest_xml(
f'<manifest-server url="{server_url}" />',
'<project name="platform/foo" path="foo" />',
)
manifest = host.create_manifest(m_xml)
manifest.commit({"default.xml": m_xml}, branch="release/16")
client.init(
host.manifest_url,
"-s",
"--smart-sync-branch=git_main-dev-warmup-abfs",
)
assert (
client.config(".repo/manifests", "repo.main.smartsyncbranch")
== "git_main-dev-warmup-abfs"
)
client.sync()
assert requests[-1]["branch"] == "git_main-dev-warmup-abfs"
assert client.head("foo") == warmup
# SYNC_BRANCH="" forces the raw manifest branch over checkout config.
saved_env = dict(client.env)
client.env["SYNC_BRANCH"] = ""
try:
client.sync()
finally:
client.env = saved_env
assert requests[-1]["branch"] == "main"
assert client.head("foo") == v1
# Switching manifest branches does not leak main's override.
client.init(host.manifest_url, "-b", "release/16")
client.sync()
assert requests[-1]["branch"] == "release/16"
assert client.head("foo") == r16
# Switching back to main restores main's configured override.
client.init(host.manifest_url, "-b", "main")
client.sync()
assert requests[-1]["branch"] == "git_main-dev-warmup-abfs"
assert client.head("foo") == warmup
# Passing an empty --smart-sync-branch clears the override.
client.init(host.manifest_url, "--smart-sync-branch=")
assert (
client.config(".repo/manifests", "repo.main.smartsyncbranch")
is None
)
client.sync()
assert requests[-1]["branch"] == "main"
assert client.head("foo") == v1
# Non-empty --smart-sync-branch requires tracking a remote branch.
manifest_sha = manifest.rev_parse("main")
res = client.init(
host.manifest_url,
"-b",
manifest_sha,
"--smart-sync-branch=git_main-dev-warmup-abfs",
check=False,
)
assert res.returncode != 0
assert "requires the manifest checkout to track a branch" in res.stderr
+132
View File
@@ -245,6 +245,93 @@ def test_nested_projects(host: GitHost, client: RepoClient) -> None:
assert (client.top / "foo/f").read_text() == "foo" assert (client.top / "foo/f").read_text() == "foo"
@pytest.mark.parametrize(
"sync_args", [[], ["--no-interleaved"]], ids=["interleaved", "phased"]
)
def test_project_replaced_by_nested_project(
host: GitHost, client: RepoClient, sync_args: List[str]
) -> None:
"""Replacing a parent project with a nested child project removes parent.
Regression test for b/574048485: in interleaved mode, checking out the new
child project `foo/bar` before deleting the obsolete parent worktree `foo`
made `foo` appear dirty (`git status` in `foo` saw `foo/bar`'s files as
modifications and untracked files under `bar/`).
"""
host.create_repo(
"platform/foo",
{
"root.txt": "parent",
"bar/shared/version.txt": "v1",
"bar/old-version/file.txt": "old",
},
)
host.create_repo(
"platform/foo_bar",
{
"shared/version.txt": "v2",
"new-version/file.txt": "new",
},
)
host.create_manifest(
manifest_xml('<project name="platform/foo" path="foo" />')
)
client.init(host.manifest_url)
client.sync(*sync_args)
assert (client.top / "foo/root.txt").read_text() == "parent"
assert (client.top / "foo/bar/shared/version.txt").read_text() == "v1"
host.update_manifest(
manifest_xml('<project name="platform/foo_bar" path="foo/bar" />')
)
client.sync(*sync_args)
assert client.list_projects() == {"foo/bar": "platform/foo_bar"}
assert (client.repodir / "project.list").read_text() == "foo/bar\n"
assert not (client.top / "foo/.git").exists()
assert not (client.top / "foo/root.txt").exists()
assert not (client.top / "foo/bar/old-version").exists()
assert (client.top / "foo/bar/shared/version.txt").read_text() == "v2"
assert (client.top / "foo/bar/new-version/file.txt").read_text() == "new"
assert client.git("foo/bar", "status", "--porcelain") == ""
@pytest.mark.parametrize(
"sync_args", [[], ["--no-interleaved"]], ids=["interleaved", "phased"]
)
def test_nested_project_merged_into_parent(
host: GitHost, client: RepoClient, sync_args: List[str]
) -> None:
"""A nested project whose files move to its parent is removed first.
Otherwise the parent's new files clash with the removed project's
checkout. The reverse of test_project_replaced_by_nested_project.
"""
foo = host.create_repo("platform/foo", {"f": "foo"})
host.create_repo("platform/bar", {"shared/version.txt": "v1"})
host.create_manifest(
manifest_xml(
'<project name="platform/foo" path="foo" />',
'<project name="platform/bar" path="foo/bar" />',
)
)
client.init(host.manifest_url)
client.sync(*sync_args)
latest = foo.commit({"bar/shared/version.txt": "v2"})
host.update_manifest(
manifest_xml('<project name="platform/foo" path="foo" />')
)
client.sync(*sync_args)
assert client.list_projects() == {"foo": "platform/foo"}
assert client.head("foo") == latest
assert not os.path.lexists(client.top / "foo/bar/.git")
assert (client.top / "foo/bar/shared/version.txt").read_text() == "v2"
assert client.git("foo", "status", "--porcelain") == ""
def test_copyfile_linkfile(host: GitHost, client: RepoClient) -> None: def test_copyfile_linkfile(host: GitHost, client: RepoClient) -> None:
"""copyfile and linkfile are created, updated and cleaned up. """copyfile and linkfile are created, updated and cleaned up.
@@ -330,6 +417,51 @@ def test_submodules(
assert (client.top / path / "sub/s").read_text() == "2" assert (client.top / path / "sub/s").read_text() == "2"
@pytest.mark.parametrize(
"sync_args", [[], ["--no-interleaved"]], ids=["interleaved", "phased"]
)
def test_pinned_parent_keeps_unfetched_submodule(
host: GitHost, client: RepoClient, sync_args: List[str]
) -> None:
"""A sync-s parent moving to an unfetched SHA keeps its submodule."""
sub = host.create_repo("platform/sub", {"s": "1"})
host.create_repo("platform/bar", {"b": "old"})
gitmodules = '[submodule "sub"]\n\tpath = sub\n\turl = ../sub\n'
foo = host.create_repo(
"platform/foo",
{".gitmodules": gitmodules, "sub": Gitlink(sub.rev_parse("main"))},
)
def manifest_for(rev: str, include_bar: bool = False) -> str:
extra = (
'\n <project name="platform/bar" path="foo/bar" />'
if include_bar
else ""
)
return manifest_xml(
f'<project name="platform/foo" path="foo" sync-s="true" '
f'revision="{rev}" />{extra}'
)
host.create_manifest(manifest_for(foo.rev_parse("main"), include_bar=True))
client.init(host.manifest_url)
client.sync(*sync_args)
(client.top / "foo/sub/s").write_text("dirty")
latest = foo.commit({"bar/b": "from-foo"})
host.update_manifest(manifest_for(latest))
client.sync("-v", *sync_args)
assert client.head("foo") == latest
assert (client.top / "foo/sub/s").read_text() == "dirty"
assert (client.top / "foo/bar/b").read_text() == "from-foo"
assert not (client.top / "foo/bar/.git").exists()
expected_sub = os.path.join("foo", "sub")
assert (
client.repodir / "project.list"
).read_text() == f"foo\n{expected_sub}\n"
# The default layout reads HEAD from the ref cache instead of GetHead(). # The default layout reads HEAD from the ref cache instead of GetHead().
@pytest.mark.parametrize("init_args", [[], ["--worktree"]]) @pytest.mark.parametrize("init_args", [[], ["--worktree"]])
def test_prune_detached_head( def test_prune_detached_head(
+111 -6
View File
@@ -275,7 +275,49 @@ def test_force_sync(host: GitHost, client: RepoClient) -> None:
assert client.git("foo", "status", "--porcelain") == "" assert client.git("foo", "status", "--porcelain") == ""
def test_force_remove_dirty(host: GitHost, client: RepoClient) -> None: @pytest.mark.parametrize("mode_args", SYNC_MODES, ids=["interleaved", "phased"])
def test_network_only_keeps_removed_project(
host: GitHost, client: RepoClient, mode_args: List[str]
) -> None:
"""`sync -n` leaves projects dropped from the manifest to `sync -l`.
Removing their checkouts and copyfiles changes the worktree, which `-n`
must not do.
"""
host.create_repo("platform/foo", {"f": "foo"})
host.create_repo("platform/bar", {"f": "bar"})
host.create_manifest(
manifest_xml(
'<project name="platform/foo" path="foo" />',
'<project name="platform/bar" path="bar">'
'<copyfile src="f" dest="bar-copy" />'
"</project>",
)
)
client.init(host.manifest_url)
client.sync(*mode_args)
host.update_manifest(
manifest_xml('<project name="platform/foo" path="foo" />')
)
assert client.sync("-n", *mode_args).stderr == ""
assert (client.top / "bar/f").read_text() == "bar"
assert (client.top / "bar-copy").read_text() == "bar"
with host.watch_requests() as requests, host.offline():
client.sync("-l", *mode_args)
assert requests == []
assert not (client.top / "bar").exists()
assert not (client.top / "bar-copy").exists()
assert client.list_projects() == {"foo": "platform/foo"}
@pytest.mark.parametrize("mode_args", SYNC_MODES, ids=["interleaved", "phased"])
def test_force_remove_dirty(
host: GitHost, client: RepoClient, mode_args: List[str]
) -> None:
"""Removed projects with uncommitted changes need `--force-remove-dirty`.""" """Removed projects with uncommitted changes need `--force-remove-dirty`."""
host.create_repo("platform/foo", {"f": "foo"}) host.create_repo("platform/foo", {"f": "foo"})
host.create_repo("platform/bar", {"f": "bar"}) host.create_repo("platform/bar", {"f": "bar"})
@@ -286,7 +328,7 @@ def test_force_remove_dirty(host: GitHost, client: RepoClient) -> None:
) )
) )
client.init(host.manifest_url) client.init(host.manifest_url)
client.sync() client.sync(*mode_args)
(client.top / "bar/f").write_text("dirty") (client.top / "bar/f").write_text("dirty")
host.update_manifest( host.update_manifest(
manifest_xml('<project name="platform/foo" path="foo" />') manifest_xml('<project name="platform/foo" path="foo" />')
@@ -295,15 +337,21 @@ def test_force_remove_dirty(host: GitHost, client: RepoClient) -> None:
# Repeat to check that the project stays tracked in project.list (rather # Repeat to check that the project stays tracked in project.list (rather
# than forgotten & left behind) after the first failure. # than forgotten & left behind) after the first failure.
for _ in range(2): for _ in range(2):
result = client.sync(check=False) result = client.sync(*mode_args, check=False)
assert result.returncode != 0 assert result.returncode != 0
# Logged once when removal fails and once in the final error summary:
# removal is not attempted (and reported) a second time in the same
# sync.
assert ( assert (
"bar: Cannot remove project: uncommitted changes are present" result.stderr.count(
in result.stderr "bar: Cannot remove project: uncommitted changes are present"
)
== 2
) )
assert result.stderr.count("Updating local project lists failed") == 1
assert (client.top / "bar/f").read_text() == "dirty" assert (client.top / "bar/f").read_text() == "dirty"
result = client.sync("--force-remove-dirty") result = client.sync("--force-remove-dirty", *mode_args)
assert "bar: Removing dirty project" in result.stderr assert "bar: Removing dirty project" in result.stderr
assert not (client.top / "bar").exists() assert not (client.top / "bar").exists()
assert client.list_projects() == {"foo": "platform/foo"} assert client.list_projects() == {"foo": "platform/foo"}
@@ -337,3 +385,60 @@ def test_detach(host: GitHost, client: RepoClient) -> None:
# A regular sync afterwards stays detached. # A regular sync afterwards stays detached.
client.sync() client.sync()
assert client.is_detached("foo") assert client.is_detached("foo")
@pytest.mark.parametrize("mode", ["detached", "commit", "tag"])
def test_smart_sync_without_tracked_branch(
host: GitHost, client: RepoClient, mode: str
) -> None:
"""Smart sync requires `.repo/manifests` to track a remote branch.
When `sync-smartsync="true"` is set in the manifest, plain `repo sync`
falls back to a regular sync without contacting the manifest server, while
an explicit `repo sync -s` fails early with an actionable error
(b/580004259).
"""
foo = host.create_repo("platform/foo", {"f": "foo"})
manifest = host.create_manifest(
manifest_xml(
f'<manifest-server url="{host.url}/manifestserver" />',
'<project name="platform/foo" path="foo" />',
default='remote="origin" revision="main" sync-smartsync="true"',
)
)
pinned = manifest.rev_parse("main")
manifest.tag("v1.0", pinned)
# Detached `.repo/manifests` falls back to `init.defaultBranch` (`master`
# under Git 2 defaults) when updating the manifest repo during `repo sync`.
manifest.set_ref("refs/heads/master", pinned)
if mode == "detached":
client.init(host.manifest_url)
client.git(".repo/manifests", "checkout", "-q", "--detach")
assert client.is_detached(".repo/manifests")
elif mode == "commit":
client.init(host.manifest_url, "-b", pinned)
assert (
client.config(".repo/manifests", "branch.default.merge") == pinned
)
else:
client.init(host.manifest_url, "-b", "refs/tags/v1.0")
assert (
client.config(".repo/manifests", "branch.default.merge")
== "refs/tags/v1.0"
)
with host.watch_requests() as requests:
explicit = client.sync("-s", check=False)
assert explicit.returncode != 0
assert "not tracking a remote branch" in explicit.stderr
assert "Using manifest server" not in explicit.output
assert requests == []
with host.watch_requests() as requests:
implicit = client.sync()
assert implicit.stderr == ""
assert "Using manifest server" not in implicit.output
assert not [r for r in requests if "manifestserver" in r.path]
assert client.head("foo") == foo.rev_parse("main")
assert (client.top / "foo/f").read_text() == "foo"
+5 -3
View File
@@ -14,6 +14,7 @@
"""Unittests for the color.py module.""" """Unittests for the color.py module."""
import os
from unittest import mock from unittest import mock
import pytest import pytest
@@ -21,6 +22,7 @@ import utils_for_test
import color import color
import git_config import git_config
import pager
@pytest.fixture @pytest.fixture
@@ -147,8 +149,8 @@ class TestColoringInit:
pager_active: bool, pager_active: bool,
expected: bool, expected: bool,
) -> None: ) -> None:
with mock.patch("os.isatty", return_value=isatty), mock.patch( with mock.patch.object(
"pager.active", pager_active os, "isatty", return_value=isatty
): ), mock.patch.object(pager, "active", pager_active):
c = _make_coloring(state) c = _make_coloring(state)
assert c.is_on is expected assert c.is_on is expected
+8 -8
View File
@@ -18,7 +18,7 @@ from typing import Iterable, List, Optional
import pytest import pytest
from command import Command import command
class FakeProject: class FakeProject:
@@ -136,7 +136,7 @@ def test_get_projects_keeps_derived_subprojects_for_repeated_repo():
sync_s=True, sync_s=True,
) )
manifest = FakeManifest([project_a, project_b]) manifest = FakeManifest([project_a, project_b])
cmd = Command(manifest=manifest) cmd = command.Command(manifest=manifest)
projects = cmd.GetProjects([]) projects = cmd.GetProjects([])
@@ -165,7 +165,7 @@ def test_get_projects_submodule_override(
derived_subprojects=[submodule], derived_subprojects=[submodule],
sync_s=sync_s, sync_s=sync_s,
) )
cmd = Command(manifest=FakeManifest([project])) cmd = command.Command(manifest=FakeManifest([project]))
projects = cmd.GetProjects([], submodules_ok=submodules_ok) projects = cmd.GetProjects([], submodules_ok=submodules_ok)
@@ -214,7 +214,7 @@ def test_get_projects_uses_groups_from_each_manifest_unless_overridden(
all_projects=[outer_project, *child_manifest.projects], all_projects=[outer_project, *child_manifest.projects],
effective_groups="outer-group", effective_groups="outer-group",
) )
cmd = Command(manifest=outer_manifest) cmd = command.Command(manifest=outer_manifest)
projects = cmd.GetProjects( projects = cmd.GetProjects(
[], [],
@@ -248,7 +248,7 @@ def test_get_projects_by_name_uses_groups_from_each_manifest() -> None:
all_projects=[outer_project, *child_manifest.projects], all_projects=[outer_project, *child_manifest.projects],
effective_groups="outer-group", effective_groups="outer-group",
) )
cmd = Command(manifest=outer_manifest) cmd = command.Command(manifest=outer_manifest)
projects = cmd.GetProjects( projects = cmd.GetProjects(
["shared"], ["shared"],
@@ -296,7 +296,7 @@ def test_find_projects_uses_groups_from_each_manifest() -> None:
) )
outer_manifest.outer_client = outer_manifest outer_manifest.outer_client = outer_manifest
cmd = Command(manifest=outer_manifest) cmd = command.Command(manifest=outer_manifest)
projects = cmd.FindProjects(["match"], all_manifests=True) projects = cmd.FindProjects(["match"], all_manifests=True)
@@ -322,7 +322,7 @@ def test_find_projects_uses_explicit_groups() -> None:
[default_project, override_project], [default_project, override_project],
effective_groups="default-group", effective_groups="default-group",
) )
cmd = Command(manifest=manifest) cmd = command.Command(manifest=manifest)
projects = cmd.FindProjects( projects = cmd.FindProjects(
["override"], ["override"],
@@ -339,7 +339,7 @@ def test_find_projects_allows_missing_projects() -> None:
"missing", "missing",
exists=False, exists=False,
) )
cmd = Command(manifest=FakeManifest([project])) cmd = command.Command(manifest=FakeManifest([project]))
projects = cmd.FindProjects( projects = cmd.FindProjects(
["missing"], ["missing"],
+12 -10
View File
@@ -14,32 +14,34 @@
"""Unittests for the editor.py module.""" """Unittests for the editor.py module."""
from typing import Iterator
import pytest import pytest
from editor import Editor import editor
@pytest.fixture(autouse=True) @pytest.fixture(autouse=True)
def reset_editor() -> None: def reset_editor() -> Iterator[None]:
"""Take care of resetting Editor state across tests.""" """Take care of resetting Editor state across tests."""
Editor._editor = None editor.Editor._editor = None
yield yield
Editor._editor = None editor.Editor._editor = None
def test_basic() -> None: def test_basic() -> None:
"""Basic checking of _GetEditor.""" """Basic checking of _GetEditor."""
Editor._editor = ":" editor.Editor._editor = ":"
assert Editor._GetEditor() == ":" assert editor.Editor._GetEditor() == ":"
def test_no_editor() -> None: def test_no_editor() -> None:
"""Check behavior when no editor is available.""" """Check behavior when no editor is available."""
Editor._editor = ":" editor.Editor._editor = ":"
assert Editor.EditString("foo") == "foo" assert editor.Editor.EditString("foo") == "foo"
def test_cat_editor() -> None: def test_cat_editor() -> None:
"""Check behavior when editor is `cat`.""" """Check behavior when editor is `cat`."""
Editor._editor = "cat" editor.Editor._editor = "cat"
assert Editor.EditString("foo") == "foo" assert editor.Editor.EditString("foo") == "foo"
+2 -2
View File
@@ -25,10 +25,10 @@ import error
import fetch import fetch
import git_command import git_command
import project import project
from subcmds import all_modules import subcmds
_IMPORTS = all_modules + [ _IMPORTS = subcmds.all_modules + [
error, error,
project, project,
git_command, git_command,
+8 -61
View File
@@ -18,6 +18,7 @@ import io
import os import os
import re import re
import subprocess import subprocess
import sys
import unittest import unittest
from unittest import mock from unittest import mock
@@ -65,60 +66,6 @@ class GitCommandTest(unittest.TestCase):
) )
class GitCommandWaitTest(unittest.TestCase):
"""Tests the GitCommand class .Wait()"""
def setUp(self):
class MockPopen:
rc = 0
def __init__(self):
self.stdout = io.BufferedReader(io.BytesIO())
self.stderr = io.BufferedReader(io.BytesIO())
def communicate(
self, input: str = None, timeout: float = None
) -> [str, str]:
"""Mock communicate fn."""
return ["", ""]
def wait(self, timeout=None):
return self.rc
self.popen = popen = MockPopen()
def popen_mock(*args, **kwargs):
return popen
def realpath_mock(val):
return val
mock.patch.object(subprocess, "Popen", side_effect=popen_mock).start()
mock.patch.object(
os.path, "realpath", side_effect=realpath_mock
).start()
def tearDown(self):
mock.patch.stopall()
def test_raises_when_verify_non_zero_result(self):
self.popen.rc = 1
r = git_command.GitCommand(None, ["status"], verify_command=True)
with self.assertRaises(git_command.GitCommandError):
r.Wait()
def test_returns_when_no_verify_non_zero_result(self):
self.popen.rc = 1
r = git_command.GitCommand(None, ["status"], verify_command=False)
self.assertEqual(1, r.Wait())
def test_default_returns_non_zero_result(self):
self.popen.rc = 1
r = git_command.GitCommand(None, ["status"])
self.assertEqual(1, r.Wait())
class GitCommandStreamLogsTest(unittest.TestCase): class GitCommandStreamLogsTest(unittest.TestCase):
"""Tests the GitCommand class stderr log streaming cases.""" """Tests the GitCommand class stderr log streaming cases."""
@@ -130,7 +77,7 @@ class GitCommandStreamLogsTest(unittest.TestCase):
self.mock_popen = mock.MagicMock() self.mock_popen = mock.MagicMock()
self.mock_popen.return_value = self.mock_process self.mock_popen.return_value = self.mock_process
mock.patch("subprocess.Popen", self.mock_popen).start() mock.patch.object(subprocess, "Popen", self.mock_popen).start()
def tearDown(self): def tearDown(self):
mock.patch.stopall() mock.patch.stopall()
@@ -199,7 +146,7 @@ class GitCommandStreamLogsTest(unittest.TestCase):
self.mock_process.communicate.assert_called_once_with(input=None) self.mock_process.communicate.assert_called_once_with(input=None)
self.mock_process.stderr.read1.assert_not_called() self.mock_process.stderr.read1.assert_not_called()
@mock.patch("sys.stderr") @mock.patch.object(sys, "stderr")
def test_streams_stderr_when_no_stream_is_set(self, mock_stderr): def test_streams_stderr_when_no_stream_is_set(self, mock_stderr):
mock_stderr.isatty.return_value = False mock_stderr.isatty.return_value = False
logs = "\n".join( logs = "\n".join(
@@ -229,7 +176,7 @@ class GitCommandStreamLogsTest(unittest.TestCase):
mock_stderr.write.assert_called_once_with(logs) mock_stderr.write.assert_called_once_with(logs)
self.assertEqual(cmd.stderr, logs) self.assertEqual(cmd.stderr, logs)
@mock.patch("sys.stderr") @mock.patch.object(sys, "stderr")
def test_push_injects_progress_only_when_tty_and_not_suppressed( def test_push_injects_progress_only_when_tty_and_not_suppressed(
self, mock_stderr: mock.MagicMock self, mock_stderr: mock.MagicMock
) -> None: ) -> None:
@@ -240,7 +187,7 @@ class GitCommandStreamLogsTest(unittest.TestCase):
cmd = self.mock_popen.call_args[0][0] cmd = self.mock_popen.call_args[0][0]
self.assertEqual(cmd, ["git", "push", "--progress"]) self.assertEqual(cmd, ["git", "push", "--progress"])
@mock.patch("sys.stderr") @mock.patch.object(sys, "stderr")
def test_push_does_not_inject_progress_when_no_progress_set( def test_push_does_not_inject_progress_when_no_progress_set(
self, mock_stderr: mock.MagicMock self, mock_stderr: mock.MagicMock
) -> None: ) -> None:
@@ -263,7 +210,7 @@ class GitCommandCaptureBytesTest(unittest.TestCase):
self.mock_popen = mock.MagicMock() self.mock_popen = mock.MagicMock()
self.mock_popen.return_value = self.mock_process self.mock_popen.return_value = self.mock_process
mock.patch("subprocess.Popen", self.mock_popen).start() mock.patch.object(subprocess, "Popen", self.mock_popen).start()
def tearDown(self) -> None: def tearDown(self) -> None:
mock.patch.stopall() mock.patch.stopall()
@@ -497,13 +444,13 @@ class GitRequireTests(unittest.TestCase):
"""Test fatal require calls with old versions.""" """Test fatal require calls with old versions."""
with self.assertRaises(git_command.GitRequireError) as e: with self.assertRaises(git_command.GitRequireError) as e:
git_command.git_require((2,), fail=True) git_command.git_require((2,), fail=True)
self.assertNotEqual(0, e.code) self.assertNotEqual(0, e.exception.exit_code)
def test_older_fatal_msg(self): def test_older_fatal_msg(self):
"""Test fatal require calls with old versions and message.""" """Test fatal require calls with old versions and message."""
with self.assertRaises(git_command.GitRequireError) as e: with self.assertRaises(git_command.GitRequireError) as e:
git_command.git_require((2,), fail=True, msg="so sad") git_command.git_require((2,), fail=True, msg="so sad")
self.assertNotEqual(0, e.code) self.assertNotEqual(0, e.exception.exit_code)
class GitCommandErrorTest(unittest.TestCase): class GitCommandErrorTest(unittest.TestCase):
+446 -1
View File
@@ -14,12 +14,16 @@
"""Unittests for the git_config.py module.""" """Unittests for the git_config.py module."""
import json
import os
from pathlib import Path from pathlib import Path
from typing import Any, Optional import types
from typing import Any, List, Optional, Tuple
import pytest import pytest
import utils_for_test import utils_for_test
import error
import git_config import git_config
@@ -211,6 +215,9 @@ def test_get_sync_analysis_state_data(rw_config_file: Path) -> None:
class Options: class Options:
"""Container for testing.""" """Container for testing."""
verbose: str
mp_update: str
options = Options() options = Options()
options.verbose = "true" options.verbose = "true"
options.mp_update = "false" options.mp_update = "false"
@@ -346,3 +353,441 @@ def test_get_boolean_git_spellings(
config_file.write_text(f"[section]\n\tkey = {value}\n") config_file.write_text(f"[section]\n\tkey = {value}\n")
config = git_config.GitConfig(config_file) config = git_config.GitConfig(config_file)
assert config.GetBoolean("section.key") is expected assert config.GetBoolean("section.key") is expected
def test_set_string_updates_json_and_section_caches(
rw_config_file: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Test SetString keeps .repo_config.json and _sections coherent."""
config = git_config.GitConfig(str(rw_config_file))
assert not config.HasSection("remote", "origin")
config.SetString("remote.origin.url", "https://example.com/repo")
assert config.HasSection("remote", "origin")
json_file = rw_config_file.parent / ".repo_config.json"
raw = json.loads(json_file.read_text(encoding="utf-8"))
assert git_config._JSON_META_KEY in raw
raw.pop(git_config._JSON_META_KEY)
assert raw == {"remote.origin.url": ["https://example.com/repo"]}
reader = git_config.GitConfig(str(rw_config_file))
def fail_read_git() -> None:
raise AssertionError("Expected _ReadJson cache hit without git config")
monkeypatch.setattr(reader, "_ReadGit", fail_read_git)
assert reader.GetString("remote.origin.url") == "https://example.com/repo"
assert git_config._JSON_META_KEY not in reader.DumpConfigDict()
def test_reload_refreshes_external_edits(rw_config_file: Path) -> None:
"""Test Reload() discards stale JSON cache and reloads external edits."""
config = git_config.GitConfig(str(rw_config_file))
config.SetString("abfs.initialized", "false")
assert config.GetBoolean("abfs.initialized") is False
rw_config_file.write_text(
"[abfs]\n\tinitialized = true\n", encoding="utf-8"
)
config.Reload()
assert config.GetBoolean("abfs.initialized") is True
json_file = rw_config_file.parent / ".repo_config.json"
raw = json.loads(json_file.read_text(encoding="utf-8"))
raw.pop(git_config._JSON_META_KEY, None)
assert raw == {"abfs.initialized": ["true"]}
def test_same_tick_external_edit_invalidates_json_cache(
rw_config_file: Path,
) -> None:
"""External edit with identical mtime_ns and size invalidates JSON."""
config = git_config.GitConfig(str(rw_config_file))
config.SetString("foo.bar", "val1")
json_file = rw_config_file.parent / ".repo_config.json"
json_stat = os.stat(json_file)
# Rewrite with identical byte length and force identical mtime_ns.
updated = rw_config_file.read_text(encoding="utf-8").replace("val1", "val2")
rw_config_file.write_text(updated, encoding="utf-8")
os.utime(rw_config_file, ns=(json_stat.st_atime_ns, json_stat.st_mtime_ns))
reader = git_config.GitConfig(str(rw_config_file))
assert reader.GetString("foo.bar") == "val2"
def test_clear_cache_and_set_string_refresh_remotes_and_branches(
rw_config_file: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""ClearCache, Reload, SetString, and Branch.Save keep caches coherent."""
config = git_config.GitConfig(str(rw_config_file))
remote = config.GetRemote("origin")
remote.url = "https://old.example.com/repo"
remote.ResetFetch()
remote.Save()
branch = config.GetBranch("main")
branch.remote = remote
branch.merge = "refs/heads/main"
branch.Save()
assert config.HasSection("branch", "main")
assert config.GetString("branch.main.remote") == "origin"
assert config.GetString("branch.main.merge") == "refs/heads/main"
# Subsequent SetString must not clobber branch.* keys in .repo_config.json.
config.SetString("custom.flag", "1")
reader = git_config.GitConfig(str(rw_config_file))
monkeypatch.setattr(
reader,
"_ReadGit",
lambda: (_ for _ in ()).throw(AssertionError("expected JSON hit")),
)
assert reader.GetString("branch.main.remote") == "origin"
assert reader.GetString("branch.main.merge") == "refs/heads/main"
# Direct SetString on remote.* invalidates cached Remote and Branch objects.
config.SetString(
"remote.origin.fetch", "+refs/heads/dev:refs/remotes/origin/dev"
)
assert (
config.GetRemote("origin").ToLocal("dev") == "refs/remotes/origin/dev"
)
assert config.GetBranch("main").remote is config.GetRemote("origin")
# External edit + Reload() refreshes both GetRemote() and GetBranch().
rw_config_file.write_text(
'[remote "origin"]\n'
"\turl = https://new.example.com/repo\n"
"\tfetch = +refs/heads/*:refs/remotes/origin/*\n"
'[branch "main"]\n'
"\tremote = origin\n"
"\tmerge = refs/heads/next\n",
encoding="utf-8",
)
config.Reload()
assert config.GetRemote("origin").url == "https://new.example.com/repo"
assert config.GetBranch("main").merge == "refs/heads/next"
def test_set_string_with_include_invalidates_instead_of_corrupting(
tmp_path: Path,
) -> None:
"""SetString with [include] avoids freezing or dropping included values."""
inc_file = tmp_path / "extra.gitconfig"
inc_file.write_text("[inc]\n\tkey = v1\n", encoding="utf-8")
cfg_file = tmp_path / "config"
config = git_config.GitConfig(str(cfg_file))
config.SetString("include.path", str(inc_file))
assert config.GetString("inc.key") == "v1"
config.SetString("local.only", "yes")
json_file = tmp_path / ".repo_config.json"
assert not json_file.exists()
inc_file.write_text("[inc]\n\tkey = v2\n", encoding="utf-8")
assert config.GetString("inc.key") == "v2"
assert config.GetString("local.only") == "yes"
# Unsetting an included-only key does not fail with exit code 5 or hide it.
config.SetString("inc.key", None)
assert config.GetString("inc.key") == "v2"
def test_set_string_failure_preserves_cache(
rw_config_file: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Failed _do() must not leave mutated values in _cache."""
config = git_config.GitConfig(str(rw_config_file))
config.SetString("foo.bar", "good")
def fail_do(*_args: Any, **_kwargs: Any) -> str:
raise error.GitError("lock contention")
monkeypatch.setattr(config, "_do", fail_do)
with pytest.raises(error.GitError):
config.SetString("foo.bar", "bad")
assert config.GetString("foo.bar") == "good"
with pytest.raises(error.GitError):
config.SetString("foo.bar", None)
assert config.GetString("foo.bar") == "good"
def test_set_string_after_external_edit_preserves_external_keys(
rw_config_file: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Warm SetString reloads external edits instead of overwriting JSON."""
config = git_config.GitConfig(str(rw_config_file))
config.SetString("init.key", "v0")
assert config._cache_dict is not None
# External edit adds other.key while config._cache_dict is warm in memory.
rw_config_file.write_text(
"[init]\n\tkey = v0\n[other]\n\tkey = ext1\n", encoding="utf-8"
)
config.SetString("user.name", "Alice")
assert config.GetString("other.key") == "ext1"
assert config.GetString("user.name") == "Alice"
reader = git_config.GitConfig(str(rw_config_file))
monkeypatch.setattr(
reader,
"_ReadGit",
lambda: (_ for _ in ()).throw(AssertionError("expected JSON hit")),
)
assert reader.GetString("other.key") == "ext1"
assert reader.GetString("user.name") == "Alice"
# Same-mtime_ns and same-size external edit while warm is also detected.
prev_stat = os.stat(rw_config_file)
same_len = rw_config_file.read_text(encoding="utf-8").replace(
"ext1", "ext2"
)
rw_config_file.write_text(same_len, encoding="utf-8")
os.utime(rw_config_file, ns=(prev_stat.st_atime_ns, prev_stat.st_mtime_ns))
config.SetString("user.email", "alice@example.com")
assert config.GetString("other.key") == "ext2"
assert config.GetString("user.email") == "alice@example.com"
reader2 = git_config.GitConfig(str(rw_config_file))
monkeypatch.setattr(
reader2,
"_ReadGit",
lambda: (_ for _ in ()).throw(AssertionError("expected JSON hit")),
)
assert reader2.GetString("other.key") == "ext2"
assert reader2.GetString("user.name") == "Alice"
assert reader2.GetString("user.email") == "alice@example.com"
def test_batch_writes_coalesces_new_keys_without_git_replace(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Test BatchWrites writes new sections atomically with one JSON save."""
cfg_path = tmp_path / "config"
cfg_path.write_text("[core]\n\trepositoryformatversion = 0\n")
config = git_config.GitConfig(str(cfg_path))
orig_do = config._do
seen_do = []
save_count = 0
orig_save = config._SaveJson
def spy_do(*args: str) -> str:
seen_do.append(args)
return orig_do(*args)
def spy_save(cache: Any) -> None:
nonlocal save_count
save_count += 1
orig_save(cache)
monkeypatch.setattr(config, "_do", spy_do)
monkeypatch.setattr(config, "_SaveJson", spy_save)
with config.BatchWrites():
config.SetString("filter.lfs.smudge", "git-lfs smudge --skip -- %f")
config.SetString("filter.lfs.process", "git-lfs filter-process --skip")
config.SetString("remote.goog.url", "https://example.com/platform/art")
config.SetString(
"remote.goog.fetch",
[
"+refs/heads/*:refs/remotes/goog/*",
"+refs/tags/*:refs/tags/*",
],
)
config.SetBoolean("maintenance.auto", False)
assert seen_do == [("--null", "--list")]
assert save_count == 1
assert not (tmp_path / "config.lock").exists()
reloaded = git_config.GitConfig(str(cfg_path))
fresh = reloaded._ReadGit()
assert fresh["filter.lfs.smudge"] == ["git-lfs smudge --skip -- %f"]
assert fresh["filter.lfs.process"] == ["git-lfs filter-process --skip"]
assert fresh["remote.goog.url"] == ["https://example.com/platform/art"]
assert fresh["remote.goog.fetch"] == [
"+refs/heads/*:refs/remotes/goog/*",
"+refs/tags/*:refs/tags/*",
]
assert fresh["maintenance.auto"] == ["false"]
def test_batch_writes_falls_back_on_lock_or_external_edit(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Test BatchWrites preserves concurrent edits and respects config.lock."""
# 1. External edit between cache load and flush falls back to git config CLI
# so neither the external key nor the batched key is lost.
cfg_path = tmp_path / "config"
cfg_path.write_text("[core]\n\tfilemode = true\n")
config = git_config.GitConfig(str(cfg_path))
with config.BatchWrites():
config.SetString("remote.origin.url", "https://example.com/a")
with open(cfg_path, "a", encoding="utf-8") as fp:
fp.write("[external]\n\tadded = yes\n")
reloaded = git_config.GitConfig(str(cfg_path))
fresh = reloaded._ReadGit()
assert fresh["external.added"] == ["yes"]
assert fresh["remote.origin.url"] == ["https://example.com/a"]
assert config.GetString("external.added") == "yes"
# 2. Pre-existing config.lock is never clobbered; BatchWrites delegates to
# _do instead.
locked_path = tmp_path / "locked_config"
locked_path.write_text("[core]\n\tfilemode = true\n")
lock_file = tmp_path / "locked_config.lock"
lock_file.write_text("held-by-other-process")
locked_cfg = git_config.GitConfig(str(locked_path))
_ = locked_cfg.GetBoolean("core.filemode")
do_calls = []
monkeypatch.setattr(
locked_cfg,
"_do",
lambda *args: do_calls.append(args) or "core.filemode\ntrue\0",
)
with locked_cfg.BatchWrites():
locked_cfg.SetString("remote.origin.url", "https://example.com/b")
assert lock_file.read_text() == "held-by-other-process"
assert ("--replace-all", "remote.origin.url", "https://example.com/b") in (
do_calls
)
def test_batch_writes_direct_do_refreshes_fingerprint_and_branch_save_batches(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Direct _do in BatchWrites updates fingerprint; Save uses BatchWrites."""
cfg_path = tmp_path / "config"
cfg_path.write_text("[core]\n\tfilemode = true\n")
config = git_config.GitConfig(str(cfg_path))
_ = config.GetBoolean("core.filemode")
orig_do = config._do
seen_do = []
def spy_do(*args: str, **kwargs: Any) -> str:
seen_do.append(args)
return orig_do(*args, **kwargs)
monkeypatch.setattr(config, "_do", spy_do)
# Updating an existing key runs _do() directly inside BatchWrites; the next
# SetString must not see a stale _cache_fingerprint or re-run --list.
with config.BatchWrites():
config.SetString("core.filemode", "false")
config.SetString("new.section.key", "val")
assert seen_do == [("--replace-all", "core.filemode", "false")]
assert config.GetString("core.filemode") == "false"
assert config.GetString("new.section.key") == "val"
# Remote.Save() and Branch.Save() coalesce new keys via BatchWrites without
# spawning git config --replace-all subprocesses.
seen_do.clear()
remote = config.GetRemote("origin")
remote.url = "https://example.com/repo"
remote.ResetFetch()
remote.Save()
branch = config.GetBranch("main")
branch.remote = remote
branch.merge = "refs/heads/main"
branch.Save()
assert seen_do == []
# When include.path is present, _CanQueueInBatch returns False so writes go
# through git config immediately instead of queueing for lockfile append.
inc_file = tmp_path / "inc.gitconfig"
inc_file.write_text("[extra]\n\tflag = 1\n")
config.SetString("include.path", str(inc_file))
_ = config.GetString("extra.flag")
seen_do.clear()
with config.BatchWrites():
config.SetString("after.include", "yes")
assert ("--replace-all", "after.include", "yes") in seen_do
def test_batch_writes_failure_paths(tmp_path: Path) -> None:
"""BatchWrites clears state on failure and preserves body errors."""
# 1. Failed fallback flush clears poisoned in-memory cache and dirty flag.
fail_path = tmp_path / "fail_config"
fail_path.write_text("[core]\n\tfilemode = true\n")
lock_path = tmp_path / "fail_config.lock"
lock_path.write_text("locked")
fail_cfg = git_config.GitConfig(str(fail_path))
assert fail_cfg.GetBoolean("core.filemode") is True
with pytest.raises(error.GitError):
with fail_cfg.BatchWrites():
fail_cfg.SetString("remote.origin.url", "https://bad.example.com")
lock_path.unlink()
assert fail_cfg.GetString("remote.origin.url") is None
with fail_cfg.BatchWrites():
pass
assert (
git_config.GitConfig(str(fail_path)).GetString("remote.origin.url")
is None
)
# 2. If body raises and flush also fails, original body error propagates.
body_path = tmp_path / "body_config"
body_path.write_text("[core]\n\tfilemode = true\n")
body_lock = tmp_path / "body_config.lock"
body_lock.write_text("locked")
body_cfg = git_config.GitConfig(str(body_path))
assert body_cfg.GetBoolean("core.filemode") is True
with pytest.raises(RuntimeError, match="body boom"):
with body_cfg.BatchWrites():
body_cfg.SetString("remote.origin.url", "https://bad.example.com")
raise RuntimeError("body boom")
body_lock.unlink()
assert body_cfg.GetString("remote.origin.url") is None
def test_batch_writes_sync_analysis_state(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""UpdateSyncAnalysisState coalesces new syncstate keys without CLI."""
state_cfg_path = tmp_path / "state_config"
state_cfg_path.write_text(
'[remote "origin"]\n\turl = https://example.com/repo\n',
encoding="utf-8",
)
state_cfg = git_config.GitConfig(str(state_cfg_path))
_ = state_cfg.GetString("remote.origin.url")
seen_do: List[Tuple[str, ...]] = []
orig_do = state_cfg._do
monkeypatch.setattr(
state_cfg,
"_do",
lambda *args, **kw: seen_do.append(args) or orig_do(*args, **kw),
)
opts = types.SimpleNamespace(jobs=8, quiet=True)
state_cfg.UpdateSyncAnalysisState(opts, {"superproject": False})
assert seen_do == []
fresh_state = git_config.GitConfig(str(state_cfg_path))._ReadGit()
assert fresh_state["repo.syncstate.main.version"] == ["1"]
assert fresh_state["repo.syncstate.options.jobs"] == ["8"]
assert fresh_state["repo.syncstate.options.quiet"] == ["true"]
assert fresh_state["repo.syncstate.remote.origin.url"] == [
"https://example.com/repo"
]
+500
View File
@@ -16,7 +16,9 @@
import os import os
from pathlib import Path from pathlib import Path
import stat
import subprocess import subprocess
import time
from typing import Any, List from typing import Any, List
from unittest import mock from unittest import mock
@@ -25,6 +27,7 @@ import utils_for_test
import git_command import git_command
import git_refs import git_refs
import platform_utils
def _run(repo: str, *args: str) -> str: def _run(repo: str, *args: str) -> str:
@@ -138,6 +141,7 @@ def test_root_ref_snapshot_falls_back_for_unborn_head(
def read_head(name: str) -> None: def read_head(name: str) -> None:
assert name == "HEAD" assert name == "HEAD"
assert refs._symref is not None
refs._symref[name] = "refs/heads/main" refs._symref[name] = "refs/heads/main"
monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head) monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head)
@@ -168,6 +172,7 @@ def test_old_git_keeps_separate_head_fallback(
def read_head(name: str) -> None: def read_head(name: str) -> None:
assert name == "HEAD" assert name == "HEAD"
assert refs._symref is not None
refs._symref[name] = "refs/heads/main" refs._symref[name] = "refs/heads/main"
monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head) monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head)
@@ -201,6 +206,7 @@ def test_for_each_ref_failure_falls_back_to_symbolic_ref(
def read_head(name: str) -> None: def read_head(name: str) -> None:
assert name == "HEAD" assert name == "HEAD"
assert refs._symref is not None
refs._symref[name] = "refs/heads/main" refs._symref[name] = "refs/heads/main"
monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head) monkeypatch.setattr(refs, "_ReadSymbolicRef", read_head)
@@ -244,3 +250,497 @@ def test_updates_when_storage_backend_toggles(tmp_path):
_run(repo, "refs", "migrate", "--ref-format=files") _run(repo, "refs", "migrate", "--ref-format=files")
_run(repo, "branch", "files-branch") _run(repo, "branch", "files-branch")
assert refs.get("refs/heads/files-branch") == head assert refs.get("refs/heads/files-branch") == head
@pytest.mark.parametrize("reftable", [False, True])
def test_unborn_head_resolution(
tmp_path: Path, reftable: bool, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Unborn HEAD reads files backend directly and uses git on reftable."""
if reftable and not utils_for_test.supports_reftable():
pytest.skip("reftable not supported")
repo = os.path.join(tmp_path, "unborn")
ref_format = "reftable" if reftable else "files"
utils_for_test.init_git_tree(repo, ref_format=ref_format)
_run(repo, "symbolic-ref", "HEAD", "refs/heads/unborn-main")
refs = git_refs.GitRefs(os.path.join(repo, ".git"))
seen_symbolic_ref = []
class SpyGitCommand(git_command.GitCommand):
def __init__(
self,
proj: Any,
cmdv: List[str],
*args: Any,
**kwargs: Any,
) -> None:
if cmdv and cmdv[0] == "symbolic-ref":
seen_symbolic_ref.append(cmdv)
super().__init__(proj, cmdv, *args, **kwargs)
monkeypatch.setattr(git_command, "GitCommand", SpyGitCommand)
assert refs.symref("HEAD") == "refs/heads/unborn-main"
if not reftable:
assert seen_symbolic_ref == []
@pytest.mark.parametrize("reftable", [False, True])
def test_update_symref_keeps_snapshot_warm(
tmp_path: Path, reftable: bool
) -> None:
"""UpdateSymref stays warm on files backend and reloads on reftable."""
if reftable and not utils_for_test.supports_reftable():
pytest.skip("reftable not supported")
repo = _init_repo(tmp_path, reftable=reftable)
branch = _run(repo, "symbolic-ref", "--short", "HEAD")
head = _run(repo, "rev-parse", "HEAD")
refs = git_refs.GitRefs(os.path.join(repo, ".git"))
assert refs.get(f"refs/heads/{branch}") == head
_run(repo, "symbolic-ref", "refs/remotes/m/main", f"refs/heads/{branch}")
refs.UpdateSymref("refs/remotes/m/main", f"refs/heads/{branch}")
if reftable:
assert not refs.is_loaded
assert refs.symref("refs/remotes/m/main") == f"refs/heads/{branch}"
assert refs.get("refs/remotes/m/main") == head
else:
with mock.patch.object(
refs,
"_LoadAll",
side_effect=AssertionError("expected warm GitRefs snapshot"),
):
assert refs.symref("refs/remotes/m/main") == f"refs/heads/{branch}"
assert refs.get("refs/remotes/m/main") == head
def test_update_symref_invalidates_when_external_ref_changed(
tmp_path: Path,
) -> None:
"""External ref writes before UpdateSymref invalidate instead of masking."""
repo = _init_repo(tmp_path, reftable=False)
branch = _run(repo, "symbolic-ref", "--short", "HEAD")
head = _run(repo, "rev-parse", "HEAD")
refs = git_refs.GitRefs(os.path.join(repo, ".git"))
assert refs.get(f"refs/heads/{branch}") == head
# External branch write happens before symbolic-ref + UpdateSymref.
_run(repo, "branch", "external-topic")
_run(repo, "symbolic-ref", "refs/remotes/m/main", f"refs/heads/{branch}")
refs.UpdateSymref("refs/remotes/m/main", f"refs/heads/{branch}")
assert not refs.is_loaded
assert refs.get("refs/heads/external-topic") == head
assert refs.get("refs/remotes/m/main") == head
# External sibling ref in the same directory is also detected.
_run(repo, "update-ref", "refs/remotes/m/sibling", head)
_run(
repo,
"symbolic-ref",
"refs/remotes/m/main",
"refs/heads/external-topic",
)
refs.UpdateSymref("refs/remotes/m/main", "refs/heads/external-topic")
assert not refs.is_loaded
assert refs.get("refs/remotes/m/sibling") == head
def test_update_symref_self_or_cyclic_clears_phyref(tmp_path: Path) -> None:
"""Self or cyclic symrefs clear any prior physical SHA from _phyref."""
repo = _init_repo(tmp_path, reftable=False)
head = _run(repo, "rev-parse", "HEAD")
gitdir = os.path.join(repo, ".git")
_run(repo, "update-ref", "refs/remotes/m/main", head)
_run(repo, "update-ref", "refs/remotes/m/other", head)
refs = git_refs.GitRefs(gitdir)
assert refs.get("refs/remotes/m/main") == head
assert refs.get("refs/remotes/m/other") == head
_run(repo, "symbolic-ref", "refs/remotes/m/main", "refs/remotes/m/main")
refs.UpdateSymref("refs/remotes/m/main", "refs/remotes/m/main")
assert refs.get("refs/remotes/m/main") == ""
_run(repo, "symbolic-ref", "refs/remotes/m/main", "refs/remotes/m/other")
refs.UpdateSymref("refs/remotes/m/main", "refs/remotes/m/other")
assert refs.get("refs/remotes/m/main") == head
_run(repo, "symbolic-ref", "refs/remotes/m/other", "refs/remotes/m/main")
refs.UpdateSymref("refs/remotes/m/other", "refs/remotes/m/main")
assert refs.get("refs/remotes/m/main") == ""
assert refs.get("refs/remotes/m/other") == ""
def test_update_symref_propagates_chained_symrefs(tmp_path: Path) -> None:
"""Re-pointing an intermediate symref updates chained symrefs cleanly."""
repo = _init_repo(tmp_path, reftable=False)
branch = _run(repo, "symbolic-ref", "--short", "HEAD")
c1 = _run(repo, "rev-parse", "HEAD")
Path(os.path.join(repo, "a")).write_text("2", encoding="utf-8")
_run(repo, "add", "a")
_run(repo, "commit", "-q", "-m", "second")
c2 = _run(repo, "rev-parse", "HEAD")
_run(repo, "update-ref", "refs/heads/first", c1)
# Point HEAD at unborn refs/remotes/m/main before warming GitRefs.
_run(repo, "symbolic-ref", "HEAD", "refs/remotes/m/main")
refs = git_refs.GitRefs(os.path.join(repo, ".git"))
assert refs.symref("HEAD") == "refs/remotes/m/main"
assert refs.get("HEAD") == ""
_run(repo, "symbolic-ref", "refs/remotes/m/main", "refs/heads/first")
refs.UpdateSymref("refs/remotes/m/main", "refs/heads/first")
_run(repo, "symbolic-ref", "refs/remotes/m/alias", "refs/remotes/m/main")
refs.UpdateSymref("refs/remotes/m/alias", "refs/remotes/m/main")
assert refs.get("HEAD") == c1
assert refs.get("refs/remotes/m/alias") == c1
_run(repo, "symbolic-ref", "refs/remotes/m/main", f"refs/heads/{branch}")
refs.UpdateSymref("refs/remotes/m/main", f"refs/heads/{branch}")
with mock.patch.object(
refs,
"_LoadAll",
side_effect=AssertionError("expected warm GitRefs snapshot"),
):
assert refs.get("refs/remotes/m/main") == c2
assert refs.get("refs/remotes/m/alias") == c2
assert refs.get("HEAD") == c2
def test_files_backend_reads_loose_and_packed_without_git(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Files backend reads packed-refs, loose refs, and symrefs in Python."""
repo = _init_repo(tmp_path, reftable=False)
branch = _run(repo, "symbolic-ref", "--short", "HEAD")
c1 = _run(repo, "rev-parse", "HEAD")
_run(repo, "tag", "-a", "-m", "v1", "v1.0")
tag_oid = _run(repo, "rev-parse", "refs/tags/v1.0")
_run(repo, "branch", "packed-only", c1)
_run(repo, "branch", "overridden", c1)
_run(repo, "pack-refs", "--all")
Path(os.path.join(repo, "a")).write_text("2", encoding="utf-8")
_run(repo, "add", "a")
_run(repo, "commit", "-q", "-m", "second")
c2 = _run(repo, "rev-parse", "HEAD")
_run(repo, "update-ref", "refs/heads/overridden", c2)
_run(repo, "symbolic-ref", "refs/remotes/m/main", "refs/heads/overridden")
monkeypatch.delenv("GIT_DIR", raising=False)
monkeypatch.delenv("GIT_COMMON_DIR", raising=False)
seen_cmds: List[List[str]] = []
class SpyGitCommand(git_command.GitCommand):
def __init__(
self,
proj: Any,
cmdv: List[str],
*args: Any,
**kwargs: Any,
) -> None:
seen_cmds.append(list(cmdv))
super().__init__(proj, cmdv, *args, **kwargs)
monkeypatch.setattr(git_command, "GitCommand", SpyGitCommand)
refs = git_refs.GitRefs(os.path.join(repo, ".git"))
assert refs.get("HEAD") == c2
assert refs.is_files_authoritative
with mock.patch.object(
refs,
"_LoadAll",
side_effect=AssertionError("expected warm GitRefs snapshot"),
):
assert refs.get(f"refs/heads/{branch}") == c2
assert refs.get("refs/heads/packed-only") == c1
assert refs.get("refs/heads/overridden") == c2
assert refs.get("refs/remotes/m/main") == c2
assert refs.get("refs/tags/v1.0") == tag_oid
assert seen_cmds == []
# A concurrent .lock file triggers clean fallback to git for-each-ref.
lock_file = Path(repo, ".git", "refs", "heads", "packed-only.lock")
lock_file.write_text("", encoding="utf-8")
refs2 = git_refs.GitRefs(os.path.join(repo, ".git"))
assert refs2.get("refs/heads/packed-only") == c1
assert not refs2.is_files_authoritative
assert any(cmd and cmd[0] == "for-each-ref" for cmd in seen_cmds)
def test_try_resolve_ref_loose_packed_symref_and_missing(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""TryResolveRef resolves loose/packed/symref refs without full scan."""
repo = _init_repo(tmp_path, reftable=False)
c1 = _run(repo, "rev-parse", "HEAD")
_run(repo, "branch", "packed-branch", c1)
_run(repo, "pack-refs", "--all")
Path(os.path.join(repo, "a")).write_text("2", encoding="utf-8")
_run(repo, "add", "a")
_run(repo, "commit", "-q", "-m", "second")
c2 = _run(repo, "rev-parse", "HEAD")
_run(repo, "update-ref", "refs/remotes/origin/main", c2)
_run(
repo,
"symbolic-ref",
"refs/remotes/m/main",
"refs/remotes/origin/main",
)
monkeypatch.delenv("GIT_DIR", raising=False)
monkeypatch.delenv("GIT_COMMON_DIR", raising=False)
gitdir = os.path.join(repo, ".git")
refs = git_refs.GitRefs(gitdir)
assert refs.TryResolveRef("refs/remotes/origin/main") == c2
assert refs.TryResolveRef("refs/heads/packed-branch") == c1
assert refs.TryResolveRef("refs/remotes/m/main") == c2
assert refs.TryResolveRef("refs/remotes/origin/missing") == ""
assert not refs.is_loaded
# A .lock file forces None so callers fall back to git.
lock_file = Path(gitdir, "refs", "remotes", "origin", "main.lock")
lock_file.write_text("", encoding="utf-8")
assert refs.TryResolveRef("refs/remotes/origin/main") is None
def test_detects_new_packed_refs_and_same_mtime_replace(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""GitRefs reloads on newly created packed-refs and same-mtime replaces."""
monkeypatch.delenv("GIT_DIR", raising=False)
monkeypatch.delenv("GIT_COMMON_DIR", raising=False)
repo = os.path.join(tmp_path, "fresh")
utils_for_test.init_git_tree(repo, ref_format="files")
gitdir = os.path.join(repo, ".git")
refs = git_refs.GitRefs(gitdir)
assert refs.all == {}
c1 = "1" * 40
c2 = "2" * 40
Path(gitdir, "packed-refs").write_text(
f"# pack-refs with: peeled fully-peeled sorted \n"
f"{c1} refs/remotes/origin/main\n",
encoding="utf-8",
)
assert refs.get("refs/remotes/origin/main") == c1
loose = Path(gitdir, "refs", "remotes", "origin", "topic")
loose.parent.mkdir(parents=True, exist_ok=True)
loose.write_text(c1 + "\n", encoding="utf-8")
assert refs.get("refs/remotes/origin/topic") == c1
# Replace the ref the way git does, by renaming a new file over it, then
# put the mtimes back so that only the new inode shows.
file_st = os.stat(loose)
dir_st = os.stat(loose.parent)
replacement = Path(gitdir, "topic.new")
replacement.write_text(c2 + "\n", encoding="utf-8")
os.replace(replacement, loose)
os.utime(loose, ns=(file_st.st_atime_ns, file_st.st_mtime_ns))
os.utime(loose.parent, ns=(dir_st.st_atime_ns, dir_st.st_mtime_ns))
assert refs.get("refs/remotes/origin/topic") == c2
@pytest.mark.skipif(
platform_utils.isWindows(),
reason="st_ctime is the creation time on Windows",
)
def test_ctime_only_change_keeps_snapshot_warm(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A ref file whose ctime alone moves does not invalidate the snapshot."""
monkeypatch.delenv("GIT_DIR", raising=False)
monkeypatch.delenv("GIT_COMMON_DIR", raising=False)
repo = _init_repo(tmp_path, reftable=False)
branch = _run(repo, "symbolic-ref", "--short", "HEAD")
head = _run(repo, "rev-parse", "HEAD")
gitdir = os.path.join(repo, ".git")
refs = git_refs.GitRefs(gitdir)
assert refs.get(f"refs/heads/{branch}") == head
# Backup agents and file system crawlers tag files with xattrs, which moves
# ctime and nothing else. Flipping a mode bit and putting it back does the
# same; repeat it until the timestamp granularity lets the change show.
loose = os.path.join(gitdir, "refs", "heads", branch)
before = os.lstat(loose)
mode = stat.S_IMODE(before.st_mode)
for _ in range(1000):
os.chmod(loose, mode ^ stat.S_IXUSR)
os.chmod(loose, mode)
if os.lstat(loose).st_ctime_ns != before.st_ctime_ns:
break
time.sleep(0.001)
else:
pytest.fail("chmod did not move ctime")
with mock.patch.object(
refs,
"_LoadAll",
side_effect=AssertionError("expected warm GitRefs snapshot"),
):
assert refs.get(f"refs/heads/{branch}") == head
@pytest.mark.parametrize("reftable", [False, True])
def test_update_ref_keeps_snapshot_warm(
tmp_path: Path, reftable: bool, monkeypatch: pytest.MonkeyPatch
) -> None:
"""UpdateRef stays warm on files backend and reloads on reftable."""
if reftable and not utils_for_test.supports_reftable():
pytest.skip("reftable not supported")
monkeypatch.delenv("GIT_DIR", raising=False)
monkeypatch.delenv("GIT_COMMON_DIR", raising=False)
repo = _init_repo(tmp_path, reftable=reftable)
branch = _run(repo, "symbolic-ref", "--short", "HEAD")
c1 = _run(repo, "rev-parse", "HEAD")
_run(repo, "symbolic-ref", "refs/remotes/m/main", f"refs/heads/{branch}")
Path(os.path.join(repo, "a")).write_text("2", encoding="utf-8")
_run(repo, "add", "a")
_run(repo, "commit", "-q", "-m", "second")
c2 = _run(repo, "rev-parse", "HEAD")
refs = git_refs.GitRefs(os.path.join(repo, ".git"))
assert refs.symref("refs/remotes/m/main") == f"refs/heads/{branch}"
assert refs.get("refs/remotes/m/main") == c2
_run(repo, "update-ref", "--no-deref", "refs/remotes/m/main", c1)
refs.UpdateRef("refs/remotes/m/main")
if reftable:
assert not refs.is_loaded
assert refs.symref("refs/remotes/m/main") == ""
assert refs.get("refs/remotes/m/main") == c1
else:
with mock.patch.object(
refs,
"_LoadAll",
side_effect=AssertionError("expected warm GitRefs snapshot"),
):
assert refs.symref("refs/remotes/m/main") == ""
assert refs.get("refs/remotes/m/main") == c1
# Setting GIT_COMMON_DIR invalidates instead of reading self._gitdir.
monkeypatch.setenv("GIT_COMMON_DIR", os.path.join(repo, ".git"))
assert refs.TryResolveRef("refs/remotes/m/main") is None
refs.UpdateRef("refs/remotes/m/main")
assert not refs.is_loaded
def test_symlinked_ref_fallback_reloads_and_breaks_cycles(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Symlink fallback reloads on target changes and breaks dir cycles."""
monkeypatch.delenv("GIT_DIR", raising=False)
monkeypatch.delenv("GIT_COMMON_DIR", raising=False)
repo = _init_repo(tmp_path, reftable=False)
c1 = _run(repo, "rev-parse", "HEAD")
Path(os.path.join(repo, "a")).write_text("2", encoding="utf-8")
_run(repo, "add", "a")
_run(repo, "commit", "-q", "-m", "second")
c2 = _run(repo, "rev-parse", "HEAD")
gitdir = os.path.join(repo, ".git")
ext_dir = tmp_path / "ext_refs"
ext_dir.mkdir()
target_ref = ext_dir / "linked"
target_ref.write_text(c1 + "\n", encoding="utf-8")
link_ref = os.path.join(gitdir, "refs", "heads", "linked")
loop_dir = os.path.join(gitdir, "refs", "heads", "loop")
refs_root = os.path.join(gitdir, "refs")
used_junction = False
try:
os.symlink(str(target_ref), link_ref)
os.symlink(refs_root, loop_dir, target_is_directory=True)
tracked_ref = "refs/heads/linked"
except OSError:
if not platform_utils.isWindows():
raise
used_junction = True
ext_link_dir = os.path.join(gitdir, "refs", "remotes", "ext")
os.makedirs(os.path.dirname(ext_link_dir), exist_ok=True)
subprocess.run(
["cmd", "/c", "mklink", "/J", ext_link_dir, str(ext_dir)],
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
subprocess.run(
["cmd", "/c", "mklink", "/J", loop_dir, refs_root],
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
tracked_ref = "refs/remotes/ext/linked"
try:
# Remove loop_dir before running git for-each-ref, then verify cycle
# handling directly on _TrackTreeMtimes so git itself does not walk
# deep cyclic paths.
if used_junction:
os.rmdir(loop_dir)
else:
os.unlink(loop_dir)
refs = git_refs.GitRefs(gitdir)
assert refs.get(tracked_ref) == c1
assert not refs.is_files_authoritative
replacement = ext_dir / "linked.new"
replacement.write_text(c2 + "\n", encoding="utf-8")
st = os.stat(target_ref)
os.replace(replacement, target_ref)
os.utime(target_ref, ns=(st.st_atime_ns, st.st_mtime_ns))
assert refs.get(tracked_ref) == c2
if used_junction:
subprocess.run(
["cmd", "/c", "mklink", "/J", loop_dir, refs_root],
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
else:
os.symlink(refs_root, loop_dir, target_is_directory=True)
refs._TrackTreeMtimes("refs")
assert os.path.join("refs", "heads", "loop") in refs._mtime
finally:
if used_junction:
for junc in (
loop_dir,
os.path.join(gitdir, "refs", "remotes", "ext"),
):
if os.path.lexists(junc):
os.rmdir(junc)
# Also exercise S_ISLNK target and dangling-link tracking directly so
# unprivileged Windows runs cover POSIX symlink stat transitions too.
lnk = stat.S_IFLNK | 0o777
reg = stat.S_IFREG | 0o644
fake_link = os.stat_result((lnk, 10, 1, 1, 0, 0, 8, 1, 1, 1))
fake_t1 = os.stat_result((reg, 20, 1, 1, 0, 0, 41, 1, 2, 1))
fake_t2 = os.stat_result((reg, 21, 1, 1, 0, 0, 41, 1, 2, 1))
with mock.patch("os.lstat", return_value=fake_link):
with mock.patch("os.stat", return_value=fake_t1):
s_t1 = git_refs.GitRefs._StatPath("sym")
with mock.patch("os.stat", return_value=fake_t2):
s_t2 = git_refs.GitRefs._StatPath("sym")
with mock.patch("os.stat", side_effect=FileNotFoundError):
s_dangling = git_refs.GitRefs._StatPath("sym")
assert s_t1 != s_t2
assert s_t1 != s_dangling
assert s_dangling is not None and not stat.S_ISDIR(s_dangling[0])
+19 -18
View File
@@ -22,8 +22,9 @@ import unittest
from unittest import mock from unittest import mock
import pytest import pytest
from test_manifest_xml import sort_attributes import test_manifest_xml
import git_command
import git_superproject import git_superproject
import git_trace2_event_log import git_trace2_event_log
import manifest_xml import manifest_xml
@@ -255,7 +256,7 @@ class SuperprojectTestCase(unittest.TestCase):
with open(manifest_path) as fp: with open(manifest_path) as fp:
manifest_xml_data = fp.read() manifest_xml_data = fp.read()
self.assertEqual( self.assertEqual(
sort_attributes(manifest_xml_data), test_manifest_xml.sort_attributes(manifest_xml_data),
'<?xml version="1.0" ?><manifest>' '<?xml version="1.0" ?><manifest>'
'<remote fetch="http://localhost" name="default-remote"/>' '<remote fetch="http://localhost" name="default-remote"/>'
'<default remote="default-remote" revision="refs/heads/main"/>' '<default remote="default-remote" revision="refs/heads/main"/>'
@@ -290,7 +291,7 @@ class SuperprojectTestCase(unittest.TestCase):
with open(update_result.manifest_path) as fp: with open(update_result.manifest_path) as fp:
manifest_xml_data = fp.read() manifest_xml_data = fp.read()
self.assertEqual( self.assertEqual(
sort_attributes(manifest_xml_data), test_manifest_xml.sort_attributes(manifest_xml_data),
'<?xml version="1.0" ?><manifest>' '<?xml version="1.0" ?><manifest>'
'<remote fetch="http://localhost" name="default-remote"/>' '<remote fetch="http://localhost" name="default-remote"/>'
'<default remote="default-remote" revision="refs/heads/main"/>' '<default remote="default-remote" revision="refs/heads/main"/>'
@@ -317,7 +318,7 @@ class SuperprojectTestCase(unittest.TestCase):
self.maxDiff = None self.maxDiff = None
self.assertIsNone(manifest.superproject) self.assertIsNone(manifest.superproject)
self.assertEqual( self.assertEqual(
sort_attributes(manifest.ToXml().toxml()), test_manifest_xml.sort_attributes(manifest.ToXml().toxml()),
'<?xml version="1.0" ?><manifest>' '<?xml version="1.0" ?><manifest>'
'<remote fetch="http://localhost" name="default-remote"/>' '<remote fetch="http://localhost" name="default-remote"/>'
'<default remote="default-remote" revision="refs/heads/main"/>' '<default remote="default-remote" revision="refs/heads/main"/>'
@@ -379,7 +380,7 @@ class SuperprojectTestCase(unittest.TestCase):
# Verify platform/vendor/x's project revision hasn't # Verify platform/vendor/x's project revision hasn't
# changed. # changed.
self.assertEqual( self.assertEqual(
sort_attributes(manifest_xml_data), test_manifest_xml.sort_attributes(manifest_xml_data),
'<?xml version="1.0" ?><manifest>' '<?xml version="1.0" ?><manifest>'
'<remote fetch="http://localhost" name="default-remote"/>' '<remote fetch="http://localhost" name="default-remote"/>'
'<remote fetch="http://localhost2" name="goog"/>' '<remote fetch="http://localhost2" name="goog"/>'
@@ -444,7 +445,7 @@ class SuperprojectTestCase(unittest.TestCase):
# Verify platform/vendor/x's project revision hasn't # Verify platform/vendor/x's project revision hasn't
# changed. # changed.
self.assertEqual( self.assertEqual(
sort_attributes(manifest_xml_data), test_manifest_xml.sort_attributes(manifest_xml_data),
'<?xml version="1.0" ?><manifest>' '<?xml version="1.0" ?><manifest>'
'<remote fetch="http://localhost" name="default-remote"/>' '<remote fetch="http://localhost" name="default-remote"/>'
'<default remote="default-remote" revision="refs/heads/main"/>' '<default remote="default-remote" revision="refs/heads/main"/>'
@@ -464,8 +465,8 @@ class SuperprojectTestCase(unittest.TestCase):
def test_Init_success(self): def test_Init_success(self):
"""Test _Init succeeds and creates the work git dir.""" """Test _Init succeeds and creates the work git dir."""
self.assertFalse(os.path.exists(self._superproject._work_git)) self.assertFalse(os.path.exists(self._superproject._work_git))
with mock.patch( with mock.patch.object(
"git_command.GitCommand", autospec=True git_command, "GitCommand", autospec=True
) as mock_git_command: ) as mock_git_command:
instance = mock_git_command.return_value instance = mock_git_command.return_value
instance.Wait.return_value = 0 instance.Wait.return_value = 0
@@ -491,16 +492,16 @@ class SuperprojectTestCase(unittest.TestCase):
"""Test _Init returns early if the work git dir already exists.""" """Test _Init returns early if the work git dir already exists."""
os.mkdir(self._superproject._superproject_path) os.mkdir(self._superproject._superproject_path)
os.mkdir(self._superproject._work_git) os.mkdir(self._superproject._work_git)
with mock.patch( with mock.patch.object(
"git_command.GitCommand", autospec=True git_command, "GitCommand", autospec=True
) as mock_git_command: ) as mock_git_command:
self.assertTrue(self._superproject._Init()) self.assertTrue(self._superproject._Init())
mock_git_command.assert_not_called() mock_git_command.assert_not_called()
def test_Init_failure_cleans_up(self): def test_Init_failure_cleans_up(self):
"""Test _Init cleans up the temporary directory if 'git init' fails.""" """Test _Init cleans up the temporary directory if 'git init' fails."""
with mock.patch( with mock.patch.object(
"git_command.GitCommand", autospec=True git_command, "GitCommand", autospec=True
) as mock_git_command: ) as mock_git_command:
instance = mock_git_command.return_value instance = mock_git_command.return_value
instance.Wait.return_value = 1 instance.Wait.return_value = 1
@@ -539,8 +540,8 @@ class SuperprojectTestCase(unittest.TestCase):
os.mkdir(self._superproject._superproject_path) os.mkdir(self._superproject._superproject_path)
os.mkdir(self._superproject._work_git) os.mkdir(self._superproject._work_git)
with mock.patch.object(self._superproject, "_Init", return_value=True): with mock.patch.object(self._superproject, "_Init", return_value=True):
with mock.patch( with mock.patch.object(
"git_command.GitCommand", autospec=True git_command, "GitCommand", autospec=True
) as mock_git_command: ) as mock_git_command:
with mock.patch.object( with mock.patch.object(
self._superproject, "_GetRef" self._superproject, "_GetRef"
@@ -599,12 +600,12 @@ class SuperprojectTestCase(unittest.TestCase):
def test_GetRef_resolves_only_the_requested_ref(self) -> None: def test_GetRef_resolves_only_the_requested_ref(self) -> None:
command = mock.MagicMock(stdout="1234\n") command = mock.MagicMock(stdout="1234\n")
command.Wait.return_value = 0 command.Wait.return_value = 0
with mock.patch( with mock.patch.object(
"git_command.GitCommand", return_value=command git_command, "GitCommand", return_value=command
) as git_command: ) as mock_git_command:
self.assertEqual("1234", self._superproject._GetRef("HEAD")) self.assertEqual("1234", self._superproject._GetRef("HEAD"))
git_command.assert_called_once_with( mock_git_command.assert_called_once_with(
None, None,
["rev-parse", "--verify", "--quiet", "HEAD"], ["rev-parse", "--verify", "--quiet", "HEAD"],
gitdir=self._superproject._work_git, gitdir=self._superproject._work_git,
+10 -7
View File
@@ -437,7 +437,9 @@ class TestEventLogVerbose:
event_log.verbose = False event_log.verbose = False
with contextlib.redirect_stderr( with contextlib.redirect_stderr(
io.StringIO() io.StringIO()
) as mock_stderr, mock.patch("socket.socket", side_effect=OSError): ) as mock_stderr, mock.patch.object(
socket, "socket", side_effect=OSError
):
event_log.Write(path="af_unix:stream:/tmp/test_sock") event_log.Write(path="af_unix:stream:/tmp/test_sock")
assert mock_stderr.getvalue() == "" assert mock_stderr.getvalue() == ""
@@ -447,8 +449,8 @@ class TestEventLogVerbose:
event_log.verbose = True event_log.verbose = True
with contextlib.redirect_stderr( with contextlib.redirect_stderr(
io.StringIO() io.StringIO()
) as mock_stderr, mock.patch( ) as mock_stderr, mock.patch.object(
"socket.socket", side_effect=OSError("Mock error") socket, "socket", side_effect=OSError("Mock error")
): ):
event_log.Write(path="af_unix:stream:/tmp/test_sock") event_log.Write(path="af_unix:stream:/tmp/test_sock")
assert ( assert (
@@ -462,8 +464,8 @@ class TestEventLogVerbose:
event_log.verbose = False event_log.verbose = False
with contextlib.redirect_stderr( with contextlib.redirect_stderr(
io.StringIO() io.StringIO()
) as mock_stderr, mock.patch( ) as mock_stderr, mock.patch.object(
"tempfile.NamedTemporaryFile", side_effect=FileExistsError tempfile, "NamedTemporaryFile", side_effect=FileExistsError
): ):
event_log.Write(path="/tmp") event_log.Write(path="/tmp")
assert mock_stderr.getvalue() == "" assert mock_stderr.getvalue() == ""
@@ -474,8 +476,9 @@ class TestEventLogVerbose:
event_log.verbose = True event_log.verbose = True
with contextlib.redirect_stderr( with contextlib.redirect_stderr(
io.StringIO() io.StringIO()
) as mock_stderr, mock.patch( ) as mock_stderr, mock.patch.object(
"tempfile.NamedTemporaryFile", tempfile,
"NamedTemporaryFile",
side_effect=FileExistsError("Mock error"), side_effect=FileExistsError("Mock error"),
): ):
event_log.Write(path="/tmp") event_log.Write(path="/tmp")
+2 -2
View File
@@ -14,7 +14,7 @@
"""Unittests for the hooks.py module.""" """Unittests for the hooks.py module."""
from io import StringIO import io
from pathlib import Path from pathlib import Path
import sys import sys
@@ -82,7 +82,7 @@ def test_post_sync_argument_validation() -> None:
) )
old_stderr = sys.stderr old_stderr = sys.stderr
sys.stderr = StringIO() sys.stderr = io.StringIO()
try: try:
# Call with missing arg `sync_duration_seconds` # Call with missing arg `sync_duration_seconds`
+64 -10
View File
@@ -14,16 +14,21 @@
"""Tests for the main repo script and subcommand routing.""" """Tests for the main repo script and subcommand routing."""
import builtins
from pathlib import Path
import time
from unittest import mock from unittest import mock
import pytest import pytest
from main import _Repo import event_log
import main
import manifest_xml
@pytest.fixture(name="repo") @pytest.fixture(name="repo")
def fixture_repo(): def fixture_repo():
repo = _Repo("repodir") repo = main._Repo("repodir")
# Overriding the command list here ensures that we are only testing # Overriding the command list here ensures that we are only testing
# against a fixed set of commands, reducing fragility to new # against a fixed set of commands, reducing fragility to new
# subcommands being added to the main repo tool. # subcommands being added to the main repo tool.
@@ -36,7 +41,7 @@ def fixture_mock_config():
return mock.MagicMock() return mock.MagicMock()
@mock.patch("time.sleep") @mock.patch.object(time, "sleep")
def test_autocorrect_delay(mock_sleep, repo, mock_config): def test_autocorrect_delay(mock_sleep, repo, mock_config):
"""Test autocorrect with positive delay.""" """Test autocorrect with positive delay."""
mock_config.GetString.return_value = "10" mock_config.GetString.return_value = "10"
@@ -48,7 +53,7 @@ def test_autocorrect_delay(mock_sleep, repo, mock_config):
assert res == "start" assert res == "start"
@mock.patch("time.sleep") @mock.patch.object(time, "sleep")
def test_autocorrect_delay_one(mock_sleep, repo, mock_config): def test_autocorrect_delay_one(mock_sleep, repo, mock_config):
"""Test autocorrect with '1' (0.1s delay, not immediate).""" """Test autocorrect with '1' (0.1s delay, not immediate)."""
mock_config.GetString.return_value = "1" mock_config.GetString.return_value = "1"
@@ -59,7 +64,7 @@ def test_autocorrect_delay_one(mock_sleep, repo, mock_config):
assert res == "start" assert res == "start"
@mock.patch("time.sleep", side_effect=KeyboardInterrupt()) @mock.patch.object(time, "sleep", side_effect=KeyboardInterrupt())
def test_autocorrect_delay_interrupt(mock_sleep, repo, mock_config): def test_autocorrect_delay_interrupt(mock_sleep, repo, mock_config):
"""Test autocorrect handles KeyboardInterrupt during delay.""" """Test autocorrect handles KeyboardInterrupt during delay."""
mock_config.GetString.return_value = "10" mock_config.GetString.return_value = "10"
@@ -70,7 +75,7 @@ def test_autocorrect_delay_interrupt(mock_sleep, repo, mock_config):
assert res is None assert res is None
@mock.patch("time.sleep") @mock.patch.object(time, "sleep")
def test_autocorrect_immediate(mock_sleep, repo, mock_config): def test_autocorrect_immediate(mock_sleep, repo, mock_config):
"""Test autocorrect with immediate/negative delay.""" """Test autocorrect with immediate/negative delay."""
# Test numeric negative. # Test numeric negative.
@@ -123,7 +128,7 @@ def test_autocorrect_never(repo, mock_config):
assert res is None assert res is None
@mock.patch("builtins.input", return_value="y") @mock.patch.object(builtins, "input", return_value="y")
def test_autocorrect_prompt_yes(mock_input, repo, mock_config): def test_autocorrect_prompt_yes(mock_input, repo, mock_config):
"""Test autocorrect with prompt and user answers yes.""" """Test autocorrect with prompt and user answers yes."""
mock_config.GetString.return_value = "prompt" mock_config.GetString.return_value = "prompt"
@@ -133,7 +138,7 @@ def test_autocorrect_prompt_yes(mock_input, repo, mock_config):
assert res == "start" assert res == "start"
@mock.patch("builtins.input", return_value="n") @mock.patch.object(builtins, "input", return_value="n")
def test_autocorrect_prompt_no(mock_input, repo, mock_config): def test_autocorrect_prompt_no(mock_input, repo, mock_config):
"""Test autocorrect with prompt and user answers no.""" """Test autocorrect with prompt and user answers no."""
mock_config.GetString.return_value = "prompt" mock_config.GetString.return_value = "prompt"
@@ -143,7 +148,7 @@ def test_autocorrect_prompt_no(mock_input, repo, mock_config):
assert res is None assert res is None
@mock.patch("builtins.input", return_value="y") @mock.patch.object(builtins, "input", return_value="y")
def test_autocorrect_multiple_candidates(mock_input, repo, mock_config): def test_autocorrect_multiple_candidates(mock_input, repo, mock_config):
"""Test autocorrect with multiple matches forces a prompt.""" """Test autocorrect with multiple matches forces a prompt."""
mock_config.GetString.return_value = "10" # Normally just delay mock_config.GetString.return_value = "10" # Normally just delay
@@ -156,7 +161,7 @@ def test_autocorrect_multiple_candidates(mock_input, repo, mock_config):
assert res == "start" assert res == "start"
@mock.patch("builtins.input", side_effect=KeyboardInterrupt()) @mock.patch.object(builtins, "input", side_effect=KeyboardInterrupt())
def test_autocorrect_prompt_interrupt(mock_input, repo, mock_config): def test_autocorrect_prompt_interrupt(mock_input, repo, mock_config):
"""Test autocorrect with prompt and user interrupts.""" """Test autocorrect with prompt and user interrupts."""
mock_config.GetString.return_value = "prompt" mock_config.GetString.return_value = "prompt"
@@ -164,3 +169,52 @@ def test_autocorrect_prompt_interrupt(mock_input, repo, mock_config):
res = repo._autocorrect_command_name("tart", mock_config) res = repo._autocorrect_command_name("tart", mock_config)
assert res is None assert res is None
def test_event_log_disabled_skips_add_and_git_hash() -> None:
"""Test disabled EventLog returns None and skips GetCommitRevisionId."""
log = event_log.EventLog(enabled=False)
project = mock.MagicMock()
assert log.Add("sync", event_log.TASK_COMMAND, 1.0) is None
assert (
log.AddSync(project, event_log.TASK_SYNC_NETWORK, 1.0, 2.0, True)
is None
)
project.GetCommitRevisionId.assert_not_called()
@mock.patch.object(manifest_xml, "RepoClient")
def test_run_long_gates_event_log_on_flag(
mock_client_cls: mock.MagicMock, tmp_path: Path
) -> None:
"""Test _RunLong enables cmd.event_log only when --event-log is set."""
client = mock_client_cls.return_value
client.manifest.IsMirror = False
client.manifest.manifestProject.config.DumpConfigDict.return_value = {}
repo = main._Repo("repodir")
for log_path, expected in (
(None, False),
(str(tmp_path / "events.log"), True),
):
cmd = mock.MagicMock()
cmd.event_log = event_log.EventLog(enabled=False)
cmd.manifest = client.manifest
cmd.MULTI_MANIFEST_SUPPORT = True
cmd.OptionParser.parse_args.return_value = (mock.MagicMock(), [])
cmd.ReadEnvironmentOptions.side_effect = lambda copts: copts
cmd.Execute.return_value = 0
repo.commands = {"sync": mock.Mock(return_value=cmd)}
gopts = mock.MagicMock(
color=None,
submanifest_path="",
pager=False,
time=False,
event_log=log_path,
git_trace2_event_log=None,
)
assert repo._RunLong("sync", gopts, [], mock.MagicMock()) == 0
assert cmd.event_log.enabled is expected
assert bool(log_path and Path(log_path).exists()) is expected
+103 -1
View File
@@ -180,7 +180,9 @@ class TestValue:
"""Check utility parsing code.""" """Check utility parsing code."""
def _get_node(self, text: str) -> xml.dom.minidom.Element: def _get_node(self, text: str) -> xml.dom.minidom.Element:
return xml.dom.minidom.parseString(text).firstChild node = xml.dom.minidom.parseString(text).firstChild
assert isinstance(node, xml.dom.minidom.Element)
return node
def test_bool_default(self) -> None: def test_bool_default(self) -> None:
"""Check XmlBool default handling.""" """Check XmlBool default handling."""
@@ -933,6 +935,106 @@ class TestProjectElement:
with pytest.raises(error.ManifestInvalidPathError): with pytest.raises(error.ManifestInvalidPathError):
parse("ok", path) parse("ok", path)
def test_sparse_checkout(self, repo_client: RepoClient) -> None:
"""Check sparse-checkout settings."""
manifest = repo_client.get_xml_manifest(
"""
<manifest>
<remote name="test-remote" fetch="http://localhost" />
<default remote="test-remote" revision="refs/heads/main" />
<project name="test-name" path="test-path">
<sparse-path path="src/main" />
<sparse-path path="docs" />
</project>
<project name="no-sparse" path="no-sparse-path" />
</manifest>
"""
)
assert len(manifest.projects) == 2
result = {p.name: p for p in manifest.projects}
assert result["test-name"].sparse_paths == ["src/main", "docs"]
assert result["no-sparse"].sparse_paths == []
def test_sparse_checkout_bad_paths(self, repo_client: RepoClient) -> None:
"""Check <sparse-path> rejects paths that escape the checkout."""
def parse(path: str) -> None:
path = repo_client.encode_xml_attr(path)
manifest = repo_client.get_xml_manifest(
f"""
<manifest>
<remote name="test-remote" fetch="http://localhost" />
<default remote="test-remote" revision="refs/heads/main" />
<project name="test-name" path="test-path">
<sparse-path path="{path}" />
</project>
</manifest>
"""
)
# Force the manifest to be parsed.
manifest.ToXml()
# Verify the parser is valid by default to avoid buggy tests below.
parse("src/main")
# Sparse paths name directories, so a trailing slash is fine.
parse("src/main/")
# A missing path is rejected by a different codepath.
with pytest.raises(error.ManifestParseError):
parse("")
for path in INVALID_FS_PATHS:
if not path or path.endswith("/") or path.endswith(os.path.sep):
continue
with pytest.raises(error.ManifestInvalidPathError):
parse(path)
def test_sparse_checkout_extend_project(
self, repo_client: RepoClient
) -> None:
"""Check sparse-path elements added via extend-project."""
manifest = repo_client.get_xml_manifest(
"""
<manifest>
<remote name="test-remote" fetch="http://localhost" />
<default remote="test-remote" revision="refs/heads/main" />
<project name="test-name" path="test-path">
<sparse-path path="src/main" />
</project>
<extend-project name="test-name">
<sparse-path path="docs" />
</extend-project>
</manifest>
"""
)
assert len(manifest.projects) == 1
assert manifest.projects[0].sparse_paths == ["src/main", "docs"]
def test_sparse_checkout_roundtrip(self, repo_client: RepoClient) -> None:
"""Check sparse-path elements survive a manifest round-trip."""
manifest = repo_client.get_xml_manifest(
"""
<manifest>
<remote name="test-remote" fetch="http://localhost" />
<default remote="test-remote" revision="refs/heads/main" />
<project name="test-name" path="test-path">
<sparse-path path="src/main" />
<sparse-path path="docs" />
</project>
</manifest>
"""
)
roundtrip = repo_client.get_xml_manifest(manifest.ToXml().toxml())
assert roundtrip.projects[0].sparse_paths == ["src/main", "docs"]
assert manifest.ToDict()["project"][0]["sparse-path"] == [
{"path": "src/main"},
{"path": "docs"},
]
class TestSuperProjectElement: class TestSuperProjectElement:
"""Tests for <superproject>.""" """Tests for <superproject>."""
+1830 -90
View File
File diff suppressed because it is too large. Load diff
+2 -2
View File
@@ -16,7 +16,7 @@
import runpy import runpy
import sys import sys
from types import ModuleType import types
import pytest import pytest
import utils_for_test import utils_for_test
@@ -31,7 +31,7 @@ def test_wrapper_does_not_run_main_for_multiprocessing_child(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
"""Do not rerun main when multiprocessing re-executes the wrapper.""" """Do not rerun main when multiprocessing re-executes the wrapper."""
fake_update_manpages = ModuleType("update_manpages") fake_update_manpages = types.ModuleType("update_manpages")
# Mock this because the real module requires newer Python versions than # Mock this because the real module requires newer Python versions than
# our unittest framework does. # our unittest framework does.
+2 -2
View File
@@ -22,7 +22,7 @@ from unittest import mock
import pytest import pytest
from color import SetDefaultColoring import color
import error import error
import repo_logging import repo_logging
@@ -86,7 +86,7 @@ def test_log_with_format_string(level: int) -> None:
# Set color output to "always" for consistent test results. # Set color output to "always" for consistent test results.
# This ensures the logger's behavior is uniform across different # This ensures the logger's behavior is uniform across different
# environments and git configurations. # environments and git configurations.
SetDefaultColoring("always") color.SetDefaultColoring("always")
# Regex pattern to match optional ANSI color codes. # Regex pattern to match optional ANSI color codes.
# \033 - Escape character # \033 - Escape character
+1
View File
@@ -27,6 +27,7 @@ def test_trace_max_size_enforced(monkeypatch: pytest.MonkeyPatch) -> None:
# Enable trace for the test, in case users have it disabled. # Enable trace for the test, in case users have it disabled.
monkeypatch.setattr(repo_trace, "_TRACE", True) monkeypatch.setattr(repo_trace, "_TRACE", True)
assert repo_trace._TRACE_FILE is not None
with repo_trace.Trace(content, first_trace=True): with repo_trace.Trace(content, first_trace=True):
pass pass
+12 -6
View File
@@ -53,7 +53,9 @@ def test_parse_ssh_version(input_str: str, expected: Tuple[int, ...]) -> None:
def test_version() -> None: def test_version() -> None:
"""Check version() handling.""" """Check version() handling."""
with mock.patch("ssh._run_ssh_version", return_value="OpenSSH_1.2\n"): with mock.patch.object(
ssh, "_run_ssh_version", return_value="OpenSSH_1.2\n"
):
assert ssh.version() == (1, 2) assert ssh.version() == (1, 2)
@@ -67,7 +69,7 @@ def test_context_manager_empty() -> None:
def test_context_manager_child_cleanup() -> None: def test_context_manager_child_cleanup() -> None:
"""Verify orphaned clients & masters get cleaned up.""" """Verify orphaned clients & masters get cleaned up."""
with multiprocessing.Manager() as manager: with multiprocessing.Manager() as manager:
with mock.patch("ssh.version", return_value=(1, 2)): with mock.patch.object(ssh, "version", return_value=(1, 2)):
with ssh.ProxyManager(manager) as ssh_proxy: with ssh.ProxyManager(manager) as ssh_proxy:
client = subprocess.Popen(["sleep", "964853320"]) client = subprocess.Popen(["sleep", "964853320"])
ssh_proxy.add_client(client) ssh_proxy.add_client(client)
@@ -87,13 +89,17 @@ def test_ssh_sock(monkeypatch: pytest.MonkeyPatch) -> None:
) )
# Old ssh version uses port. # Old ssh version uses port.
with mock.patch("ssh.version", return_value=(6, 6)): with mock.patch.object(ssh, "version", return_value=(6, 6)):
with proxy as ssh_proxy: with proxy as ssh_proxy:
assert ssh_proxy.sock().endswith("%p") sock = ssh_proxy.sock()
assert sock is not None
assert sock.endswith("%p")
proxy._sock_path = None proxy._sock_path = None
# New ssh version uses hash. # New ssh version uses hash.
with mock.patch("ssh.version", return_value=(6, 7)): with mock.patch.object(ssh, "version", return_value=(6, 7)):
with proxy as ssh_proxy: with proxy as ssh_proxy:
assert ssh_proxy.sock().endswith("%C") sock = ssh_proxy.sock()
assert sock is not None
assert sock.endswith("%C")
proxy._sock_path = None proxy._sock_path = None
+7 -7
View File
@@ -19,8 +19,9 @@ from typing import Type
import pytest import pytest
from command import Command import command
import subcmds import subcmds
from subcmds import sync
# NB: We don't test all subcommands as we want to avoid "change detection" # NB: We don't test all subcommands as we want to avoid "change detection"
@@ -48,7 +49,7 @@ def test_naming(name: str) -> None:
@pytest.mark.parametrize("name, cls", subcmds.all_commands.items()) @pytest.mark.parametrize("name, cls", subcmds.all_commands.items())
def test_help_desc_style(name: str, cls: Type[Command]) -> None: def test_help_desc_style(name: str, cls: Type[command.Command]) -> None:
"""Force some consistency in option descriptions. """Force some consistency in option descriptions.
Python's optparse & argparse has a few default options like --help. Python's optparse & argparse has a few default options like --help.
@@ -78,7 +79,7 @@ def test_help_desc_style(name: str, cls: Type[Command]) -> None:
@pytest.mark.parametrize("name, cls", subcmds.all_commands.items()) @pytest.mark.parametrize("name, cls", subcmds.all_commands.items())
def test_cli_option_style(name: str, cls: Type[Command]) -> None: def test_cli_option_style(name: str, cls: Type[command.Command]) -> None:
"""Force some consistency in option flags.""" """Force some consistency in option flags."""
cmd = cls() cmd = cls()
parser = cmd.OptionParser parser = cmd.OptionParser
@@ -111,6 +112,7 @@ def test_cli_option_dest() -> None:
bad_opts.append((str(opt), opt.dest)) bad_opts.append((str(opt), opt.dest))
# Hook the option check list. # Hook the option check list.
assert optparse.Option.CHECK_METHODS is not None
optparse.Option.CHECK_METHODS.insert(0, _check_dest) optparse.Option.CHECK_METHODS.insert(0, _check_dest)
try: try:
@@ -141,7 +143,7 @@ def test_cli_option_dest() -> None:
@pytest.mark.parametrize("name, cls", subcmds.all_commands.items()) @pytest.mark.parametrize("name, cls", subcmds.all_commands.items())
def test_common_validate_options(name: str, cls: Type[Command]) -> None: def test_common_validate_options(name: str, cls: Type[command.Command]) -> None:
"""Verify CommonValidateOptions sets up expected fields.""" """Verify CommonValidateOptions sets up expected fields."""
cmd = cls() cmd = cls()
opts, args = cmd.OptionParser.parse_args([]) opts, args = cmd.OptionParser.parse_args([])
@@ -164,9 +166,7 @@ def test_common_validate_options(name: str, cls: Type[Command]) -> None:
def test_attribute_error_repro() -> None: def test_attribute_error_repro() -> None:
"""Confirm that accessing verbose before CommonValidateOptions fails.""" """Confirm that accessing verbose before CommonValidateOptions fails."""
from subcmds.sync import Sync cmd = sync.Sync()
cmd = Sync()
opts, args = cmd.OptionParser.parse_args([]) opts, args = cmd.OptionParser.parse_args([])
# This confirms that without the fix in main.py, an AttributeError # This confirms that without the fix in main.py, an AttributeError
+23 -1
View File
@@ -14,7 +14,7 @@
"""Unittests for the subcmds/init.py module.""" """Unittests for the subcmds/init.py module."""
from typing import List from typing import List, Optional
import pytest import pytest
@@ -29,6 +29,8 @@ from subcmds import init
["--smart-sync"], ["--smart-sync"],
["--no-smart-sync"], ["--no-smart-sync"],
["--head"], ["--head"],
["--smart-sync-branch=git_main-warmup"],
["--smart-sync-branch="],
), ),
) )
def test_cli_parser_good(argv: List[str]) -> None: def test_cli_parser_good(argv: List[str]) -> None:
@@ -38,6 +40,24 @@ def test_cli_parser_good(argv: List[str]) -> None:
cmd.ValidateOptions(opts, args) cmd.ValidateOptions(opts, args)
@pytest.mark.parametrize(
"argv,expected",
(
([], None),
(["--smart-sync-branch=git_main-warmup"], "git_main-warmup"),
(["--smart-sync-branch="], ""),
),
)
def test_cli_smart_sync_branch_value(
argv: List[str], expected: Optional[str]
) -> None:
"""Check --smart-sync-branch parsed values."""
cmd = init.Init()
opts, args = cmd.OptionParser.parse_args(argv)
cmd.ValidateOptions(opts, args)
assert opts.smart_sync_branch == expected
@pytest.mark.parametrize( @pytest.mark.parametrize(
"argv", "argv",
( (
@@ -45,6 +65,8 @@ def test_cli_parser_good(argv: List[str]) -> None:
["url", "asdf"], ["url", "asdf"],
# Conflicting options. # Conflicting options.
["--mirror", "--archive"], ["--mirror", "--archive"],
["--standalone-manifest", "--smart-sync-branch=git_main-warmup"],
["--standalone-manifest", "--smart-sync-branch="],
), ),
) )
def test_cli_parser_bad(argv: List[str]) -> None: def test_cli_parser_bad(argv: List[str]) -> None:
Loaded 100 of 107 files, more files were not shown because too many files have changed in this diff. Show more