During _CustomFetch, GetEnvVars resolves REPO_LREV via GetRevisionId
before fetchcmd runs, and postcondition checks afterward run ResolveRef
on the tracking ref and FETCH_HEAD. On cold clones, missing branches, or
packed-refs tracking refs, GetRevisionId fell back to ResolveCommit,
spawning `git rev-parse <rev>^{commit}` and peeling objects from lazily
populated object stores such as ABFS.
Resolve non-tag refs/* in GetRevisionId via GitRefs.TryResolveRef and
authoritative files snapshots so loose, packed, and missing refs resolve
without spawning git or reading .git/objects. In _CustomFetch,
invalidate bare_ref after fetchcmd runs, resolve tracking refs through
bare_ref to warm the snapshot for _InitMRef and Sync_LocalHalf, and read
FETCH_HEAD's leading object ID directly when unambiguous.
Bug: 543101585
Change-Id: Ifb5c0f182e539a2836672a47da348fbeb41e5068
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655964
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
When creating a gitdir, _InitGitDir spawned `git config --unset-all
core.bare` for every non-mirror project. That is needed when `git init`
wrote `core.bare = true`, as it does for a gitdir under
.repo/project-objects. For a local worktree gitdir (<worktree>/.git, as
with --use-local-gitdirs) `git init` already writes `core.bare = false`,
so the spawn only removed a setting that was already correct. Checked
on git 1.9.1 (the oldest version repo supports), 2.7.4, 2.20.1 and
2.56.
Unset core.bare on non-mirror repositories only when it is currently
true. Local worktree gitdirs now keep the `core.bare = false` that
`git init` wrote.
Bug: 543101585
Change-Id: I662ffce61dc68ebca921941c1a5e5ebce2d22338
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655965
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
This change attempts to skip sequential git config calls in order to
further speed up git-repo's performance for VFS integration. On
git-main we currently call git config 26K+ times, causing roughly 5~6
seconds of latency (see b/543101585#comment23's TRACE_FILE for data).
Given the expected time of VFS integration is targeting 30s, 6 seconds
makes a huge difference.
During an initial `repo sync`, initializing each project (`_InitGitDir`
and `_InitRemote`) invokes `SetString` multiple times per project, and
`SyncAnalysisState` writes `repo.syncstate.*` keys once per sync.
Because the `git config` CLI has no batch-write mode to set multiple
keys in a single invocation, each `SetString` call previously spawned a
separate `git config` subprocess and rewrote `.repo_config.json` per key
(see b/543101585#comment23 and b/543101585#comment25 for trace profiles
and benchmark data).
Bug: 543101585
Bug: 564817989
Change-Id: Ia1d4613266fb6b7742e8db637039e72ed26a13a2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/657083
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
When initializing missing project object directories in parallel, each
sync worker cold-misses the init.templateDir lookup cache and spawns
`git config --get init.templatedir`. Relying on parent process memory
alone also misses workers started via `spawn` or `forkserver`.
Resolve the init.templateDir lookup once in the parent process when
parallel sync workers may initialize an object directory (either
because an objdir is missing or `--force-sync` is active), pass the
cached entry through `parallel_context`, and seed it in `InitWorker`.
Bug: 543101585
Change-Id: I6bac2352a9fd4b62d25ed2a89b670ac21aa70f4e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/657084
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Only set preciousObjects when objdir differs from gitdir, check actual
objdir sharing in _GetPreciousObjectsState, and skip post-sync
preciousObjects scans on unshared --use-local-gitdirs projects.
Bug: 543101585
Change-Id: I5077aed128f22cf19f520c7866cfd62f0c2724be
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/652266
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
This suppresses a lot of findings in most files so we can run pyrefly
by default and prevent new findings from showing up. This will need
time to slowly chip away file by file.
Concretely, it's enabled for ~60% of files, and disabled for the other
~40% files. Not the worst start.
I've opted for disabling files completely at this point rather than
using a baseline because the generated baseline is >200KiB in size.
I think we need to get through a good chunk of that before we switch.
We're going with pyrefly because it's what Google has adopted, and it
evaluated alternatives including mypy, pyright, and its own pytype.
Change-Id: Ib73752dd240a9a84ce0f56059a82292141b6af66
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656364
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The base Project class defined HasChanges as a function, but the
MetaProject & RepoProject subclasses defined it as a property.
Obviously this is bad & confusing, so standardize it.
Change-Id: Ia3ac9d0592307366296b8ef991dc5d7bb2204490
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656723
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
By default, pytest keeps the temp dirs of every test for the last 3
sessions. Each e2e test clones & checks out a snapshot of the tree
under its `tmp_path`, so these add up, and /tmp is often a tmpfs.
Set `tmp_path_retention_policy` to `failed` so a test's `tmp_path` is
removed as soon as the test passes, and the session's temp dir is
removed when all tests pass. The `tmp_path` of a failed test is still
kept for debugging. Pass `-o tmp_path_retention_policy=all` to keep
the dirs of passing tests too.
Test: vpython3 run_tests
Change-Id: I2cf464b063d62f155ffdbfed24a12ea9774086d6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656093
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
GitRefs._LoadAll spawns `git for-each-ref` to load HEAD and refs/*, and
Project._InitAnyMRef leaves warm snapshots stale after direct UpdateRef
writes. On virtualized checkouts with thousands of repositories, these
spawns add avoidable process overhead.
Read HEAD, packed-refs, and loose refs directly in GitRefs when the
repository uses a plain files ref backend, recording (mode, mtime, size,
inode) stat tuples during the single directory pass and rechecking
packed-refs afterward to guard against concurrent pack-refs updates.
The tuples leave out ctime, which backup agents and file system crawlers
move without changing a ref. Fall back to `git for-each-ref` when
GIT_DIR, GIT_COMMON_DIR, reftable, commondir, custom refStorage config,
symlinks, or lockfiles are present.
Also add GitRefs.TryResolveRef for single-ref lookups without scanning
all of refs/, GitRefs.Invalidate, and GitRefs.UpdateRef to keep cached
snapshots warm after _InitAnyMRef writes a direct ref.
Bug: 543101585
Change-Id: I804ac45712208e9f9e6437012661bc1de58b9825
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655963
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
If the manifest checkout in `.repo/manifests` is on a detached HEAD, or
on a local branch without an upstream, `Sync._GetBranch` read a missing
`merge` config and crashed with `'NoneType' object has no attribute
'startswith'`. When smart sync is enabled by default, this broke plain
`repo sync` for such checkouts because `AttributeError` bypassed the
fallback to a regular ToT sync in `_ExecuteHelper`, which only catches
`SmartSyncError` and `ManifestParseError`. Pinning the manifest via
`repo init -b <sha>` or `repo init -b refs/tags/<tag>` also sent the
raw revision to the manifest server as a branch name.
Add `ManifestProject.GetMergeBranch(heads_only=...)` (shared with
`XmlManifest._Load`) to resolve the upstream ref of the current
manifest branch, returning `None` under `heads_only=True` unless it
tracks `refs/heads/*`, and raise `SmartSyncError` in `_SmartSyncSetup`
before running the remote helper or contacting the manifest server.
Implicit smart sync now falls back to a regular ToT sync, and explicit
smart sync fails early with an actionable error message.
Bug: 580004259
Test: vpython3 run_tests
Change-Id: Ice1f9bfe7ea8c923a283bcbd0ac7576cf33e42f6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655123
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
The testsuite has grown in size quite a bit. It takes ~3 min on
my cloudtop with ~48 cores. Running it in parallel makes it take
~20 seconds.
I've capped it to 32 jobs because it seems to complete in the same
amount of time as 128 jobs, and there's startup cost to spinning
up each node.
Change-Id: I6daa456a92e9db213118c9d16c9dee646db448e0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655887
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Since vpython no longer does this automatically, I upgraded flake8
incorrectly by only updating the script file. Add a note to remind
people they need to regen the lock file themselves.
Change-Id: I7884714a110bbf843c76aca6e3a2ff14b932b943
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655845
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
The flake8 version hasn't updated properly due the uv.lock file
being out of date. Fix newer warnings before unwedging that.
We have a lot of B042 errors that probably should be fixed by reworking
the code, but for now suppress them since we've been doing this for a
long time in repo.
error.py:25:5: B042 Exception class with `__init__` should pass all args to `super().__init__()` to work in edge cases of `pickle` and `copy.copy()`. It should also not take any kwargs.
The only other warning is fairly simple:
subcmds/sync.py:1154:9: B018 Found useless Call expression. Consider either assigning it to a variable or removing it.
Change-Id: I532000a66a5d8493a490fa30080b137aff5f706e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655844
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
RepoSourceVersion() and _build_env() are module-level functions that
called GitCommand._GetBasicEnv(), while GitCommand itself never called
it directly. When tests mock git_command.GitCommand before
RepoSourceVersion() has cached its result (such as running
test_upload_for_review_real_event_log in isolation), calling
GitCommand._GetBasicEnv() returned a MagicMock instead of an env dict,
causing subprocess.run to fail with ValueError.
Change-Id: Ib8b866b012602c549ce2d59105526d847b0d7d51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656383
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
In `_SyncInterleaved`, `_UpdateManifestLists` only ran after fetching
and checking out all projects. When a manifest replaced a parent project
`foo` with a nested child project `foo/bar`, `foo/bar` was checked out
inside the still-existing obsolete `foo` worktree before
`UpdateProjectList` deleted `foo`. Because `bar/` was tracked by `foo`,
`git status` in `foo` saw `foo/bar`'s newly checked-out files as
modified and untracked files belonging to `foo`, causing
`DeleteWorktree` to fail with `DeleteDirtyWorktreeError` (and similarly,
when `foo/bar` was merged back into `foo`, `foo`'s checkout collided
with the not-yet-deleted `foo/bar` worktree):
error: foo: Cannot remove project: uncommitted changes are present.
error: Unable to fully sync the tree
error: Updating local project lists failed.
Split `_UpdateManifestLists` into `_UpdateProjectLists` and
`_UpdateCopyLinkfileLists`, and run `_UpdateProjectLists` before the
sync loop in `_SyncInterleaved` so obsolete manifest projects are
deleted before new project checkouts begin (and `--fail-fast` aborts
before fetching if an obsolete project cannot be removed). Run
`_UpdateProjectLists` again after the sync loop (passing the set of
paths that already failed removal pre-sync so they are not retried or
reported twice) together with `_UpdateCopyLinkfileLists` so derived
submodules discovered or dropped when `_ReloadManifest` runs during the
loop are updated in `project.list` and copyfiles/linkfiles are updated.
Because the pre-sync `_UpdateProjectLists` pass in `_SyncInterleaved`
runs before projects are fetched (unlike `_SyncPhased`, which runs it
after `_FetchMain`), a `sync-s="true"` parent project whose manifest
revision moved to an unfetched commit cannot derive its submodules yet
(`GetSubmoduleRevisions() is None`). Retain existing submodule checkouts
under such parents until a post-fetch pass can determine whether their
submodules were removed. Also continue past individual
`DeleteWorktreeError` failures in `UpdateProjectList` (skipping obsolete
ancestors of any retained child worktree) so remaining clean obsolete
projects are still removed while failed paths stay tracked in
`project.list`, and skip project list and copy/linkfile updates when
`--network-only` is set so `-n` does not mutate local worktrees.
Bug: 574048485
Test: vpython3 run_tests
Change-Id: I530337977f063173a9a8a48df3250627cd4bc57f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656283
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Align with Google style guides and `import command` as a module
rather than import symbols out of it.
The help subcmd needed a minor rewrite as it had a "command"
variable already.
Change-Id: I9b465c85246670d63827df0744b00e59033e87b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656243
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
The Popen mocking is incomplete & buggy:
* It uses a nested class which hinders pickling.
* Nested classes should rarely be used in the first place, stylewise.
* It does not mock the full Popen API causing failures.
For example, this fails:
$ pytest -vvv tests/test_git_command.py::GitCommandWaitTest
...
FAILED tests/test_git_command.py::GitCommandWaitTest::test_default_returns_non_zero_result - AttributeError: 'MockPopen' object has no attribute 'returncode'
FAILED tests/test_git_command.py::GitCommandWaitTest::test_raises_when_verify_non_zero_result - AttributeError: 'MockPopen' object has no attribute 'returncode'
FAILED tests/test_git_command.py::GitCommandWaitTest::test_returns_when_no_verify_non_zero_result - AttributeError: 'MockPopen' object has no attribute 'returncode'
...
Delete the tests since they don't seem to add much value currently.
Bug: b/293344017
Change-Id: I3d613a84657e30cbc7eac0b363ebfcb46b95eb49
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655886
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
After repo.reprojectcmd exits 0, _Reproject ran `git diff-index --cached
--name-only <revid>^{tree}` to verify that the index matches the target
commit's tree.
Peeling <revid>^{tree} and comparing every entry against the index
forces Git to read the target commit and walk every tree object in the
target revision from .git/objects.
reprojectcmd exists for virtualized filesystems such as ABFS whose
checkout command materializes the worktree and writes .git/index from
its own tree metadata without reading .git/objects/pack/*-trees-*.pack.
Running diff-index against <revid>^{tree} afterward forces Git to fault
in the tree packfiles over the network (260 FetchTrees pack downloads /
1.64 GiB across a 3,296-project Android checkout).
Trust reprojectcmd's exit status for index materialization instead of
re-walking the target tree in Git, while keeping the HEAD postcondition
check that verifies the command did not modify refs.
Bug: 543101585
Change-Id: Ie5feea41a4191245232c6763a489d01596370c55
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643004
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
After repo.fetchcmd exits, _CustomFetch verified the tracking ref and
FETCH_HEAD with ResolveCommit, i.e. `git rev-parse --verify
<ref>^{commit}`. Peeling to a commit makes Git look the object up: it
opens every pack index in objects/pack/ until it finds the commit, then
reads the commit from its packfile.
fetchcmd exists for object stores that are populated lazily, such as
ABFS, whose `abfs git fetch` registers packfiles as virtual files and
returns at once. There the peel is the first read of the packs the
command registered, and it blocks until they are downloaded. A project
whose revision did not change needs no objects to sync, yet paid that
download whenever fetchcmd registered new packs for it.
Compare the object IDs that the refs record instead (ResolveRef, which
runs the same rev-parse without the peel). Only when they differ peel
both sides to commits, so an annotated tag still matches the commit it
points to. A commit missing from the object store now fails the checkout
that reads it instead of the network half.
On a first sync the checkout reads the commit anyway, so there the
download only moves to the next command that needs the object.
Bug: 543101585
Change-Id: I51248ff1e9674b16e3ff8cee69572dbc8c7a8e99
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643003
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Skip projects fetched via repo.fetchcmd in _CheckForBloatedProjects,
and expose Project.UseFetchCmd alongside Project.UseReprojectCmd.
The bloat check exists because Project._RemoteFetch passes --no-auto-gc
for clone-depth="1" repositories. Projects using repo.fetchcmd bypass
_RemoteFetch and manage object storage externally, so scanning their
worktrees for bloat is unnecessary and recommending git repack can
materialize virtualized objects locally.
Bug: 543101585
Change-Id: I16510ad3b5b39c3a5e2a1a75df706439252a088e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655083
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Sparse checkout allows Git to check out only a portion of the
repository. This is particularly helpful for monorepository workflows
where CI jobs only need a portion of the repository to run tests.
This speeds up the CI jobs, and puts less load on the Git server as it
relieves it from having to pack up all the objects.
In the Git project, though it is still labeled experimental, the feature
has been stable for a number of years and is unlikely to change at this
point.
This change allows configuring Git to use sparse checkout and provide a
list of directories for the sync command to retrieve.
Projects with sparse paths default to a blob:none partial clone so the
excluded paths are never fetched, unless a filter is already configured
or the project is excluded from partial clones. Both the plain and the
git-worktree checkout flows configure sparse-checkout before populating
the working tree.
Change-Id: I772c7db5209b345306be378771c8f2040832bca1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/535061
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: John Cai <johncai86@gmail.com>
Commit-Queue: John Cai <johncai86@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Import modules at the top of the module in most cases. Many of these
weren't actually saving any time:
* editor: shlex is already imported by the main module
* platform_utils: On Windows, the win32 module is almost always needed
* tests: There's no point in optimizing these.
The trace import in the main module is about the only one worth allowing
since that module itself is rarely used, and only when in a specific
debug mode.
Change-Id: I5d3fcecbe51e0ce78d4e856d7033565150ef4526
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654103
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Add an enabled flag to EventLog (defaulting to False on
Command.event_log and enabled by _Repo._RunLong when --event-log is
passed) and return None early from EventLog.Add when disabled.
Because EventLog.AddSync only resolves project.GetCommitRevisionId()
when EventLog.Add returns an event, disabling Command.event_log by
default eliminates 3,300+ synchronous git subprocess calls in
MainProcess result callbacks during normal repo sync runs without
touching subcmds/sync.py.
Bug: 564817989
Change-Id: I79dd66d1c392ba5ee562f104759e36d1dd6eefea
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649606
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Read plain files-backend unborn HEAD symrefs directly before spawning
git symbolic-ref, update already-loaded bare_ref snapshots in
_InitAnyMRef when no external ref changes occurred, and reuse warm
bare_ref snapshots in GetRevisionId for non-tag refs.
Also skip redundant rev-parse and empty-index ls-files subprocesses in
_Reproject when HEAD is guaranteed to be an unborn local ref.
Bug: 564817989
Change-Id: I0da10cefb14753ce6e1e22e192d652790d59a5b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649605
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Gavin Mak
Hung-Wei Chen
Mike Frysinger
Brian Gan
John Cai