Align with Google style guides and `import command` as a module
rather than import symbols out of it.
The help subcmd needed a minor rewrite as it had a "command"
variable already.
Change-Id: I9b465c85246670d63827df0744b00e59033e87b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/656243
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
The Popen mocking is incomplete & buggy:
* It uses a nested class which hinders pickling.
* Nested classes should rarely be used in the first place, stylewise.
* It does not mock the full Popen API causing failures.
For example, this fails:
$ pytest -vvv tests/test_git_command.py::GitCommandWaitTest
...
FAILED tests/test_git_command.py::GitCommandWaitTest::test_default_returns_non_zero_result - AttributeError: 'MockPopen' object has no attribute 'returncode'
FAILED tests/test_git_command.py::GitCommandWaitTest::test_raises_when_verify_non_zero_result - AttributeError: 'MockPopen' object has no attribute 'returncode'
FAILED tests/test_git_command.py::GitCommandWaitTest::test_returns_when_no_verify_non_zero_result - AttributeError: 'MockPopen' object has no attribute 'returncode'
...
Delete the tests since they don't seem to add much value currently.
Bug: b/293344017
Change-Id: I3d613a84657e30cbc7eac0b363ebfcb46b95eb49
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655886
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
After repo.reprojectcmd exits 0, _Reproject ran `git diff-index --cached
--name-only <revid>^{tree}` to verify that the index matches the target
commit's tree.
Peeling <revid>^{tree} and comparing every entry against the index
forces Git to read the target commit and walk every tree object in the
target revision from .git/objects.
reprojectcmd exists for virtualized filesystems such as ABFS whose
checkout command materializes the worktree and writes .git/index from
its own tree metadata without reading .git/objects/pack/*-trees-*.pack.
Running diff-index against <revid>^{tree} afterward forces Git to fault
in the tree packfiles over the network (260 FetchTrees pack downloads /
1.64 GiB across a 3,296-project Android checkout).
Trust reprojectcmd's exit status for index materialization instead of
re-walking the target tree in Git, while keeping the HEAD postcondition
check that verifies the command did not modify refs.
Bug: 543101585
Change-Id: Ie5feea41a4191245232c6763a489d01596370c55
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643004
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
After repo.fetchcmd exits, _CustomFetch verified the tracking ref and
FETCH_HEAD with ResolveCommit, i.e. `git rev-parse --verify
<ref>^{commit}`. Peeling to a commit makes Git look the object up: it
opens every pack index in objects/pack/ until it finds the commit, then
reads the commit from its packfile.
fetchcmd exists for object stores that are populated lazily, such as
ABFS, whose `abfs git fetch` registers packfiles as virtual files and
returns at once. There the peel is the first read of the packs the
command registered, and it blocks until they are downloaded. A project
whose revision did not change needs no objects to sync, yet paid that
download whenever fetchcmd registered new packs for it.
Compare the object IDs that the refs record instead (ResolveRef, which
runs the same rev-parse without the peel). Only when they differ peel
both sides to commits, so an annotated tag still matches the commit it
points to. A commit missing from the object store now fails the checkout
that reads it instead of the network half.
On a first sync the checkout reads the commit anyway, so there the
download only moves to the next command that needs the object.
Bug: 543101585
Change-Id: I51248ff1e9674b16e3ff8cee69572dbc8c7a8e99
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643003
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Skip projects fetched via repo.fetchcmd in _CheckForBloatedProjects,
and expose Project.UseFetchCmd alongside Project.UseReprojectCmd.
The bloat check exists because Project._RemoteFetch passes --no-auto-gc
for clone-depth="1" repositories. Projects using repo.fetchcmd bypass
_RemoteFetch and manage object storage externally, so scanning their
worktrees for bloat is unnecessary and recommending git repack can
materialize virtualized objects locally.
Bug: 543101585
Change-Id: I16510ad3b5b39c3a5e2a1a75df706439252a088e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655083
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Sparse checkout allows Git to check out only a portion of the
repository. This is particularly helpful for monorepository workflows
where CI jobs only need a portion of the repository to run tests.
This speeds up the CI jobs, and puts less load on the Git server as it
relieves it from having to pack up all the objects.
In the Git project, though it is still labeled experimental, the feature
has been stable for a number of years and is unlikely to change at this
point.
This change allows configuring Git to use sparse checkout and provide a
list of directories for the sync command to retrieve.
Projects with sparse paths default to a blob:none partial clone so the
excluded paths are never fetched, unless a filter is already configured
or the project is excluded from partial clones. Both the plain and the
git-worktree checkout flows configure sparse-checkout before populating
the working tree.
Change-Id: I772c7db5209b345306be378771c8f2040832bca1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/535061
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: John Cai <johncai86@gmail.com>
Commit-Queue: John Cai <johncai86@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Import modules at the top of the module in most cases. Many of these
weren't actually saving any time:
* editor: shlex is already imported by the main module
* platform_utils: On Windows, the win32 module is almost always needed
* tests: There's no point in optimizing these.
The trace import in the main module is about the only one worth allowing
since that module itself is rarely used, and only when in a specific
debug mode.
Change-Id: I5d3fcecbe51e0ce78d4e856d7033565150ef4526
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654103
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Add an enabled flag to EventLog (defaulting to False on
Command.event_log and enabled by _Repo._RunLong when --event-log is
passed) and return None early from EventLog.Add when disabled.
Because EventLog.AddSync only resolves project.GetCommitRevisionId()
when EventLog.Add returns an event, disabling Command.event_log by
default eliminates 3,300+ synchronous git subprocess calls in
MainProcess result callbacks during normal repo sync runs without
touching subcmds/sync.py.
Bug: 564817989
Change-Id: I79dd66d1c392ba5ee562f104759e36d1dd6eefea
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649606
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Read plain files-backend unborn HEAD symrefs directly before spawning
git symbolic-ref, update already-loaded bare_ref snapshots in
_InitAnyMRef when no external ref changes occurred, and reuse warm
bare_ref snapshots in GetRevisionId for non-tag refs.
Also skip redundant rev-parse and empty-index ls-files subprocesses in
_Reproject when HEAD is guaranteed to be an unborn local ref.
Bug: 564817989
Change-Id: I0da10cefb14753ce6e1e22e192d652790d59a5b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649605
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Update .repo_config.json atomically with file metadata and keep cached
sections, remotes, and branches coherent across SetString, Branch.Save,
ClearCache, and Reload. Also reload Project.config in workers when
fetchcmd or reprojectcmd modifies .git/config externally so subsequent
reads hit the JSON cache instead of re-spawning git config --list in
MainProcess.
Also pass --template= to git init when no custom template directory is
configured so Git does not copy sample hooks only to delete them.
Bug: 564817989
Change-Id: I5ff655b0ab0b34a4f77f13617bbe5a1fff488c8c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649604
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
When uploading a branch that merges another remote-tracking branch (such
as an upstream branch into the destination branch), ReviewableBranch
only excludes the destination tracking branch (^base). As a result, git
rev-list walks the merged parent and counts all merged commits even
though they already exist on a remote branch, triggering the unusual
commit count warning in _VerifyPendingCommits.
Add --ignore-remote-commits to `repo upload` to pass --not --remotes to
git rev-list in ReviewableBranch so commits already reachable from
refs/remotes/* are excluded from the pending commit list.
Test: ./run_tests tests/test_project.py tests/test_subcmds_upload.py
Change-Id: Ib1fb864982729e20a621144580fe95daa478eaee
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/651284
Tested-by: Romaric Jodin <rjodin@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Romaric Jodin <rjodin@google.com>
The Ubuntu and Debian release wiki pages added in
https://gerrit-review.googlesource.com/c/git-repo/+/649983 only list
distro release and EOL dates, not which versions of Python, Git, or
OpenSSH shipped in each distro release.
Add Repology package version links alongside the distro release pages
in python-support.md and requirements.json so maintainers can check
which versions shipped across historical Ubuntu and Debian releases.
Bug: 575551885
Change-Id: Ib256fd9b364074b52d3e15d185af519d8376cb28
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/650868
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
The documented process has drifted from how releases are actually done:
* Releases are now cut weekly by an automated pipeline, so the manual
steps are only a fallback, and for rollbacks.
* Releases are tested (both updating an existing checkout and a fresh
init) before pushing anything.
* The tag is pushed & announced first, and only promoted to stable after
a day.
* Direct pushes to the stable branch are rejected, so it has to be
deleted & recreated via the Gerrit REST API instead.
Bug: 575551885
Change-Id: Ib36df07811c225558aba1a925b9703f3b2017e6f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649984
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
The table of git/python/ssh/distro versions hasn't been kept up to date
and duplicates information maintained upstream by the distros and
projects themselves. Drop it, and point the places that referenced it
(python-support.md & requirements.json) at the Ubuntu & Debian release
pages instead.
Bug: 575551885
Change-Id: Id1cb2225b3852273ca717fc7a443449fb2fd006e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649983
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
`Project.UploadForReview()` unconditionally passed `--progress` to
`git push`, forcing client and remote progress meters (`Counting
objects`, `remote: Resolving deltas: 3%..100%`) even on non-TTY
stderr or when `-q`/`--quiet` was passed. Meanwhile, `repo upload`
ignored `opt.quiet` completely.
Move `--progress` injection for `push` into `GitCommand` alongside
`fetch` and `clone`, enabling it only when `sys.stderr.isatty()` and
none of `--progress`, `--no-progress`, `--quiet`, or `-q` are set.
Forward `opt.quiet` from `repo upload` to `Project.UploadForReview()`
and pass `--no-progress` to `git push` rather than `--quiet` so
Gerrit `remote: https://...` review URLs are still emitted and parsed
by `_FindGerritUrls()`. Also suppress the non-error `[OK]` summary
banner when `-q`/`--quiet` is set.
Bug: 578114287
Change-Id: I88b7b055dcca0db80dae6962886a693fe717c741
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649663
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Cover `--reference` from a local mirror (with & without `--dissociate`),
`--standalone-manifest` with a manifest file served over HTTP, and
`--manifest-depth`.
Teach the server to serve static files (`GitHost.publish_file`), let
`manifest_xml` take an absolute `fetch` URL, and add `move_aside` to
make git dirs disappear mid-test (deleting them fails on Windows).
Bug: 566270944
Change-Id: Id60a19613a627f76d8d9125570dd5ce0cdfff9aa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648549
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Cover projects missing on the server (incl. `--fail-fast`), missing
project & manifest revisions, and local changes that conflict with an
incoming change. The tests check that repo exits non-zero with a useful
error, leaves the rest of the tree usable, and recovers once the problem
is fixed.
Let `RepoClient.init` return failures instead of raising.
Bug: 566270944
Change-Id: I84e2b8c0bb6012678add77360b446d4bea010c78
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648548
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Cover re-running `repo init -g` to add & remove projects,
`<include groups=...>`, `platform-*` groups & `--platform`, and groups
combined with a superproject.
Let `RepoClient.list_projects` pass args to `repo list`.
Bug: 566270944
Change-Id: Id333078fd18d38962d8b725ae9bd87f4976c3112
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648547
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Cover the go/repo-init flags used across Android (superproject, `-c`,
`--partial-clone`, `--partial-clone-exclude` & `--clone-filter`)
followed by the go/sync flags, under Git 2 & Git 3 defaults: projects
are pinned to the superproject, filtered blobs stay on the server,
excluded projects are full clones, and incremental syncs skip projects
whose pin didn't move. Also cover the default filter, and `sync -n`
then `sync -l` with a partial clone (where `-l` has to fetch blobs on
demand).
Add harness helpers to read a checkout's git config, list the repos a
client contacted, and create blobs that only exist in history.
Bug: 566270944
Change-Id: I77d2cabc1d4f5db2ea139c7825ba87638c6db1e7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648546
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Cover `repo sync -n` & `-l` on existing checkouts (interleaved &
phased), projects that were never fetched, and two-phase syncs with a
superproject. `-l` must not touch the network, which the tests check by
taking the server offline. Also cover tags (`--no-tags` & `sync-tags`),
`--prune`, `--force-sync`, `--force-remove-dirty` & `-d`.
Teach the harness to simulate outages (`GitHost.offline`) and to collect
the requests made during a block (`GitHost.watch_requests`), and add a
few `RepoClient` helpers.
Bug: 566270944
Change-Id: I96b4729a7d4e5b58bc57fb46a060beda35549979
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648545
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Cover the core `repo init` & `repo sync` options (current branch only,
clone bundles, shallow clones, groups, local manifests & manifest
names), multi-step workflows (two-phase syncs, mirrors, incremental
syncs, local work, switching manifest branches, selfupdate, ...), and
manifests that broke users in the past. Some tests also run under Git 3
defaults.
Add `RepoClient` helpers to list remote-tracking branches & checkouts,
and document the test layout in the README.
Bug: 566270944
Change-Id: Iac333216e0f27e489e64f4eb853189f860a91f20
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648544
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Normalize the `\r\n` that Python on Windows writes to pipes in repo's
output so tests can match multi-line messages. Pass a str to
`os.readlink`, which only accepts Path objects on Windows since Python
3.8. Set `GIT_CEILING_DIRECTORIES` for the client so git never finds a
repo above the test dir (e.g. when running with `--basetemp=./tmp` from
a checkout).
Run the real git rather than a `git` wrapper that's first on PATH: the
real git's exec dir (from `git --exec-path`) has a copy of git, so put
it first on PATH. LUCI's wrapper, for one, retries server errors for
minutes, which hangs tests that simulate them.
Bug: 566270944
Change-Id: Ia0ed5285852638e533632a7272f154239f1e834f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648543
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Add tests/e2e/conftest.py with fixtures for writing end-to-end tests: a
session-wide git server, the published code under test, a per-test
namespace of server repos, and a repo client with an isolated HOME &
git config. Parametrize `git_defaults` to emulate Git 3 defaults
(reftable & `main` as the default branch) on older git.
Tests under tests/e2e/ are marked `e2e` and run after the unit tests in
`./run_tests`. Use `-m e2e` or `-m 'not e2e'` to select them. They're
skipped when `git http-backend` is unavailable, git is older than 2.28,
or not running from a git checkout.
Bug: 566270944
Change-Id: Ib61e4dd7acc41a65beec4d74f1635a25dce4e3be
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/647643
Commit-Queue: Brian Gan <brgan@google.com>
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Mike Frysinger
Gavin Mak
John Cai
hungweichen
Romaric Jodin
Brian Gan