After repo.reprojectcmd exits 0, _Reproject ran `git diff-index --cached
--name-only <revid>^{tree}` to verify that the index matches the target
commit's tree.
Peeling <revid>^{tree} and comparing every entry against the index
forces Git to read the target commit and walk every tree object in the
target revision from .git/objects.
reprojectcmd exists for virtualized filesystems such as ABFS whose
checkout command materializes the worktree and writes .git/index from
its own tree metadata without reading .git/objects/pack/*-trees-*.pack.
Running diff-index against <revid>^{tree} afterward forces Git to fault
in the tree packfiles over the network (260 FetchTrees pack downloads /
1.64 GiB across a 3,296-project Android checkout).
Trust reprojectcmd's exit status for index materialization instead of
re-walking the target tree in Git, while keeping the HEAD postcondition
check that verifies the command did not modify refs.
Bug: 543101585
Change-Id: Ie5feea41a4191245232c6763a489d01596370c55
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643004
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
After repo.fetchcmd exits, _CustomFetch verified the tracking ref and
FETCH_HEAD with ResolveCommit, i.e. `git rev-parse --verify
<ref>^{commit}`. Peeling to a commit makes Git look the object up: it
opens every pack index in objects/pack/ until it finds the commit, then
reads the commit from its packfile.
fetchcmd exists for object stores that are populated lazily, such as
ABFS, whose `abfs git fetch` registers packfiles as virtual files and
returns at once. There the peel is the first read of the packs the
command registered, and it blocks until they are downloaded. A project
whose revision did not change needs no objects to sync, yet paid that
download whenever fetchcmd registered new packs for it.
Compare the object IDs that the refs record instead (ResolveRef, which
runs the same rev-parse without the peel). Only when they differ peel
both sides to commits, so an annotated tag still matches the commit it
points to. A commit missing from the object store now fails the checkout
that reads it instead of the network half.
On a first sync the checkout reads the commit anyway, so there the
download only moves to the next command that needs the object.
Bug: 543101585
Change-Id: I51248ff1e9674b16e3ff8cee69572dbc8c7a8e99
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643003
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Skip projects fetched via repo.fetchcmd in _CheckForBloatedProjects,
and expose Project.UseFetchCmd alongside Project.UseReprojectCmd.
The bloat check exists because Project._RemoteFetch passes --no-auto-gc
for clone-depth="1" repositories. Projects using repo.fetchcmd bypass
_RemoteFetch and manage object storage externally, so scanning their
worktrees for bloat is unnecessary and recommending git repack can
materialize virtualized objects locally.
Bug: 543101585
Change-Id: I16510ad3b5b39c3a5e2a1a75df706439252a088e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/655083
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Sparse checkout allows Git to check out only a portion of the
repository. This is particularly helpful for monorepository workflows
where CI jobs only need a portion of the repository to run tests.
This speeds up the CI jobs, and puts less load on the Git server as it
relieves it from having to pack up all the objects.
In the Git project, though it is still labeled experimental, the feature
has been stable for a number of years and is unlikely to change at this
point.
This change allows configuring Git to use sparse checkout and provide a
list of directories for the sync command to retrieve.
Projects with sparse paths default to a blob:none partial clone so the
excluded paths are never fetched, unless a filter is already configured
or the project is excluded from partial clones. Both the plain and the
git-worktree checkout flows configure sparse-checkout before populating
the working tree.
Change-Id: I772c7db5209b345306be378771c8f2040832bca1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/535061
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: John Cai <johncai86@gmail.com>
Commit-Queue: John Cai <johncai86@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Import modules at the top of the module in most cases. Many of these
weren't actually saving any time:
* editor: shlex is already imported by the main module
* platform_utils: On Windows, the win32 module is almost always needed
* tests: There's no point in optimizing these.
The trace import in the main module is about the only one worth allowing
since that module itself is rarely used, and only when in a specific
debug mode.
Change-Id: I5d3fcecbe51e0ce78d4e856d7033565150ef4526
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/654103
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Add an enabled flag to EventLog (defaulting to False on
Command.event_log and enabled by _Repo._RunLong when --event-log is
passed) and return None early from EventLog.Add when disabled.
Because EventLog.AddSync only resolves project.GetCommitRevisionId()
when EventLog.Add returns an event, disabling Command.event_log by
default eliminates 3,300+ synchronous git subprocess calls in
MainProcess result callbacks during normal repo sync runs without
touching subcmds/sync.py.
Bug: 564817989
Change-Id: I79dd66d1c392ba5ee562f104759e36d1dd6eefea
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649606
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Read plain files-backend unborn HEAD symrefs directly before spawning
git symbolic-ref, update already-loaded bare_ref snapshots in
_InitAnyMRef when no external ref changes occurred, and reuse warm
bare_ref snapshots in GetRevisionId for non-tag refs.
Also skip redundant rev-parse and empty-index ls-files subprocesses in
_Reproject when HEAD is guaranteed to be an unborn local ref.
Bug: 564817989
Change-Id: I0da10cefb14753ce6e1e22e192d652790d59a5b0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649605
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Update .repo_config.json atomically with file metadata and keep cached
sections, remotes, and branches coherent across SetString, Branch.Save,
ClearCache, and Reload. Also reload Project.config in workers when
fetchcmd or reprojectcmd modifies .git/config externally so subsequent
reads hit the JSON cache instead of re-spawning git config --list in
MainProcess.
Also pass --template= to git init when no custom template directory is
configured so Git does not copy sample hooks only to delete them.
Bug: 564817989
Change-Id: I5ff655b0ab0b34a4f77f13617bbe5a1fff488c8c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649604
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
When uploading a branch that merges another remote-tracking branch (such
as an upstream branch into the destination branch), ReviewableBranch
only excludes the destination tracking branch (^base). As a result, git
rev-list walks the merged parent and counts all merged commits even
though they already exist on a remote branch, triggering the unusual
commit count warning in _VerifyPendingCommits.
Add --ignore-remote-commits to `repo upload` to pass --not --remotes to
git rev-list in ReviewableBranch so commits already reachable from
refs/remotes/* are excluded from the pending commit list.
Test: ./run_tests tests/test_project.py tests/test_subcmds_upload.py
Change-Id: Ib1fb864982729e20a621144580fe95daa478eaee
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/651284
Tested-by: Romaric Jodin <rjodin@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Romaric Jodin <rjodin@google.com>
The Ubuntu and Debian release wiki pages added in
https://gerrit-review.googlesource.com/c/git-repo/+/649983 only list
distro release and EOL dates, not which versions of Python, Git, or
OpenSSH shipped in each distro release.
Add Repology package version links alongside the distro release pages
in python-support.md and requirements.json so maintainers can check
which versions shipped across historical Ubuntu and Debian releases.
Bug: 575551885
Change-Id: Ib256fd9b364074b52d3e15d185af519d8376cb28
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/650868
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
The documented process has drifted from how releases are actually done:
* Releases are now cut weekly by an automated pipeline, so the manual
steps are only a fallback, and for rollbacks.
* Releases are tested (both updating an existing checkout and a fresh
init) before pushing anything.
* The tag is pushed & announced first, and only promoted to stable after
a day.
* Direct pushes to the stable branch are rejected, so it has to be
deleted & recreated via the Gerrit REST API instead.
Bug: 575551885
Change-Id: Ib36df07811c225558aba1a925b9703f3b2017e6f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649984
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
The table of git/python/ssh/distro versions hasn't been kept up to date
and duplicates information maintained upstream by the distros and
projects themselves. Drop it, and point the places that referenced it
(python-support.md & requirements.json) at the Ubuntu & Debian release
pages instead.
Bug: 575551885
Change-Id: Id1cb2225b3852273ca717fc7a443449fb2fd006e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649983
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
`Project.UploadForReview()` unconditionally passed `--progress` to
`git push`, forcing client and remote progress meters (`Counting
objects`, `remote: Resolving deltas: 3%..100%`) even on non-TTY
stderr or when `-q`/`--quiet` was passed. Meanwhile, `repo upload`
ignored `opt.quiet` completely.
Move `--progress` injection for `push` into `GitCommand` alongside
`fetch` and `clone`, enabling it only when `sys.stderr.isatty()` and
none of `--progress`, `--no-progress`, `--quiet`, or `-q` are set.
Forward `opt.quiet` from `repo upload` to `Project.UploadForReview()`
and pass `--no-progress` to `git push` rather than `--quiet` so
Gerrit `remote: https://...` review URLs are still emitted and parsed
by `_FindGerritUrls()`. Also suppress the non-error `[OK]` summary
banner when `-q`/`--quiet` is set.
Bug: 578114287
Change-Id: I88b7b055dcca0db80dae6962886a693fe717c741
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/649663
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Cover `--reference` from a local mirror (with & without `--dissociate`),
`--standalone-manifest` with a manifest file served over HTTP, and
`--manifest-depth`.
Teach the server to serve static files (`GitHost.publish_file`), let
`manifest_xml` take an absolute `fetch` URL, and add `move_aside` to
make git dirs disappear mid-test (deleting them fails on Windows).
Bug: 566270944
Change-Id: Id60a19613a627f76d8d9125570dd5ce0cdfff9aa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648549
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Cover projects missing on the server (incl. `--fail-fast`), missing
project & manifest revisions, and local changes that conflict with an
incoming change. The tests check that repo exits non-zero with a useful
error, leaves the rest of the tree usable, and recovers once the problem
is fixed.
Let `RepoClient.init` return failures instead of raising.
Bug: 566270944
Change-Id: I84e2b8c0bb6012678add77360b446d4bea010c78
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648548
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Cover re-running `repo init -g` to add & remove projects,
`<include groups=...>`, `platform-*` groups & `--platform`, and groups
combined with a superproject.
Let `RepoClient.list_projects` pass args to `repo list`.
Bug: 566270944
Change-Id: Id333078fd18d38962d8b725ae9bd87f4976c3112
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648547
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Cover the go/repo-init flags used across Android (superproject, `-c`,
`--partial-clone`, `--partial-clone-exclude` & `--clone-filter`)
followed by the go/sync flags, under Git 2 & Git 3 defaults: projects
are pinned to the superproject, filtered blobs stay on the server,
excluded projects are full clones, and incremental syncs skip projects
whose pin didn't move. Also cover the default filter, and `sync -n`
then `sync -l` with a partial clone (where `-l` has to fetch blobs on
demand).
Add harness helpers to read a checkout's git config, list the repos a
client contacted, and create blobs that only exist in history.
Bug: 566270944
Change-Id: I77d2cabc1d4f5db2ea139c7825ba87638c6db1e7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648546
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Cover `repo sync -n` & `-l` on existing checkouts (interleaved &
phased), projects that were never fetched, and two-phase syncs with a
superproject. `-l` must not touch the network, which the tests check by
taking the server offline. Also cover tags (`--no-tags` & `sync-tags`),
`--prune`, `--force-sync`, `--force-remove-dirty` & `-d`.
Teach the harness to simulate outages (`GitHost.offline`) and to collect
the requests made during a block (`GitHost.watch_requests`), and add a
few `RepoClient` helpers.
Bug: 566270944
Change-Id: I96b4729a7d4e5b58bc57fb46a060beda35549979
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648545
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Cover the core `repo init` & `repo sync` options (current branch only,
clone bundles, shallow clones, groups, local manifests & manifest
names), multi-step workflows (two-phase syncs, mirrors, incremental
syncs, local work, switching manifest branches, selfupdate, ...), and
manifests that broke users in the past. Some tests also run under Git 3
defaults.
Add `RepoClient` helpers to list remote-tracking branches & checkouts,
and document the test layout in the README.
Bug: 566270944
Change-Id: Iac333216e0f27e489e64f4eb853189f860a91f20
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648544
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Normalize the `\r\n` that Python on Windows writes to pipes in repo's
output so tests can match multi-line messages. Pass a str to
`os.readlink`, which only accepts Path objects on Windows since Python
3.8. Set `GIT_CEILING_DIRECTORIES` for the client so git never finds a
repo above the test dir (e.g. when running with `--basetemp=./tmp` from
a checkout).
Run the real git rather than a `git` wrapper that's first on PATH: the
real git's exec dir (from `git --exec-path`) has a copy of git, so put
it first on PATH. LUCI's wrapper, for one, retries server errors for
minutes, which hangs tests that simulate them.
Bug: 566270944
Change-Id: Ia0ed5285852638e533632a7272f154239f1e834f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/648543
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Add tests/e2e/conftest.py with fixtures for writing end-to-end tests: a
session-wide git server, the published code under test, a per-test
namespace of server repos, and a repo client with an isolated HOME &
git config. Parametrize `git_defaults` to emulate Git 3 defaults
(reftable & `main` as the default branch) on older git.
Tests under tests/e2e/ are marked `e2e` and run after the unit tests in
`./run_tests`. Use `-m e2e` or `-m 'not e2e'` to select them. They're
skipped when `git http-backend` is unavailable, git is older than 2.28,
or not running from a git checkout.
Bug: 566270944
Change-Id: Ib61e4dd7acc41a65beec4d74f1635a25dce4e3be
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/647643
Commit-Queue: Brian Gan <brgan@google.com>
Tested-by: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Add `RepoClient`, which runs a standalone copy of the `repo` launcher as
a subprocess the same way end users do, and `publish_repo_source`, which
publishes the code under test (the current checkout, including
uncommitted changes) as a repo on the server so `repo init` installs it
into .repo/repo/ over HTTP.
Also add `GIT_DEFAULTS` so tests can emulate Git 3 defaults (reftable &
`main` as the default branch) on older git.
Bug: 566270944
Change-Id: I01ef53362b7812f4cb3e012509350980413cf0a8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/647505
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Add `GitHttpServer`, which serves the bare repositories under a
directory over smart HTTP by running `git http-backend` behind a small
threaded HTTP server on 127.0.0.1. Like Gerrit/GoB, it also serves
static `<repo>/clone.bundle` files. It records the requests it receives
so tests can check what clients fetched.
Add `GitHost`, which gives each test its own namespace of repos on a
shared server, with helpers to create & update a manifest repo.
Bug: 566270944
Change-Id: I35ff0b9ae743ed7a8d3c775525986e662ef03de0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/647504
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
This is the first piece of a hermetic end-to-end test harness for
running the real `repo` against a local git server.
Add tests/e2e/harness.py with `ServerRepo`, which creates bare
repositories, commits (including gitlinks for superprojects), branches,
tags & clone bundles without needing a worktree. Commits are written
with `git fast-import` using a fixed identity & a monotonic fake clock
so they're cheap & strictly ordered. Also add helpers to run git with
the user's git environment stripped & to write git config files
directly.
Bug: 566270944
Change-Id: Ifa9cf84529a699987bf6b21bbe94bd43483d1e73
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/647503
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Tested-by: Brian Gan <brgan@google.com>
A skip in CQ usually means the builder is missing a dependency (e.g. git
without reftable support), which silently drops coverage while the check
still passes. Add a `--fail-on-skip` pytest option that turns skips
into failures, and have run_tests pass it in CI alongside the existing
`skip_cq` deselection. Tests that shouldn't run in CQ keep using the
`skip_cq` marker.
Bug: 576728033
Change-Id: I2eb8ab168859a9a13d05ba7c771bb455c082f512
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/647626
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Per-user settings such as sync.smartsync live in ~/.repoconfig/config,
but reading or changing them required knowing the file location and
running `git config --file ~/.repoconfig/config ...`.
Add `repo config` as a thin wrapper around RepoConfig.ForUser(),
following the subcommand style of modern `git config`:
* `repo config get <name>` prints the value(s), exiting 1 if unset.
* `repo config set <name> <value>` sets it.
* `repo config unset <name>` removes it; a missing key is a no-op.
* `repo config list` shows all settings.
Values are stored as strings; `--type=bool|int` validates and
canonicalizes them on set and get, like `git config --type`. `-z` makes
get and list NUL-terminated for scripting. Setting names are validated
by `git config` itself when writing.
Factor the value parsing out of GitConfig.GetBoolean()/GetInt() into
ParseBoolean()/ParseInt() so the command and the readers agree, and
accept on/off/1/0 as booleans like git does.
Only the per-user file is handled; per-checkout settings in
.repo/manifests.git/config are out of scope. Like other subcommands, it
runs from inside a repo checkout.
Update docs/repoconfig.md to use the new command in its examples and
add zsh completion and the man page.
Bug: 569927750
Test: ./run_tests
Change-Id: I940d0318549fe9d79f6f9173139f0b96a706f1bf
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/645727
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Insaf Latypov <saferif@google.com>
Commit-Queue: Insaf Latypov <saferif@google.com>
Mike Frysinger
Gavin Mak
John Cai
hungweichen
Romaric Jodin
Brian Gan
Insaf Latypov