Compare commits

...
20 Commits
Author SHA1 Message Date
Insaf Latypov a0bc2b14a3 git_config: Fix bugs in processing RepoConfig.
- Automatically create all the parent directories for `git confing
  --file ...`
- Prevent `RepoConfig` from inheriting caches from `GitConfig`. These
  two are different files.

Bug: 569927750
Test: ./run_tests
Change-Id: I9c60cecff75c1fb62be3ff63425d9618c0f56b06
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643263
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Insaf Latypov <saferif@google.com>
Commit-Queue: Insaf Latypov <saferif@google.com>
2026-10-14 07:49:21 -07:00
Rahul Yadav 3448df3954 sync: Add GetApprovedManifestV2 with named request options
Smart sync calls GetApprovedManifest(branch[, target]) over XML-RPC,
which only supports positional arguments.  Adding new request options
that way does not scale.

Add a GetApprovedManifestV2 method that takes a single struct of named
options and returns a struct:

  request:  {branch, target?, best_effort?}
  response: {success, manifest_xml?, error_message?}

Servers must ignore unknown options, so new options (e.g. criteria for
selecting a good build) can be added without another method.

The first new option is `best_effort`.  repo sets it when smart sync was
enabled implicitly by the manifest's sync-smartsync="true" default
rather than by -s.  The server may decline a best_effort request (e.g.
for rollout or cohort gating) by returning success=false.  The client
still owns the fallback policy: declined implicit requests fall back to
a regular sync, while explicit -s requests fail as before.

If the server does not implement GetApprovedManifestV2, repo falls back
to the legacy GetApprovedManifest call.

Bug: 545137090
Test: ./run_tests
Change-Id: I9297b728df26516012cb93dffc91d39c1fb0458f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/636623
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Rahul Yadav <yadavrah@google.com>
Tested-by: Rahul Yadav <yadavrah@google.com>
2026-10-14 03:30:36 -07:00
Brian Gan a110680ff3 repo: tolerate a transiently missing stable branch
The automated release process deletes the stable branch and then
recreates it. A client that fetches during that window can't find it:
`repo init` fails to resolve --repo-rev, and the daily self-update in
`repo sync` fails with "Cannot fetch repo tool".

In the launcher, if --repo-rev doesn't resolve after the initial fetch,
wait 10 seconds and refetch once before giving up. `resolve_repo_rev`
gains a `report_errors` flag so this first check doesn't print an error
that the refetch then fixes. Bump the launcher version to 2.69 so
existing users pick this up.

For fetches of the repo project itself (sync's self-update and
`repo selfupdate`), wait 10 seconds instead of 4 before retrying. The
delay is now a `RETRY_SLEEP_INITIAL_SEC` class attribute that
`RepoProject` overrides, so other projects keep the 4 second delay. This
replaces the unused `retry_sleep_initial_sec` parameter of
`_RemoteFetch`.

Bug: 566270786
Change-Id: I23cb810258522b5aa9eae9d7013dbb7904ebcad8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/643265
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-10-13 13:55:40 -07:00
Rahul Yadav 178b66ee3e sync: keep smart sync override with -l
Splitting a sync into `repo sync -n` followed by `repo sync -l` should
end up in the same state as a single `repo sync`. With smart sync it
does not always: -n fetches the revisions pinned by the smart sync
manifest and saves it as smart_sync_override.xml, but the following -l
deletes that override and checks out ToT instead, unless the manifest
sets sync-smartsync="true". So `repo sync -s -n` + `repo sync -l` lands
on ToT, while `repo sync -s` lands on the pinned revisions.

Without an explicit -s or -t, -l does not contact the manifest server,
so it cannot choose new pins itself; the only consistent choice is to
keep the override the previous -n left behind. Do that regardless of
the manifest default.

Syncs that choose a manifest themselves still replace or remove the
override as before: any sync without -l, and -l combined with -s, -t,
-m, --superproject-revision or --no-smart-sync. Smart sync stays
non-sticky for regular syncs, and `repo sync -l --no-smart-sync` still
discards the override to check out ToT.

Bug: 569927750
Change-Id: Id06ff88e9f3183fae32e993e86630fc3e8de71a1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/640965
Commit-Queue: Rahul Yadav <yadavrah@google.com>
Tested-by: Rahul Yadav <yadavrah@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-10-12 02:14:30 -07:00
Gavin Mak beda65323e sync: skip clone.bundle when repo.fetchcmd is configured
In Sync_NetworkHalf, clone_bundle defaults to True on initial sync when
self.objdir does not exist yet, and _ApplyCloneBundle is called before
custom_fetch_configured is checked.

For remotes with HTTP/HTTPS URLs that serve /clone.bundle, this causes
repo sync to download and unpack the full clone.bundle into .git/objects
before invoking repo.fetchcmd.

Skip _ApplyCloneBundle when custom_fetch_configured is enabled so
repo.fetchcmd exclusively handles object fetching.

Bug: 543101585
Change-Id: If41a775a75589ad5a23106163f9932e2940b72d2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/641685
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2026-10-09 14:09:50 -07:00
Gavin Mak 02e07525f9 project: peel annotated tags in GetRevisionId with cached refs
When all_refs (e.g. self.bare_ref.all in Sync_LocalHalf) is passed to
GetRevisionId, looking up a tag ref in all_refs returns the tag object
SHA for annotated tags rather than the peeled commit SHA.

In Sync_LocalHalf, this causes repo.reprojectcmd to receive a tag SHA in
REPO_TREV and DetachHead to fail with 'trying to write non-commit object
to branch HEAD', and causes standard checkouts to re-checkout on every
subsequent sync because HEAD (the commit SHA) never equals revid (the
tag object SHA).

Skip the all_refs fast path for refs/tags/* so tags always fall through
to bare_git.ResolveCommit, which peels to the underlying commit SHA.

Bug: 543101585
Change-Id: Ie4afa3cee9149e2a8438bd8fb4a8026a210848d8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/641683
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2026-10-09 14:09:44 -07:00
Gavin Mak 2e8a3805bb sync: don't run git ls-remote before fetchcmd on unpinned refs
When a project's revision is an unpinned branch or tag and superprojects
are not used (so self.revisionId is unset), _CustomFetch previously ran
git ls-remote against the Git remote before invoking repo.fetchcmd just
to pre-populate REPO_TREV.

This added a per-project network round-trip before fetchcmd and forced
branch resolution against the Git remote rather than letting fetchcmd
resolve REPO_RREV from its own server (just as standard git fetch does
in _RemoteFetch).

Only populate REPO_TREV before fetchcmd when the target commit SHA is
already known (via self.revisionId from superproject or a SHA-pinned
revisionExpr). When REPO_RREV is an unpinned branch or tag, leave
REPO_TREV empty, let fetchcmd resolve REPO_RREV and update the local
tracking ref and FETCH_HEAD, and verify afterward that the tracking ref
and FETCH_HEAD resolve to the same valid commit in the object store.

Bug: 543101585
Change-Id: I482bf4c022d0e669e5620d4517581ac193983da0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/640423
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2026-10-09 14:09:37 -07:00
Becky Siegel 277759b9e1 agents: enable generic multi-perspective code review agent
Configure git-repo-code-review-agent with use_default_subagents: true in
agents/agent_configs.txtpb so groupagent-grev-review uses the generic
gerrit_reviewer multi-perspective review framework.

Bug: 555706086
Change-Id: I079681b00e71cb886dc2343692d88b01d58a51ea
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/641904
Reviewed-by: Brian Gan <brgan@google.com>
Tested-by: Becky Siegel <beckysiegel@google.com>
2026-10-09 13:10:58 -07:00
Brian Gan 9593a76fd1 sync: keep --superproject-revision across manifest reloads
When --superproject-revision moves `.repo/manifests`, the manifest is
reloaded, discarding the pinned superproject. _UpdateProjectsRevisionId
then read gitlinks from the superproject's default branch instead,
pinning failed, and projects synced unpinned.

Apply the revision in _ConfigureSuperproject so it is re-applied after
every reload.

Verified with https://paste.googleplex.com/5889974172516352

Bug: 571245807
Change-Id: I4d782d229b128c0f15081d11b722d8901709458c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/641743
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
2026-10-09 12:56:07 -07:00
Brian Gan 9293cda144 sync: roll back repo when stable is rewound
The documented way to roll back a bad release is to force push an older
release to stable, and docs/release-process.md says clients will then
automatically downgrade. _PostRepoFetch already supports switching to an
older commit, but it only runs when `RepoProject.HasChanges` is true,
and that used the `MetaProject` check of whether the remote has commits
that HEAD lacks. After a rewind it doesn't, so clients that had already
upgraded stayed on the bad release.

Make `RepoProject.HasChanges` true whenever the remote revision differs
from HEAD, in either direction. `ManifestProject` keeps the forward-only
check so local manifest commits are still rebased rather than dropped.

Bug: 571204396
Change-Id: I248507f434d2552ce11a9afabe9a6335cb24ddbd
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/640130
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Brian Gan <brgan@google.com>
Commit-Queue: Brian Gan <brgan@google.com>
2026-10-09 09:49:49 -07:00
Rahul Yadav 340b587e94 sync: Support SYNC_BRANCH to override the smart sync branch
`repo sync -s` always sends the manifest branch to the manifest server
as the `branch` argument of GetApprovedManifest. Some servers map one
manifest branch to several server-side (CI) branches, and the only way
to pick among them was indirectly through the target.

Honor a SYNC_BRANCH environment variable, mirroring SYNC_TARGET: if set
and non-empty, send it instead of the manifest branch. An empty value
is treated as unset. -t/--smart-tag is unaffected.

Bug: 568962967
Test: ./run_tests
Change-Id: I26700e8997c21487f83011258100fd91a5c87508
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/639123
Commit-Queue: Rahul Yadav <yadavrah@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Rahul Yadav <yadavrah@google.com>
2026-10-09 07:00:47 -07:00
Rahul Yadav 0a4e68f9d2 sync: don't let --nmu affect smart sync
--no-manifest-update has always meant "use the existing manifest
checkout as-is": it skips updating .repo/manifests and nothing else.
Smart sync pins do not come from that checkout but from the manifest
server, and an explicit `-s --nmu` already works that way.

The manifest-driven default smart sync (sync-smartsync="true") instead
treated --nmu like -l: it disabled the implicit smart sync and kept an
existing smart_sync_override.xml. So on such manifests `repo sync --nmu`
fetched and checked out the previous pins (or ToT if there were none)
instead of doing what `repo sync` does minus the manifest checkout
update.

Make --nmu orthogonal to smart sync again: it no longer disables the
default smart sync, and it no longer keeps the override. Both have to
change together; otherwise a failed implicit smart sync under --nmu
would fall back to ToT while leaving the override on disk. Only -l,
which skips fetching, keeps the override.

Bug: 569927750
Change-Id: I33730363df485d72cd86ae99bf5eb8399b58e81a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/641264
Reviewed-by: Brian Gan <brgan@google.com>
Commit-Queue: Rahul Yadav <yadavrah@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Rahul Yadav <yadavrah@google.com>
2026-10-09 07:00:22 -07:00
Mike Frysinger 7bba4ee47e vpython: update to uv spec format
Change-Id: I18dca6f8c9c1f9234cfee640f780f1f2e4462ab8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/635623
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-10-05 10:40:31 -07:00
Rahul Yadav d53a4278f5 sync: Support manifest-driven default smart sync
Add support for enabling smart sync by default via the
sync-smartsync="true" attribute on the manifest <default> element.

When sync-smartsync is set to true in the manifest defaults:
- Plain `repo sync` triggers Smart Sync automatically.
- If that implicit Smart Sync fails (manifest server unreachable, or
  the returned manifest cannot be parsed), repo silently falls back to
  a regular ToT sync. The reason is logged with --verbose.
- `--no-smart-sync` CLI flag overrides the manifest default and
  syncs to ToT.
- Passing explicit `-s` / `--smart-sync` forces Smart Sync as before,
  and its failures are still fatal.
- `-t`, `--smart-tag`, `-m`, `--manifest-name`, and
  `--superproject-revision` disable the default smart sync.
- `-l` / `--local-only` and `--no-manifest-update` skip the manifest
  server and reuse the existing smart_sync_override.xml, if any.

Update docs, completion, regenerate man pages, and add tests for XML
parsing, serialization, CLI option resolution, and the fallback and
reuse paths.

Bug: 545137090
Change-Id: If75569c58dd525d170e9801e70ccbf02206a52f2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/617641
Tested-by: Rahul Yadav <yadavrah@google.com>
Commit-Queue: Rahul Yadav <yadavrah@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-10-01 03:13:33 -07:00
Victor Pushkarev 5d4149f32e gc: add --keep-manifest-projects
By default, `repo gc` treats projects excluded by the configured
manifest groups as unused. This matches the active project set, but it
can remove cached Git data for projects that might be used again after
changing groups.

Add `--keep-manifest-projects` to keep Git data for projects that are
still in the manifests, even if the current groups exclude them or their
checkout is missing. Git data for projects no longer in the manifests
can still be removed.

Change-Id: I63ae18ce1c6acc2463be86f81921476b12fb27d7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/629741
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Victor Pushkarev <corvinus.v@gmail.com>
Commit-Queue: Victor Pushkarev <corvinus.v@gmail.com>
2026-09-30 01:11:40 -07:00
Ram Peri cf499e312a sync: record superproject sync analysis state again
Since the multi-manifest change (bdcba7d), _UpdateProjectsRevisionId
rebinds its superproject_logging_data parameter to a new local dict.
The caller keeps its own empty dict and passes it to
UpdateSyncAnalysisState, so repo.syncstate.superproject.* is never
written to the manifest config or logged to trace2.

Drop the rebinding so the caller's dict is populated.

Also record superproject, haslocalmanifests and hassuperprojecttag
when no manifest declares a superproject. Before bdcba7d these keys
were logged on every sync. SyncAnalysisState does not remove old
keys, so without this a checkout that moves to a manifest without a
superproject keeps reporting the values from its last superproject
sync.

Bug: 565957530
Test: ./run_tests
Change-Id: I8796e3a34b9cf4786383d25ecc3ebf0c5409caa3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/634843
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Ram Peri <ramperi@google.com>
Tested-by: Ram Peri <ramperi@google.com>
2026-09-28 15:34:54 -07:00
Mike Frysinger cc0546bc71 isort: print diff on failures
This should help users figure out how to fix problems.

Change-Id: I6346e163d06f9a0e52147689c5b329fe7f1366d2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/635343
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-09-28 10:55:16 -07:00
Victor Pushkarev ccf9ce39b8 sync: keep local sync state paths relative to repo root
`LocalSyncState` uses `Project.relpath` to identify projects, but that
path is relative to each project's own manifest. Projects in different
manifests can have the same relpath, so their fetch and checkout
timestamps overwrite each other. This can make a later partial sync look
like a full sync.

Pruning has a related problem. It treats the stored paths as relative to
the current manifest's topdir, so when sync starts from the outermost
manifest it can look for submanifest projects in the wrong place and
remove valid state.

Use `Project.RelPath(local=False)` to identify projects in local sync
state and check stored paths from the repo root when pruning.

Add tests for same-relpath projects, partial-sync detection, and pruning
when sync starts from either the outermost manifest or a submanifest.

Change-Id: I24485c315f1277a14913443674245cedd20820fe
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/632421
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Victor Pushkarev <corvinus.v@gmail.com>
Commit-Queue: Victor Pushkarev <corvinus.v@gmail.com>
2026-09-26 01:17:45 -07:00
Ram Peri 9066409c2d init: don't suggest --no-use-superproject on failure
When the superproject can't be fetched, repo init warns and tells users
to re-run repo init with --no-use-superproject. That choice is saved in
the checkout config (repo.superproject), so following it disables the
superproject for that checkout until it is re-inited, even though these
failures are often transient.

Drop the recommendation and keep the rest of the warning.

This is the repo init counterpart of reverting
https://gerrit-review.googlesource.com/c/git-repo/+/305622 for repo sync
(https://gerrit-review.googlesource.com/c/git-repo/+/634804).

Bug: 565845573
Change-Id: I91a6e682ef1d1c6dee07bc7969de88a681e1d239
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/634784
Tested-by: Ram Peri <ramperi@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
Reviewed-by: Arif Kasim <arifkasim@google.com>
Commit-Queue: Ram Peri <ramperi@google.com>
2026-09-24 13:54:06 -07:00
Ram Peri 2908f596c5 Revert "sync: Recommend using --no-use-superproject if sync fails."
This reverts commit f907ced0fe.
Original change: https://gerrit-review.googlesource.com/c/git-repo/+/305622

Reason for revert: the hint was added when a superproject failure made
repo sync exit with an error. Since 8db30d6 the failure is non-fatal
and sync falls back to fetching normally, and these failures are often
transient (e.g. on 2026-09-23 a manifest change added
external/slackhq/compose-lints before the superproject had a commit
for it, until the change was reverted ~2h later). Following the hint
means users keep passing --no-use-superproject and bypass the
superproject even after the failure is gone.

The revert does not apply cleanly after the black reformat, so the
conflict was resolved by restoring the original message text.

Bug: 565845573
Change-Id: I5efadedd4d3d0002df50493fa29554ba9626fe82
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/634804
Commit-Queue: Ram Peri <ramperi@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
Tested-by: Ram Peri <ramperi@google.com>
2026-09-24 13:53:54 -07:00
23 changed files with 1983 additions and 333 deletions

No files matched your search

+13
View File
@@ -24,3 +24,16 @@ configs {
}
automatic: false
}
# Default Multi-Perspective Code Review Agent
configs {
id: "git-repo-code-review-agent"
display_name: "Code Review Agent"
description: "Default multi-perspective review agent for git-repo."
include_filters {
project: "git-repo"
path_regex: ".*"
}
use_default_subagents: true
tags: "code-review-agent"
}
+1
View File
@@ -337,6 +337,7 @@ _repo() {
'--auto-gc[Run auto gc]' \
'--no-auto-gc[Do not run auto gc]' \
'(-s --smart-sync)'{-s,--smart-sync}'[Smart sync]' \
'--no-smart-sync[Do not smart sync]' \
'(-t --smart-tag)'{-t,--smart-tag=}'[Smart tag]:tag:' \
'--no-repo-verify[Do not verify repo]' \
'--no-verify[Do not verify]' \
+18 -14
View File
@@ -28,7 +28,10 @@ Environment section in `repo help forall` or `subcmds/forall.py`.
The following environment variable is specific to `repo.fetchcmd`:
* `REPO_TREV`: The target revision resolved to a full commit hash.
* `REPO_TREV`: The target commit hash when the revision is already pinned to
a commit (e.g. via a superproject or a commit SHA in the manifest), or
empty when `REPO_RREV` is an unpinned branch or tag to be resolved by the
fetch command.
## Contract
@@ -37,22 +40,23 @@ The following environment variable is specific to `repo.fetchcmd`:
After the fetch command exits with status 0, `repo` expects the following
postconditions to be met:
1. `git cat-file -e REPO_TREV` succeeds (the commit must exist in the object
store).
2. The mapped local tracking ref (e.g. `refs/remotes/REPO_REMOTE/<branch>`
for a branch revision, or the tag ref itself for a tag) must point to
`REPO_TREV`.
3. `FETCH_HEAD` must point to `REPO_TREV`.
4. The commit graph from `REPO_TREV` must be reachable far enough to compute
merge bases with local branches.
1. When the manifest revision names a branch or tag, the mapped local tracking
ref (e.g. `refs/remotes/REPO_REMOTE/<branch>` for a branch, or
`refs/tags/<tag>` for a tag) must resolve to a valid commit in the object
store (`git rev-parse --verify <ref>^{commit}`), and must equal `REPO_TREV`
if `REPO_TREV` was non-empty.
2. `FETCH_HEAD` must resolve to the same target commit in the object store
(`git rev-parse --verify FETCH_HEAD^{commit}`).
3. The commit graph from the target commit must be reachable far enough to
compute merge bases with local branches.
### Invariants
* The command should be idempotent; fetching the same `REPO_TREV` twice should
be a no-op.
* Only `FETCH_HEAD` and `refs/remotes/*` should be modified to preserve
`repo sync --network-only` semantics. `HEAD` and local branches must not be
touched by the fetch command.
* The command should be idempotent; repeating the same fetch should be a
no-op.
* Only `FETCH_HEAD`, `refs/remotes/*`, and `refs/tags/*` should be modified to
preserve `repo sync --network-only` semantics. `HEAD` and local branches
must not be touched by the fetch command.
* Dirty worktree state must be preserved.
* The command is **not** executed for `MetaProject`s (i.e. the internal `repo`
repository itself at `.repo/repo` and the `manifests` repository at
+13 -10
View File
@@ -46,18 +46,19 @@ following DTD:
<!ATTLIST remote revision CDATA #IMPLIED>
<!ELEMENT default EMPTY>
<!ATTLIST default remote IDREF #IMPLIED>
<!ATTLIST default revision CDATA #IMPLIED>
<!ATTLIST default dest-branch CDATA #IMPLIED>
<!ATTLIST default upstream CDATA #IMPLIED>
<!ATTLIST default sync-j CDATA #IMPLIED>
<!ATTLIST default sync-j-max CDATA #IMPLIED>
<!ATTLIST default sync-c CDATA #IMPLIED>
<!ATTLIST default sync-s CDATA #IMPLIED>
<!ATTLIST default sync-tags CDATA #IMPLIED>
<!ATTLIST default remote IDREF #IMPLIED>
<!ATTLIST default revision CDATA #IMPLIED>
<!ATTLIST default dest-branch CDATA #IMPLIED>
<!ATTLIST default upstream CDATA #IMPLIED>
<!ATTLIST default sync-j CDATA #IMPLIED>
<!ATTLIST default sync-j-max CDATA #IMPLIED>
<!ATTLIST default sync-c CDATA #IMPLIED>
<!ATTLIST default sync-s CDATA #IMPLIED>
<!ATTLIST default sync-tags CDATA #IMPLIED>
<!ATTLIST default sync-smartsync CDATA #IMPLIED>
<!ELEMENT manifest-server EMPTY>
<!ATTLIST manifest-server url CDATA #REQUIRED>
<!ATTLIST manifest-server url CDATA #REQUIRED>
<!ATTLIST manifest-server helper CDATA #IMPLIED>
<!ELEMENT submanifest EMPTY>
@@ -231,6 +232,8 @@ Attribute `sync-tags`: Set to false to only sync the given Git
branch (specified in the `revision` attribute) rather than
the other ref tags.
Attribute `sync-smartsync`: Set to true to enable smart sync by default.
### Element manifest-server
+80 -4
View File
@@ -95,6 +95,36 @@ example would do. Google's internal server uses Python's
The manifest server should implement the following RPC methods.
### GetApprovedManifestV2
> `GetApprovedManifestV2(request: dict) -> dict`
Like [GetApprovedManifest](#GetApprovedManifest), but takes a single struct of
named options so new options can be added without changing the method signature.
Unset options are omitted from the struct. Servers should ignore options they do
not understand.
Request options:
* `branch` (`str`, required): Same as in `GetApprovedManifest`.
* `target` (`str`, optional): Same as in `GetApprovedManifest`.
* `best_effort` (`bool`, optional, default `False`): The client can proceed
without a manifest, so the server may decline the request (e.g. during a
staged rollout or to shed load). Repo sets this for Smart Syncs enabled by
the manifest's `sync-smartsync` default, and falls back to a regular ToT
sync if the request is declined. Servers must not decline requests without
this option for these reasons.
Response fields:
* `success` (`bool`): Whether a manifest was returned.
* `manifest_xml` (`str`): The manifest. Required if `success` is true.
* `error_message` (`str`): Why the request failed or was declined. Repo shows
it to the user like `GetApprovedManifest` errors.
If the server does not implement this method, repo calls `GetApprovedManifest`
instead and drops the options it cannot express.
### GetApprovedManifest
> `GetApprovedManifest(branch: str, target: Optional[str]) -> str`
@@ -138,12 +168,53 @@ copy into `.repo/manifests/smart_sync_override.xml` so users can examine it.
The next time `repo sync` is run, this file is automatically replaced or removed
based on the current set of options.
### --smart-sync
With `-l`/`--local-only` and no explicit `-s`/`--smart-sync` or
`-t`/`--smart-tag`, repo does not contact the manifest server. Instead, it keeps
using the existing `smart_sync_override.xml`, if there is one, whether it came
from an explicit `-s`/`--smart-sync` or from the manifest's `sync-smartsync`
default. This way, `repo sync -n` followed by `repo sync -l` ends up on the same
revisions as the equivalent single `repo sync` (e.g. `repo sync -s -n` +
`repo sync -l` matches `repo sync -s`). Pass `--no-smart-sync` to discard it and
use the default manifest instead.
Repo will call `GetApprovedManifest(branch[, target])`.
`--no-manifest-update` only skips updating the manifest checkout. It does not
affect Smart Sync: the default Smart Sync from `sync-smartsync` (see below)
still runs, and an existing `smart_sync_override.xml` is replaced or removed as
in a regular sync.
### sync-smartsync (Manifest Attribute)
The manifest can enable Smart Sync by default for all `repo sync` invocations
by setting `sync-smartsync="true"` on the `<default>` element:
```xml
<default sync-smartsync="true" ... />
```
When enabled in the manifest, `repo sync` will automatically perform a Smart
Sync unless explicitly overridden on the command line. Since the user did not
ask for a Smart Sync, any failure to obtain a manifest from the manifest server
(e.g. the server is unreachable, or it returns a manifest that cannot be parsed)
is silently ignored and `repo sync` falls back to a regular ToT sync; pass
`--verbose` to see the reason. Repo marks these requests as `best_effort` (see
[GetApprovedManifestV2](#GetApprovedManifestV2)), so the server may also decline
them, e.g. during a staged rollout. This is only a fallback for the
manifest-driven default: an explicit `-s`/`--smart-sync` or `-t`/`--smart-tag`
still fails the sync, as before. Options that specify an explicit target or
manifest source (such as `-t`/`--smart-tag`, `-m`/`--manifest-name` or
`--superproject-revision`) will also disable the default Smart Sync behavior.
### --smart-sync / -s
Explicitly enables Smart Sync. Repo will call
`GetApprovedManifestV2({branch[, target]})`, or
`GetApprovedManifest(branch[, target])` if the server does not implement it.
The `branch` is determined by the current manifest branch as specified by
`--manifest-branch=BRANCH` when running `repo init`.
`--manifest-branch=BRANCH` when running `repo init`. If `${SYNC_BRANCH}` is
defined and non-empty, its value is sent instead. This allows selecting a
branch other than the one the checkout tracks, e.g. a server-side (CI) branch
name that builds from the same manifest branch.
The `target` is defined by environment variables in the order below. If none
of them match, then `target` is omitted. These variables were decided as they
@@ -155,6 +226,11 @@ match the settings Android build environments automatically setup.
3. `${TARGET_PRODUCT}-${TARGET_BUILD_VARIANT}`: If these variables are all
defined, then they are merged with `-` and used.
### --smart-tag=TAG
### --no-smart-sync
Explicitly disables Smart Sync, overriding any `sync-smartsync="true"` setting
declared in the manifest `<default>` element.
### --smart-tag=TAG / -t
Repo will call `GetManifest(TAG)`.
+12
View File
@@ -266,6 +266,13 @@ class GitConfig:
except KeyError:
old = []
if self.file != self._SYSTEM_CONFIG:
# `git config --file` does not create missing parent directories
# (e.g. ~/.repoconfig/ on first use).
parent_dir = os.path.dirname(self.file)
if parent_dir:
os.makedirs(parent_dir, exist_ok=True)
if value is None:
if old:
del self._cache[key]
@@ -444,6 +451,11 @@ class GitConfig:
class RepoConfig(GitConfig):
"""User settings for repo itself."""
# Keep separate caches from GitConfig, otherwise ForUser()/ForSystem()
# would return the already-cached ~/.gitconfig instance of the parent.
_ForUser = None
_ForSystem = None
@staticmethod
def _getUserConfig():
repo_config_dir = os.getenv("REPO_CONFIG_DIR", os.path.expanduser("~"))
+12 -1
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "June 2026" "repo gc" "Repo Manual"
.TH REPO "1" "September 2026" "repo gc" "Repo Manual"
.SH NAME
repo \- repo gc - manual page for repo gc
.SH SYNOPSIS
@@ -22,6 +22,9 @@ answer yes to all safe prompts
.TP
\fB\-\-repack\fR
repack all projects that use partial clone with filter=blob:none
.TP
\fB\-\-keep\-manifest\-projects\fR
keep Git data for projects still in the manifests
.SS Logging options:
.TP
\fB\-v\fR, \fB\-\-verbose\fR
@@ -44,3 +47,11 @@ only operate on this (sub)manifest
operate on this manifest and its submanifests
.PP
Run `repo help gc` to view the detailed manual.
.SH DETAILS
.PP
The 'repo gc' command removes internal Git data for projects that are no longer
active in the current configuration.
.PP
Projects excluded by the configured manifest groups are inactive by default. Use
\fB\-\-keep\-manifest\-projects\fR to keep Git data for projects that are still in the
manifests.
+14 -11
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "June 2026" "repo manifest" "Repo Manual"
.TH REPO "1" "August 2026" "repo manifest" "Repo Manual"
.SH NAME
repo \- repo manifest - manual page for repo manifest
.SH SYNOPSIS
@@ -121,18 +121,19 @@ include*)>
<!ATTLIST remote revision CDATA #IMPLIED>
.IP
<!ELEMENT default EMPTY>
<!ATTLIST default remote IDREF #IMPLIED>
<!ATTLIST default revision CDATA #IMPLIED>
<!ATTLIST default dest\-branch CDATA #IMPLIED>
<!ATTLIST default upstream CDATA #IMPLIED>
<!ATTLIST default sync\-j CDATA #IMPLIED>
<!ATTLIST default sync\-j\-max CDATA #IMPLIED>
<!ATTLIST default sync\-c CDATA #IMPLIED>
<!ATTLIST default sync\-s CDATA #IMPLIED>
<!ATTLIST default sync\-tags CDATA #IMPLIED>
<!ATTLIST default remote IDREF #IMPLIED>
<!ATTLIST default revision CDATA #IMPLIED>
<!ATTLIST default dest\-branch CDATA #IMPLIED>
<!ATTLIST default upstream CDATA #IMPLIED>
<!ATTLIST default sync\-j CDATA #IMPLIED>
<!ATTLIST default sync\-j\-max CDATA #IMPLIED>
<!ATTLIST default sync\-c CDATA #IMPLIED>
<!ATTLIST default sync\-s CDATA #IMPLIED>
<!ATTLIST default sync\-tags CDATA #IMPLIED>
<!ATTLIST default sync\-smartsync CDATA #IMPLIED>
.IP
<!ELEMENT manifest\-server EMPTY>
<!ATTLIST manifest\-server url CDATA #REQUIRED>
<!ATTLIST manifest\-server url CDATA #REQUIRED>
<!ATTLIST manifest\-server helper CDATA #IMPLIED>
.IP
<!ELEMENT submanifest EMPTY>
@@ -336,6 +337,8 @@ Attribute `sync\-s`: Set to true to also sync sub\-projects.
Attribute `sync\-tags`: Set to false to only sync the given Git branch (specified
in the `revision` attribute) rather than the other ref tags.
.PP
Attribute `sync\-smartsync`: Set to true to enable smart sync by default.
.PP
Element manifest\-server
.PP
At most one manifest\-server may be specified. The url attribute is used to
+3
View File
@@ -125,6 +125,9 @@ do not run garbage collection on any projects (default)
\fB\-s\fR, \fB\-\-smart\-sync\fR
smart sync using manifest from the latest known good build
.TP
\fB\-\-no\-smart\-sync\fR
disable smart sync and sync to ToT instead
.TP
\fB\-t\fR SMART_TAG, \fB\-\-smart\-tag\fR=\fI\,SMART_TAG\/\fR
smart sync using manifest from a known tag
.SS Logging options:
+16
View File
@@ -159,6 +159,7 @@ class _Default:
sync_c = False
sync_s = False
sync_tags = True
sync_smartsync = False
def __eq__(self, other):
if not isinstance(other, _Default):
@@ -483,6 +484,17 @@ class XmlManifest:
self.Unload()
self._Load()
def ClearOverride(self) -> None:
"""Stop overriding the manifest, reverting to the default one.
This is the inverse of Override(), and is a no-op if no override is in
effect. It is useful to recover from a failed Override(), which
registers the override before parsing it.
"""
self._outer_client.manifestFileOverrides.pop(self.path_prefix, None)
self._load_local_manifests = True
self.Unload()
def Link(self, name):
"""Update the repo metadata to use a different manifest."""
self.Override(name)
@@ -644,6 +656,9 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
if not d.sync_tags:
have_default = True
e.setAttribute("sync-tags", "false")
if d.sync_smartsync:
have_default = True
e.setAttribute("sync-smartsync", "true")
if have_default:
root.appendChild(e)
root.appendChild(doc.createTextNode(""))
@@ -1789,6 +1804,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
d.sync_c = XmlBool(node, "sync-c", False)
d.sync_s = XmlBool(node, "sync-s", False)
d.sync_tags = XmlBool(node, "sync-tags", True)
d.sync_smartsync = XmlBool(node, "sync-smartsync", False)
return d
def _ParseNotice(self, node):
+73 -67
View File
@@ -559,6 +559,9 @@ class RemoteSpec:
class Project:
# How long to wait before the first retry of a failed fetch.
RETRY_SLEEP_INITIAL_SEC = 4.0
# These objects can be shared between several working trees.
@property
def shareable_dirs(self):
@@ -1855,8 +1858,15 @@ class Project:
else:
alt_dir = None
custom_fetch_configured = (
self.manifest.manifestProject.use_local_gitdirs
and self.manifest.manifestProject.fetch_cmd
and not isinstance(self, MetaProject)
)
if (
clone_bundle
and not custom_fetch_configured
and alt_dir is None
and self._ApplyCloneBundle(
initial=is_new, quiet=quiet, verbose=verbose
@@ -1890,12 +1900,6 @@ class Project:
depth = None
clone_filter = clone_filter_for_depth
custom_fetch_configured = (
self.manifest.manifestProject.use_local_gitdirs
and self.manifest.manifestProject.fetch_cmd
and not isinstance(self, MetaProject)
)
remote_fetched = False
if custom_fetch_configured:
# The custom fetch command is executed unconditionally on every sync
@@ -2012,25 +2016,12 @@ class Project:
def GetCommitRevisionId(
self, all_refs: Optional[Dict[str, str]] = None
) -> str:
"""Get revisionId of a commit.
Use this method instead of GetRevisionId to get the id of the commit
rather than the id of the current git object (for example, a tag)
"""
"""Get revisionId of a commit."""
if self.revisionId:
return self.revisionId
if not self.revisionExpr.startswith(R_TAGS):
if all_refs is None:
all_refs = self._allrefs
return self.GetRevisionId(all_refs)
try:
return self.bare_git.ResolveCommit(self.revisionExpr)
except GitError:
raise ManifestInvalidRevisionError(
f"revision {self.revisionExpr} in {self.name} not found"
)
if all_refs is None and not self.revisionExpr.startswith(R_TAGS):
all_refs = self._allrefs
return self.GetRevisionId(all_refs)
def GetHeadRevisionId(self) -> Optional[str]:
"""Get the commit revision of the checked out HEAD.
@@ -2055,7 +2046,14 @@ class Project:
rem = self.GetRemote()
rev = rem.ToLocal(self.revisionExpr)
if all_refs is not None and rev in all_refs:
# Tags may be annotated, in which case all_refs holds the tag object
# SHA rather than the peeled commit SHA. Fall through to ResolveCommit
# so GetRevisionId always returns a commit SHA.
if (
all_refs is not None
and not rev.startswith(R_TAGS)
and rev in all_refs
):
return all_refs[rev]
try:
@@ -3228,7 +3226,8 @@ class Project:
"""Fetch using a custom command specified by repo.fetchcmd.
Populates the subshell with project-context environment variables,
including REPO_TREV (target revision resolved to a commit hash).
including REPO_TREV when the target revision is already resolved to a
commit hash.
Expects the target commit to be reachable and tracking refs/FETCH_HEAD
to be updated upon a successful 0 exit code.
@@ -3236,27 +3235,20 @@ class Project:
For a detailed contract, environment variables, and postconditions,
see docs/fetch-cmd.md.
"""
# Resolve REPO_TREV (target revision resolved to a full commit hash).
# Populate REPO_TREV when the target commit hash is already known
# (e.g. from a superproject or a SHA-pinned manifest revision). When the
# manifest names an unpinned branch or tag, leave REPO_TREV empty and
# let fetchcmd resolve REPO_RREV from its remote without a separate
# git ls-remote round-trip.
repo_trev = None
if self.revisionId and IsId(self.revisionId):
repo_trev = self.revisionId
if not repo_trev:
output = self._LsRemote(self.upstream or self.revisionExpr)
if output:
lines = output.splitlines()
if lines:
parts = lines[0].split()
if parts:
repo_trev = parts[0]
if not repo_trev:
logger.error("error: Cannot resolve REPO_TREV for %s", self.name)
return False
elif self.revisionExpr and IsId(self.revisionExpr):
repo_trev = self.revisionExpr
env = os.environ.copy()
env.update(self.GetEnvVars())
env["REPO_TREV"] = repo_trev
env["REPO_TREV"] = repo_trev or ""
cmd_str = self.manifest.manifestProject.fetch_cmd
if verbose:
@@ -3288,23 +3280,22 @@ class Project:
return False
# Verify postconditions.
try:
self.bare_git.cat_file("-e", repo_trev)
except GitError:
logger.error(
"error: Postcondition failed: %s not found in object store",
repo_trev,
)
return False
try:
ref_name = self.GetRemote().ToLocal(self.revisionExpr)
except GitError as e:
logger.error("error: Failed to resolve tracking ref: %s", e)
return False
try:
resolved_ref = self.bare_git.rev_parse(ref_name)
if resolved_ref != repo_trev:
if not IsId(self.revisionExpr):
try:
ref_name = self.GetRemote().ToLocal(self.revisionExpr)
except GitError as e:
logger.error("error: Failed to resolve tracking ref: %s", e)
return False
try:
resolved_ref = self.bare_git.ResolveCommit(ref_name)
except GitError:
logger.error(
"error: Postcondition failed: %s does not resolve to a "
"commit in object store",
ref_name,
)
return False
if repo_trev and resolved_ref != repo_trev:
logger.error(
"error: Postcondition failed: %s is %s, expected %s",
ref_name,
@@ -3312,12 +3303,10 @@ class Project:
repo_trev,
)
return False
except GitError:
logger.error("error: Postcondition failed: %s not found", ref_name)
return False
repo_trev = resolved_ref
try:
fetch_head = self.bare_git.rev_parse("FETCH_HEAD")
fetch_head = self.bare_git.ResolveCommit("FETCH_HEAD")
if fetch_head != repo_trev:
logger.error(
"error: Postcondition failed: FETCH_HEAD is %s, "
@@ -3327,7 +3316,10 @@ class Project:
)
return False
except GitError:
logger.error("error: Postcondition failed: FETCH_HEAD not found")
logger.error(
"error: Postcondition failed: FETCH_HEAD does not resolve to "
"a commit in object store"
)
return False
return True
@@ -3364,7 +3356,6 @@ class Project:
force_sync=False,
clone_filter=None,
retry_fetches=2,
retry_sleep_initial_sec=4.0,
retry_exp_factor=2.0,
) -> bool:
tag_name = None
@@ -3557,7 +3548,7 @@ class Project:
# At least one retry minimum due to git remote prune.
retry_fetches = max(retry_fetches, 2)
retry_cur_sleep = retry_sleep_initial_sec
retry_cur_sleep = self.RETRY_SLEEP_INITIAL_SEC
ok = prune_tried = False
try:
for try_n in range(retry_fetches):
@@ -5499,6 +5490,10 @@ class MetaProject(Project):
class RepoProject(MetaProject):
"""The MetaProject for repo itself."""
# The automated release process deletes the stable branch and then
# recreates it, so wait long enough for it to come back before retrying.
RETRY_SLEEP_INITIAL_SEC = 10.0
@property
def LastFetch(self):
try:
@@ -5507,6 +5502,18 @@ class RepoProject(MetaProject):
except OSError:
return 0
@property
def HasChanges(self) -> bool:
"""Does the remote revision differ from what is checked out?
Unlike MetaProject, this is also true when the remote moved backwards:
a release rollback force pushes an older release to stable, and users
who already upgraded have to move back to it too.
"""
# Not passing all_refs makes GetRevisionId peel annotated tags (from
# e.g. --repo-rev=v2.50) to the commit that HEAD points at.
return self.GetRevisionId() != self.GetHeadRevisionId()
class ManifestProject(MetaProject):
"""The MetaProject for manifests."""
@@ -6187,9 +6194,8 @@ class ManifestProject(MetaProject):
logger.warning(
"warning: git update of superproject %s failed, "
"repo sync will not use superproject to fetch source; "
"while this error is not fatal, and you can continue to "
"run repo sync, please run repo init with the "
"--no-use-superproject option to stop seeing this warning",
"this error is not fatal, and you can continue to "
"run repo sync",
submanifest,
)
if sync_result.fatal and use_superproject is not None:
+54 -18
View File
@@ -26,7 +26,7 @@ import platform
import shlex
import subprocess
import sys
from typing import NamedTuple
from typing import NamedTuple, Tuple
# These should never be newer than the main.py version since this needs to be a
@@ -129,7 +129,7 @@ if not REPO_REV:
BUG_URL = "https://issues.gerritcodereview.com/issues/new?component=1370071"
# increment this whenever we make important changes to this script
VERSION = (2, 65)
VERSION = (2, 69)
# increment this if the MAINTAINER_KEYS block is modified
KEYRING_VERSION = (2, 3)
@@ -220,6 +220,10 @@ repodir = ".repo" # name of repo's private directory
S_repo = "repo" # special repo repository
S_manifests = "manifests" # special manifest repository
REPO_MAIN = S_repo + "/main.py" # main script
# The automated release process deletes the stable branch and then recreates it
# at the new release, so a fetch can briefly miss it. Wait this long and
# refetch once before giving up on --repo-rev.
REV_REFETCH_DELAY_SEC = 10
import errno
@@ -228,6 +232,7 @@ import optparse
import re
import shutil
import stat
import time
import urllib.error
import urllib.request
@@ -631,6 +636,7 @@ def _Init(args):
dst = dst_final + ".tmp"
shutil.rmtree(dst, ignore_errors=True)
_Clone(url, dst, opt.clone_bundle, opt.quiet, opt.verbose)
_RefetchIfRevMissing(url, dst, rev, opt.quiet, opt.verbose)
remote_ref, rev = check_repo_rev(
dst, rev, opt.repo_verify, quiet=opt.quiet
@@ -1005,7 +1011,43 @@ def _Clone(url, cwd, clone_bundle, quiet, verbose):
_Fetch(url, cwd, "origin", quiet, verbose)
def resolve_repo_rev(cwd, committish):
def _RefetchIfRevMissing(
url: str, cwd: str, rev: str, quiet: bool, verbose: bool
) -> None:
"""Refetch once if |rev| doesn't resolve.
This doesn't fail if |rev| still doesn't resolve; check_repo_rev reports
that.
Any unresolvable |rev| is retried, not just branches. Telling a branch
apart from a tag or commit isn't possible while it's missing, and the only
cost for a bad --repo-rev is a short delay before the same error.
Args:
url: The URL of the repo source.
cwd: The git checkout to fetch into.
rev: The --repo-rev to wait for.
quiet: Whether to suppress fetch output.
verbose: Whether to show verbose fetch output.
"""
try:
resolve_repo_rev(cwd, rev, report_errors=False)
return
except CloneFailure:
pass
# Print even with --quiet since this explains the delay.
print(
f'repo: warning: unable to resolve "{rev}"; '
f"refetching in {REV_REFETCH_DELAY_SEC} seconds",
file=sys.stderr,
)
time.sleep(REV_REFETCH_DELAY_SEC)
_Fetch(url, cwd, "origin", quiet, verbose)
def resolve_repo_rev(
cwd: str, committish: str, report_errors: bool = True
) -> Tuple[str, str]:
"""Figure out what REPO_REV represents.
We support:
@@ -1016,6 +1058,7 @@ def resolve_repo_rev(cwd, committish):
Args:
cwd: The git checkout to run in.
committish: The REPO_REV argument to resolve.
report_errors: Whether to print an error if it can't be resolved.
Returns:
A tuple of (remote ref, commit) as makes sense for the committish.
@@ -1034,17 +1077,18 @@ def resolve_repo_rev(cwd, committish):
)
return None if ret.returncode else ret.stdout.strip()
def error(msg: str) -> CloneFailure:
if report_errors:
print(f"repo: error: {msg}", file=sys.stderr)
return CloneFailure()
# An explicit branch.
if committish.startswith("refs/heads/"):
remote_ref = committish
committish = committish[len("refs/heads/") :]
rev = resolve("refs/remotes/origin/%s" % committish)
if rev is None:
print(
f'repo: error: unknown branch "{committish}"',
file=sys.stderr,
)
raise CloneFailure()
raise error(f'unknown branch "{committish}"')
return (remote_ref, rev)
# An explicit tag.
@@ -1053,11 +1097,7 @@ def resolve_repo_rev(cwd, committish):
committish = committish[len("refs/tags/") :]
rev = resolve(remote_ref)
if rev is None:
print(
f'repo: error: unknown tag "{committish}"',
file=sys.stderr,
)
raise CloneFailure()
raise error(f'unknown tag "{committish}"')
return (remote_ref, rev)
# See if it's a short branch name.
@@ -1076,11 +1116,7 @@ def resolve_repo_rev(cwd, committish):
return (rev, rev)
# Give up!
print(
f'repo: error: unable to resolve "{committish}"',
file=sys.stderr,
)
raise CloneFailure()
raise error(f'unable to resolve "{committish}"')
def verify_rev(cwd, remote_ref, rev, quiet):
+13 -1
View File
@@ -13,6 +13,18 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# /// script
# requires-python = '>=3.11,<3.12'
# dependencies = [
# 'pytest==8.3.4',
# 'black==25.1.0',
# 'flake8==6.0.0',
# 'flake8-bugbear==24.12.12',
# 'flake8-comprehensions==3.16.0',
# 'isort==5.10.1'
# ]
# ///
"""Wrapper to run linters and pytest with the right settings."""
import functools
@@ -135,7 +147,7 @@ def run_isort():
cwd=ROOT_DIR,
)
argv = ["--check", ROOT_DIR]
argv = ["--check", "--diff", ROOT_DIR]
log_cmd("isort", argv)
return subprocess.run(
[sys.executable, "-m", "isort"] + argv,
+101
View File
@@ -0,0 +1,101 @@
attrs==24.2.0 \
--hash=sha256:37c4653e51383dc1249dba15635d59011ca11714a8e61db8cd6dc73011e4b603 \
--hash=sha256:81921eb96de3191c8258c199618104dd27ac608d9366f5e35d011eae1867ede2
# via flake8-bugbear
black==25.1.0 \
--hash=sha256:95e8176dae143ba9097f351d174fdaf0ccd29efb414b362ae3fd72bf0f710717 \
--hash=sha256:c9ed033e41998dd45d6f19bf9923e2444cb401185ccb26d1a2101a9049637c1f
click==8.0.3 \
--hash=sha256:353f466495adaeb40b6b5f592f9f91cb22372351c84caeb068132442a4518ef3 \
--hash=sha256:353f466495adaeb40b6b5f592f9f91cb22372351c84caeb068132442a4518ef3 \
--hash=sha256:b0a0492d07c8e7e4a88ee153ca6553d357a0746792d6868bfca79dc1a344bb1b
# via black
colorama==0.4.6 ; sys_platform == 'win32' \
--hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 \
--hash=sha256:8a5a9494346d7ae2d8f599e4661738007d2df56dbbf4ccbbf7a6a23679a3a302
# via
# click
# pytest
flake8==6.0.0 \
--hash=sha256:9b5bf0ebd06aff1601e38dfc4e91f9b99ba3cdf446e74ee25a9cbbe01aef81ee \
--hash=sha256:f34b5cb8bf908ff49e4673d3dbc557b2381a14132b637d53e9d3a59816f4b6ba
# via
# flake8-bugbear
# flake8-comprehensions
flake8-bugbear==24.12.12 \
--hash=sha256:1b6967436f65ca22a42e5373aaa6f2d87966ade9aa38d4baf2a1be550767545e
flake8-comprehensions==3.16.0 \
--hash=sha256:7c1eadc9d22e765f39857798febe7766b4d9c519793c6c149e3e13bf99693f70
iniconfig==1.1.1 \
--hash=sha256:011e24c64b7f47f6ebd835bb12a743f2fbe9a26d4cecaa7f53bc4f35ee9da8b3 \
--hash=sha256:9d81514370059186276e9a1799e5a9752e4c060233c254d6954b53fad2ff8a5e
# via pytest
isort==5.10.1 \
--hash=sha256:6f62d78e2f89b4500b080fe3a81690850cd254227f27f75c3a0c491a1f351ba7 \
--hash=sha256:ec0da0d88470e89ade75bf1e2ea7b4349ec906f95c84aff08e4d41dea6864d25
mccabe==0.7.0 \
--hash=sha256:6c2d30ab6be0e4a46919781807b4f0d834ebdd6c6e3dca0bda5a15f863427b6e \
--hash=sha256:b85abd0a321cdd357d0be34adbf49cf01453a857a23e94d44612ec6f159cb261
# via flake8
mypy-extensions==0.4.3 \
--hash=sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d \
--hash=sha256:38cbac35a502c4f13b9d9b1d06258fa51e01da4f5828b171b4d1200d00378e7c
# via black
packaging==23.0 \
--hash=sha256:37cb5b4759d17ae6bad78ee3347f0ee1057452895fcd1bbff48fb12908ce00a1 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2 \
--hash=sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2
# via
# black
# pytest
pathspec==0.9.0 \
--hash=sha256:7d15c4ddb0b5c802d161efc417ec1a2558ea2653c2e8ad9c19098201dc1c993a \
--hash=sha256:aaf90f4ad38a16bd3ebe0fa64da7f6ee01819c881120db68cc6440941a9adc08
# via black
platformdirs==2.5.2 \
--hash=sha256:027d8e83a2d7de06bbac4e5ef7e023c02b863d7ea5d079477e722bb41ab25788 \
--hash=sha256:e257a04b5d0528cb9836b766c8a16a85368e249cba7ae42888a403976ccd7d2d
# via black
pluggy==1.5.0 \
--hash=sha256:40da5bf96b15e359e280fd760a80c11d7a8ea20c3c296c3f3e09f8be67cec10b \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669 \
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669
# via pytest
py==1.11.0 \
--hash=sha256:607c53218732647dff4acdfcd50cb62615cedf612e72d1724fb1a0cc6405b378 \
--hash=sha256:607c53218732647dff4acdfcd50cb62615cedf612e72d1724fb1a0cc6405b378
pycodestyle==2.10.0 \
--hash=sha256:8a4eaf0d0495c7395bdab3589ac2db602797d76207242c17d470186815706610 \
--hash=sha256:aacf8d28ecea10fad31b2d14943a42d85d007f4d2fee8f212fc05751c8460647
# via flake8
pyflakes==3.0.1 \
--hash=sha256:a968b8baa43926124b692c7e76279bc5fa8c6ef9a2281e4a2a475be69f954bdf \
--hash=sha256:ec55bf7fe21fff7f1ad2f7da62363d749e2a470500eab1b555334b67aa1ef8cf
# via flake8
pyparsing==3.0.7 \
--hash=sha256:943c220ca64e93940b74dd070152ed9ef1402bc3b28fd1f899843bee34e83708 \
--hash=sha256:a6c06a88f252e6c322f65faf8f418b16213b51bdfaece0524c1c1bc30c63c484
pytest==8.3.4 \
--hash=sha256:00cba7845d2dccfe5403025a688d60ceae228b1ddaa866d2eb86b2edb6923e73 \
--hash=sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6
toml==0.10.1 \
--hash=sha256:4099063d9b50219bf09bfa693ab8dcd96ce585cee9fdadce5a48a08ee23b0646 \
--hash=sha256:bda89d5935c2eac546d648028b9901107a595863cb36bae0c73ac804a9b4ce88
tomli==2.0.1 \
--hash=sha256:864f310e021d68415d6ba45b23e29808f99703b471d884267d6942eecdf8d6c5 \
--hash=sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc
typing-extensions==4.3.0 \
--hash=sha256:09126dab54422810a541e078c37e62024e7260a022d4f67c075a6707d9553649 \
--hash=sha256:25642c956049920a5aa49edcdd6ab1e06d7e5d467fc00e0506c44ac86fbfca02
-135
View File
@@ -1,135 +0,0 @@
# This is a vpython "spec" file.
#
# Read more about `vpython` and how to modify this file here:
# https://chromium.googlesource.com/infra/infra/+/main/doc/users/vpython.md
# List of available wheels:
# https://chromium.googlesource.com/infra/infra/+/main/infra/tools/dockerbuild/wheels.md
python_version: "3.11"
wheel: <
name: "infra/python/wheels/pytest-py3"
version: "version:8.3.4"
>
# Required by pytest==8.3.4
wheel: <
name: "infra/python/wheels/py-py2_py3"
version: "version:1.11.0"
>
# Required by pytest==8.3.4
wheel: <
name: "infra/python/wheels/iniconfig-py3"
version: "version:1.1.1"
>
# Required by pytest==8.3.4
wheel: <
name: "infra/python/wheels/packaging-py3"
version: "version:23.0"
>
# Required by pytest==8.3.4
wheel: <
name: "infra/python/wheels/pluggy-py3"
version: "version:1.5.0"
>
# Required by pytest==8.3.4
wheel: <
name: "infra/python/wheels/toml-py3"
version: "version:0.10.1"
>
# Required by pytest==8.3.4
wheel: <
name: "infra/python/wheels/pyparsing-py3"
version: "version:3.0.7"
>
# Required by pytest==8.3.4 and flake8-bugbear==24.12.12
wheel: <
name: "infra/python/wheels/attrs-py3"
version: "version:24.2.0"
>
# NB: Keep in sync with constraints.txt.
wheel: <
name: "infra/python/wheels/black-py3"
version: "version:25.1.0"
>
# Required by black==25.1.0
wheel: <
name: "infra/python/wheels/mypy-extensions-py3"
version: "version:0.4.3"
>
# Required by black==25.1.0
wheel: <
name: "infra/python/wheels/tomli-py3"
version: "version:2.0.1"
>
# Required by black==25.1.0
wheel: <
name: "infra/python/wheels/platformdirs-py3"
version: "version:2.5.2"
>
# Required by black==25.1.0
wheel: <
name: "infra/python/wheels/pathspec-py3"
version: "version:0.9.0"
>
# Required by black==25.1.0
wheel: <
name: "infra/python/wheels/typing-extensions-py3"
version: "version:4.3.0"
>
# Required by black==25.1.0
wheel: <
name: "infra/python/wheels/click-py3"
version: "version:8.0.3"
>
wheel: <
name: "infra/python/wheels/flake8-py2_py3"
version: "version:6.0.0"
>
# Required by flake8==6.0.0
wheel: <
name: "infra/python/wheels/mccabe-py2_py3"
version: "version:0.7.0"
>
# Required by flake8==6.0.0
wheel: <
name: "infra/python/wheels/pyflakes-py2_py3"
version: "version:3.0.1"
>
# Required by flake8==6.0.0
wheel: <
name: "infra/python/wheels/pycodestyle-py2_py3"
version: "version:2.10.0"
>
wheel: <
name: "infra/python/wheels/flake8-bugbear-py3"
version: "version:24.12.12"
>
wheel: <
name: "infra/python/wheels/flake8-comprehensions-py3"
version: "version:3.16.0"
>
wheel: <
name: "infra/python/wheels/isort-py3"
version: "version:5.10.1"
>
+25 -6
View File
@@ -28,6 +28,14 @@ class Gc(Command):
helpSummary = "Cleaning up internal repo and Git state."
helpUsage = """
%prog
"""
helpDescription = """
The '%prog' command removes internal Git data for projects that are no
longer active in the current configuration.
Projects excluded by the configured manifest groups are inactive by
default. Use --keep-manifest-projects to keep Git data for projects that
are still in the manifests.
"""
def _Options(self, p):
@@ -53,6 +61,12 @@ class Gc(Command):
help="repack all projects that use partial clone with "
"filter=blob:none",
)
p.add_option(
"--keep-manifest-projects",
default=False,
action="store_true",
help="keep Git data for projects still in the manifests",
)
def _find_git_to_delete(
self, to_keep: Set[str], start_dir: str
@@ -292,16 +306,21 @@ class Gc(Command):
args, all_manifests=not opt.this_manifest_only
)
# If the user specified projects, fetch the global list separately
# to avoid deleting untargeted projects.
if args:
all_projects = self.GetProjects(
if opt.keep_manifest_projects:
projects_to_keep = self.GetProjects(
[],
groups="all",
missing_ok=True,
all_manifests=not opt.this_manifest_only,
)
elif args:
projects_to_keep = self.GetProjects(
[], all_manifests=not opt.this_manifest_only
)
else:
all_projects = projects
projects_to_keep = projects
ret = self.delete_unused_projects(all_projects, opt)
ret = self.delete_unused_projects(projects_to_keep, opt)
if ret != 0:
return ret
+222 -37
View File
@@ -58,6 +58,7 @@ from command import DEFAULT_LOCAL_JOBS
from command import MirrorSafeCommand
from command import WORKER_BATCH_SIZE
from error import GitError
from error import ManifestParseError
from error import RepoChangedException
from error import RepoError
from error import RepoExitError
@@ -97,6 +98,69 @@ _BLOAT_SIZE_GARBAGE_THRESHOLD_KB = 1 * 1024 * 1024 # 1 GiB in KiB
logger = RepoLogger(__file__)
# Manifest server RPC that takes a single struct of named options. Servers
# that do not implement it are called via the positional GetApprovedManifest.
_GET_APPROVED_MANIFEST_V2 = "GetApprovedManifestV2"
def _IsMethodNotFoundFault(fault: xmlrpc.client.Fault, method: str) -> bool:
"""Whether |fault| means the server does not implement |method|."""
if fault.faultCode == xmlrpc.client.METHOD_NOT_FOUND:
return True
# Python's SimpleXMLRPCDispatcher uses faultCode 1 with this message.
return f'method "{method}" is not supported' in str(fault.faultString)
def _GetApprovedManifest(
server: xmlrpc.client.ServerProxy,
branch: str,
target: Optional[str],
best_effort: bool,
) -> Tuple[bool, str]:
"""Requests the approved manifest from the manifest server.
Args:
server: The manifest server proxy.
branch: The manifest branch.
target: The build target, if any.
best_effort: The client can proceed without a manifest (e.g. by
falling back to ToT), so the server may decline the request.
Returns:
A (success, manifest XML or error message) tuple.
"""
request = {"branch": branch}
if target:
request["target"] = target
if best_effort:
request["best_effort"] = True
try:
response = server.GetApprovedManifestV2(request)
except xmlrpc.client.Fault as e:
if not _IsMethodNotFoundFault(e, _GET_APPROVED_MANIFEST_V2):
raise
# Older server: |best_effort| has no equivalent and is dropped.
if target:
success, manifest_str = server.GetApprovedManifest(branch, target)
else:
success, manifest_str = server.GetApprovedManifest(branch)
return success, manifest_str
if not isinstance(response, dict):
raise SmartSyncError(
f"error: invalid {_GET_APPROVED_MANIFEST_V2} response: {response!r}"
)
if response.get("success"):
manifest_xml = response.get("manifest_xml")
if not isinstance(manifest_xml, str):
raise SmartSyncError(
f"error: {_GET_APPROVED_MANIFEST_V2} response has invalid "
f"manifest_xml: {manifest_xml!r}"
)
return True, manifest_xml
return False, response.get("error_message", "unknown error")
def _SafeCheckoutOrder(checkouts: List[Project]) -> List[List[Project]]:
"""Generate a sequence of checkouts that is safe to perform.
@@ -764,9 +828,16 @@ later is required to fix a server side protocol bug.
"-s",
"--smart-sync",
action="store_true",
default=None,
help="smart sync using manifest from the latest known good "
"build",
)
p.add_option(
"--no-smart-sync",
dest="smart_sync",
action="store_false",
help="disable smart sync and sync to ToT instead",
)
p.add_option(
"-t",
"--smart-tag",
@@ -819,13 +890,42 @@ later is required to fix a server side protocol bug.
or opt.current_branch_only
)
def _GetSuperprojectRevision(
self, opt: optparse.Values, manifest: XmlManifest
) -> Optional[str]:
"""Returns the --superproject-revision to apply to manifest.
The option only applies to the outer manifest.
Args:
opt: Program options returned from optparse. See _Options().
manifest: The manifest to check.
Returns:
The requested superproject revision, or None if manifest's
superproject should not be pinned.
"""
if manifest == self.outer_manifest:
return opt.superproject_revision
return None
def _ConfigureSuperproject(
self,
opt: optparse.Values,
manifest,
revision: Optional[str] = None,
self, opt: optparse.Values, manifest: XmlManifest
) -> bool:
"""Configure superproject with options."""
"""Configure superproject with options.
The configuration lives on the manifest's Superproject object, which
is discarded whenever the manifest is unloaded (see _ReloadManifest
and XmlManifest.Unload). Callers must therefore configure the
superproject again after every manifest reload.
Args:
opt: Program options returned from optparse. See _Options().
manifest: The manifest whose superproject should be configured.
Returns:
Whether superproject messages should be printed.
"""
if not manifest.superproject:
return False
manifest.superproject.SetQuiet(not opt.verbose)
@@ -833,6 +933,7 @@ later is required to fix a server side protocol bug.
opt.use_superproject, manifest
)
manifest.superproject.SetPrintMessages(print_messages)
revision = self._GetSuperprojectRevision(opt, manifest)
if revision:
manifest.superproject.SetRevisionId(revision)
return print_messages
@@ -858,6 +959,11 @@ later is required to fix a server side protocol bug.
m.superproject for m in manifest.all_children
)
if not have_superproject:
superproject_logging_data.update(
superproject=False,
haslocalmanifests=bool(manifest.HasLocalManifests),
hassuperprojecttag=False,
)
return
if opt.local_only and manifest.superproject:
@@ -882,7 +988,6 @@ later is required to fix a server side protocol bug.
for p in all_projects:
per_manifest[p.manifest.path_prefix].append(p)
superproject_logging_data = {}
need_unload = False
for m in self.ManifestList(opt):
if m.path_prefix not in per_manifest:
@@ -923,12 +1028,8 @@ later is required to fix a server side protocol bug.
else:
if print_messages:
logger.warning(
"%s: warning: Update of revisionId from superproject "
"has failed, repo sync will not use superproject to "
"fetch the source. Please resync with the "
"--no-use-superproject option to avoid this repo "
"warning.",
m.path_prefix,
"error: Update of revsionId from superproject "
"has failed"
)
if update_result.fatal and opt.use_superproject is not None:
raise SuperprojectError()
@@ -2051,7 +2152,13 @@ later is required to fix a server side protocol bug.
return server_url, transport
def _SmartSyncSetup(self, opt, smart_sync_manifest_path, manifest):
def _SmartSyncSetup(
self,
opt: optparse.Values,
smart_sync_manifest_path: str,
manifest: XmlManifest,
best_effort: bool = False,
) -> str:
if not manifest.manifest_server:
raise SmartSyncError(
"error: cannot smart sync: no manifest server defined in "
@@ -2069,7 +2176,12 @@ later is required to fix a server side protocol bug.
try:
server = xmlrpc.client.Server(server_url, transport=transport)
if opt.smart_sync:
branch = self._GetBranch(manifest.manifestProject)
# SYNC_BRANCH overrides the branch sent to the manifest server
# (e.g. to select a specific CI branch that builds the same
# manifest branch). An empty value is treated as unset.
branch = os.environ.get("SYNC_BRANCH")
if not branch:
branch = self._GetBranch(manifest.manifestProject)
target = None
if "SYNC_TARGET" in os.environ:
target = os.environ["SYNC_TARGET"]
@@ -2092,12 +2204,9 @@ later is required to fix a server side protocol bug.
os.environ["TARGET_BUILD_VARIANT"],
)
if target:
[success, manifest_str] = server.GetApprovedManifest(
branch, target
)
else:
[success, manifest_str] = server.GetApprovedManifest(branch)
success, manifest_str = _GetApprovedManifest(
server, branch, target, best_effort
)
else:
assert opt.smart_tag
[success, manifest_str] = server.GetManifest(opt.smart_tag)
@@ -2153,7 +2262,7 @@ later is required to fix a server side protocol bug.
mp: the manifestProject to query.
manifest_name: Manifest file to be reloaded.
"""
if opt.superproject_revision and mp.manifest == self.outer_manifest:
if self._GetSuperprojectRevision(opt, mp.manifest):
self._SyncToSuperprojectRev(
opt, mp.manifest, mp, manifest_name, errors
)
@@ -2249,10 +2358,6 @@ later is required to fix a server side protocol bug.
if opt.manifest_name and opt.smart_tag:
self.OptionParser.error("cannot combine -m and -t")
if opt.manifest_server_username or opt.manifest_server_password:
if not (opt.smart_sync or opt.smart_tag):
self.OptionParser.error(
"-u and -p may only be combined with -s or -t"
)
if None in [
opt.manifest_server_username,
opt.manifest_server_password,
@@ -2369,18 +2474,19 @@ later is required to fix a server side protocol bug.
def _SyncToSuperprojectRev(
self,
opt: optparse.Values,
manifest,
manifest: XmlManifest,
mp: Project,
manifest_name: Optional[str],
errors: List[Exception],
) -> None:
"""Sync to a specific superproject commit."""
"""Sync to a specific superproject commit.
Only valid for the outer manifest; see _GetSuperprojectRevision.
"""
if not manifest.superproject:
raise SyncError("superproject not defined in manifest")
self._ConfigureSuperproject(
opt, manifest, revision=opt.superproject_revision
)
self._ConfigureSuperproject(opt, manifest)
sync_result = manifest.superproject.Sync(self.git_event_log)
if not sync_result.success:
@@ -2413,6 +2519,33 @@ later is required to fix a server side protocol bug.
"failed to sync manifest project", aggregate_errors=[e]
)
def _ResolveSmartSyncOption(
self, opt: optparse.Values, manifest: XmlManifest
) -> bool:
"""Resolve opt.smart_sync from the CLI flags and manifest default.
Returns:
True if smart_sync was enabled implicitly by the manifest default,
False otherwise.
"""
implicit = (
opt.smart_sync is None
and not opt.smart_tag
and not opt.manifest_name
and not opt.superproject_revision
and not opt.local_only
and getattr(manifest.default, "sync_smartsync", False) is True
)
if opt.smart_sync is None:
opt.smart_sync = implicit
if (
opt.manifest_server_username or opt.manifest_server_password
) and not (opt.smart_sync or opt.smart_tag):
raise SmartSyncError("-u and -p may only be combined with -s or -t")
return implicit
def _ExecuteHelper(self, opt, args, errors):
manifest = self.outer_manifest
if not opt.outer_manifest:
@@ -2427,11 +2560,62 @@ later is required to fix a server side protocol bug.
if opt.clone_bundle is None:
opt.clone_bundle = manifest.CloneBundle
# Without an explicit smart sync option, -l does not contact the
# manifest server, so it keeps the smart sync manifest the client last
# synced to, whether that came from -s or from the manifest's
# sync-smartsync default. This keeps `sync -n` + `sync -l` equivalent
# to a single sync. Computed before _ResolveSmartSyncOption, which
# makes an unset opt.smart_sync indistinguishable from --no-smart-sync.
reuse_smart_sync_override = (
opt.local_only
and opt.smart_sync is None
and not opt.smart_tag
and not opt.manifest_name
and not opt.superproject_revision
and os.path.isfile(smart_sync_manifest_path)
)
implicit_smart_sync = self._ResolveSmartSyncOption(opt, manifest)
if opt.smart_sync or opt.smart_tag:
manifest_name = self._SmartSyncSetup(
opt, smart_sync_manifest_path, manifest
)
else:
try:
manifest_name = self._SmartSyncSetup(
opt,
smart_sync_manifest_path,
manifest,
best_effort=implicit_smart_sync,
)
except (SmartSyncError, ManifestParseError) as e:
if not implicit_smart_sync:
raise
# The user did not ask for a smart sync, so failing to get a
# usable manifest from the server should not fail the sync.
# Fall back to a regular ToT sync instead.
opt.smart_sync = False
# Override() records the override before parsing it, so an
# unparsable manifest would otherwise leave the client pinned
# to a file it cannot load.
manifest.ClearOverride()
if opt.verbose:
logger.warning(
"warning: smart sync failed; falling back to ToT: %s", e
)
elif reuse_smart_sync_override:
manifest_name = os.path.basename(smart_sync_manifest_path)
try:
self._ReloadManifest(manifest_name, manifest)
except ManifestParseError as e:
reuse_smart_sync_override = False
manifest_name = None
manifest.ClearOverride()
if opt.verbose:
logger.warning(
"warning: failed to load existing smart sync manifest; "
"falling back to ToT: %s",
e,
)
if not (opt.smart_sync or opt.smart_tag or reuse_smart_sync_override):
if os.path.isfile(smart_sync_manifest_path):
try:
platform_utils.remove(smart_sync_manifest_path)
@@ -3369,13 +3553,13 @@ class LocalSyncState:
def _Get(self, project, key):
self._Load()
p = project.relpath
p = project.RelPath(local=False)
if p not in self._state:
return
return self._state[p].get(key)
def _Set(self, project, key):
p = project.relpath
p = project.RelPath(local=False)
if p not in self._state:
self._state[p] = {}
self._state[p][key] = self._time
@@ -3403,8 +3587,9 @@ class LocalSyncState:
if not self._state:
return
delete = set()
outer_topdir = self._manifest.outer_client.topdir
for path in self._state:
gitdir = os.path.join(self._manifest.topdir, path, ".git")
gitdir = os.path.join(outer_topdir, path, ".git")
if not os.path.exists(gitdir) or os.path.islink(gitdir):
delete.add(path)
if not delete:
+20
View File
@@ -268,3 +268,23 @@ def test_remote_save_with_push_url_without_projectname(
def test_is_id(rev: str, expected: bool) -> None:
"""Test IsId identifies both SHA-1 and SHA-256 hashes."""
assert git_config.IsId(rev) == expected
def test_repo_config_for_user_is_not_shared_with_git_config(
monkeypatch: pytest.MonkeyPatch, tmp_home_dir: Path
) -> None:
"""RepoConfig.ForUser() must not reuse GitConfig.ForUser()'s cache."""
monkeypatch.setattr(git_config.GitConfig, "_ForUser", None)
monkeypatch.setattr(git_config.RepoConfig, "_ForUser", None)
monkeypatch.delenv("REPO_CONFIG_DIR", raising=False)
# Populate the GitConfig cache first, as manifest loading does.
git_cfg = git_config.GitConfig.ForUser()
repo_cfg = git_config.RepoConfig.ForUser()
assert repo_cfg is not git_cfg
assert git_cfg.file == str(tmp_home_dir / ".gitconfig")
assert repo_cfg.file == str(tmp_home_dir / ".repoconfig" / "config")
# And the RepoConfig instance is cached on its own class.
assert git_config.RepoConfig.ForUser() is repo_cfg
assert git_config.GitConfig.ForUser() is git_cfg
+56
View File
@@ -420,6 +420,62 @@ class TestXmlManifest:
)
manifest.ToXml()
def test_sync_smartsync(self, repo_client: RepoClient) -> None:
"""Check sync-smartsync handling."""
manifest = repo_client.get_xml_manifest(
"<manifest><default /></manifest>"
)
assert manifest.default.sync_smartsync is False
manifest = repo_client.get_xml_manifest(
'<manifest><default sync-smartsync="true" /></manifest>'
)
assert manifest.default.sync_smartsync is True
assert (
manifest.ToXml().toxml() == '<?xml version="1.0" ?>'
'<manifest><default sync-smartsync="true"/></manifest>'
)
manifest = repo_client.get_xml_manifest(
'<manifest><default sync-smartsync="false" /></manifest>'
)
assert manifest.default.sync_smartsync is False
assert manifest.ToXml().toxml() == '<?xml version="1.0" ?><manifest/>'
def test_clear_override(self, repo_client: RepoClient) -> None:
"""Check ClearOverride reverts to the default manifest."""
manifest_xml_fmt = (
'<manifest><remote name="r" fetch="." />'
'<default remote="r" revision="main" />'
'<project name="%s" /></manifest>'
)
manifest = repo_client.get_xml_manifest(manifest_xml_fmt % "default")
(repo_client.manifest_dir / "good.xml").write_text(
manifest_xml_fmt % "override", encoding="utf-8"
)
(repo_client.manifest_dir / "bad.xml").write_text(
"<manifest>", encoding="utf-8"
)
# No override in effect: a no-op.
manifest.ClearOverride()
assert list(manifest.paths) == ["default"]
manifest.Override("good.xml")
assert list(manifest.paths) == ["override"]
manifest.ClearOverride()
assert not manifest.manifestFileOverrides
assert list(manifest.paths) == ["default"]
# Override() registers the override before parsing it, so a failed
# Override() leaves the client pinned until ClearOverride().
with pytest.raises(error.ManifestParseError):
manifest.Override("bad.xml")
with pytest.raises(error.ManifestParseError):
manifest.Load()
manifest.ClearOverride()
assert list(manifest.paths) == ["default"]
class TestIncludeElement:
"""Tests for <include>."""
+307 -12
View File
@@ -15,12 +15,13 @@
"""Unittests for the project.py module."""
import contextlib
import io
import os
from pathlib import Path
import shutil
import subprocess
import tempfile
from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple
from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Type
import unittest
from unittest import mock
@@ -2061,6 +2062,87 @@ def test_metaproject_has_changes_bounds_revision_walk() -> None:
meta._revlist.assert_called_once_with("-1", "^HEAD", "remote")
@pytest.mark.parametrize(
"project_cls,revision,expected",
(
(project.RepoProject, "refs/remotes/origin/current", False),
(project.RepoProject, "refs/remotes/origin/newer", True),
(project.RepoProject, "refs/remotes/origin/divergent", True),
# A release rollback rewinds stable, and repo itself has to follow.
(project.RepoProject, "refs/remotes/origin/older", True),
# --repo-rev can pin repo to an annotated tag, which must be peeled.
(project.RepoProject, "refs/tags/current-tag", False),
# Other MetaProjects only update when the remote has new commits.
(project.ManifestProject, "refs/remotes/origin/current", False),
(project.ManifestProject, "refs/remotes/origin/newer", True),
(project.ManifestProject, "refs/remotes/origin/divergent", True),
(project.ManifestProject, "refs/remotes/origin/older", False),
),
ids=lambda v: v.__name__ if isinstance(v, type) else None,
)
def test_metaproject_has_changes(
project_cls: Type[project.MetaProject], revision: str, expected: bool
) -> None:
"""HasChanges compares the tracked revision with the checked out one."""
with utils_for_test.TempGitTree() as tempdir:
def git(*args: str) -> str:
return subprocess.run(
["git", "-C", tempdir, *args],
check=True,
capture_output=True,
text=True,
).stdout.strip()
def commit(msg: str) -> str:
git("commit", "-q", "--allow-empty", "-m", msg)
return git("rev-parse", "HEAD")
# older -> current -> newer, plus divergent branching off older.
revs = {name: commit(name) for name in ("older", "current", "newer")}
git("checkout", "-q", "--detach", revs["older"])
revs["divergent"] = commit("divergent")
git("checkout", "-q", "main")
git("reset", "-q", "--hard", revs["current"])
for name, rev in revs.items():
git("update-ref", f"refs/remotes/origin/{name}", rev)
git("tag", "-a", "-m", "current-tag", "current-tag", revs["current"])
proj = _create_meta_project(tempdir, project_cls)
proj.revisionExpr = revision
assert proj.HasChanges is expected
@pytest.mark.parametrize(
"project_cls,expected_sleep",
(
(project.ManifestProject, 4.0),
# Repo waits longer for a briefly deleted stable branch to come back.
(project.RepoProject, 10.0),
),
ids=lambda v: v.__name__ if isinstance(v, type) else None,
)
def test_remote_fetch_retry_sleep(
project_cls: Type[project.MetaProject], expected_sleep: float
) -> None:
"""A failed fetch waits RETRY_SLEEP_INITIAL_SEC before retrying."""
with utils_for_test.TempGitTree() as tempdir:
proj = _create_meta_project(tempdir, project_cls)
proj.config.GetRemote("origin").ResetFetch()
proj.revisionExpr = "refs/heads/stable"
with mock.patch.object(project, "GitCommand") as mock_git:
# What git does while stable is deleted.
mock_git.return_value.Wait.return_value = 128
mock_git.return_value.stdout = (
"fatal: couldn't find remote ref refs/heads/stable\n"
)
with mock.patch.object(project.time, "sleep") as mock_sleep:
assert not proj._RemoteFetch(output_redir=io.StringIO())
assert mock_sleep.call_args_list == [mock.call(expected_sleep)]
@pytest.mark.parametrize(
"state,expect_failure",
[
@@ -3225,8 +3307,11 @@ class GetEnvVarsTests(unittest.TestCase):
self.assertEqual(env["REPO_LREV"], "")
def _create_manifest_project(tempdir: str) -> project.ManifestProject:
"""Return a ManifestProject for a new .repo/ under |tempdir|."""
def _create_meta_project(
tempdir: str,
project_cls: Type[project.MetaProject] = project.ManifestProject,
) -> project.MetaProject:
"""Return a |project_cls| for a new .repo/ under |tempdir|."""
repodir = os.path.join(tempdir, ".repo")
manifest_dir = os.path.join(repodir, "manifests")
manifest_file = os.path.join(repodir, manifest_xml.MANIFEST_FILE_NAME)
@@ -3234,7 +3319,7 @@ def _create_manifest_project(tempdir: str) -> project.ManifestProject:
os.mkdir(manifest_dir)
manifest = manifest_xml.XmlManifest(repodir, manifest_file)
return project.ManifestProject(
return project_cls(
manifest, "test/manifest", os.path.join(tempdir, ".git"), tempdir
)
@@ -3243,7 +3328,7 @@ class FetchCmdTests(unittest.TestCase):
"""Tests for fetch_cmd feature."""
def setUpManifest(self, tempdir):
return _create_manifest_project(tempdir)
return _create_meta_project(tempdir)
def _get_project(self, tempdir):
proj = _create_mock_project(
@@ -3252,12 +3337,13 @@ class FetchCmdTests(unittest.TestCase):
proj.GetRevisionId = mock.MagicMock(return_value="1234abcd")
return proj
def test_fetch_cmd_execution(self):
"""Test that fetch_cmd is executed with correct environment."""
def test_fetch_cmd_execution_unpinned_branch(self) -> None:
"""Test fetch_cmd with an unpinned branch does not call ls-remote."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir)
proj.bare_git.rev_parse.return_value = "1234abcd"
proj.revisionId = None
proj.revisionExpr = "main"
proj.bare_git.ResolveCommit.return_value = "1234abcd"
mock_remote = mock.MagicMock()
mock_remote.ToLocal.return_value = "refs/remotes/origin/main"
proj.GetRemote = mock.MagicMock(return_value=mock_remote)
@@ -3268,15 +3354,115 @@ class FetchCmdTests(unittest.TestCase):
self.assertTrue(res)
mock_run.assert_called_once()
proj._LsRemote.assert_not_called()
args, kwargs = mock_run.call_args
self.assertEqual(args[0], "echo hi")
self.assertEqual(kwargs["shell"], True)
self.assertEqual(kwargs["cwd"], tempdir)
self.assertEqual(kwargs["env"]["REPO_TREV"], "1234abcd")
self.assertEqual(kwargs["env"]["REPO_TREV"], "")
self.assertEqual(kwargs["env"]["REPO_RREV"], "main")
self.assertEqual(
kwargs["env"]["REPO_PROJECT_FETCH_URL"],
"http://example.com/repo",
)
proj.bare_git.ResolveCommit.assert_any_call(
"refs/remotes/origin/main"
)
proj.bare_git.ResolveCommit.assert_any_call("FETCH_HEAD")
def test_fetch_cmd_execution_pinned_revision_id(self) -> None:
"""Test fetch_cmd populates REPO_TREV when revisionId is set."""
with utils_for_test.TempGitTree() as tempdir:
sha = "0123456789abcdef0123456789abcdef01234567"
proj = self._get_project(tempdir)
proj.revisionId = sha
proj.revisionExpr = "main"
proj.bare_git.ResolveCommit.return_value = sha
mock_remote = mock.MagicMock()
mock_remote.ToLocal.return_value = "refs/remotes/origin/main"
proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("subprocess.run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
res = proj._CustomFetch()
self.assertTrue(res)
proj._LsRemote.assert_not_called()
_, kwargs = mock_run.call_args
self.assertEqual(kwargs["env"]["REPO_TREV"], sha)
def test_fetch_cmd_execution_pinned_revision_expr_sha(self) -> None:
"""Test fetch_cmd populates REPO_TREV when revisionExpr is a SHA."""
with utils_for_test.TempGitTree() as tempdir:
sha = "0123456789abcdef0123456789abcdef01234567"
proj = self._get_project(tempdir)
proj.revisionId = None
proj.revisionExpr = sha
proj.bare_git.ResolveCommit.return_value = sha
proj.GetRemote = mock.MagicMock()
with mock.patch("subprocess.run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
res = proj._CustomFetch()
self.assertTrue(res)
proj._LsRemote.assert_not_called()
proj.GetRemote.assert_not_called()
_, kwargs = mock_run.call_args
self.assertEqual(kwargs["env"]["REPO_TREV"], sha)
proj.bare_git.ResolveCommit.assert_called_once_with("FETCH_HEAD")
def test_fetch_cmd_postcondition_fails_when_fetch_head_mismatches_ref(
self,
) -> None:
"""Test _CustomFetch fails when FETCH_HEAD disagrees with the ref."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir)
proj.revisionId = None
proj.revisionExpr = "main"
proj.bare_git.ResolveCommit.side_effect = [
"11111111" * 5,
"22222222" * 5,
]
mock_remote = mock.MagicMock()
mock_remote.ToLocal.return_value = "refs/remotes/origin/main"
proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("subprocess.run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
self.assertFalse(proj._CustomFetch())
def test_fetch_cmd_postcondition_fails_when_ref_mismatches_pinned_trev(
self,
) -> None:
"""Test _CustomFetch fails when the ref disagrees with REPO_TREV."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir)
proj.revisionId = "11111111" * 5
proj.revisionExpr = "main"
proj.bare_git.ResolveCommit.return_value = "22222222" * 5
mock_remote = mock.MagicMock()
mock_remote.ToLocal.return_value = "refs/remotes/origin/main"
proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("subprocess.run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
self.assertFalse(proj._CustomFetch())
def test_fetch_cmd_postcondition_fails_when_ref_missing(self) -> None:
"""Test _CustomFetch fails when tracking ref or FETCH_HEAD fails."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir)
proj.revisionId = None
proj.revisionExpr = "main"
proj.bare_git.ResolveCommit.side_effect = error.GitError("missing")
mock_remote = mock.MagicMock()
mock_remote.ToLocal.return_value = "refs/remotes/origin/main"
proj.GetRemote = mock.MagicMock(return_value=mock_remote)
with mock.patch("subprocess.run") as mock_run:
mock_run.return_value = mock.MagicMock(returncode=0, stderr="")
self.assertFalse(proj._CustomFetch())
def test_sync_fetch_cmd_requires_use_local_gitdirs(self):
"""Test that fetch_cmd requires use_local_gitdirs."""
@@ -3297,6 +3483,31 @@ class FetchCmdTests(unittest.TestCase):
result = fakeproj.Sync(use_local_gitdirs=False)
self.assertFalse(result)
def test_sync_network_half_skips_clone_bundle_with_fetch_cmd(self) -> None:
"""Test Sync_NetworkHalf skips clone.bundle when fetchcmd is set."""
with tempfile.TemporaryDirectory() as tempdir:
worktree = os.path.join(tempdir, "new-proj")
gitdir = os.path.join(worktree, ".git")
proj = _create_mock_project(
worktree,
use_local_gitdirs=True,
fetch_cmd="echo hi",
gitdir=gitdir,
objdir=gitdir,
)
proj._InitGitDir = mock.MagicMock()
proj._InitRemote = mock.MagicMock()
proj._InitMRef = mock.MagicMock()
proj._CustomFetch = mock.MagicMock(return_value=True)
proj._ApplyCloneBundle = mock.MagicMock(return_value=True)
res = proj.Sync_NetworkHalf(clone_bundle=True, is_new=True)
self.assertIsNone(res.error)
self.assertTrue(res.remote_fetched)
proj._ApplyCloneBundle.assert_not_called()
proj._CustomFetch.assert_called_once()
class ReprojectCmdTests(unittest.TestCase):
"""Tests for the repo.reprojectcmd feature."""
@@ -3750,7 +3961,7 @@ class ReprojectCmdTests(unittest.TestCase):
def test_metaproject_never_uses_the_command(self) -> None:
"""Test .repo/manifests and .repo/repo are checked out by Git."""
with utils_for_test.TempGitTree() as tempdir:
fakeproj = _create_manifest_project(tempdir)
fakeproj = _create_meta_project(tempdir)
fakeproj.config.SetString("repo.reprojectcmd", "echo hi")
fakeproj.config.SetBoolean("repo.uselocalgitdirs", True)
self.assertFalse(fakeproj.UseReprojectCmd)
@@ -3758,7 +3969,7 @@ class ReprojectCmdTests(unittest.TestCase):
def test_sync_reproject_cmd_requires_use_local_gitdirs(self) -> None:
"""Test that repo.reprojectcmd requires repo.uselocalgitdirs."""
with utils_for_test.TempGitTree() as tempdir:
fakeproj = _create_manifest_project(tempdir)
fakeproj = _create_meta_project(tempdir)
class DummyManifest:
is_submanifest = False
@@ -3776,6 +3987,51 @@ class ReprojectCmdTests(unittest.TestCase):
result = fakeproj.Sync(use_local_gitdirs=False)
self.assertFalse(result)
def test_get_revision_id_peels_tags_even_with_all_refs(self) -> None:
"""Test GetRevisionId resolves tags via ResolveCommit, not all_refs."""
with utils_for_test.TempGitTree() as tempdir:
proj = _create_mock_project(tempdir, use_local_gitdirs=True)
proj.revisionId = None
mock_remote = mock.MagicMock()
proj.GetRemote = mock.MagicMock(return_value=mock_remote)
proj.bare_git.ResolveCommit.return_value = self.REVID
# Branch ref in all_refs returns cached commit SHA directly.
proj.revisionExpr = "main"
mock_remote.ToLocal.return_value = "refs/remotes/origin/main"
all_refs = {
"refs/remotes/origin/main": self.HEAD_ID,
"refs/tags/v1.0": self.OTHER_ID,
}
self.assertEqual(proj.GetRevisionId(all_refs), self.HEAD_ID)
proj.bare_git.ResolveCommit.assert_not_called()
# Tag ref in all_refs may be an annotated tag object SHA, so it
# must fall through to ResolveCommit to peel to a commit SHA.
proj.revisionExpr = "refs/tags/v1.0"
mock_remote.ToLocal.return_value = "refs/tags/v1.0"
self.assertEqual(proj.GetRevisionId(all_refs), self.REVID)
proj.bare_git.ResolveCommit.assert_called_once_with(
"refs/tags/v1.0"
)
# GetCommitRevisionId skips _allrefs for tags and pinned revisionId.
with mock.patch.object(
project.Project, "_allrefs", new_callable=mock.PropertyMock
) as mock_allrefs:
proj.bare_git.ResolveCommit.reset_mock()
self.assertEqual(proj.GetCommitRevisionId(), self.REVID)
mock_allrefs.assert_not_called()
proj.bare_git.ResolveCommit.assert_called_once_with(
"refs/tags/v1.0"
)
proj.revisionId = self.HEAD_ID
proj.bare_git.ResolveCommit.reset_mock()
self.assertEqual(proj.GetCommitRevisionId(), self.HEAD_ID)
mock_allrefs.assert_not_called()
proj.bare_git.ResolveCommit.assert_not_called()
class ReprojectCmdGitTests(unittest.TestCase):
"""Tests running the reprojectcmd contract against a real Git checkout."""
@@ -4048,6 +4304,45 @@ class ReprojectCmdGitTests(unittest.TestCase):
[" M keep.txt", "?? junk"],
)
def test_sync_local_half_checks_out_an_annotated_tag(self) -> None:
"""Test Sync_LocalHalf peels an annotated tag to its commit SHA."""
with tempfile.TemporaryDirectory(prefix="repo-tests") as topdir:
marker = os.path.join(topdir, "ran")
proj, revid = self._make_client(
topdir, f"{self.READ_TREE} && touch {marker}"
)
remote = proj.remote.url
worktree = proj.worktree
self._git(
remote,
"-c",
"user.name=Test",
"-c",
"user.email=test@example.com",
"tag",
"-a",
"-m",
"release v1.0",
"v1.0",
"HEAD",
)
self._git(
worktree, "fetch", "-q", remote, "refs/tags/v1.0:refs/tags/v1.0"
)
proj.revisionExpr = "refs/tags/v1.0"
proj.revisionId = None
clean, errors = self._sync(proj)
self.assertTrue(clean, errors)
self.assertTrue(os.path.exists(marker))
self.assertEqual(self._git(worktree, "rev-parse", "HEAD"), revid)
# Syncing a second time sees HEAD == revid and skips reprojectcmd.
os.remove(marker)
clean, errors = self._sync(proj)
self.assertTrue(clean, errors)
self.assertFalse(os.path.exists(marker))
class DownloadPatchSetTests(unittest.TestCase):
"""Tests for Project.DownloadPatchSet."""
+107
View File
@@ -14,9 +14,11 @@
"""Unittests for the subcmds/gc.py module."""
from pathlib import Path
import unittest
from unittest import mock
import manifest_xml
from subcmds import gc
@@ -80,3 +82,108 @@ class GcCommand(unittest.TestCase):
ret = self.cmd.Execute(self.opt, [])
self.assertEqual(ret, 1)
self.mock_repack.assert_not_called()
def test_gc_keep_manifest_projects(tmp_path: Path) -> None:
"""Keep inactive projects that are still in the manifest."""
repodir = tmp_path / ".repo"
manifest_dir = repodir / "manifests"
manifest_file = repodir / manifest_xml.MANIFEST_FILE_NAME
repodir.mkdir()
manifest_dir.mkdir()
manifest_gitdir = repodir / "manifests.git"
manifest_gitdir.mkdir()
(manifest_gitdir / "config").write_text(
"""[remote "origin"]
url = https://localhost:0/manifest
""",
encoding="utf-8",
)
manifest_file.write_text(
"""\
<manifest>
<remote name="origin" fetch="http://localhost" />
<default remote="origin" revision="refs/heads/main" />
<project name="included" groups="active-group" />
<project name="excluded" groups="other-group" />
</manifest>
""",
encoding="utf-8",
)
manifest = manifest_xml.XmlManifest(str(repodir), str(manifest_file))
manifest.manifestProject.config.SetString(
"manifest.groups",
"active-group",
)
projects = {project.name: project for project in manifest.projects}
included = projects["included"]
excluded = projects["excluded"]
Path(included.gitdir).mkdir(parents=True)
Path(included.objdir).mkdir(parents=True)
# Simulate an inactive project with only its object cache left.
Path(excluded.objdir).mkdir(parents=True)
assert not excluded.Exists
orphan_gitdir = repodir / "projects" / "orphan.git"
orphan_objdir = repodir / "project-objects" / "orphan.git"
orphan_gitdir.mkdir()
orphan_objdir.mkdir()
cmd = gc.Gc(repodir=str(repodir), manifest=manifest)
opt, args = cmd.OptionParser.parse_args(
["--yes", "--keep-manifest-projects"]
)
opt.quiet = True
selected_projects = cmd.GetProjects([], all_manifests=True)
assert included in selected_projects
assert excluded not in selected_projects
cmd.Execute(opt, args)
assert included.Exists
assert Path(excluded.objdir).exists()
assert not orphan_gitdir.exists()
assert not orphan_objdir.exists()
def test_gc_keep_manifest_projects_preserves_repack_targets() -> None:
"""Keep manifest projects without expanding repack targets."""
cmd = gc.Gc()
opt, _ = cmd.OptionParser.parse_args(
["--keep-manifest-projects", "--repack"]
)
opt.this_manifest_only = False
with mock.patch.object(
cmd,
"GetProjects",
side_effect=[["projA"], ["manifest_projects"]],
) as get_projects, mock.patch.object(
cmd, "delete_unused_projects", return_value=0
) as delete, mock.patch.object(
cmd, "repack_projects", return_value=0
) as repack:
cmd.Execute(opt, ["projA"])
get_projects.assert_has_calls(
[
mock.call(["projA"], all_manifests=True),
mock.call(
[],
groups="all",
missing_ok=True,
all_manifests=True,
),
]
)
delete.assert_called_once_with(["manifest_projects"], opt)
repack.assert_called_once_with(["projA"], opt)
+767 -17
View File
@@ -20,15 +20,19 @@ import os
from pathlib import Path
import shutil
import tempfile
import threading
import time
from typing import Dict, List, Optional, Tuple
from typing import Any, Dict, Iterator, List, Optional, Tuple, Type
import unittest
from unittest import mock
import xmlrpc.client
import xmlrpc.server
import pytest
import command
from error import GitError
from error import ManifestParseError
from error import RepoExitError
import manifest_xml
from project import SyncNetworkHalfResult
@@ -215,6 +219,132 @@ def test_sync_update_projects_revision_id_respects_groups(tmp_path: Path):
assert kwargs.get("groups") == "group1"
def test_sync_update_projects_revision_id_populates_logging_data(
tmp_path: Path,
) -> None:
"""Test that _UpdateProjectsRevisionId fills in the caller's dict."""
manifest = _create_manifest_with_groups(tmp_path)
cmd = sync.Sync()
cmd.manifest = manifest
superproject = mock.MagicMock()
superproject.UpdateProjectsRevisionId.return_value = mock.MagicMock(
manifest_path=None
)
manifest._superproject = superproject
opts, args = cmd.OptionParser.parse_args([])
opts.this_manifest_only = True
opts.local_only = False
superproject_logging_data = {}
with mock.patch.object(
cmd, "ManifestList", return_value=[manifest]
), mock.patch.object(
cmd, "_ConfigureSuperproject", return_value=False
), mock.patch.object(
sync.git_superproject, "UseSuperproject", return_value=True
):
cmd._UpdateProjectsRevisionId(
opts, args, superproject_logging_data, manifest
)
assert superproject_logging_data == {
"superproject": True,
"haslocalmanifests": False,
"hassuperprojecttag": True,
"updatedrevisionid": False,
}
@pytest.mark.parametrize(
"superproject_revision, is_outer_manifest, expected_revision",
[
("deadbeef", True, "deadbeef"),
("deadbeef", False, None),
(None, True, None),
],
ids=["outer_manifest", "not_outer_manifest", "no_superproject_revision"],
)
def test_sync_update_projects_revision_id_pins_superproject_revision(
tmp_path: Path,
superproject_revision: Optional[str],
is_outer_manifest: bool,
expected_revision: Optional[str],
) -> None:
"""Test that --superproject-revision is applied to a reloaded manifest.
_SyncToSuperprojectRev pins the superproject, but updating the manifest
project unloads the manifest, so _UpdateProjectsRevisionId must pin the
freshly loaded superproject again before using it.
"""
manifest = _create_manifest_with_groups(tmp_path)
cmd = sync.Sync()
cmd.manifest = manifest
cmd.outer_manifest = manifest if is_outer_manifest else mock.MagicMock()
# Simulate the state after a manifest reload: a new, unpinned superproject.
superproject = mock.MagicMock()
superproject.UpdateProjectsRevisionId.return_value = mock.MagicMock(
manifest_path=None, fatal=False
)
manifest._superproject = superproject
opts, args = cmd.OptionParser.parse_args([])
opts.verbose = False
opts.this_manifest_only = True
opts.local_only = False
opts.superproject_revision = superproject_revision
with mock.patch.object(
cmd, "ManifestList", return_value=[manifest]
), mock.patch.object(
sync.git_superproject, "UseSuperproject", return_value=True
), mock.patch.object(
sync.git_superproject, "PrintMessages", return_value=False
):
cmd._UpdateProjectsRevisionId(opts, args, {}, manifest)
if expected_revision is not None:
superproject.SetRevisionId.assert_called_once_with(expected_revision)
# The pin must be in place before the superproject is used.
method_calls = [name for name, _, _ in superproject.mock_calls]
assert method_calls.index("SetRevisionId") < method_calls.index(
"UpdateProjectsRevisionId"
)
else:
superproject.SetRevisionId.assert_not_called()
superproject.UpdateProjectsRevisionId.assert_called_once()
def test_sync_update_projects_revision_id_logs_without_superproject_tag(
tmp_path: Path,
) -> None:
"""Test that sync state is recorded when the manifest has no superproject.
SyncAnalysisState never removes keys, so values from an earlier sync would
otherwise persist in the config.
"""
manifest = _create_manifest_with_groups(tmp_path)
cmd = sync.Sync()
cmd.manifest = manifest
opts, args = cmd.OptionParser.parse_args([])
opts.this_manifest_only = True
opts.local_only = False
superproject_logging_data = {}
cmd._UpdateProjectsRevisionId(
opts, args, superproject_logging_data, manifest
)
assert superproject_logging_data == {
"superproject": False,
"haslocalmanifests": False,
"hassuperprojecttag": False,
}
@pytest.mark.parametrize(
"generate_manpages, expected_default",
[
@@ -324,6 +454,92 @@ def test_cli_jobs_sync_j_max(
assert opts.jobs_checkout == jobs_check
@pytest.mark.parametrize(
"argv, sync_smartsync_manifest, expected_smart_sync, expected_implicit",
[
([], False, False, False),
([], None, False, False),
([], True, True, True),
(["-s"], False, True, False),
(["--smart-sync"], False, True, False),
(["--smart-sync"], True, True, False),
(["--no-smart-sync"], True, False, False),
(["--no-smart-sync"], False, False, False),
(["-t", "tag123"], True, False, False),
(["--smart-tag=tag123"], True, False, False),
(["-m", "other.xml"], True, False, False),
(["--manifest-name=other.xml"], True, False, False),
(["-l"], True, False, False),
(["--local-only"], True, False, False),
(["--nmu"], True, True, True),
(["--no-manifest-update"], True, True, True),
(["--nmu"], False, False, False),
(["--superproject-revision=abc"], True, False, False),
(["-s", "-l"], True, True, False),
(["-s", "--nmu"], True, True, False),
(["-s", "--superproject-revision=abc"], True, True, False),
],
)
def test_cli_smart_sync(
argv: List[str],
sync_smartsync_manifest: Optional[bool],
expected_smart_sync: bool,
expected_implicit: bool,
) -> None:
"""Tests --smart-sync and --no-smart-sync option behavior with manifest
default.
"""
manifest = mock.MagicMock()
manifest.default.sync_smartsync = sync_smartsync_manifest
cmd = sync.Sync(manifest=manifest)
opts, args = cmd.OptionParser.parse_args(argv)
cmd.ValidateOptions(opts, args)
implicit = cmd._ResolveSmartSyncOption(opts, manifest)
assert opts.smart_sync == expected_smart_sync
assert implicit == expected_implicit
@pytest.mark.parametrize(
"argv, sync_smartsync_manifest, expected_exception",
[
(["-u", "user", "-p", "pass"], False, sync.SmartSyncError),
(["-u", "user", "-p", "pass"], True, None),
(["-s", "-u", "user", "-p", "pass"], False, None),
(["-t", "tag", "-u", "user", "-p", "pass"], False, None),
(
["--no-smart-sync", "-u", "user", "-p", "pass"],
True,
sync.SmartSyncError,
),
(["-l", "-u", "user", "-p", "pass"], True, sync.SmartSyncError),
(["--nmu", "-u", "user", "-p", "pass"], True, None),
(["--nmu", "-u", "user", "-p", "pass"], False, sync.SmartSyncError),
(["-s", "-l", "-u", "user", "-p", "pass"], True, None),
(["-u", "user"], False, SystemExit),
(["-p", "pass"], False, SystemExit),
],
)
def test_cli_manifest_server_credentials(
argv: List[str],
sync_smartsync_manifest: bool,
expected_exception: Optional[Type[BaseException]],
) -> None:
"""Tests -u and -p validation rules."""
manifest = mock.MagicMock()
manifest.default.sync_smartsync = sync_smartsync_manifest
cmd = sync.Sync(manifest=manifest)
opts, args = cmd.OptionParser.parse_args(argv)
if expected_exception:
with pytest.raises(expected_exception):
cmd.ValidateOptions(opts, args)
cmd._ResolveSmartSyncOption(opts, manifest)
else:
cmd.ValidateOptions(opts, args)
cmd._ResolveSmartSyncOption(opts, manifest)
class LocalSyncState(unittest.TestCase):
"""Tests for LocalSyncState."""
@@ -338,8 +554,9 @@ class LocalSyncState(unittest.TestCase):
self.manifest = mock.MagicMock(
topdir=self.topdir,
repodir=self.repodir,
repoProject=mock.MagicMock(relpath=".repo/repo"),
repoProject=FakeProject(".repo/repo"),
)
self.manifest.outer_client = self.manifest
self.state = self._new_state()
def tearDown(self):
@@ -352,7 +569,7 @@ class LocalSyncState(unittest.TestCase):
def test_set(self):
"""Times are set."""
p = mock.MagicMock(relpath="projA")
p = FakeProject("projA")
self.state.SetFetchTime(p)
self.state.SetCheckoutTime(p)
self.assertEqual(self.state.GetFetchTime(p), self._TIME)
@@ -374,8 +591,8 @@ class LocalSyncState(unittest.TestCase):
# Initialize state to read from the new file.
self.state = self._new_state()
projA = mock.MagicMock(relpath="projA")
projB = mock.MagicMock(relpath="projB")
projA = FakeProject("projA")
projB = FakeProject("projB")
self.assertEqual(self.state.GetFetchTime(projA), None)
self.assertEqual(self.state.GetFetchTime(projB), 5)
self.assertEqual(self.state.GetCheckoutTime(projB), 7)
@@ -386,9 +603,43 @@ class LocalSyncState(unittest.TestCase):
self.assertEqual(self.state.GetFetchTime(projB), self._TIME)
self.assertEqual(self.state.GetCheckoutTime(projB), 7)
def test_same_relpath_projects_keep_separate_state(self) -> None:
"""Projects with the same relpath keep separate sync state."""
outer = FakeProject("proj")
child = FakeProject("proj", path_prefix="sub")
self.state.SetFetchTime(outer)
self.state.Save()
self.state = self._new_state(self._TIME + 1)
self.state.SetFetchTime(child)
self.assertEqual(self.state.GetFetchTime(outer), self._TIME)
self.assertEqual(
self.state.GetFetchTime(child),
self._TIME + 1,
)
def test_partial_sync_with_same_relpath_projects(self) -> None:
"""Projects with the same relpath keep independent partial sync."""
outer = FakeProject("proj")
child = FakeProject("proj", path_prefix="sub")
for project in (outer, child):
self.state.SetFetchTime(project)
self.state.SetCheckoutTime(project)
self.state.Save()
self.assertFalse(self.state.IsPartiallySynced())
self.state = self._new_state(self._TIME + 1)
self.state.SetFetchTime(outer)
self.state.SetCheckoutTime(outer)
self.assertTrue(self.state.IsPartiallySynced())
def test_save_to_file(self):
"""Data is saved under repodir."""
p = mock.MagicMock(relpath="projA")
p = FakeProject("projA")
self.state.SetFetchTime(p)
self.state.Save()
self.assertEqual(
@@ -415,7 +666,7 @@ class LocalSyncState(unittest.TestCase):
# Initialize state to read from the new file.
self.state = self._new_state()
projB = mock.MagicMock(relpath="projB")
projB = FakeProject("projB")
self.assertEqual(self.state.IsPartiallySynced(), False)
self.state.SetFetchTime(projB)
@@ -424,7 +675,7 @@ class LocalSyncState(unittest.TestCase):
def test_ignore_repo_project(self):
"""Sync data for repo project is ignored when checking partial sync."""
p = mock.MagicMock(relpath="projA")
p = FakeProject("projA")
self.state.SetFetchTime(p)
self.state.SetCheckoutTime(p)
self.state.SetFetchTime(self.manifest.repoProject)
@@ -441,7 +692,7 @@ class LocalSyncState(unittest.TestCase):
def test_nonexistent_project(self):
"""Unsaved projects don't have data."""
p = mock.MagicMock(relpath="projC")
p = FakeProject("projC")
self.assertEqual(self.state.GetFetchTime(p), None)
self.assertEqual(self.state.GetCheckoutTime(p), None)
@@ -466,8 +717,8 @@ class LocalSyncState(unittest.TestCase):
return False
return True
projA = mock.MagicMock(relpath="projA")
projB = mock.MagicMock(relpath="projB")
projA = FakeProject("projA")
projB = FakeProject("projB")
self.state = self._new_state()
self.assertEqual(self.state.GetFetchTime(projA), 5)
self.assertEqual(self.state.GetFetchTime(projB), 7)
@@ -479,6 +730,38 @@ class LocalSyncState(unittest.TestCase):
self.assertIsNone(self.state.GetFetchTime(projA))
self.assertEqual(self.state.GetFetchTime(projB), 7)
def test_prune_keeps_submanifest_project(self) -> None:
"""Existing submanifest projects are not pruned."""
project = FakeProject("proj", path_prefix="sub")
os.makedirs(
os.path.join(self.topdir, "sub", "proj", ".git"),
)
self.state.SetFetchTime(project)
self.state.PruneRemovedProjects()
self.assertEqual(self.state.GetFetchTime(project), self._TIME)
def test_prune_from_submanifest_keeps_project(self) -> None:
"""Existing projects are not pruned during a submanifest sync."""
child_topdir = os.path.join(self.topdir, "sub")
project = FakeProject("proj", path_prefix="sub")
os.makedirs(os.path.join(child_topdir, "proj", ".git"))
child_manifest = mock.MagicMock(
topdir=child_topdir,
repodir=self.repodir,
outer_client=self.manifest,
)
with mock.patch("time.time", return_value=self._TIME):
state = sync.LocalSyncState(child_manifest)
state.SetFetchTime(project)
state.PruneRemovedProjects()
self.assertEqual(state.GetFetchTime(project), self._TIME)
def test_prune_removed_and_symlinked_projects(self):
"""Removed projects that still exists on disk as symlink are pruned."""
with open(self.state._path, "w") as f:
@@ -503,8 +786,8 @@ class LocalSyncState(unittest.TestCase):
return True
return False
projA = mock.MagicMock(relpath="projA")
projB = mock.MagicMock(relpath="projB")
projA = FakeProject("projA")
projB = FakeProject("projB")
self.state = self._new_state()
self.assertEqual(self.state.GetFetchTime(projA), 5)
self.assertEqual(self.state.GetFetchTime(projB), 7)
@@ -1351,6 +1634,231 @@ class SyncCommand(unittest.TestCase):
phased, interleaved = self._ExecuteUntilSync([])
self.assertTrue(phased.called or interleaved.called)
def test_implicit_smart_sync_fallback(self) -> None:
"""Ensure implicit smart sync silently falls back to ToT."""
self.manifest.default.sync_smartsync = True
self.opt.smart_sync = None
mock.patch.object(self.cmd, "_UpdateAllManifestProjects").start()
self.opt.verbose = False
with mock.patch.object(
self.cmd,
"_SmartSyncSetup",
side_effect=sync.SmartSyncError("unreachable"),
) as mock_setup:
with mock.patch.object(self.cmd, "_UpdateRepoProject"):
with mock.patch.object(
self.cmd, "_ValidateOptionsWithManifest"
):
with mock.patch.object(self.cmd, "_SyncInterleaved"):
with mock.patch.object(self.cmd, "_RunPostSyncHook"):
with mock.patch.object(
sync.logger, "warning"
) as mock_warn:
self.cmd.Execute(self.opt, [])
mock_setup.assert_called_once()
self.assertTrue(mock_setup.call_args.kwargs.get("best_effort"))
self.assertFalse(self.opt.smart_sync)
self.assertFalse(mock_warn.called)
def test_implicit_smart_sync_fallback_verbose(self) -> None:
"""Ensure the fallback reason is logged when --verbose is given."""
self.manifest.default.sync_smartsync = True
self.opt.smart_sync = None
mock.patch.object(self.cmd, "_UpdateAllManifestProjects").start()
self.opt.verbose = True
with mock.patch.object(
self.cmd,
"_SmartSyncSetup",
side_effect=sync.SmartSyncError("unreachable"),
):
with mock.patch.object(self.cmd, "_UpdateRepoProject"):
with mock.patch.object(
self.cmd, "_ValidateOptionsWithManifest"
):
with mock.patch.object(self.cmd, "_SyncInterleaved"):
with mock.patch.object(self.cmd, "_RunPostSyncHook"):
with mock.patch.object(
sync.logger, "warning"
) as mock_warn:
self.cmd.Execute(self.opt, [])
self.assertFalse(self.opt.smart_sync)
self.assertTrue(mock_warn.called)
def test_explicit_smart_sync_error(self) -> None:
"""Ensure explicit smart sync raises SmartSyncError on failure."""
self.manifest.default.sync_smartsync = True
self.opt.smart_sync = True
self.opt.mp_update = False
with mock.patch.object(
self.cmd,
"_SmartSyncSetup",
side_effect=sync.SmartSyncError("unreachable"),
) as mock_setup:
with self.assertRaises(sync.SmartSyncError):
self.cmd.Execute(self.opt, [])
mock_setup.assert_called_once()
self.assertFalse(mock_setup.call_args.kwargs.get("best_effort"))
def test_implicit_smart_sync_unparsable_manifest(self) -> None:
"""Ensure an unparsable smart sync manifest falls back to ToT."""
self.manifest.default.sync_smartsync = True
self.opt.smart_sync = None
mock.patch.object(self.cmd, "_UpdateAllManifestProjects").start()
self.opt.verbose = False
with mock.patch.object(
self.cmd,
"_SmartSyncSetup",
side_effect=ManifestParseError("mismatched tag"),
):
with mock.patch.object(self.cmd, "_UpdateRepoProject"):
with mock.patch.object(
self.cmd, "_ValidateOptionsWithManifest"
):
with mock.patch.object(self.cmd, "_SyncInterleaved"):
with mock.patch.object(self.cmd, "_RunPostSyncHook"):
self.cmd.Execute(self.opt, [])
self.assertFalse(self.opt.smart_sync)
# A half-applied override would pin the client to a broken manifest.
self.manifest.ClearOverride.assert_called_once_with()
def test_explicit_smart_sync_unparsable_manifest(self) -> None:
"""Ensure explicit smart sync still reports the parse error."""
self.manifest.default.sync_smartsync = True
self.opt.smart_sync = True
self.opt.mp_update = False
with mock.patch.object(
self.cmd,
"_SmartSyncSetup",
side_effect=ManifestParseError("mismatched tag"),
):
with self.assertRaises(ManifestParseError):
self.cmd.Execute(self.opt, [])
def _ExecuteWithSmartSyncOverride(
self,
reload_error: Optional[Exception] = None,
setup_error: Optional[Exception] = None,
) -> Tuple[str, mock.MagicMock, mock.MagicMock, mock.MagicMock]:
"""Run Execute with an existing smart sync override on disk.
Returns:
The override path, and the _SmartSyncSetup, _ReloadManifest and
_UpdateAllManifestProjects mocks.
"""
worktree = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, worktree)
self.manifest.manifestProject.worktree = worktree
# -l reloads the superproject manifest; keep that out of the way.
self.manifest.superproject = None
self.opt.verbose = False
override_path = os.path.join(worktree, "smart_sync_override.xml")
with open(override_path, "w") as f:
f.write("<manifest />")
with contextlib.ExitStack() as stack:
for name in (
"_UpdateRepoProject",
"_ValidateOptionsWithManifest",
"_SyncInterleaved",
"_RunPostSyncHook",
):
stack.enter_context(mock.patch.object(self.cmd, name))
setup = stack.enter_context(
mock.patch.object(
self.cmd, "_SmartSyncSetup", side_effect=setup_error
)
)
reload = stack.enter_context(
mock.patch.object(
self.cmd, "_ReloadManifest", side_effect=reload_error
)
)
update_mps = stack.enter_context(
mock.patch.object(self.cmd, "_UpdateAllManifestProjects")
)
self.cmd.Execute(self.opt, [])
return override_path, setup, reload, update_mps
def test_smart_sync_override_reused_offline(self) -> None:
"""Ensure -l reuses an existing smart sync manifest.
This applies whether the override came from the manifest's
sync-smartsync default or from an explicit -s, so that `sync -n`
followed by `sync -l` matches a single sync.
"""
for sync_smartsync in (True, False):
for mp_update in (True, False):
with self.subTest(
sync_smartsync=sync_smartsync, mp_update=mp_update
):
self.manifest.default.sync_smartsync = sync_smartsync
self.opt.smart_sync = None
self.opt.local_only = True
self.opt.mp_update = mp_update
path, setup, reload, update_mps = (
self._ExecuteWithSmartSyncOverride()
)
setup.assert_not_called()
reload.assert_called_once_with(
"smart_sync_override.xml", self.manifest
)
self.assertTrue(os.path.isfile(path))
if mp_update:
self.assertEqual(
update_mps.call_args[0][2],
"smart_sync_override.xml",
)
def test_no_manifest_update_keeps_default_smart_sync(self) -> None:
"""Ensure --nmu still runs the manifest-driven default smart sync."""
self.manifest.default.sync_smartsync = True
self.opt.smart_sync = None
self.opt.mp_update = False
_, setup, _, update_mps = self._ExecuteWithSmartSyncOverride()
setup.assert_called_once()
update_mps.assert_not_called()
def test_no_manifest_update_does_not_reuse_override(self) -> None:
"""Ensure --nmu handles an existing override like a regular sync."""
cases = (
("manifest without sync-smartsync", False, None),
("failed default smart sync", True, sync.SmartSyncError("down")),
)
for name, sync_smartsync, setup_error in cases:
with self.subTest(name):
self.manifest.default.sync_smartsync = sync_smartsync
self.opt.smart_sync = None
self.opt.mp_update = False
path, _, reload, _ = self._ExecuteWithSmartSyncOverride(
setup_error=setup_error
)
reload.assert_not_called()
self.assertFalse(os.path.isfile(path))
def test_smart_sync_override_removed_offline(self) -> None:
"""Ensure -l --no-smart-sync still drops the override."""
for sync_smartsync in (True, False):
with self.subTest(sync_smartsync=sync_smartsync):
self.manifest.default.sync_smartsync = sync_smartsync
self.opt.smart_sync = False
self.opt.local_only = True
path, setup, reload, _ = self._ExecuteWithSmartSyncOverride()
setup.assert_not_called()
reload.assert_not_called()
self.assertFalse(os.path.isfile(path))
def test_smart_sync_override_unparsable_offline(self) -> None:
"""Ensure an unloadable reused override falls back to ToT."""
self.manifest.default.sync_smartsync = True
self.opt.smart_sync = None
self.opt.local_only = True
path, _, _, update_mps = self._ExecuteWithSmartSyncOverride(
reload_error=ManifestParseError("mismatched tag")
)
self.manifest.ClearOverride.assert_called_once_with()
self.assertFalse(os.path.isfile(path))
self.assertIsNone(update_mps.call_args[0][2])
class SyncUpdateRepoProject(unittest.TestCase):
"""Tests for Sync._UpdateRepoProject."""
@@ -2328,10 +2836,10 @@ class TestSmartSyncSetupRemoteHelper(unittest.TestCase):
# Mock XML-RPC server call
mock_server = mock.MagicMock()
mock_server.GetApprovedManifest.return_value = [
True,
"<manifest></manifest>",
]
mock_server.GetApprovedManifestV2.return_value = {
"success": True,
"manifest_xml": "<manifest></manifest>",
}
mock_server_class.return_value = mock_server
# Mock manifest project branch
@@ -2368,6 +2876,38 @@ class TestSmartSyncSetupRemoteHelper(unittest.TestCase):
self.assertEqual(manifest_name, "manifest.xml")
@mock.patch("xmlrpc.client.Server")
def test_smart_sync_setup_unparsable_manifest(
self, mock_server_class: mock.MagicMock
) -> None:
"""Test _SmartSyncSetup when the server returns a bad manifest.
The parse error is left alone so that an explicit smart sync keeps
reporting it as-is; recovering from it is up to the caller.
"""
self.manifest.manifest_server = (
"http://android-smartsync.corp.google.com/manifestserver"
)
self.manifest.manifest_server_helper = None
mock_server = mock.MagicMock()
mock_server.GetApprovedManifestV2.return_value = {
"success": True,
"manifest_xml": "<manifest>",
}
mock_server_class.return_value = mock_server
self.cmd._GetBranch = mock.MagicMock(return_value="main")
self.cmd._ReloadManifest = mock.MagicMock(
side_effect=ManifestParseError("mismatched tag")
)
with mock.patch("builtins.open", mock.mock_open()):
with self.assertRaises(ManifestParseError):
self.cmd._SmartSyncSetup(
self.opt, self.smart_sync_manifest_path, self.manifest
)
@mock.patch("shutil.which")
@mock.patch("subprocess.Popen")
def test_smart_sync_setup_helper_error(self, mock_popen, mock_which):
@@ -2516,3 +3056,213 @@ class TestSmartSyncSetupRemoteHelper(unittest.TestCase):
"Stderr was: debugging logs",
str(context.exception),
)
class TestSmartSyncSetupBranch(unittest.TestCase):
"""Tests for how _SmartSyncSetup picks the branch sent to the server."""
def setUp(self) -> None:
self.cmd = sync.Sync()
self.opt = mock.MagicMock()
self.opt.quiet = True
self.opt.smart_sync = True
self.opt.smart_tag = None
self.manifest = mock.MagicMock()
self.manifest.manifest_server = (
"http://android-smartsync.corp.google.com/manifestserver"
)
self.manifest.manifest_server_helper = None
self.smart_sync_manifest_path = "/fake/path/to/manifest.xml"
self.mock_server = mock.MagicMock()
self.mock_server.GetApprovedManifestV2.return_value = {
"success": True,
"manifest_xml": "<manifest></manifest>",
}
self.cmd._GetBranch = mock.MagicMock(return_value="main")
self.cmd._ReloadManifest = mock.MagicMock()
self.cmd._ResolveManifestServerTransport = mock.MagicMock(
return_value=(self.manifest.manifest_server, mock.MagicMock())
)
def _RunSetup(self, env: Dict[str, str]) -> str:
with mock.patch.dict(os.environ, env, clear=True), mock.patch(
"xmlrpc.client.Server", return_value=self.mock_server
), mock.patch("builtins.open", mock.mock_open()):
return self.cmd._SmartSyncSetup(
self.opt, self.smart_sync_manifest_path, self.manifest
)
def test_default_branch(self) -> None:
"""Without SYNC_BRANCH the manifest branch is used."""
self._RunSetup({})
self.cmd._GetBranch.assert_called_once_with(
self.manifest.manifestProject
)
self.mock_server.GetApprovedManifestV2.assert_called_once_with(
{"branch": "main"}
)
def test_sync_branch(self) -> None:
"""SYNC_BRANCH replaces the manifest branch."""
self._RunSetup({"SYNC_BRANCH": "git_main-abfs"})
self.cmd._GetBranch.assert_not_called()
self.mock_server.GetApprovedManifestV2.assert_called_once_with(
{"branch": "git_main-abfs"}
)
def test_sync_branch_with_target(self) -> None:
"""SYNC_BRANCH and SYNC_TARGET are both forwarded."""
self._RunSetup(
{
"SYNC_BRANCH": "git_main-abfs",
"SYNC_TARGET": "cf_x86_64_phone-trunk_staging-eng",
}
)
self.mock_server.GetApprovedManifestV2.assert_called_once_with(
{
"branch": "git_main-abfs",
"target": "cf_x86_64_phone-trunk_staging-eng",
}
)
def test_empty_sync_branch(self) -> None:
"""An empty SYNC_BRANCH falls back to the manifest branch."""
self._RunSetup({"SYNC_BRANCH": ""})
self.cmd._GetBranch.assert_called_once_with(
self.manifest.manifestProject
)
self.mock_server.GetApprovedManifestV2.assert_called_once_with(
{"branch": "main"}
)
class GetApprovedManifestTests(unittest.TestCase):
"""Tests for _GetApprovedManifest."""
def test_request_with_target_and_best_effort(self) -> None:
server = mock.MagicMock()
server.GetApprovedManifestV2.return_value = {
"success": True,
"manifest_xml": "<manifest/>",
}
result = sync._GetApprovedManifest(server, "main", "tgt", True)
self.assertEqual(result, (True, "<manifest/>"))
server.GetApprovedManifestV2.assert_called_once_with(
{"branch": "main", "target": "tgt", "best_effort": True}
)
server.GetApprovedManifest.assert_not_called()
def test_request_omits_unset_options(self) -> None:
server = mock.MagicMock()
server.GetApprovedManifestV2.return_value = {
"success": True,
"manifest_xml": "<manifest/>",
}
sync._GetApprovedManifest(server, "main", None, False)
server.GetApprovedManifestV2.assert_called_once_with({"branch": "main"})
def test_declined(self) -> None:
server = mock.MagicMock()
server.GetApprovedManifestV2.return_value = {
"success": False,
"error_message": "not in rollout",
}
result = sync._GetApprovedManifest(server, "main", None, True)
self.assertEqual(result, (False, "not in rollout"))
def test_invalid_response(self) -> None:
server = mock.MagicMock()
server.GetApprovedManifestV2.return_value = [True, "<manifest/>"]
with self.assertRaises(sync.SmartSyncError):
sync._GetApprovedManifest(server, "main", None, False)
def test_success_missing_manifest(self) -> None:
server = mock.MagicMock()
server.GetApprovedManifestV2.return_value = {"success": True}
with self.assertRaises(sync.SmartSyncError):
sync._GetApprovedManifest(server, "main", None, False)
def test_success_non_string_manifest(self) -> None:
server = mock.MagicMock()
for value in (None, 42, ["<manifest/>"]):
with self.subTest(manifest_xml=value):
server.GetApprovedManifestV2.return_value = {
"success": True,
"manifest_xml": value,
}
with self.assertRaises(sync.SmartSyncError):
sync._GetApprovedManifest(server, "main", None, False)
def test_fallback_on_method_not_found_code(self) -> None:
server = mock.MagicMock()
server.GetApprovedManifestV2.side_effect = xmlrpc.client.Fault(
-32601, "no such method"
)
server.GetApprovedManifest.return_value = [True, "<manifest/>"]
result = sync._GetApprovedManifest(server, "main", "tgt", True)
self.assertEqual(result, (True, "<manifest/>"))
server.GetApprovedManifest.assert_called_once_with("main", "tgt")
def test_other_fault_is_not_retried(self) -> None:
server = mock.MagicMock()
server.GetApprovedManifestV2.side_effect = xmlrpc.client.Fault(
1, "internal error"
)
with self.assertRaises(xmlrpc.client.Fault):
sync._GetApprovedManifest(server, "main", "tgt", False)
server.GetApprovedManifest.assert_not_called()
@contextlib.contextmanager
def _LiveManifestServer(**funcs: Any) -> Iterator[xmlrpc.client.ServerProxy]:
"""Runs a local XML-RPC server exposing |funcs|."""
server = xmlrpc.server.SimpleXMLRPCServer(
("127.0.0.1", 0), logRequests=False
)
for name, func in funcs.items():
server.register_function(func, name)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
host, port = server.server_address
yield xmlrpc.client.ServerProxy(f"http://{host}:{port}/")
finally:
server.shutdown()
server.server_close()
thread.join()
class GetApprovedManifestLiveTests(unittest.TestCase):
"""Tests _GetApprovedManifest against a real XML-RPC server."""
def test_legacy_server_fallback(self) -> None:
calls = []
def legacy(*args: str) -> List[Any]:
calls.append(args)
return [True, "<manifest/>"]
with _LiveManifestServer(GetApprovedManifest=legacy) as proxy:
self.assertEqual(
sync._GetApprovedManifest(proxy, "main", "tgt", True),
(True, "<manifest/>"),
)
self.assertEqual(
sync._GetApprovedManifest(proxy, "main", None, True),
(True, "<manifest/>"),
)
self.assertEqual(calls, [("main", "tgt"), ("main",)])
def test_v2_server(self) -> None:
requests = []
def v2(request: Dict[str, Any]) -> Dict[str, Any]:
requests.append(request)
return {"success": False, "error_message": "declined"}
with _LiveManifestServer(GetApprovedManifestV2=v2) as proxy:
self.assertEqual(
sync._GetApprovedManifest(proxy, "main", None, True),
(False, "declined"),
)
self.assertEqual(requests, [{"branch": "main", "best_effort": True}])
+56
View File
@@ -515,6 +515,62 @@ class TestResolveRepoRev:
with pytest.raises(repo_wrapper.CloneFailure):
repo_wrapper.resolve_repo_rev(git_checkout.git_dir, "boooooooya")
@pytest.mark.parametrize(
"committish",
["refs/heads/unknown", "refs/tags/unknown", "boooooooya"],
)
@pytest.mark.parametrize("report_errors", [True, False])
def test_report_errors(
self,
repo_wrapper: wrapper.Wrapper,
git_checkout: GitCheckout,
capsys: pytest.CaptureFixture,
committish: str,
report_errors: bool,
) -> None:
"""Check errors are only printed when requested."""
with pytest.raises(repo_wrapper.CloneFailure):
repo_wrapper.resolve_repo_rev(
git_checkout.git_dir, committish, report_errors=report_errors
)
assert ("repo: error:" in capsys.readouterr().err) == report_errors
class TestRefetchIfRevMissing:
"""Check _RefetchIfRevMissing behavior."""
@pytest.mark.parametrize("missing", [False, True])
def test_refetch(
self,
repo_wrapper: wrapper.Wrapper,
capsys: pytest.CaptureFixture,
missing: bool,
) -> None:
"""Check it refetches once only if the rev doesn't resolve."""
mock_resolve = mock.Mock(
side_effect=repo_wrapper.CloneFailure() if missing else None
)
with mock.patch.object(repo_wrapper, "_Fetch") as mock_fetch:
with mock.patch.object(repo_wrapper.time, "sleep") as mock_sleep:
with mock.patch.object(
repo_wrapper, "resolve_repo_rev", mock_resolve
):
repo_wrapper._RefetchIfRevMissing(
"url", "/dst", "stable", False, False
)
# It resolves quietly; check_repo_rev reports the error.
mock_resolve.assert_called_once_with(
"/dst", "stable", report_errors=False
)
fetches = int(missing)
fetch = mock.call("url", "/dst", "origin", False, False)
assert mock_fetch.call_args_list == [fetch] * fetches
delay = repo_wrapper.REV_REFETCH_DELAY_SEC
assert mock_sleep.call_args_list == [mock.call(delay)] * fetches
warnings = capsys.readouterr().err.count(f"refetching in {delay} ")
assert warnings == fetches
class TestCheckRepoVerify:
"""Check check_repo_verify behavior."""