Compare commits

..

77 Commits

Author SHA1 Message Date
Gavin Mak 2d54384a5e sync: Add --superproject-rev flag to sync to specific revision
Allow syncing the outer manifest to a state defined by a specific
superproject revision. It updates the superproject, reads the manifest
commit from .supermanifest, and checks out the outer manifest project
to that commit.

Submanifests are then processed normally, allowing them to be updated
to the revisions specified in the new outer manifest state.

Bug: 416589884
Change-Id: I304c37a2b8794f9b74cb7e5e209a8a93762bdb52
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/576321
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-05-20 12:28:55 -07:00
Josef Malmström 1b4e7a04be Fix submodules not synced for repeated repo
If I check out e.g. multiple branches of the same repo in one
manifest, only the submodules of one checkout are synced.

Fix this by adding the relative path  of the checkout as a key
to the mapping of derived projects, preventing overwrite.

The fix was originally proposed here:
https://issues.gerritcodereview.com/issues/40013218

Bug: 40013218
Change-Id: Ia86518ccf2c0af7bd7e4daf8d703a55b7e10ba52
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/581062
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josef Malmstrom <Josef.Malmstrom@arm.com>
Tested-by: Josef Malmstrom <Josef.Malmstrom@arm.com>
2026-05-20 06:05:57 -07:00
gurusai-voleti a94c9e2b52 Automated: Migrate gerrit/git-repo from gsutil to gcloud storage
Bug: 486536908
Change-Id: I248b093e189a3784b8959e837a5d66857f1ce0fc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/577201
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Guru sai rama subbarao Voleti (xWF) <gvoleti@google.com>
Commit-Queue: Guru sai rama subbarao Voleti (xWF) <gvoleti@google.com>
2026-05-14 21:31:49 -07:00
Gavin Mak 51021fb209 abandon/start/info: Make them respect smart sync override
Call TryOverrideManifestWithSmartSync in start, abandon, and info
commands.

This ensures they pick up the pinned revisions from the smart sync
override manifest if it exists, rather than falling back to ToT from
the default manifest.

Bug: 279204331
Change-Id: I637f054a77773805daf0bf9cf5a712d82a5959b4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/582503
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-05-14 13:19:55 -07:00
Gavin Mak d453273f06 command: Move smart sync override logic to Command base class
Deduplicate the logic for loading `smart_sync_override.xml` by
moving it from `forall.py` to the `Command` base class. This allows
other commands to reuse it to respect smart tags.

Bug: 279204331
Change-Id: I6f2f03995266c2a68c3225cacb92b2f580a89178
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/582502
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-05-14 13:19:26 -07:00
Mike Frysinger 0129ac1c46 docs: drop period in headers
We don't do this in most places, so standardize the outliers.

Change-Id: I80c1ef16c5fa7355dcb7797bc8e56852b22c24d5
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/583641
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-05-14 09:03:56 -07:00
Mike Frysinger bf8c59f4a0 run_tests: help2man: update to latest release
Change-Id: I30e4a6452ac5504b782fe545a59ca1f58ac70385
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/583621
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-05-14 09:03:52 -07:00
Josef Malmström 003f0407fc Fix missing None check in Remote.Save
This code was causing an exception in cases where `pushUrl` was set
but `projectname` was not.

This can happen when `pushUrl` is set for the manifest repo which does
not have a `projectname`. For example:

repo init --manifest-url ssh://url.to/my/manifest

cd .repo/manifests
git config remote.origin.pushurl ssh://url.to/my/manifest

repo init --manifest-url ssh://url.to/my/manifest --repo-rev main

The last `repo init` invocation causes an error.

Change-Id: Ibb68c8446880cfbac22feee595d1fd1b678c7ade
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/579162
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josef Malmstrom <Josef.Malmstrom@arm.com>
Tested-by: Josef Malmstrom <Josef.Malmstrom@arm.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-05-13 00:00:27 -07:00
Josef Malmström 1db50e49fb Add support for self referencing submodules
When working with relative submodule paths, The "./" needs special
handling similar to "../".

See information on:
https://git-scm.com/docs/git-submodule
Which currently states:
"<repository> is the URL of the new submodule’s origin repository.
This may be either an absolute URL, or (if it begins with ./ or ../),
the location relative to the superproject’s default remote
repository (Please note that to specify a repository foo.git which is
located right next to a superproject bar.git, you’ll have to use
../foo.git instead of ./foo.git - as one might expect when following
the rules for relative URLs - because the evaluation of relative URLs
in Git is identical to that of relative directories)."

The implementation also was not handling file/directory names
starting with "." or "..". Explicitly look for "./" and "../"
instead.

Change-Id: I8ae68d61fb0cbb1624183b175236e98a36e4afdb
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/579182
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josef Malmstrom <Josef.Malmstrom@arm.com>
Tested-by: Josef Malmstrom <Josef.Malmstrom@arm.com>
2026-05-13 00:00:11 -07:00
Mike Frysinger c97a306fe7 release: update-manpages: revert color filtering
We pull a pinned help2man from cipd now, so we should get stable
behavior between developers.  That means the color filtering should
not be necessary anymore, and we can drop the logic & unittest.

Change-Id: Ib53e1ce7f8d610d7f624c9a019c79dc5f438ac0d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/582402
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-05-12 11:50:38 -07:00
Carlos Fernandez 5534f164d6 linkfile: Handle directory-to-symlink transitions safely
When a manifest changes from individual linkfiles inside a directory
(e.g. dest=".llms/rules", dest=".llms/skills") to a single linkfile
for the whole directory (e.g. dest=".llms", src="dot-llms"), two
things need to happen:

1. __linkIt must replace a real directory with a symlink.  Use
   os.rmdir() instead of platform_utils.remove() for real directories.
   rmdir only removes empty directories, so user-created content is
   never deleted.

2. UpdateCopyLinkfileList must handle the cleanup correctly:
   - Use os.rmdir() for directories (safe for non-empty)
   - Remove empty parent directories after cleaning old dests
   - Retry _CopyAndLinkFiles for all projects, since in interleaved
     sync mode _CopyAndLinkFiles runs before cleanup and may have
     failed because the directory was not yet empty

Change-Id: I0437b80beab98bce064cea81c11c47d699be91aa
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/569243
Tested-by: Carlos Fernandez <carlosfsanz@meta.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Carlos Fernandez <carlosfsanz@meta.com>
2026-05-12 11:10:43 -07:00
Mike Frysinger 67e52a120b run_tests: leverage cipd when available for help2man
This tool isn't installed on CI bot images so we've been skipping it,
but this is causing people to not run tests locally, and ignore errors.
Use cipd to pull the tool in when available.

Then revert a recent man change that the tool rejects.

Change-Id: I1030d0070fd5a624656eba7434ae6ec99b2e3f2d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/582401
Reviewed-by: Greg Edelston <gredelston@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-05-12 10:45:13 -07:00
Gavin Mak d5037230e9 sync: Re-raise KeyboardInterrupt in main process
When running sync -j1, worker functions run directly in the main
process. Swallowing KeyboardInterrupt causes the loop to continue to the
next project instead of aborting.

Re-raise KeyboardInterrupt if running in the MainProcess, while
maintaining the suppression of stack traces in worker processes.

Bug: 468170157
Change-Id: I156d66bc209a265f7fa25eea0eb88737d1b51a34
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/581342
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2026-05-12 10:41:47 -07:00
Mike Frysinger 7cc99b24c1 git_config: fix error message command output
str() inside an f-string is redundant.  Actually join the array together
as a string to show a proper error message, and include the full command
line, not the extra options passed in.

Before:
error.GitError: git config ('--null', '--list'): fatal: unable to read config file 'xxx': No such file or directory

After:
error.GitError: git config --system --includes --null --list: fatal: unable to read config file 'xxx': No such file or directory

Change-Id: I8983389aa2e0de7808991e73e636b77810f04c4b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/582341
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-05-12 10:04:15 -07:00
Greg Edelston 12ad396f67 info: Parallelize repo info to improve performance
For a large checkout like chromiumos or android, `repo info` takes a
really long time! On my machine it took ~6 minutes. On a randomly
selected ChromiumOS cq-orchestrator build it took 4.1 minutes:
https://ci.chromium.org/b/8682060180498819729. This adds up to a lot of
wasted runtime for both humans and bots.

The problem is that `repo info` was single-threaded, which causes poor
performance when the checkout has 1000+ projects. We already have a
pattern for parallelization; let's use it.

BUG=None
TEST=Manually run, ensure no diff

Change-Id: I6b82b9495eb2a0e602a142dd3a16f09217871e1b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/581921
Tested-by: Greg Edelston <gredelston@google.com>
Commit-Queue: Greg Edelston <gredelston@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-05-12 09:59:07 -07:00
Gavin Mak a6bc1b7cf0 sync: Exclude stateless sync pruned projects from bloat check
If a project has been pruned for stateless sync, it has already
undergone aggressive garbage collection. There should be no bloat
to check for.

Change-Id: I9233a4611e05b7a0b7c097827d6f408144f7380d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/581841
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Becky Siegel <beckysiegel@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-05-11 19:36:59 -07:00
Gavin Mak a98e422113 completion: document installation and usage in README
Add a "Shell Completion" section to README.md to document how to install
and use `completion.bash` and `completion.zsh`.

Change-Id: Ibf6c81043af6c24d45ea2dc6a6caa26d98ab7374
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/581261
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-05-08 17:41:44 -07:00
Gavin Mak 5d8585012f sync: Suggest "git repack -a -d" in bloat warning
The warning suggests running "repo sync --auto-gc" which runs "git gc
--auto". Git may decide not to repack if its internal thresholds are not
met, leaving the warning active even after running the suggested
command.

"git repack -a -d" forces all reachable objects into a single pack and
deletes redundant packs, reducing the pack count to 1. This guarantees
that the warning (which triggers when pack count exceeds 10) goes away.

Bug: 505755299
Change-Id: I10163ef8efb7f3b7c5055378ad95051974d11b88
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/580821
Reviewed-by: Becky Siegel <beckysiegel@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2026-05-08 14:12:22 -07:00
Kamal Sacranie e8deeabf2a Add zsh completion
This PR adds `completion.zsh` to allow for relatively intelligent
completion of `repo` command-line arguments for all commands of the
`repo` CLI.

The `completion.zsh` can be added to your zsh completions by copying
(and renaming) the completion script to a directory that you have added
to your zsh completion path.

```
cp completion.zsh $ZSH_COMPLETION_DIRECTORY/_repo
```

You must name the file `_repo` for it to register as completion for
`repo`.

You can add a directory to your zsh completion by updating your `fpath`
variable before calling `compinit` as follows:

```
fpath=( /path/to/zsh-completions/dir $fpath )
```

Future work: Generate this file using the subcommand classes and python
reflection.

Change-Id: I6a4e785c1efcf9076bd693976ac03578836b691b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/579561
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Kamal Sacranie <sacranie@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-05-08 07:46:00 -07:00
Gavin Mak 11428ae984 forall: Document REPO_UPSTREAM and REPO_DEST_BRANCH envvars
Change-Id: I74365295152f8828587c6b4ed93029efc6000881
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/580761
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-05-07 20:27:08 -07:00
Carlos Fernandez 27d2232eb3 info: add --format and --include-summary/--include-projects options
Add --format={text,json} to produce machine-readable output, and
boolean options to control which sections are displayed:
  --include-summary / --no-include-summary (default: on)
  --include-projects / --no-include-projects (default: on)

The JSON output respects the include flags, so callers can request
only the fields they need (e.g. `repo info --format=json
--no-include-projects` for manifest metadata only).

Change-Id: I9641bc4023b630d9c61c5170eb86e5f3b787236f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/569203
Commit-Queue: Carlos Fernandez <carlosfsanz@meta.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Carlos Fernandez <carlosfsanz@meta.com>
Tested-by: Carlos Fernandez <carlosfsanz@meta.com>
2026-05-07 19:23:24 -07:00
Josef Malmström e3eadd3728 sync: Fix force_checkout propagation
The force_checkout parameter was not propagated in all calls to
Checkout in Sync_LocalHalf.

Without this, repo sync --force-checkout can still fail for projects
currently on a local branch with no upstream/tracking configuration,
because the detach-to-manifest checkout was executed without -f,
leaving local modifications or untracked files able to block sync.

Change-Id: I58551388e2f906c4db96e220707a369057a71c24
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/579181
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josef Malmstrom <Josef.Malmstrom@arm.com>
Tested-by: Josef Malmstrom <Josef.Malmstrom@arm.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-05-06 08:53:27 -07:00
Gavin Mak 12cfc6036a git_superproject: Remove redundant _branch variable
Both variables were initialized to the same value and never modified.

Bug: 416589884
Change-Id: Iaa1ffffb4543d4cd9391ac679d9234fe01678861
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/576921
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-04-28 10:49:54 -07:00
Nasser Grainawi 134eeb024b tests: Add tests for repo status output
Build out status subcommand unit coverage using a minimal fake repo
checkout wired through XmlManifest.

The new tests verify:
- clean status output prints the expected project header
- modified tracked files appear with the expected status marker
- `-o` output includes the orphan section and orphan entries
- branch names shown in status reflect a started non-default branch

Change-Id: Ia7c22593d0bbdc4aed81faeb168b846f3e4016ab
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/558501
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
Commit-Queue: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-04-23 17:52:09 -07:00
Nasser Grainawi 03fb18109f color: type SetDefaultColoring and drop bool states
Also add a test fixture to always disable coloring so that we get
reproducible test behavior. The few tests that want to check color
behavior (e.g. test_color.py) can still reset/change color values as
needed.

Change-Id: I1808139a63e0862c29bf81d7097e885ca8561040
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/573621
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
Tested-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
2026-04-22 17:05:55 -07:00
Ram Peri 5af71ce907 Add timing keyword argument for hooks
Measure the duration of the sync operation in the Execute method of the
Sync command and pass it to post-sync hooks as a standard keyword
argument (`sync_duration_seconds`).

Updates based on code review:
- Update _API_ARGS in hooks.py to allow sync_duration_seconds for post-sync hooks.
- Do not cast sync_duration_seconds to int for better granularity.
- Update docs/repo-hooks.md to document sync_duration_seconds.
- Add unit test for argument validation in test_hooks.py.

Test: Ran run_tests using venv python, all 554 tests passed.
Bug: TBD
Change-Id: Ie29e002a5d283460d993ad96c224dbf4b6d7985c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/575021
Tested-by: Arif Kasim <arifkasim@google.com>
Commit-Queue: Ram Peri <ramperi@google.com>
Reviewed-by: Arif Kasim <arifkasim@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-04-21 17:10:07 -07:00
Gavin Mak baa281d99e sync: Refactor to use _RunOneGC and fix config leakage
Extract _RunOneGC to handle GC on a single project. This refactoring
makes it easier to invoke GC from parallel worker tasks.

Also, avoid modifying the passed-in config dictionary in _RunOneGC by
creating a local copy, preventing unintended side effects on other
commands sharing the same config.

Bug: 498290329
Change-Id: I7b77ed6629b14b5ee3322870b9c6c8ce2bfd6ea2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574923
Reviewed-by: Becky Siegel <beckysiegel@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-04-20 15:34:07 -07:00
Gavin Mak 7e9079b7cf sync: Switch to using self._bloated_projects
Store bloated projects in self._bloated_projects and print warnings at
the end of execution. This sets up for moving the check to workers.

Bug: 498290329
Change-Id: I993f1fd741db2994d480994861588eb18f6c5503
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574922
Reviewed-by: Becky Siegel <beckysiegel@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-04-20 14:07:14 -07:00
Gavin Mak 61bd6b3ccb tests: Add tests for _CheckForBloatedProjects and _GCProjects
Change-Id: I6790a13929b8fc06a3197ca405dedf08d39d54c9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574926
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Becky Siegel <beckysiegel@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-04-20 13:31:41 -07:00
Marty Heavey 32e7327ca6 upload: Clarify partial sync message on hook failure
Demote the partial sync warning from error to info and rephrase it to
be a tip rather than an error. This prevents users from thinking that
a partial sync is the cause of their hook failures when it is often a linting failure.

The message now suggests that a full sync might help if there are
cross-project dependencies, instead of implying it will fix any issue.

Change-Id: I5d8c52b53ac315aa9f145ed069798bf201fa0815
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574262
Tested-by: Marty Heavey <mheavey@google.com>
Commit-Queue: Marty Heavey <mheavey@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-04-20 01:52:25 -07:00
Gavin Mak 4f707ff91e sync: Provide feedback during post-sync operations
After the main sync progress bar finishes, there's a pause while some
post-sync operations run. Print something to provide feedback so the
user doesn't think repo has hung.

Bug: 503869525
Change-Id: I695fd560e60dcb394e6844a56c8a336ca1f71c74
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574425
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Becky Siegel <beckysiegel@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2026-04-17 15:47:15 -07:00
Gavin Mak e2671c184e progress: Ignore updates after progress ends
This addresses the occasional "..working.." message at the end of sync.

Bug: 503869525
Change-Id: I489d29fa8ae588d77abb7fee5f157800d4abb368
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574482
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Becky Siegel <beckysiegel@google.com>
2026-04-17 15:43:14 -07:00
Becky Siegel b43a20b859 project: Avoid skipping fetches for shallow clones without .git/shallow
When optimizing fetches for projects with immutable revisions, the fetch
should not be skipped if the project is configured for a shallow clone
(depth > 0) but the .git/shallow file is missing. The absence of the
.git/shallow file means the repository is not a shallow clone, or the
shallow clone is incomplete, so a fetch is necessary to ensure the
revision is present.

Bug: 503081454
Change-Id: Ic3549612bcd69050a926652ee4e522c79ad8124c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/573821
Tested-by: Becky Siegel <beckysiegel@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Becky Siegel <beckysiegel@google.com>
2026-04-17 09:16:36 -07:00
Miyako.Enei 8869a30283 project: Drop --no-deref from update-ref --stdin
repo calls `git update-ref --stdin` when updating multiple refs during
repo init and repo sync. Historically, `--no-deref` was also passed.

Older Git 2.17 which we still support rejects the combination of
`--stdin` and `--no-deref`, emitting a usage error even when the stdin
input is valid.

The `--no-deref` option is only meaningful when updating symbolic refs
such as HEAD. The stdin-based update-ref path only operates on explicit
refs (tags, remote refs, alternates) and never symbolic refs.

Remove the unnecessary option to restore compatibility with Git 2.17
while preserving identical behavior on newer Git versions.

Tested with:
  - Git 2.17.1
  - Git 2.34.1

Change-Id: I22001de03800f5699b26a40bc1fb1fec002ed048
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/571721
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Enei <miyako.enei@alpsalpine.com>
Tested-by: Enei <miyako.enei@alpsalpine.com>
2026-04-12 16:39:28 -07:00
Gavin Mak 3b0eebeccf project: implement stateless sync pruning logic
Implement in-situ shallow re-fetching and garbage collection logic.
Enables repositories with sync-strategy="stateless" to reclaim disk
space by running reflog expire and git gc --prune=now if the working
tree is clean and has no local commits.

Bug: 498730431
Change-Id: I940bdc9b74da29d3f7b13566667dcddea769ebd3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/568463
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-04-09 12:09:40 -07:00
Gavin Mak 00991bfb42 manifest: Add sync-strategy attribute to project elements
The only supported sync-strategy is "stateless". The intent is to keep
the local workspace as small as possible by not keeping history during
syncs. This prevents disk space waste for projects with large binaries
where we only care about the current version.

A follow up change will implement the logic.

Bug: 498730431
Change-Id: I84a436a9ca2492893163c6cfda6c28dc62a568f0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/568462
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-04-09 12:09:28 -07:00
Nasser Grainawi e8338b54bd tests: Convert forall subcmd test to pytest
Rewrite tests/test_subcmds_forall.py from unittest.TestCase to pytest
function-style tests to match the surrounding test suite conventions.

Replace setUp/tearDown and class-based helpers with tmp_path-based
setup, switch stdout capture to contextlib.redirect_stdout, and keep the
existing behavior checks intact (all eight projects are invoked exactly
once).

Change-Id: I9243f3461aa6850f867bdb864f4a34c442f817f6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/569821
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
Tested-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-04-09 11:51:47 -07:00
Gavin Mak 951666fb23 gc: Fix hang during repack in partial clones
Add `--missing=allow-promisor` to `git rev-list` calls in
`repack_projects`. This prevents Git from auto-fetching missing objects
from the promisor remote, which can cause stalls due to sequential
network requests.

Also add a Git version check to ensure Git is at least 2.17.0 before
running `--repack`, as `--missing=allow-promisor` was introduced in that
version.

Bug: 500133631
Change-Id: I2dcf9b46fac4c6a53a3c2a46f06f61d6aec40f2f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/570361
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
2026-04-08 12:54:39 -07:00
Carlos Fernandez 854b330967 test_wrapper: add test for repo script executable permission
Add a test to verify that the repo launcher script has the
executable bit set, guarding against accidental permission changes.

Change-Id: I314658b57ed174673188fbbc5962d9fdeefac97d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/569242
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Carlos Fernandez <carlosfsanz@meta.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Carlos Fernandez <carlosfsanz@meta.com>
2026-04-06 14:16:04 -07:00
Mike Frysinger 654690e1b8 tests: convert more tests to pytest
Change-Id: Id4d48b61dc435564c336385bbc4944eb475d1942
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/569443
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-04-06 11:36:39 -07:00
Mike Frysinger ac2be4c089 tests: convert __file__ usage to pathlib
Change-Id: I2408b0ac97629f0d5fc92779b78bf1ff159a6f83
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/569442
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-04-06 11:15:58 -07:00
Mike Frysinger 3d819e8e3e tests: unify fixture() helper with Path constant
Change-Id: I63751042391f5cc3e06af7067bc83d67bd0716dc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/569441
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-04-06 11:15:14 -07:00
Carlos Fernandez 573983948a Fix all flake8 warnings from newer flake8-bugbear and flake8-comprehensions
Address warnings introduced by flake8-bugbear 24.12.12 and
flake8-comprehensions 3.16.0:

- C408: Replace dict()/list() calls with literal {} and []
- C413: Remove unnecessary list() around sorted()
- C414: Remove unnecessary list() inside sorted()
- C419: Suppress intentional list comprehension in all() (noqa)
- B001: Replace bare except with except Exception
- B006: Replace mutable default arguments with None
- B010: Replace setattr() with direct attribute assignment
- B017: Use RuntimeError instead of Exception in tests
- B019: Suppress lru_cache on methods for long-lived objects (noqa)
- B033: Remove duplicate item in set literal

Change-Id: If4693d3e946200bbc22f689f7b94da604addcb80
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/566321
Tested-by: Carlos Fernandez <carlosfsanz@meta.com>
Commit-Queue: Carlos Fernandez <carlosfsanz@meta.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-04-03 07:50:52 -07:00
Gavin Mak 3f3c681a02 project: Refactor GetHead to use symbolic-ref first
Simplify branch resolution and optimize unborn branch detection by
prioritizing symbolic-ref over rev-parse.

Change-Id: Ic62dcb87cd051dafb00d520b1157be2e32abc2ab
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/563222
Reviewed-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-04-02 13:57:05 -07:00
Sam Saccone 242e97d9dd Implement command forgiveness with autocorrect
Similar to `git`, when a user types an unknown command like `repo tart`,
we now use `difflib.get_close_matches` to suggest similar commands.

If `help.autocorrect` is set in the git config, it will optionally
prompt the user to automatically run the assumed command, or wait
for a configured delay before executing it.

Verification Steps:
1. Created a dummy repo project locally.
2. Verified `help.autocorrect=0|false|off|no|show` suggests
   command and exits.
3. Verified `help.autocorrect=1|true|on|yes|immediate`
   automatically runs suggestion.
4. Verified `help.autocorrect=<number>` runs after
   `<number>*0.1` seconds.
5. Verified `help.autocorrect=never` exits immediately without
   suggestions.
6. Verified `help.autocorrect=prompt` asks user to accept [y/n]
   and handles correctly.

BUG: b/489753302

Change-Id: I6dcd63229cbd7badf5404459b48690c68f5b4857
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/558021
Tested-by: Sam Saccone <samccone@google.com>
Commit-Queue: Sam Saccone <samccone@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-03-24 15:47:37 -07:00
Mike Frysinger ade45de770 docs: windows: mention Developer Mode for symlinks
This is probably better than recommending Administrator access.

Change-Id: Ic916f15fe03f7fa1e03c685265b4774bfc1279c2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/563581
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-03-20 14:07:06 -07:00
Lucas Tanure 0251fb33c4 project: don't re-shallow manually unshallowed repos during sync
If a user has manually unshallowed a repo (e.g. via
`git fetch --unshallow`), the absence of the `shallow` file in the
gitdir indicates a full clone. Re-applying depth during a subsequent
sync would undo the user's intent. Skip re-shallowing in this case
by clearing depth when the project is not new and no shallow file
is present.

Change-Id: I4ee0e78018de9078fe1bd77a9615613ef0c40d33
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/558743
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Carlos Fernandez <carlosfsanz@meta.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Lucas Alves <ltanure@gmail.com>
Tested-by: Lucas Alves <ltanure@gmail.com>
Reviewed-by: Lucas Alves <ltanure@gmail.com>
2026-03-20 10:08:40 -07:00
Jacky Liu 0176586544 Use git_superproject.UseSuperproject() everywhere
Currently somewhere use git_superproject.UseSuperproject(), which checks
both the manifest config and user's config, and otherwhere use
manifest.manifestProject.use_superproject, which only checks the
manifest config. This causes Inconsistent behaviors for users who do not
set --use-superproject when doing repo init but have
repo.superprojectChoice in their git config.

Replace where using manifest.manifestProject.use_superproject with
git_superproject.UseSuperproject() to respect user's config and avoid
inconsistency.

Bug: 454514213
Change-Id: I1f734235cdd67b8a6915f1d05967d1aaa4d03f2a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/561801
Commit-Queue: Jacky Liu <qsliu@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jacky Liu <qsliu@google.com>
2026-03-18 21:03:07 -07:00
Mike Frysinger 582804a59e pydev: drop Python 2 reference
Not sure who uses this anymore, but might as well delete obviously
wrong content.

Change-Id: I5cdf1cf699c81b7db32b400f371134d21f474743
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/563161
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-03-18 12:01:37 -07:00
Mike Frysinger afc3d55d39 isort: merge config into pyproject.toml
Change-Id: I3a50de04897789c7b2f291882faf1c862645b054
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/563141
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-03-18 12:01:25 -07:00
Mike Frysinger f24bc7aed5 tests: switch some test modules to pytest
Change-Id: I524b5ff2d77f8232f94e21921b00ba4027d2ac4f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/563081
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-03-18 11:13:26 -07:00
Gavin Mak 83b8ebdbbe git_superproject: avoid re-initing bare repo
Running sync with reftable on a files-backed workspace fails to re-init
the superproject dir with:
```
fatal: could not open
'.../.repo/exp-superproject/<hash>-superproject.git/refs/heads' for writing:
Is a directory
```

Bug: 476209856
Change-Id: Ie8473d66069aafefa5661bd3ea8e73b2b27c6a38
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/550981
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2026-03-17 14:30:35 -07:00
Gavin Mak a0abfd7339 project: resolve unborn HEAD robustly in reftable repos
Use `git symbolic-ref` to resolve HEAD before trying to parse .git/HEAD
directly which is unreliable for reftable repos.

Bug: 476209856
Change-Id: I60185d945c5b43c871945c0126cfdf52194e745d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/550762
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2026-03-17 14:30:20 -07:00
Gavin Mak 403fedfeb5 project: support reftable anchors in worktree .git migration
The reftable backend creates real refs/ and reftable/ dirs. Update
_MigrateOldWorkTreeGitDir to expect these dirs and remove them.

Bug: 476209856
Change-Id: I4700da70cb466e25ecbc51ba4de9a906b8716bd8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/550761
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-03-17 14:30:01 -07:00
Gavin Mak f14c577fce project: avoid direct packed-refs writes during fetch
Replace raw file manipulation with native `git update-ref` commands
inside a try/finally block to ensure temp refs are created/cleaned up
regardless of storage format.

Bug: 476209856
Change-Id: I228e81d3d3b323328260f6672075193421c8dc47
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/550421
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-03-17 14:29:38 -07:00
Gavin Mak 67881c0c3b git_refs: read refs via git plumbing for files/reftable
Replace direct `packed-refs` file parsing with `git for-each-ref`
plumbing to support both `files` and `reftable` backends.

Bug: 476209856
Change-Id: I2ad8ff8f3382426600f15370c997f9bc17165485
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/550401
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2026-03-17 14:28:39 -07:00
Mike Frysinger 551087cd98 tests: add a util module for sharing code
We've started duplicating code among test modules.  Start a common
utils module to hold that, and migrate over TempGitTree to start.

Change-Id: I10b2abd133535c90fbda4d6686602d7e5861d875
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/559041
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-03-12 20:09:19 -07:00
Mike Frysinger 8da56a0cc5 man: refresh after recent changes
Change-Id: Ibd60f89406e89255b3284413442b1d9c0ccbfb6d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/559601
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jeffery Miller <jefferymiller@google.com>
2026-03-10 07:35:40 -07:00
Jeffery Miller 0f01cd24e9 docs: Document support for child elements in extend-project
Clarify the existence and behavior of child elements when added to
extend-project.

Change-Id: Id9f270166c8498d4051495b9a1f68360f66e9143
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/553742
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jeffery Miller <jefferymiller@google.com>
Commit-Queue: Jeffery Miller <jefferymiller@google.com>
2026-02-19 08:38:56 -08:00
Jeffery Miller 1ee98667cc tests: Add extend-project test for additional annotations
Multiple annotations can exist for the same name when
extending projects. Add a test case to show this behavior.

Change-Id: I12bbd25e642c7e615e32f66a1c364a39ac81902c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/553906
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Jeffery Miller <jefferymiller@google.com>
Tested-by: Jeffery Miller <jefferymiller@google.com>
2026-02-19 08:35:03 -08:00
Jordan Esh 6f9622fe1c sync: Remove dependency on ssh if not needed
When running on a machine without the `ssh` command, repo sync would fail even if no ssh or ssh proxy was required. Use exception handling inside ssh.ProxyManager to more gracefully handle the case where ssh is not installed.

Bug: 467714011
Change-Id: I602a0819638ead4d02de88b750839bc3d70549ce
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/535141
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Jordan Esh <esh.jordan@gmail.com>
Commit-Queue: Jordan Esh <esh.jordan@gmail.com>
2026-02-11 17:00:07 -08:00
Gavin Mak 5cb0251248 gc: fix untargeted projects being deleted
`delete_unused_projects` needs a full list of active projects to figure
out which orphaned .git dirs need to be deleted. Otherwise it thinks
that only the projects specified in args are active.

Bug: 447626164
Change-Id: I02beebf6a01c77742a8db78221452d71cd78ea73
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/550061
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-02-09 08:34:42 -08:00
Gavin Mak a214fd31bd manifest: Introduce sync-j-max attribute to cap sync jobs
Add a way for manifest owners to limit how many sync jobs run in
parallel.

Bug: 481100878
Change-Id: Ia6cbe02cbc83c9e414b53b8d14fe5e7e1b802505
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/548963
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-02-06 13:32:28 -08:00
Sam Saccone 62cd0de6cf Make git trace2 logging errors conditional on verbose mode.
Add a verbose attribute to the EventLog class, defaulting to False.
Error messages printed to sys.stderr within the EventLog.Write method
are now guarded by this verbose flag. In main.py, set EventLog.verbose
to True if the command-line --verbose option is used. This prevents
trace2 logging failures from being printed to stderr unless verbose
output is explicitly requested.

PROMPT=convert all git trace2 logging print messages to verbose only
logging

BUG: b/479811034
Change-Id: I8757ee52117d766f2f3ec47856db64cc4f51143c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/547542
Tested-by: Sam Saccone <samccone@google.com>
Reviewed-by: Julia Tuttle <juliatuttle@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-02-03 11:53:48 -08:00
Mike Frysinger b60512a75a run_tests: log tool versions
Change-Id: I4eee58786bae6d442773c63fa937fb11eda1e2f0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/547863
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-01-29 12:32:46 -08:00
Mike Frysinger 5d88972390 Revert "init: change --manifest-depth default to 1"
This reverts commit 622a5bf9c2.

CrOS infra is failing to sync code now for some reason.
Until we can investigate further, pull this back out.

Bug: 475668525
Bug: 468033850
Change-Id: I35a8623a95336df1be27ea870afbfc8065609f01
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/545141
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-01-21 12:32:59 -08:00
Gavin Mak 3c0e67bbc5 manifest_xml: prevent extend-project from inheriting local groups
When extending a project in a local manifest, the project inherits the
`local:` group. This causes the superproject override logic (which omits
projects with `local:` groups) to incorrectly exclude the project from
the override manifest. This leads to "extend-project element specifies
non-existent project" errors during sync reload.

Fix this by stripping `local:` groups from extended projects, ensuring
they remain visible to superproject overrides while still allowing other
inherited groups to persist.

Bug: 470374343
Change-Id: I1a057ebffebc11a19dc14dde7cc13b9f18cdd0a3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/543222
Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-01-15 10:27:32 -08:00
Nico Wald 3b7b20ac1d CONTRIBUTING: fix HTTP password URL
Change-Id: I7ae085896fe951c2b1c662689fa111a0661f988d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/539762
Tested-by: Nico Wald <nicowald@mac.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Nico Wald <nicowald@mac.com>
2026-01-12 09:48:47 -08:00
Gavin Mak e71a8c6dd8 project: disable auto-gc for depth=1 in git config
During sync, `git checkout` can trigger fetch for missing objects in
partial clones. This internal fetch can trigger `git maintenance` or
`git gc` and cause delays during the local checkout phase. Set
maintenance.auto to false and gc.auto to 0 in during `_InitRemote` if
`depth=1` to ensure that implicit fetches spawned by git skip GC.

Bug: 379111283
Change-Id: I6b22a4867f29b6e9598746cb752820a84dc2aeb6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/540681
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-01-08 11:33:40 -08:00
Mike Frysinger c687b5df9e run_tests/release: require Python 3.9+
While we support running `repo` on clients with older Python versions,
we don't need to hold the runners & release code back.  These are only
used by repo devs on their systems to develop & release repo.

Python 3.9 was picked due to its typing changs which we've already
started using in this code.

Change-Id: I6f8885c84298760514c25abeb1fccb0338947bf4
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/539801
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-01-06 11:36:26 -08:00
Mike Frysinger 1dd9c57a28 tests: drop tox support
This hasn't been working out as well as we'd hope.  Tox relies on
the system having Python versions installed which distros don't
tend to carry anymore.  Our custom run_tests leverages vpython
when possible to run stable Python 3.8 & 3.11 versions which is
providing an OK level of coverage in practice.

Change-Id: Ida517f7be47ca95703e43bc0af5a24dd70c0467e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/540001
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-01-06 11:32:42 -08:00
Mike Frysinger 4525c2e0ad github: add black check action
Change-Id: Ic87c1c5c72fb8a01108146c1f9d78466acb57278
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/540021
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-01-06 11:00:32 -08:00
Mike Frysinger 45dcd738b7 tests: skip AF_UNIX tests when unavailable
UNIX sockets aren't available under Windows, so skip the test.

Change-Id: Ic4ca22d161c6dee628352aad07ac6aaceb472ac2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/540002
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2026-01-06 10:17:53 -08:00
Mike Frysinger 1dad86dc00 check-metadata: skip files that do not exist
If the files don't exist, then they can't have errors, so skip checking.

Change-Id: I3ed4be4912b253c5454df41d690cb33dfe191289
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/540003
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-01-06 10:17:32 -08:00
Mike Frysinger 622a5bf9c2 init: change --manifest-depth default to 1
Most users do not care about the manifest history in .repo/manifests/.
Let's change the default to 1 so things work smoothly for most people
most of the time.  For the rare folks who want the full history, they
can add --manifest-depth=0 to their `repo init`.

This has no effect on existing checkouts.

Spot checking Android & CrOS manifests shows significant speedups.
Full history can take O(10's seconds) to O(minutes) while depth of 1
takes constant time of O(~5 seconds).

Bug: 468033850
Change-Id: I4b8ed62a8a636babcc5226552badb69600d0c353
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/535481
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-01-05 06:36:08 -08:00
Gavin Mak 871e4c7ed1 sync: skip bloat check if fresh sync
Initial syncs won't have accumulated any garbage.

Bug: 379111283
Change-Id: I04b2ecde3e33f1f055038861a2705ab6aabb36d1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/536083
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2025-12-15 15:24:45 -08:00
Gavin Mak 5b0b5513d6 project: only use --no-auto-gc for git 2.23.0+
The flag for git fetch was introduced in git 2.23.0. Also skip the bloat
check after sync if using an older version.

Bug: 468589976
Bug: 379111283
Change-Id: Ib53e5494350c71a83906e5219d3a8c2b654e531f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/536082
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2025-12-15 11:32:49 -08:00
80 changed files with 6448 additions and 2583 deletions
+16
View File
@@ -0,0 +1,16 @@
# GitHub actions workflow.
# https://help.github.com/en/actions/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions
# https://black.readthedocs.io/en/stable/integrations/github_actions.html
name: Format
on:
push:
branches: [main]
jobs:
format:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: psf/black@stable
+3 -3
View File
@@ -27,6 +27,6 @@ jobs:
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install tox tox-gh-actions
- name: Test with tox
run: tox
python -m pip install pytest
- name: Run tests
run: python -m pytest
+1
View File
@@ -6,6 +6,7 @@ __pycache__
/dist
.repopickle_*
/repoc
/.cipd_bin
/.tox
/.venv
-41
View File
@@ -1,41 +0,0 @@
# Copyright (C) 2023 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Config file for the isort python module.
# This is used to enforce import sorting standards.
#
# https://pycqa.github.io/isort/docs/configuration/options.html
[settings]
# Be compatible with `black` since it also matches what we want.
profile = black
line_length = 80
length_sort = false
force_single_line = true
lines_after_imports = 2
from_first = false
case_sensitive = false
force_sort_within_sections = true
order_by_type = false
# Ignore generated files.
extend_skip_glob = *_pb2.py
# Allow importing multiple classes on a single line from these modules.
# https://google.github.io/styleguide/pyguide#s2.2-imports
single_line_exclusions =
abc,
collections.abc,
typing,
-1
View File
@@ -5,6 +5,5 @@
<pydev_pathproperty name="org.python.pydev.PROJECT_SOURCE_PATH">
<path>/git-repo</path>
</pydev_pathproperty>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python 2.7</pydev_property>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>
</pydev_project>
+8 -17
View File
@@ -24,7 +24,7 @@ However there are some differences, so please review and familiarize
yourself with the following relevant bits.
## Make separate commits for logically separate changes.
## Make separate commits for logically separate changes
Unless your patch is really trivial, you should not be sending out a patch that
was generated between your working tree and your commit head.
@@ -43,17 +43,12 @@ probably need to split up your commit to finer grained pieces.
Lint any changes by running:
```sh
$ tox -e lint -- file.py
$ flake8
```
And format with:
```sh
$ tox -e format -- file.py
```
Or format everything:
```sh
$ tox -e format
$ black file.py
```
Repo uses [black](https://black.readthedocs.io/) with line length of 80 as its
@@ -73,15 +68,11 @@ the entire project in the included `.flake8` file.
[PEP 8]: https://www.python.org/dev/peps/pep-0008/
[flake8 documentation]: https://flake8.pycqa.org/en/3.1.1/user/ignoring-errors.html#in-line-ignoring-errors
## Running tests
We use [pytest](https://pytest.org/) and [tox](https://tox.readthedocs.io/) for
running tests. You should make sure to install those first.
To run the full suite against all supported Python versions, simply execute:
```sh
$ tox -p auto
```
We use [pytest](https://pytest.org/) for running tests. You should make sure to
install that first.
We have [`./run_tests`](./run_tests) which is a simple wrapper around `pytest`:
```sh
@@ -127,7 +118,7 @@ your patch. It is virtually impossible to remove a patch once it
has been applied and pushed out.
## Sending your patches.
## Sending your patches
Do not email your patches to anyone.
@@ -143,7 +134,7 @@ they have right to redistribute your work under the Apache License:
Ensure you have obtained an HTTP password to authenticate:
https://gerrit-review.googlesource.com/new-password
https://www.googlesource.com/new-password
Ensure that you have the local commit hook installed to automatically
add a ChangeId to your commits:
+37
View File
@@ -49,6 +49,43 @@ $ curl https://storage.googleapis.com/git-repo-downloads/repo > ~/.bin/repo
$ chmod a+rx ~/.bin/repo
```
## Shell Completion
Repo includes completion scripts for Bash and Zsh.
### Bash
To enable completion in Bash, source `completion.bash` in your `~/.bashrc`:
```sh
source /path/to/git-repo/completion.bash
```
### Zsh
To enable completion in Zsh, you can either:
1. Copy or symlink `completion.zsh` to a file named `_repo` in a directory in your `$fpath`:
```sh
mkdir -p ~/.zsh/completion
# You can copy the file:
cp /path/to/git-repo/completion.zsh ~/.zsh/completion/_repo
# Or symlink it:
ln -s /path/to/git-repo/completion.zsh ~/.zsh/completion/_repo
```
Then add that directory to your `fpath` in `~/.zshrc` before `compinit`:
```zsh
fpath=(~/.zsh/completion $fpath)
autoload -Uz compinit
compinit
```
2. Or source the file directly and call `compdef` in your `~/.zshrc`:
```zsh
source /path/to/git-repo/completion.zsh
compdef _repo repo
```
[new-bug]: https://issues.gerritcodereview.com/issues/new?component=1370071
[issue tracker]: https://issues.gerritcodereview.com/issues?q=is:open%20componentid:1370071
+32
View File
@@ -0,0 +1,32 @@
# Copyright (C) 2026 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This file contains version pins of a few tools used by run_tests.
# Pin resolved versions in the repo, to reduce trust in the CIPD backend.
#
# Most of these tools are generated via builders at
# https://ci.chromium.org/p/infra/g/infra/console
#
# For these, the git revision is the one of
# https://chromium.googlesource.com/infra/infra.git.
#
# To regenerate them (after modifying this file):
# cipd ensure-file-resolve -ensure-file cipd_manifest.txt
$ResolvedVersions cipd_manifest.versions
# Supported platforms. Feel free to add more as long as the tools work.
$VerifiedPlatform linux-amd64 linux-arm64 mac-amd64 mac-arm64
infra/3pp/tools/help2man/${platform} version:3@1.49.3
+18
View File
@@ -0,0 +1,18 @@
# This file is auto-generated by 'cipd ensure-file-resolve'.
# Do not modify manually. All changes will be overwritten.
infra/3pp/tools/help2man/linux-amd64
version:3@1.49.3
82ySLC4--mvjnmuwj_j_2odHLj5zFaNFpWvfCyMunY0C
infra/3pp/tools/help2man/linux-arm64
version:3@1.49.3
szfWfjI4RyCT6gIB48CeUOGG6_Un4wfbIb0Vvs5M9MUC
infra/3pp/tools/help2man/mac-amd64
version:3@1.49.3
t7iyQx5TnGbuJfj6zMTvjPdBl-Cb_w5papUh80NI2S8C
infra/3pp/tools/help2man/mac-arm64
version:3@1.49.3
t5eD-eLKRpnSifZdbi072U8bnPfGye1pIMmQ07nxJ7kC
+4 -3
View File
@@ -14,6 +14,7 @@
import os
import sys
from typing import Optional
import pager
@@ -84,7 +85,7 @@ def _Color(fg=None, bg=None, attr=None):
DEFAULT = None
def SetDefaultColoring(state):
def SetDefaultColoring(state: Optional[str]) -> None:
"""Set coloring behavior to |state|.
This is useful for overriding config options via the command line.
@@ -97,9 +98,9 @@ def SetDefaultColoring(state):
state = state.lower()
if state in ("auto",):
DEFAULT = state
elif state in ("always", "yes", "true", True):
elif state in ("always", "yes", "true"):
DEFAULT = "always"
elif state in ("never", "no", "false", False):
elif state in ("never", "no", "false"):
DEFAULT = "never"
+23 -1
View File
@@ -101,6 +101,11 @@ class Command:
def WantPager(self, _opt):
return False
@staticmethod
def is_multiprocessing_active() -> bool:
"""Whether the current process is a worker in a pool."""
return multiprocessing.current_process().name != "MainProcess"
def ReadEnvironmentOptions(self, opts):
"""Set options from environment variables."""
@@ -407,7 +412,8 @@ class Command:
for project in all_projects_list:
if submodules_ok or project.sync_s:
derived_projects.update(
(p.name, p) for p in project.GetDerivedSubprojects()
(p.RelPath(local=False), p)
for p in project.GetDerivedSubprojects()
)
all_projects_list.extend(derived_projects.values())
for project in all_projects_list:
@@ -511,6 +517,22 @@ class Command:
)
return result
def GetSmartSyncOverridePath(self, manifest=None) -> str:
"""Return the path where smart sync writes its override manifest."""
if manifest is None:
manifest = self.manifest
return os.path.join(
manifest.manifestProject.worktree, "smart_sync_override.xml"
)
def TryOverrideManifestWithSmartSync(self, manifest=None) -> None:
"""Override manifest with smart_sync_override.xml if it exists."""
if manifest is None:
manifest = self.manifest
smart_sync_manifest_path = self.GetSmartSyncOverridePath(manifest)
if os.path.isfile(smart_sync_manifest_path):
manifest.Override(smart_sync_manifest_path)
def ManifestList(self, opt):
"""Yields all of the manifests to traverse.
+449
View File
@@ -0,0 +1,449 @@
#compdef _repo repo
# Copyright (C) 2026 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
_repo() {
local context state line
typeset -A opt_args
local -a subcommands
subcommands=(
'abandon:Abandon a development branch'
'branches:View current topic branches'
'checkout:Checkout a branch'
'cherry-pick:Cherry-pick a change'
'diff:Show changes between commit and working tree'
'diffmanifests:Show differences between two manifests'
'download:Download and find a change'
'forall:Run a shell command in each project'
'gc:Garbage collect'
'grep:Print lines matching a pattern'
'help:Display help about a command'
'info:Get info about the manifest'
'init:Initialize a repo client'
'list:List projects and their names'
'manifest:Manifest management'
'overview:Display overview of topic branches'
'prune:Prune topic branches'
'rebase:Rebase local branches'
'selfupdate:Update repo'
'smartsync:Update working tree to latest known good revision'
'stage:Stage file(s) for commit'
'start:Start a new branch for development'
'status:Show the working tree status'
'sync:Update working tree'
'upload:Upload changes to the review server'
'version:Display version'
'wipe:Wipe uncommitted changes'
)
local -a global_opts
global_opts=(
'(-h --help)'{-h,--help}'[Show help message]'
'--help-all[Show help message for all commands]'
'(-p --paginate)'{-p,--paginate}'[Pipe all output into less]'
'--no-pager[Do not pipe output into a pager]'
'--color=[Control color output]:color:(always never auto)'
'--trace[Trace git command execution]'
'--trace-to-stderr[Trace git command execution to stderr]'
'--trace-python[Trace python execution]'
'--time[Time execute of command]'
'--version[Show version]'
'--show-toplevel[Show top level of workspace]'
'--event-log=[File to log events to]:file:_files'
'--git-trace2-event-log=[File to log git trace2 events to]:file:_files'
'--submanifest-path=[Path to submanifest]:path:_files'
)
_arguments -C \
${global_opts} \
'1: :->cmds' \
'*:: :->subcmds'
case ${state} in
cmds)
_describe -t commands 'repo command' subcommands
;;
subcmds)
local cmd=${words[1]}
curcontext="${curcontext%:*}-${cmd}:"
local -a common_opts
common_opts=(
'(-h --help)'{-h,--help}'[Show help message]'
'(-v --verbose)'{-v,--verbose}'[Show verbose messages]'
'(-q --quiet)'{-q,--quiet}'[Show only errors]'
'(-j --jobs)'{-j,--jobs=}'[Number of jobs to run in parallel]:jobs:'
'--outer-manifest[Use outer manifest]'
'--no-outer-manifest[Do not use outer manifest]'
'--this-manifest-only[Only use this manifest]'
'--no-this-manifest-only[Do not only use this manifest]'
'--all-manifests[Use all manifests]'
)
case ${cmd} in
abandon)
_arguments \
${common_opts} \
'--all[Abandon all branches]' \
'1: :->branch' \
'*: :->project'
;;
branches)
_arguments \
${common_opts} \
'*: :->project'
;;
checkout)
_arguments \
${common_opts} \
'1: :->branch' \
'*: :->project'
;;
cherry-pick)
_arguments \
${common_opts} \
'1: :_message "sha1"'
;;
diff)
_arguments \
${common_opts} \
'(-u --absolute)'{-u,--absolute}'[Show absolute paths]' \
'*: :->project'
;;
diffmanifests)
_arguments \
'--raw[Show raw diff]' \
'--no-color[Do not show color]' \
'--pretty-format=[Pretty format]:format:' \
'1: :_files -g "*.xml"' \
'2: :_files -g "*.xml"'
;;
download)
_arguments \
${common_opts} \
'(-b --branch)'{-b,--branch=}'[One or more branches to check]:branch:' \
'(-c --cherry-pick)'{-c,--cherry-pick}'[Cherry-pick the change]' \
'(-x --record-origin)'{-x,--record-origin}'[Record origin of cherry-pick]' \
'(-r --revert)'{-r,--revert}'[Revert the change]' \
'(-f --ff-only)'{-f,--ff-only}'[Force fast-forward]' \
'*: :_message "change[/patchset]"'
;;
forall)
_arguments \
${common_opts} \
'(-r --regex)'{-r,--regex}'[Execute command only on projects matching regex]' \
'(-i --inverse-regex)'{-i,--inverse-regex}'[Execute command only on projects not matching regex]' \
'(-g --groups)'{-g,--groups=}'[Execute command only on projects matching groups]:groups:' \
'(-c --command)'{-c,--command}'[Command to execute]' \
'(-e --abort-on-errors)'{-e,--abort-on-errors}'[Abort on errors]' \
'--ignore-missing[Ignore missing projects]' \
'--interactive[Run interactively]' \
'-p[Show project headers]' \
'*: :->project'
;;
gc)
_arguments \
${common_opts} \
'(-n --dry-run)'{-n,--dry-run}'[Dry run]' \
'(-y --yes)'{-y,--yes}'[Answer yes to all prompts]' \
'--repack[Repack objects]'
;;
grep)
_arguments \
${common_opts} \
'--cached[Search cached files]' \
'(-r --revision)'{-r,--revision=}'[Search in revision]:revision:' \
'-e[Pattern]' \
'(-i --ignore-case)'{-i,--ignore-case}'[Ignore case]' \
'(-a --text)'{-a,--text}'[Treat all files as text]' \
'-I[Do not match binary files]' \
'(-w --word-regexp)'{-w,--word-regexp}'[Match word boundaries]' \
'(-v --invert-match)'{-v,--invert-match}'[Invert match]' \
'(-G --basic-regexp)'{-G,--basic-regexp}'[Basic regexp]' \
'(-E --extended-regexp)'{-E,--extended-regexp}'[Extended regexp]' \
'(-F --fixed-strings)'{-F,--fixed-strings}'[Fixed strings]' \
'--all-match[All match]' \
'--and[And]' \
'--or[Or]' \
'--not[Not]' \
'-([Open paren]' \
'-)[Close paren]' \
'-n[Line number]' \
'-C[Context]:lines:' \
'-B[Before context]:lines:' \
'-A[After context]:lines:' \
'-l[Name only]' \
'--name-only[Name only]' \
'--files-with-matches[Files with matches]' \
'-L[Files without match]' \
'--files-without-match[Files without match]' \
'1: :->pattern' \
'*: :->project'
;;
help)
_arguments \
'(-a --all)'{-a,--all}'[Show all commands]' \
'--help-all[Show help message for all commands]' \
'1: :->help_cmds'
;;
info)
_arguments \
${common_opts} \
'(-d --diff)'{-d,--diff}'[Show diff]' \
'(-o --overview)'{-o,--overview}'[Show overview]' \
'(-c --current-branch)'{-c,--current-branch}'[Show current branch]' \
'--no-current-branch[Do not show current branch]' \
'(-l --local-only)'{-l,--local-only}'[Local only]' \
'*: :->project'
;;
init)
_arguments \
'(-u --manifest-url)'{-u,--manifest-url=}'[Manifest URL]:url:' \
'(-b --manifest-branch)'{-b,--manifest-branch=}'[Manifest branch]:branch:' \
'--manifest-upstream-branch=[Manifest upstream branch]:branch:' \
'(-m --manifest-name)'{-m,--manifest-name=}'[Manifest name]:file:_files -g "*.xml"' \
'(-g --groups)'{-g,--groups=}'[Restrict manifest projects to groups]:groups:' \
'(-p --platform)'{-p,--platform=}'[Restrict manifest projects to platform]:platform:' \
'--submodules[Sync submodules]' \
'--standalone-manifest[Standalone manifest]' \
'--manifest-depth=[Manifest depth]:depth:' \
'(-c --current-branch)'{-c,--current-branch}'[Sync current branch only]' \
'--no-current-branch[Do not sync current branch only]' \
'--tags[Sync tags]' \
'--no-tags[Do not sync tags]' \
'--mirror[Mirror]' \
'--archive[Archive]' \
'--worktree[Worktree]' \
'--reference=[Reference repository]:repository:_files -/' \
'--dissociate[Dissociate from reference]' \
'--depth=[Depth]:depth:' \
'--partial-clone[Partial clone]' \
'--no-partial-clone[Do not partial clone]' \
'--partial-clone-exclude=[Exclude from partial clone]:projects:' \
'--clone-filter=[Clone filter]:filter:' \
'--use-superproject[Use superproject]' \
'--no-use-superproject[Do not use superproject]' \
'--clone-bundle[Use clone bundle]' \
'--no-clone-bundle[Do not use clone bundle]' \
'--git-lfs[Use git lfs]' \
'--no-git-lfs[Do not use git lfs]' \
'--repo-url=[Repo URL]:url:' \
'--repo-rev=[Repo revision]:revision:' \
'--no-repo-verify[Do not verify repo]' \
'--config-name[Use config name]'
;;
list)
_arguments \
${common_opts} \
'(-r --regex)'{-r,--regex}'[Filter by regex]' \
'(-g --groups)'{-g,--groups=}'[Filter by groups]:groups:' \
'(-a --all)'{-a,--all}'[Show all projects]' \
'(-n --name-only)'{-n,--name-only}'[Show only names]' \
'(-p --path-only)'{-p,--path-only}'[Show only paths]' \
'(-f --fullpath)'{-f,--fullpath}'[Show full paths]' \
'--relative-to=[Show paths relative to]:path:_files -/' \
'*: :->project'
;;
manifest)
_arguments \
'(-r --revision-as-HEAD)'{-r,--revision-as-HEAD}'[Save revisions as current HEAD]' \
'(-m --manifest-name)'{-m,--manifest-name=}'[Manifest name]:file:_files -g "*.xml"' \
'--suppress-upstream-revision[Suppress upstream revision]' \
'--suppress-dest-branch[Suppress dest branch]' \
'--format=[Output format]:format:(xml json)' \
'--pretty[Pretty print]' \
'--no-local-manifests[Ignore local manifests]' \
'(-o --output-file)'{-o,--output-file=}'[Output file]:file:_files'
;;
overview)
_arguments \
${common_opts} \
'(-c --current-branch)'{-c,--current-branch}'[Show current branch]' \
'--no-current-branch[Do not show current branch]' \
'*: :->project'
;;
prune)
_arguments \
${common_opts} \
'*: :->project'
;;
rebase)
_arguments \
${common_opts} \
'--fail-fast[Fail fast]' \
'(-f --force-rebase)'{-f,--force-rebase}'[Force rebase]' \
'--no-ff[No fast forward]' \
'--autosquash[Autosquash]' \
'--whitespace=[Whitespace option]:option:' \
'--auto-stash[Auto stash]' \
'(-m --onto-manifest)'{-m,--onto-manifest}'[Rebase onto manifest]' \
'(-i --interactive)'{-i,--interactive}'[Interactive rebase]' \
'*: :->project'
;;
selfupdate)
_arguments \
'--no-repo-verify[Do not verify repo]'
;;
smartsync | sync)
_arguments \
${common_opts} \
'--jobs-network=[Number of network jobs]:jobs:' \
'--jobs-checkout=[Number of checkout jobs]:jobs:' \
'(-f --force-broken)'{-f,--force-broken}'[Continue sync even if a project fails]' \
'--fail-fast[Fail fast]' \
'--force-sync[Overwrite existing git directories]' \
'--force-checkout[Force checkout]' \
'--force-remove-dirty[Force remove dirty]' \
'--rebase[Rebase local branches]' \
'(-l --local-only)'{-l,--local-only}'[Only use local files]' \
'--no-manifest-update[Do not update manifest]' \
'--nmu[Do not update manifest]' \
'--interleaved[Interleave output]' \
'--no-interleaved[Do not interleave output]' \
'(-n --network-only)'{-n,--network-only}'[Only fetch]' \
'(-d --detach)'{-d,--detach}'[Detach HEAD]' \
'(-c --current-branch)'{-c,--current-branch}'[Fetch only current branch]' \
'--no-current-branch[Do not fetch only current branch]' \
'(-m --manifest-name)'{-m,--manifest-name=}'[Manifest name]:file:_files -g "*.xml"' \
'--clone-bundle[Use clone bundle]' \
'--no-clone-bundle[Do not use clone bundle]' \
'(-u --manifest-server-username)'{-u,--manifest-server-username=}'[Username for manifest server]:username:' \
'(-p --manifest-server-password)'{-p,--manifest-server-password=}'[Password for manifest server]:password:' \
'--fetch-submodules[Fetch submodules]' \
'--use-superproject[Use superproject]' \
'--no-use-superproject[Do not use superproject]' \
'--tags[Sync tags]' \
'--no-tags[Do not sync tags]' \
'--optimized-fetch[Optimized fetch]' \
'--retry-fetches=[Retry fetches]:count:' \
'--prune[Prune]' \
'--no-prune[Do not prune]' \
'--auto-gc[Run auto gc]' \
'--no-auto-gc[Do not run auto gc]' \
'(-s --smart-sync)'{-s,--smart-sync}'[Smart sync]' \
'(-t --smart-tag)'{-t,--smart-tag=}'[Smart tag]:tag:' \
'--no-repo-verify[Do not verify repo]' \
'--no-verify[Do not verify]' \
'--verify[Verify]' \
'--ignore-hooks[Ignore hooks]' \
'*: :->project'
;;
stage)
_arguments \
${common_opts} \
'(-i --interactive)'{-i,--interactive}'[Interactive staging]' \
'*: :->project'
;;
start)
_arguments \
${common_opts} \
'--all[Start branch in all projects]' \
'(-r --rev --revision)'{-r,--rev=,--revision=}'[Revision]:revision:' \
'--head[Head]' \
'--HEAD[Head]' \
'1: :->newbranch' \
'*: :->project'
;;
status)
_arguments \
${common_opts} \
'(-o --orphans)'{-o,--orphans}'[Show orphans]' \
'*: :->project'
;;
upload)
_arguments \
${common_opts} \
'(-t --topic-branch)'{-t,--topic-branch}'[Topic branch]' \
'--topic=[Topic]:topic:' \
'--hashtag=[Hashtag]:hashtag:' \
'--ht=[Hashtag]:hashtag:' \
'--hashtag-branch[Hashtag branch]' \
'--htb[Hashtag branch]' \
'(-l --label)'{-l,--label=}'[Label]:label:' \
'--pd=[Patchset description]:description:' \
'--patchset-description=[Patchset description]:description:' \
'--re=[Reviewers]:reviewers:' \
'--reviewers=[Reviewers]:reviewers:' \
'--cc=[CC]:cc:' \
'--br=[Branch]:branch:' \
'--branch=[Branch]:branch:' \
'(-c --current-branch)'{-c,--current-branch}'[Upload current branch]' \
'--no-current-branch[Do not upload current branch]' \
'--ne[No emails]' \
'--no-emails[No emails]' \
'(-p --private)'{-p,--private}'[Private]' \
'(-w --wip)'{-w,--wip}'[Work in progress]' \
'(-r --ready)'{-r,--ready}'[Ready]' \
'(-o --push-option)'{-o,--push-option=}'[Push option]:option:' \
'(-D --destination --dest)'{-D,--destination=,--dest=}'[Destination]:destination:' \
'(-n --dry-run)'{-n,--dry-run}'[Dry run]' \
'(-y --yes)'{-y,--yes}'[Answer yes to all prompts]' \
'--ignore-untracked-files[Ignore untracked files]' \
'--no-ignore-untracked-files[Do not ignore untracked files]' \
'--no-cert-checks[Do not check certificates]' \
'--no-verify[Do not verify]' \
'--verify[Verify]' \
'--ignore-hooks[Ignore hooks]' \
'*: :->project'
;;
version)
_arguments ${common_opts}
;;
wipe)
_arguments \
${common_opts} \
'(-f --force)'{-f,--force}'[Force wipe]' \
'--force-uncommitted[Force uncommitted]' \
'--force-shared[Force shared]' \
'*: :->project'
;;
esac
# Handle states for positional arguments.
case ${state} in
branch)
[[ ${PREFIX} != -* ]] && _repo_branches
;;
project)
[[ ${PREFIX} != -* ]] && _repo_projects
;;
pattern)
_message 'pattern'
;;
newbranch)
_message 'new branch name'
;;
help_cmds)
_describe -t commands 'repo command' subcommands
;;
esac
;;
esac
}
_repo_branches() {
local -a branches
branches=(${(f)"$(_call_program branches repo branches 2>/dev/null | sed -E 's/^.*[[:space:]]([^[:space:]]+)[[:space:]]+\|.*$/\1/' | awk '{print $1}')"})
_describe -t branches 'branch' branches
}
_repo_projects() {
local -a projects
projects=(${(f)"$(_call_program projects repo list -n 2>/dev/null)"})
_describe -t projects 'project' projects
}
_repo "$@"
+38 -13
View File
@@ -51,6 +51,7 @@ following DTD:
<!ATTLIST default dest-branch CDATA #IMPLIED>
<!ATTLIST default upstream CDATA #IMPLIED>
<!ATTLIST default sync-j CDATA #IMPLIED>
<!ATTLIST default sync-j-max CDATA #IMPLIED>
<!ATTLIST default sync-c CDATA #IMPLIED>
<!ATTLIST default sync-s CDATA #IMPLIED>
<!ATTLIST default sync-tags CDATA #IMPLIED>
@@ -72,18 +73,19 @@ following DTD:
project*,
copyfile*,
linkfile*)>
<!ATTLIST project name CDATA #REQUIRED>
<!ATTLIST project path CDATA #IMPLIED>
<!ATTLIST project remote IDREF #IMPLIED>
<!ATTLIST project revision CDATA #IMPLIED>
<!ATTLIST project dest-branch CDATA #IMPLIED>
<!ATTLIST project groups CDATA #IMPLIED>
<!ATTLIST project sync-c CDATA #IMPLIED>
<!ATTLIST project sync-s CDATA #IMPLIED>
<!ATTLIST project sync-tags CDATA #IMPLIED>
<!ATTLIST project upstream CDATA #IMPLIED>
<!ATTLIST project clone-depth CDATA #IMPLIED>
<!ATTLIST project force-path CDATA #IMPLIED>
<!ATTLIST project name CDATA #REQUIRED>
<!ATTLIST project path CDATA #IMPLIED>
<!ATTLIST project remote IDREF #IMPLIED>
<!ATTLIST project revision CDATA #IMPLIED>
<!ATTLIST project dest-branch CDATA #IMPLIED>
<!ATTLIST project groups CDATA #IMPLIED>
<!ATTLIST project sync-c CDATA #IMPLIED>
<!ATTLIST project sync-s CDATA #IMPLIED>
<!ATTLIST project sync-tags CDATA #IMPLIED>
<!ATTLIST project upstream CDATA #IMPLIED>
<!ATTLIST project clone-depth CDATA #IMPLIED>
<!ATTLIST project force-path CDATA #IMPLIED>
<!ATTLIST project sync-strategy CDATA #IMPLIED>
<!ELEMENT annotation EMPTY>
<!ATTLIST annotation name CDATA #REQUIRED>
@@ -213,7 +215,9 @@ can be found. Used when syncing a revision locked manifest in
-c mode to avoid having to sync the entire ref space. Project elements
not setting their own `upstream` will inherit this value.
Attribute `sync-j`: Number of parallel jobs to use when synching.
Attribute `sync-j`: Number of parallel jobs to use when syncing.
Attribute `sync-j-max`: Maximum number of parallel jobs to use when syncing.
Attribute `sync-c`: Set to true to only sync the given Git
branch (specified in the `revision` attribute) rather than the
@@ -386,6 +390,22 @@ rather than the `name` attribute. This attribute only applies to the
local mirrors syncing, it will be ignored when syncing the projects in a
client working directory.
Attribute `sync-strategy`: Set the sync strategy used when fetching this
project. Currently the only supported value is `stateless`. When set to
`stateless`, repo will run a reflog expiration and aggressive garbage collection
at the end of the sync process. This is useful for projects that contain
large binary files and use `clone-depth="1"`, where garbage can accumulate
as binaries are added, deleted, or modified across successive syncs.
During a stateless sync, repo checks the following before cleaning up:
1. The project does not share an object directory with other projects.
2. The working tree is clean (no uncommitted changes, no untracked files).
3. There are no unpushed local commits.
4. There is no Git stash.
If any of these conditions are not met, repo falls back to a standard
sync without garbage collection.
### Element extend-project
Modify the attributes of the named project.
@@ -395,6 +415,11 @@ attributes of an existing project without completely replacing the
existing project definition. This makes the local manifest more robust
against changes to the original manifest.
The `extend-project` element can also contain `annotation`, `copyfile`, and
`linkfile` child elements. These are added to the project's definition. A
`copyfile` or `linkfile` with a `dest` that already exists in the project
will overwrite the original.
Attribute `path`: If specified, limit the change to projects checked out
at the specified path, rather than all projects with the given name.
+2 -1
View File
@@ -163,13 +163,14 @@ Example:
The `post-sync.py` file should be defined like:
```py
def main(repo_topdir=None, **kwargs):
def main(repo_topdir=None, sync_duration_seconds=None, **kwargs):
"""Main function invoked directly by repo.
We must use the name "main" as that is what repo requires.
Args:
repo_topdir: The absolute path to the top-level directory of the repo workspace.
sync_duration_seconds: The duration of the sync operation in seconds.
kwargs: Leave this here for forward-compatibility.
"""
```
+6 -3
View File
@@ -50,8 +50,11 @@ Git worktrees (see the previous section for more info).
Repo will use symlinks heavily internally.
On *NIX platforms, this isn't an issue, but Windows makes it a bit difficult.
There are some documents out there for how to do this, but usually the easiest
answer is to run your shell as an Administrator and invoke repo/git in that.
The easiest method to allow users to create symlinks is by enabling
[Windows Developer Mode](https://learn.microsoft.com/en-us/windows/advanced-settings/developer-mode).
The next easiest answer is to run your shell as an Administrator and invoke
repo/git in that.
This isn't a great solution, but Windows doesn't make this easy, so here we are.
@@ -142,7 +145,7 @@ You will have to specify this flag every time you upload.
[mintty]: https://github.com/mintty/mintty/issues/56
### repohooks always fail with an close_fds error.
### repohooks always fail with an close_fds error
When using the [reference repohooks project][repohooks] included in AOSP,
you might see errors like this when running `repo upload`:
+4 -3
View File
@@ -34,7 +34,7 @@ def fetch_file(url, verbose=False):
"""
scheme = urlparse(url).scheme
if scheme == "gs":
cmd = ["gsutil", "cat", url]
cmd = ["gcloud", "storage", "cat", url]
errors = []
try:
result = subprocess.run(
@@ -42,7 +42,7 @@ def fetch_file(url, verbose=False):
)
if result.stderr and verbose:
print(
'warning: non-fatal error running "gsutil": %s'
'warning: non-fatal error running "gcloud storage": %s'
% result.stderr,
file=sys.stderr,
)
@@ -50,7 +50,8 @@ def fetch_file(url, verbose=False):
except subprocess.CalledProcessError as e:
errors.append(e)
print(
'fatal: error running "gsutil": %s' % e.stderr, file=sys.stderr
'fatal: error running "gcloud storage": %s' % e.stderr,
file=sys.stderr,
)
raise FetchFileError(aggregate_errors=errors)
with urlopen(url) as f:
+4 -5
View File
@@ -22,7 +22,6 @@ from typing import Any, Optional
from error import GitError
from error import RepoExitError
from git_refs import HEAD
from git_trace2_event_log_base import BaseEventLog
import platform_utils
from repo_logging import RepoLogger
@@ -48,7 +47,7 @@ logger = RepoLogger(__file__)
class _GitCall:
@functools.lru_cache(maxsize=None)
@functools.lru_cache(maxsize=None) # noqa: B019
def version_tuple(self):
ret = Wrapper().ParseGitVersion()
if ret is None:
@@ -83,7 +82,7 @@ def RepoSourceVersion():
proj = os.path.dirname(os.path.abspath(__file__))
env[GIT_DIR] = os.path.join(proj, ".git")
result = subprocess.run(
[GIT, "describe", HEAD],
[GIT, "describe", "HEAD"],
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
encoding="utf-8",
@@ -96,7 +95,7 @@ def RepoSourceVersion():
ver = ver[1:]
else:
ver = "unknown"
setattr(RepoSourceVersion, "version", ver)
RepoSourceVersion.version = ver
return ver
@@ -612,7 +611,7 @@ class GitCommandError(GitError):
self.git_stderr = git_stderr
@property
@functools.lru_cache(maxsize=None)
@functools.lru_cache(maxsize=None) # noqa: B019
def suggestion(self):
"""Returns helpful next steps for the given stderr."""
if not self.git_stderr:
+13 -4
View File
@@ -42,7 +42,7 @@ SYNC_STATE_PREFIX = "repo.syncstate."
ID_RE = re.compile(r"^[0-9a-f]{40}$")
REVIEW_CACHE = dict()
REVIEW_CACHE = {}
def IsChange(rev):
@@ -111,7 +111,7 @@ class GitConfig:
return cls(configfile=os.path.join(gitdir, "config"), defaults=defaults)
def __init__(self, configfile, defaults=None, jsonFile=None):
self.file = configfile
self.file = str(configfile)
self.defaults = defaults
self._cache_dict = None
self._section_dict = None
@@ -222,6 +222,12 @@ class GitConfig:
value = "true" if value else "false"
self.SetString(name, value)
def SetInt(self, name: str, value: int) -> None:
"""Set an integer value for a key."""
if value is not None:
value = str(value)
self.SetString(name, value)
def GetString(self, name: str, all_keys: bool = False) -> Union[str, None]:
"""Get the first value for a key, or None if it is not defined.
@@ -432,7 +438,7 @@ class GitConfig:
if p.Wait() == 0:
return p.stdout
else:
raise GitError(f"git config {str(args)}: {p.stderr}")
raise GitError(f"git {' '.join(command)}: {p.stderr}")
class RepoConfig(GitConfig):
@@ -718,7 +724,10 @@ class Remote:
def Save(self):
"""Save this remote to the configuration."""
self._Set("url", self.url)
if self.pushUrl is not None:
# projectname is initialized for projects listed in the manifest, but
# not for others (e.g. the manifest project). This class is used for
# all of them.
if self.pushUrl is not None and self.projectname is not None:
self._Set("pushurl", self.pushUrl + "/" + self.projectname)
else:
self._Set("pushurl", self.pushUrl)
+88 -57
View File
@@ -14,6 +14,7 @@
import os
from git_command import GitCommand
import platform_utils
from repo_trace import Trace
@@ -86,9 +87,8 @@ class GitRefs:
self._symref = {}
self._mtime = {}
self._ReadPackedRefs()
self._ReadLoose("refs/")
self._ReadLoose1(os.path.join(self._gitdir, HEAD), HEAD)
self._ReadRefs()
self._ReadSymbolicRef(HEAD)
scan = self._symref
attempts = 0
@@ -102,64 +102,95 @@ class GitRefs:
scan = scan_next
attempts += 1
def _ReadPackedRefs(self):
path = os.path.join(self._gitdir, "packed-refs")
self._TrackMtime(HEAD)
self._TrackMtime("config")
self._TrackMtime("packed-refs")
self._TrackTreeMtimes("refs")
self._TrackTreeMtimes("reftable")
@staticmethod
def _IsNullRef(ref_id: str) -> bool:
"""Check if a ref_id is a null object ID."""
return ref_id and all(ch == "0" for ch in ref_id)
def _ReadRefs(self) -> None:
"""Read all references using git for-each-ref."""
p = GitCommand(
None,
["for-each-ref", "--format=%(objectname)%00%(refname)%00%(symref)"],
capture_stdout=True,
capture_stderr=True,
bare=True,
gitdir=self._gitdir,
)
if p.Wait() != 0:
return
for line in p.stdout.splitlines():
ref_id, name, symref = line.split("\0")
if symref:
self._symref[name] = symref
elif ref_id and not self._IsNullRef(ref_id):
self._phyref[name] = ref_id
def _ReadSymbolicRef(self, name: str) -> None:
"""Read a symbolic reference."""
p = GitCommand(
None,
["symbolic-ref", "-q", name],
capture_stdout=True,
capture_stderr=True,
bare=True,
gitdir=self._gitdir,
)
if p.Wait() == 0:
ref = p.stdout.strip()
if ref:
self._symref[name] = ref
return
p = GitCommand(
None,
["rev-parse", "--verify", "-q", name],
capture_stdout=True,
capture_stderr=True,
bare=True,
gitdir=self._gitdir,
)
if p.Wait() == 0:
ref_id = p.stdout.strip()
if ref_id:
self._phyref[name] = ref_id
def _TrackMtime(self, name: str) -> None:
"""Track the modification time of a specific gitdir path."""
path = os.path.join(self._gitdir, name)
try:
fd = open(path)
mtime = os.path.getmtime(path)
self._mtime[name] = os.path.getmtime(path)
except OSError:
return
def _TrackTreeMtimes(self, root: str) -> None:
"""Recursively track modification times for a directory tree."""
root_path = os.path.join(self._gitdir, root)
try:
for line in fd:
line = str(line)
if line[0] == "#":
continue
if line[0] == "^":
continue
line = line[:-1]
p = line.split(" ")
ref_id = p[0]
name = p[1]
self._phyref[name] = ref_id
finally:
fd.close()
self._mtime["packed-refs"] = mtime
def _ReadLoose(self, prefix):
base = os.path.join(self._gitdir, prefix)
for name in platform_utils.listdir(base):
p = os.path.join(base, name)
# We don't implement the full ref validation algorithm, just the
# simple rules that would show up in local filesystems.
# https://git-scm.com/docs/git-check-ref-format
if name.startswith(".") or name.endswith(".lock"):
pass
elif platform_utils.isdir(p):
self._mtime[prefix] = os.path.getmtime(base)
self._ReadLoose(prefix + name + "/")
else:
self._ReadLoose1(p, prefix + name)
def _ReadLoose1(self, path, name):
try:
with open(path) as fd:
mtime = os.path.getmtime(path)
ref_id = fd.readline()
except (OSError, UnicodeError):
if not platform_utils.isdir(root_path):
return
except OSError:
return
try:
ref_id = ref_id.decode()
except AttributeError:
pass
if not ref_id:
return
ref_id = ref_id[:-1]
to_scan = [root]
while to_scan:
name = to_scan.pop()
self._TrackMtime(name)
path = os.path.join(self._gitdir, name)
if not platform_utils.isdir(path):
continue
if ref_id.startswith("ref: "):
self._symref[name] = ref_id[5:]
else:
self._phyref[name] = ref_id
self._mtime[name] = mtime
for child in platform_utils.listdir(path):
child_name = os.path.join(name, child)
child_path = os.path.join(self._gitdir, child_name)
if platform_utils.isdir(child_path):
to_scan.append(child_name)
else:
self._TrackMtime(child_name)
+73 -22
View File
@@ -23,17 +23,21 @@ Examples:
"""
import functools
import glob
import hashlib
import os
import sys
import tempfile
import time
from typing import NamedTuple
import urllib.parse
from git_command import git_require
from git_command import GitCommand
from git_config import IsId
from git_config import RepoConfig
from git_refs import GitRefs
import platform_utils
_SUPERPROJECT_GIT_NAME = "superproject.git"
@@ -97,7 +101,7 @@ class Superproject:
self._manifest = manifest
self.name = name
self.remote = remote
self.revision = self._branch = revision
self.revision = revision
self._repodir = manifest.repodir
self._superproject_dir = superproject_dir
self._superproject_path = manifest.SubmanifestInfoDir(
@@ -129,6 +133,10 @@ class Superproject:
"""Set the _print_messages attribute."""
self._print_messages = value
def SetRevisionId(self, revision_id: str) -> None:
"""Set the revisionId of the superproject to sync to."""
self.revision = revision_id
@property
def commit_id(self):
"""Returns the commit ID of the superproject checkout."""
@@ -196,7 +204,8 @@ class Superproject:
def _LogMessagePrefix(self):
"""Returns the prefix string to be logged in each log message"""
return (
f"repo superproject branch: {self._branch} url: {self._remote_url}"
f"repo superproject revision: {self.revision} "
f"url: {self._remote_url}"
)
def _LogError(self, fmt, *inputs):
@@ -215,30 +224,63 @@ class Superproject:
"""
if not os.path.exists(self._superproject_path):
os.mkdir(self._superproject_path)
if not self._quiet and not os.path.exists(self._work_git):
if os.path.exists(self._work_git):
return True
if not self._quiet:
print(
"%s: Performing initial setup for superproject; this might "
"take several minutes." % self._work_git
)
cmd = ["init", "--bare", self._work_git_name]
p = GitCommand(
None,
cmd,
cwd=self._superproject_path,
capture_stdout=True,
capture_stderr=True,
tmp_gitdir_prefix = ".tmp-superproject-initgitdir-"
tmp_gitdir = tempfile.mkdtemp(
prefix=tmp_gitdir_prefix,
dir=self._superproject_path,
)
retval = p.Wait()
if retval:
self._LogWarning(
"git init call failed, command: git {}, "
"return code: {}, stderr: {}",
tmp_git_name = os.path.basename(tmp_gitdir)
try:
cmd = ["init", "--bare", tmp_git_name]
p = GitCommand(
None,
cmd,
retval,
p.stderr,
cwd=self._superproject_path,
capture_stdout=True,
capture_stderr=True,
)
return False
return True
retval = p.Wait()
if retval:
self._LogWarning(
"git init call failed, command: git {}, "
"return code: {}, stderr: {}",
cmd,
retval,
p.stderr,
)
return False
platform_utils.rename(tmp_gitdir, self._work_git)
tmp_gitdir = None
return True
finally:
# Clean up the temporary directory created during the process,
# as well as any stale ones left over from previous attempts.
if tmp_gitdir and os.path.exists(tmp_gitdir):
platform_utils.rmtree(tmp_gitdir)
age_threshold = 60 * 60 * 24 # 1 day in seconds
now = time.time()
for tmp_dir in glob.glob(
os.path.join(self._superproject_path, f"{tmp_gitdir_prefix}*")
):
try:
mtime = os.path.getmtime(tmp_dir)
if now - mtime > age_threshold:
platform_utils.rmtree(tmp_dir)
except OSError:
pass
def _Fetch(self):
"""Fetches a superproject for the manifest based on |_remote_url|.
@@ -276,8 +318,15 @@ class Superproject:
if rev_commit:
cmd.extend(["--negotiation-tip", rev_commit])
if self._branch:
cmd += [self._branch + ":" + self._branch]
if self.revision:
# If revision is a commit hash, fetch it directly to avoid
# creating a local branch of the same name.
refspec = (
self.revision
if IsId(self.revision)
else f"{self.revision}:{self.revision}"
)
cmd.append(refspec)
p = GitCommand(
None,
cmd,
@@ -312,7 +361,7 @@ class Superproject:
)
return None
data = None
branch = "HEAD" if not self._branch else self._branch
branch = "HEAD" if not self.revision else self.revision
cmd = ["ls-tree", "-z", "-r", branch]
p = GitCommand(
@@ -364,6 +413,8 @@ class Superproject:
if not self._Init():
return SyncResult(False, should_exit)
if IsId(self.revision) and self.commit_id:
return SyncResult(True, False)
if not self._Fetch():
return SyncResult(False, should_exit)
if not self._quiet:
+23 -17
View File
@@ -68,6 +68,7 @@ class BaseEventLog:
global p_init_count
p_init_count += 1
self._log = []
self.verbose = False
# Try to get session-id (sid) from environment (setup in repo launcher).
KEY = "GIT_TRACE2_PARENT_SID"
if env is None:
@@ -309,10 +310,12 @@ class BaseEventLog:
# ignore the attempt and continue to DGRAM below. Otherwise,
# issue a warning.
if err.errno != errno.EPROTOTYPE:
print(
f"repo: warning: git trace2 logging failed: {err}",
file=sys.stderr,
)
if self.verbose:
print(
"repo: warning: git trace2 logging failed:",
f"{err}",
file=sys.stderr,
)
return None
if socket_type == socket.SOCK_DGRAM or socket_type is None:
try:
@@ -322,18 +325,20 @@ class BaseEventLog:
self._WriteLog(lambda bs: sock.sendto(bs, path))
return f"af_unix:dgram:{path}"
except OSError as err:
print(
f"repo: warning: git trace2 logging failed: {err}",
file=sys.stderr,
)
if self.verbose:
print(
f"repo: warning: git trace2 logging failed: {err}",
file=sys.stderr,
)
return None
# Tried to open a socket but couldn't connect (SOCK_STREAM) or write
# (SOCK_DGRAM).
print(
"repo: warning: git trace2 logging failed: could not write to "
"socket",
file=sys.stderr,
)
if self.verbose:
print(
"repo: warning: git trace2 logging failed: could not"
"write to socket",
file=sys.stderr,
)
return None
# Path is an absolute path
@@ -348,9 +353,10 @@ class BaseEventLog:
self._WriteLog(f.write)
log_path = f.name
except FileExistsError as err:
print(
"repo: warning: git trace2 logging failed: %r" % err,
file=sys.stderr,
)
if self.verbose:
print(
"repo: warning: git trace2 logging failed: %r" % err,
file=sys.stderr,
)
return None
return log_path
+1 -1
View File
@@ -25,7 +25,7 @@ from git_refs import HEAD
# The API we've documented to hook authors. Keep in sync with repo-hooks.md.
_API_ARGS = {
"pre-upload": {"project_list", "worktree_list"},
"post-sync": {"repo_topdir"},
"post-sync": {"repo_topdir", "sync_duration_seconds"},
}
+116 -14
View File
@@ -19,6 +19,7 @@ People shouldn't run this directly; instead, they should use the `repo` wrapper
which takes care of execing this entry point.
"""
import difflib
import getpass
import json
import netrc
@@ -29,6 +30,7 @@ import signal
import sys
import textwrap
import time
from typing import Optional
import urllib.request
from repo_logging import RepoLogger
@@ -292,6 +294,102 @@ class _Repo:
result = run()
return result
def _autocorrect_command_name(
self, name: str, config: RepoConfig
) -> Optional[str]:
"""Autocorrect command name based on user's git config."""
close_commands = difflib.get_close_matches(
name, self.commands.keys(), n=5, cutoff=0.7
)
if not close_commands:
logger.error(
"repo: '%s' is not a repo command. See 'repo help'.", name
)
return None
assumed = close_commands[0]
autocorrect = config.GetString("help.autocorrect")
# If there are multiple close matches, git won't automatically run one.
# We'll always prompt instead of guessing.
if len(close_commands) > 1:
autocorrect = "prompt"
# Handle git configuration boolean values:
# 0, "false", "off", "no", "show": show suggestion (default)
# 1, "true", "on", "yes", "immediate": run suggestion immediately
# "never": don't run or show any suggested command
# "prompt": show the suggestion and prompt for confirmation
# positive number > 1: run suggestion after specified deciseconds
if autocorrect is None:
autocorrect = "0"
autocorrect = autocorrect.lower()
if autocorrect in ("0", "false", "off", "no", "show"):
autocorrect = 0
elif autocorrect in ("true", "on", "yes", "immediate"):
autocorrect = -1 # immediate
elif autocorrect == "never":
return None
elif autocorrect == "prompt":
logger.warning(
"You called a repo command named "
"'%s', which does not exist.",
name,
)
try:
resp = input(f"Run '{assumed}' instead [y/N]? ")
if resp.lower().startswith("y"):
return assumed
except (KeyboardInterrupt, EOFError):
pass
return None
else:
try:
autocorrect = int(autocorrect)
except ValueError:
autocorrect = 0
if autocorrect != 0:
if autocorrect < 0:
logger.warning(
"You called a repo command named "
"'%s', which does not exist.\n"
"Continuing assuming that "
"you meant '%s'.",
name,
assumed,
)
else:
delay = autocorrect * 0.1
logger.warning(
"You called a repo command named "
"'%s', which does not exist.\n"
"Continuing in %.1f seconds, assuming "
"that you meant '%s'.",
name,
delay,
assumed,
)
try:
time.sleep(delay)
except KeyboardInterrupt:
return None
return assumed
logger.error(
"repo: '%s' is not a repo command. See 'repo help'.", name
)
logger.warning(
"The most similar command%s\n\t%s",
"s are" if len(close_commands) > 1 else " is",
"\n\t".join(close_commands),
)
return None
def _RunLong(self, name, gopts, argv, git_trace2_event_log):
"""Execute the (longer running) requested subcommand."""
result = 0
@@ -306,20 +404,22 @@ class _Repo:
outer_client=outer_client,
)
try:
cmd = self.commands[name](
repodir=self.repodir,
client=repo_client,
manifest=repo_client.manifest,
outer_client=outer_client,
outer_manifest=outer_client.manifest,
git_event_log=git_trace2_event_log,
if name not in self.commands:
corrected_name = self._autocorrect_command_name(
name, outer_client.globalConfig
)
except KeyError:
logger.error(
"repo: '%s' is not a repo command. See 'repo help'.", name
)
return 1
if not corrected_name:
return 1
name = corrected_name
cmd = self.commands[name](
repodir=self.repodir,
client=repo_client,
manifest=repo_client.manifest,
outer_client=outer_client,
outer_manifest=outer_client.manifest,
git_event_log=git_trace2_event_log,
)
Editor.globalConfig = cmd.client.globalConfig
@@ -337,6 +437,9 @@ class _Repo:
)
return 1
cmd.CommonValidateOptions(copts, cargs)
git_trace2_event_log.verbose = copts.verbose
if gopts.pager is not False and not isinstance(cmd, InteractiveCommand):
config = cmd.client.globalConfig
if gopts.pager:
@@ -359,7 +462,6 @@ class _Repo:
Execute the subcommand.
"""
nonlocal result
cmd.CommonValidateOptions(copts, cargs)
cmd.ValidateOptions(copts, cargs)
this_manifest_only = copts.this_manifest_only
+6 -1
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "July 2022" "repo forall" "Repo Manual"
.TH REPO "1" "May 2026" "repo forall" "Repo Manual"
.SH NAME
repo \- repo forall - manual page for repo forall
.SH SYNOPSIS
@@ -120,6 +120,11 @@ git command, use REPO_LREV.
REPO_RREV is the name of the revision from the manifest, exactly as written in
the manifest.
.PP
REPO_UPSTREAM is the name of the upstream branch as specified in the manifest.
.PP
REPO_DEST_BRANCH is the name of the destination branch for code review, as
specified in the manifest.
.PP
REPO_COUNT is the total number of projects being iterated.
.PP
REPO_I is the current (1\-based) iteration count. Can be used in conjunction with
+21 -2
View File
@@ -1,10 +1,10 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "July 2022" "repo info" "Repo Manual"
.TH REPO "1" "May 2026" "repo info" "Repo Manual"
.SH NAME
repo \- repo info - manual page for repo info
.SH SYNOPSIS
.B repo
\fI\,info \/\fR[\fI\,-dl\/\fR] [\fI\,-o \/\fR[\fI\,-c\/\fR]] [\fI\,<project>\/\fR...]
\fI\,info \/\fR[\fI\,-dl\/\fR] [\fI\,-o \/\fR[\fI\,-c\/\fR]] [\fI\,--format=<format>\/\fR] [\fI\,<project>\/\fR...]
.SH DESCRIPTION
Summary
.PP
@@ -14,6 +14,10 @@ Get info on the manifest branch, current branch or unmerged branches
\fB\-h\fR, \fB\-\-help\fR
show this help message and exit
.TP
\fB\-j\fR JOBS, \fB\-\-jobs\fR=\fI\,JOBS\/\fR
number of jobs to run in parallel (default: based on
number of CPU cores)
.TP
\fB\-d\fR, \fB\-\-diff\fR
show full info and commit diff including remote
branches
@@ -21,6 +25,18 @@ branches
\fB\-o\fR, \fB\-\-overview\fR
show overview of all local commits
.TP
\fB\-\-include\-summary\fR
include manifest summary (default: true)
.TP
\fB\-\-no\-include\-summary\fR
exclude manifest summary
.TP
\fB\-\-include\-projects\fR
include project details (default: true)
.TP
\fB\-\-no\-include\-projects\fR
exclude project details
.TP
\fB\-c\fR, \fB\-\-current\-branch\fR
consider only checked out branches
.TP
@@ -29,6 +45,9 @@ consider all local branches
.TP
\fB\-l\fR, \fB\-\-local\-only\fR
disable all remote operations
.TP
\fB\-\-format\fR=\fI\,FORMAT\/\fR
output format: text, json (default: text)
.SS Logging options:
.TP
\fB\-v\fR, \fB\-\-verbose\fR
+50 -10
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "December 2025" "repo manifest" "Repo Manual"
.TH REPO "1" "April 2026" "repo manifest" "Repo Manual"
.SH NAME
repo \- repo manifest - manual page for repo manifest
.SH SYNOPSIS
@@ -131,6 +131,7 @@ include*)>
<!ATTLIST default dest\-branch CDATA #IMPLIED>
<!ATTLIST default upstream CDATA #IMPLIED>
<!ATTLIST default sync\-j CDATA #IMPLIED>
<!ATTLIST default sync\-j\-max CDATA #IMPLIED>
<!ATTLIST default sync\-c CDATA #IMPLIED>
<!ATTLIST default sync\-s CDATA #IMPLIED>
<!ATTLIST default sync\-tags CDATA #IMPLIED>
@@ -164,15 +165,32 @@ IDREF #IMPLIED>
.TP
<!ATTLIST project revision
CDATA #IMPLIED>
.TP
<!ATTLIST project dest\-branch
CDATA #IMPLIED>
.TP
<!ATTLIST project groups
CDATA #IMPLIED>
.TP
<!ATTLIST project sync\-c
CDATA #IMPLIED>
.TP
<!ATTLIST project sync\-s
CDATA #IMPLIED>
.TP
<!ATTLIST project sync\-tags
CDATA #IMPLIED>
.TP
<!ATTLIST project upstream
CDATA #IMPLIED>
.TP
<!ATTLIST project clone\-depth
CDATA #IMPLIED>
.TP
<!ATTLIST project force\-path
CDATA #IMPLIED>
.IP
<!ATTLIST project dest\-branch CDATA #IMPLIED>
<!ATTLIST project groups CDATA #IMPLIED>
<!ATTLIST project sync\-c CDATA #IMPLIED>
<!ATTLIST project sync\-s CDATA #IMPLIED>
<!ATTLIST project sync\-tags CDATA #IMPLIED>
<!ATTLIST project upstream CDATA #IMPLIED>
<!ATTLIST project clone\-depth CDATA #IMPLIED>
<!ATTLIST project force\-path CDATA #IMPLIED>
<!ATTLIST project sync\-strategy CDATA #IMPLIED>
.IP
<!ELEMENT annotation EMPTY>
<!ATTLIST annotation name CDATA #REQUIRED>
@@ -309,7 +327,9 @@ when syncing a revision locked manifest in \fB\-c\fR mode to avoid having to syn
entire ref space. Project elements not setting their own `upstream` will inherit
this value.
.PP
Attribute `sync\-j`: Number of parallel jobs to use when synching.
Attribute `sync\-j`: Number of parallel jobs to use when syncing.
.PP
Attribute `sync\-j\-max`: Maximum number of parallel jobs to use when syncing.
.PP
Attribute `sync\-c`: Set to true to only sync the given Git branch (specified in
the `revision` attribute) rather than the whole ref space. Project elements
@@ -466,6 +486,21 @@ mirror repository according to its `path` attribute (if supplied) rather than
the `name` attribute. This attribute only applies to the local mirrors syncing,
it will be ignored when syncing the projects in a client working directory.
.PP
Attribute `sync\-strategy`: Set the sync strategy used when fetching this
project. Currently the only supported value is `stateless`. When set to
`stateless`, repo will run a reflog expiration and aggressive garbage collection
at the end of the sync process. This is useful for projects that contain large
binary files and use `clone\-depth="1"`, where garbage can accumulate as binaries
are added, deleted, or modified across successive syncs.
.PP
During a stateless sync, repo checks the following before cleaning up: 1. The
project does not share an object directory with other projects. 2. The working
tree is clean (no uncommitted changes, no untracked files). 3. There are no
unpushed local commits. 4. There is no Git stash.
.PP
If any of these conditions are not met, repo falls back to a standard sync
without garbage collection.
.PP
Element extend\-project
.PP
Modify the attributes of the named project.
@@ -475,6 +510,11 @@ of an existing project without completely replacing the existing project
definition. This makes the local manifest more robust against changes to the
original manifest.
.PP
The `extend\-project` element can also contain `annotation`, `copyfile`, and
`linkfile` child elements. These are added to the project's definition. A
`copyfile` or `linkfile` with a `dest` that already exists in the project will
overwrite the original.
.PP
Attribute `path`: If specified, limit the change to projects checked out at the
specified path, rather than all projects with the given name.
.PP
+5 -1
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "August 2025" "repo smartsync" "Repo Manual"
.TH REPO "1" "May 2026" "repo smartsync" "Repo Manual"
.SH NAME
repo \- repo smartsync - manual page for repo smartsync
.SH SYNOPSIS
@@ -101,6 +101,10 @@ implies \fB\-c\fR
\fB\-\-no\-use\-superproject\fR
disable use of manifest superprojects
.TP
\fB\-\-superproject\-revision\fR=\fI\,SUPERPROJECT_REVISION\/\fR
sync to superproject revision (applies to outer
manifest)
.TP
\fB\-\-tags\fR
fetch tags
.TP
+5 -1
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "August 2025" "repo sync" "Repo Manual"
.TH REPO "1" "May 2026" "repo sync" "Repo Manual"
.SH NAME
repo \- repo sync - manual page for repo sync
.SH SYNOPSIS
@@ -101,6 +101,10 @@ implies \fB\-c\fR
\fB\-\-no\-use\-superproject\fR
disable use of manifest superprojects
.TP
\fB\-\-superproject\-revision\fR=\fI\,SUPERPROJECT_REVISION\/\fR
sync to superproject revision (applies to outer
manifest)
.TP
\fB\-\-tags\fR
fetch tags
.TP
+27 -3
View File
@@ -155,6 +155,7 @@ class _Default:
upstreamExpr = None
remote = None
sync_j = None
sync_j_max = None
sync_c = False
sync_s = False
sync_tags = True
@@ -631,6 +632,9 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
if d.sync_j is not None:
have_default = True
e.setAttribute("sync-j", "%d" % d.sync_j)
if d.sync_j_max is not None:
have_default = True
e.setAttribute("sync-j-max", "%d" % d.sync_j_max)
if d.sync_c:
have_default = True
e.setAttribute("sync-c", "true")
@@ -755,14 +759,17 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
if p.clone_depth:
e.setAttribute("clone-depth", str(p.clone_depth))
if p.sync_strategy:
e.setAttribute("sync-strategy", str(p.sync_strategy))
self._output_manifest_project_extras(p, e)
if p.subprojects:
subprojects = {subp.name for subp in p.subprojects}
output_projects(p, e, list(sorted(subprojects)))
output_projects(p, e, sorted(subprojects))
projects = {p.name for p in self._paths.values() if not p.parent}
output_projects(None, root, list(sorted(projects)))
output_projects(None, root, sorted(projects))
if self._repo_hooks_project:
root.appendChild(doc.createTextNode(""))
@@ -819,7 +826,6 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
"submanifest",
# These are children of 'project' nodes.
"annotation",
"project",
"copyfile",
"linkfile",
}
@@ -1487,6 +1493,14 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
continue
if groups:
p.groups |= groups
# Drop local groups so we don't mistakenly omit this
# project from the superproject override manifest.
p.groups = {
g
for g in p.groups
if not g.startswith(LOCAL_MANIFEST_GROUP_PREFIX)
}
if revision:
if base_revision:
if p.revisionExpr != base_revision:
@@ -1755,6 +1769,13 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
% (self.manifestFile, d.sync_j)
)
d.sync_j_max = XmlInt(node, "sync-j-max", None)
if d.sync_j_max is not None and d.sync_j_max <= 0:
raise ManifestParseError(
'%s: sync-j-max must be greater than 0, not "%s"'
% (self.manifestFile, d.sync_j_max)
)
d.sync_c = XmlBool(node, "sync-c", False)
d.sync_s = XmlBool(node, "sync-s", False)
d.sync_tags = XmlBool(node, "sync-tags", True)
@@ -1920,6 +1941,8 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
% (self.manifestFile, clone_depth)
)
sync_strategy = node.getAttribute("sync-strategy") or None
dest_branch = (
node.getAttribute("dest-branch") or self._default.destBranchExpr
)
@@ -1966,6 +1989,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
sync_s=sync_s,
sync_tags=sync_tags,
clone_depth=clone_depth,
sync_strategy=sync_strategy,
upstream=upstream,
parent=parent,
dest_branch=dest_branch,
+37
View File
@@ -222,6 +222,43 @@ def rmdir(path):
os.rmdir(_makelongpath(path))
def removedirs(path: str) -> None:
"""Remove a directory tree of symlinks and empty directories.
Walks |path| bottom-up without following symlinks. Removes symlinks
and empty directories. Stops at (and preserves) regular files and
non-empty directories. Silently succeeds if |path| does not exist.
Availability: Unix, Windows.
"""
if islink(path):
remove(path)
return
if not isdir(path):
return
for dirpath, dirnames, filenames in walk(path, topdown=False):
for name in dirnames:
entry = os.path.join(dirpath, name)
if islink(entry):
remove(entry)
else:
try:
rmdir(entry)
except OSError:
pass
for name in filenames:
entry = os.path.join(dirpath, name)
if islink(entry):
remove(entry)
try:
rmdir(path)
except OSError:
pass
def isdir(path):
"""os.path.isdir(path) wrapper with support for long paths on Windows.
+2
View File
@@ -159,6 +159,8 @@ class Progress:
inc: The number of items completed.
msg: The message to display. If None, use the last message.
"""
if self._ended:
return
self._done += inc
if msg is None:
msg = self._last_msg
+363 -175
View File
@@ -28,7 +28,7 @@ import sys
import tarfile
import tempfile
import time
from typing import List, NamedTuple
from typing import List, NamedTuple, Optional
import urllib.parse
from color import Coloring
@@ -225,7 +225,7 @@ class ReviewableBranch:
@property
def unabbrev_commits(self):
r = dict()
r = {}
for commit in self.project.bare_git.rev_list(
not_rev(self.base), R_HEADS + self.name, "--"
):
@@ -453,9 +453,13 @@ class _LinkFile(NamedTuple):
platform_utils.readlink(absDest) != relSrc
):
try:
# Remove existing file first, since it might be read-only.
# Remove existing path first, since it might be read-only.
if os.path.lexists(absDest):
platform_utils.remove(absDest)
# removedirs handles symlinks, empty dirs, and nested
# trees of stale linkfile dests without deleting user
# content. Falls through to symlink() if the path was
# fully removed, or raises OSError if not.
platform_utils.removedirs(absDest)
else:
dest_dir = os.path.dirname(absDest)
if not platform_utils.isdir(dest_dir):
@@ -553,11 +557,12 @@ class Project:
revisionExpr,
revisionId,
rebase=True,
groups=set(),
groups=None,
sync_c=False,
sync_s=False,
sync_tags=True,
clone_depth=None,
sync_strategy=None,
upstream=None,
parent=None,
use_git_worktrees=False,
@@ -605,11 +610,12 @@ class Project:
self.SetRevision(revisionExpr, revisionId=revisionId)
self.rebase = rebase
self.groups = groups
self.groups = groups if groups is not None else set()
self.sync_c = sync_c
self.sync_s = sync_s
self.sync_tags = sync_tags
self.clone_depth = clone_depth
self.sync_strategy = sync_strategy
self.upstream = upstream
self.parent = parent
# NB: Do not use this setting in __init__ to change behavior so that the
@@ -627,6 +633,7 @@ class Project:
self.linkfiles = {}
self.annotations = []
self.dest_branch = dest_branch
self.stateless_prune_needed = False
# This will be filled in if a project is later identified to be the
# project containing repo hooks.
@@ -756,6 +763,18 @@ class Project:
return True
return False
def HasStash(self) -> bool:
"""Returns True if there is a stash in the repository."""
p = GitCommand(
self,
["rev-parse", "--verify", "refs/stash"],
bare=True,
capture_stdout=True,
capture_stderr=True,
log_as_error=False,
)
return p.Wait() == 0
_userident_name = None
_userident_email = None
@@ -943,7 +962,7 @@ class Project:
out.important("prior sync failed; rebase still in progress")
out.nl()
paths = list()
paths = []
paths.extend(di.keys())
paths.extend(df.keys())
paths.extend(do)
@@ -1239,12 +1258,74 @@ class Project:
logger.error("error: Cannot extract archive %s: %s", tarpath, e)
return False
def _ShouldStatelessPrune(
self, use_superproject: Optional[bool] = None
) -> bool:
"""Determines if a stateless prune should be performed.
Stateless pruning reclaims space by running a reflog expiration and
garbage collection instead of an incremental fetch. It is only performed
if the repository is clean and has no local-only state.
"""
if not self.Exists:
return False
if self._CheckForImmutableRevision(use_superproject=use_superproject):
return False
# Query the target hash from remote to see if we are up-to-date.
target_hash = None
if IsId(self.revisionExpr):
target_hash = self.revisionExpr
else:
output = self._LsRemote(self.upstream or self.revisionExpr)
if output:
target_hash = output.splitlines()[0].split()[0]
if not target_hash:
return False
try:
local_head = self.bare_git.rev_parse("HEAD")
except GitError:
local_head = None
if target_hash == local_head:
return False
# Skip if sharing objects with other projects.
shares_objdir = self.UseAlternates or self.use_git_worktrees
if not shares_objdir:
for p in self.manifest.GetProjectsWithName(self.name):
if p != self and p.objdir == self.objdir:
shares_objdir = True
break
if shares_objdir:
return False
# Skip if HEAD contains any unpushed local commits.
try:
local_commits = self.bare_git.rev_list(
"--count", "HEAD", "--not", "--remotes", "--tags"
)
if int(local_commits[0]) > 0:
return False
except (GitError, IndexError, ValueError):
return False
if self.IsDirty(consider_untracked=True) or self.HasStash():
return False
return True
def Sync_NetworkHalf(
self,
quiet=False,
verbose=False,
output_redir=None,
is_new=None,
use_superproject=None,
current_branch_only=None,
force_sync=False,
clone_bundle=True,
@@ -1256,7 +1337,7 @@ class Project:
submodules=False,
ssh_proxy=None,
clone_filter=None,
partial_clone_exclude=set(),
partial_clone_exclude=None,
clone_filter_for_depth=None,
):
"""Perform only the network IO portion of the sync process.
@@ -1309,10 +1390,17 @@ class Project:
if clone_bundle and os.path.exists(self.objdir):
clone_bundle = False
if partial_clone_exclude is None:
partial_clone_exclude = set()
if self.name in partial_clone_exclude:
clone_bundle = True
clone_filter = None
if self.sync_strategy == "stateless" and self._ShouldStatelessPrune(
use_superproject
):
self.stateless_prune_needed = True
if is_new is None:
is_new = not self.Exists
if is_new:
@@ -1390,6 +1478,15 @@ class Project:
else:
depth = self.manifest.manifestProject.depth
# If the project has been manually unshallowed (e.g. via
# `git fetch --unshallow`), don't re-shallow it during sync.
if (
depth
and not is_new
and not os.path.exists(os.path.join(self.gitdir, "shallow"))
):
depth = None
if depth and clone_filter_for_depth:
depth = None
clone_filter = clone_filter_for_depth
@@ -1399,7 +1496,13 @@ class Project:
if not (
optimized_fetch
and IsId(self.revisionExpr)
and self._CheckForImmutableRevision()
and self._CheckForImmutableRevision(
use_superproject=use_superproject
)
and (
not depth
or os.path.exists(os.path.join(self.gitdir, "shallow"))
)
):
remote_fetched = True
try:
@@ -1409,6 +1512,7 @@ class Project:
verbose=verbose,
output_redir=output_redir,
alt_dir=alt_dir,
use_superproject=use_superproject,
current_branch_only=current_branch_only,
tags=tags,
prune=prune,
@@ -1585,6 +1689,23 @@ class Project:
def _dosubmodules():
self._SyncSubmodules(quiet=True)
def _doprune() -> None:
"""Expire reflogs and run prune-now GC for stateless sync."""
GitCommand(
self,
["reflog", "expire", "--expire=all", "--all"],
bare=True,
).Wait()
p = GitCommand(
self,
["gc", "--prune=now"],
bare=True,
capture_stdout=True,
capture_stderr=True,
)
if p.Wait() != 0:
logger.warning("warn: %s: stateless gc failed", self.name)
head = self.work_git.GetHead()
if head.startswith(R_HEADS):
branch = head[len(R_HEADS) :]
@@ -1630,6 +1751,8 @@ class Project:
fail(e)
return
self._CopyAndLinkFiles()
if self.stateless_prune_needed:
syncbuf.later2(self, _doprune, not verbose)
return
if head == revid:
@@ -1648,7 +1771,7 @@ class Project:
self, "leaving %s; does not track upstream", branch.name
)
try:
self._Checkout(revid, quiet=True)
self._Checkout(revid, force_checkout=force_checkout, quiet=True)
if submodules:
self._SyncSubmodules(quiet=True)
except GitError as e:
@@ -1776,6 +1899,9 @@ class Project:
if submodules:
syncbuf.later1(self, _dosubmodules, not verbose)
if self.stateless_prune_needed:
syncbuf.later2(self, _doprune, not verbose)
def AddCopyFile(self, src, dest, topdir):
"""Mark |src| for copying to |dest| (relative to |topdir|).
@@ -2322,7 +2448,7 @@ class Project:
result.extend(project.GetDerivedSubprojects())
continue
if url.startswith(".."):
if url.startswith(("./", "../")):
url = urllib.parse.urljoin("%s/" % self.remote.url, url)
remote = RemoteSpec(
self.remote.name,
@@ -2397,7 +2523,9 @@ class Project:
return None
def _CheckForImmutableRevision(self):
def _CheckForImmutableRevision(
self, use_superproject: Optional[bool] = None
) -> bool:
try:
# if revision (sha or tag) is not present then following function
# throws an error.
@@ -2405,7 +2533,9 @@ class Project:
upstream_rev = None
# Only check upstream when using superproject.
if self.upstream and self.manifest.manifestProject.use_superproject:
if self.upstream and git_superproject.UseSuperproject(
use_superproject, self.manifest
):
upstream_rev = self.GetRemote().ToLocal(self.upstream)
revs.append(upstream_rev)
@@ -2419,7 +2549,9 @@ class Project:
# Only verify upstream relationship for superproject scenarios
# without affecting plain usage.
if self.upstream and self.manifest.manifestProject.use_superproject:
if self.upstream and git_superproject.UseSuperproject(
use_superproject, self.manifest
):
self.bare_git.merge_base(
"--is-ancestor",
self.revisionExpr,
@@ -2450,6 +2582,7 @@ class Project:
def _RemoteFetch(
self,
name=None,
use_superproject=None,
current_branch_only=False,
initial=False,
quiet=False,
@@ -2489,7 +2622,12 @@ class Project:
tag_name = self.upstream[len(R_TAGS) :]
if is_sha1 or tag_name is not None:
if self._CheckForImmutableRevision():
if self._CheckForImmutableRevision(
use_superproject=use_superproject
) and (
not depth
or os.path.exists(os.path.join(self.gitdir, "shallow"))
):
if verbose:
print(
"Skipped fetching project %s (already have "
@@ -2516,18 +2654,20 @@ class Project:
if not remote.PreConnectFetch(ssh_proxy):
ssh_proxy = None
alt_tmp_refs = []
if initial:
if alt_dir and "objects" == os.path.basename(alt_dir):
ref_dir = os.path.dirname(alt_dir)
packed_refs = os.path.join(self.gitdir, "packed-refs")
all_refs = self.bare_ref.all
ids = set(all_refs.values())
tmp = set()
update_ref_cmds = []
for r, ref_id in GitRefs(ref_dir).all.items():
if r not in all_refs:
if r.startswith(R_TAGS) or remote.WritesTo(r):
update_ref_cmds.append(f"create {r} {ref_id}\n")
all_refs[r] = ref_id
ids.add(ref_id)
continue
@@ -2536,22 +2676,18 @@ class Project:
continue
r = "refs/_alt/%s" % ref_id
update_ref_cmds.append(f"create {r} {ref_id}\n")
all_refs[r] = ref_id
ids.add(ref_id)
tmp.add(r)
alt_tmp_refs.append(r)
tmp_packed_lines = []
old_packed_lines = []
for r in sorted(all_refs):
line = f"{all_refs[r]} {r}\n"
tmp_packed_lines.append(line)
if r not in tmp:
old_packed_lines.append(line)
tmp_packed = "".join(tmp_packed_lines)
old_packed = "".join(old_packed_lines)
_lwrite(packed_refs, tmp_packed)
if update_ref_cmds:
GitCommand(
self,
["update-ref", "--stdin"],
bare=True,
input="".join(update_ref_cmds),
).Wait()
else:
alt_dir = None
@@ -2579,7 +2715,7 @@ class Project:
effective_depth = (
self.clone_depth or self.manifest.manifestProject.depth
)
if effective_depth == 1:
if effective_depth == 1 and git_require((2, 23, 0)):
cmd.append("--no-auto-gc")
if not verbose:
@@ -2652,153 +2788,168 @@ class Project:
retry_fetches = max(retry_fetches, 2)
retry_cur_sleep = retry_sleep_initial_sec
ok = prune_tried = False
for try_n in range(retry_fetches):
verify_command = try_n == retry_fetches - 1
gitcmd = GitCommand(
self,
cmd,
bare=True,
objdir=os.path.join(self.objdir, "objects"),
ssh_proxy=ssh_proxy,
merge_output=True,
capture_stdout=quiet or bool(output_redir),
verify_command=verify_command,
)
if gitcmd.stdout and not quiet and output_redir:
output_redir.write(gitcmd.stdout)
ret = gitcmd.Wait()
if ret == 0:
ok = True
break
# Retry later due to HTTP 429 Too Many Requests.
elif (
gitcmd.stdout
and "error:" in gitcmd.stdout
and "HTTP 429" in gitcmd.stdout
):
# Fallthru to sleep+retry logic at the bottom.
pass
# TODO(b/360889369#comment24): git may gc commits incorrectly.
# Until the root cause is fixed, retry fetch with --refetch which
# will bring the repository into a good state.
elif gitcmd.stdout and (
"could not parse commit" in gitcmd.stdout
or "unable to parse commit" in gitcmd.stdout
):
cmd.insert(1, "--refetch")
print(
"could not parse commit error, retrying with refetch",
file=output_redir,
)
continue
# Try to prune remote branches once in case there are conflicts.
# For example, if the remote had refs/heads/upstream, but deleted
# that and now has refs/heads/upstream/foo.
elif (
gitcmd.stdout
and "error:" in gitcmd.stdout
and "git remote prune" in gitcmd.stdout
and not prune_tried
):
prune_tried = True
prunecmd = GitCommand(
try:
for try_n in range(retry_fetches):
verify_command = try_n == retry_fetches - 1
gitcmd = GitCommand(
self,
["remote", "prune", name],
cmd,
bare=True,
objdir=os.path.join(self.objdir, "objects"),
ssh_proxy=ssh_proxy,
merge_output=True,
capture_stdout=quiet or bool(output_redir),
verify_command=verify_command,
)
ret = prunecmd.Wait()
if ret:
if gitcmd.stdout and not quiet and output_redir:
output_redir.write(gitcmd.stdout)
ret = gitcmd.Wait()
if ret == 0:
ok = True
break
print(
"retrying fetch after pruning remote branches",
file=output_redir,
)
# Continue right away so we don't sleep as we shouldn't need to.
continue
elif (
ret == 128
and gitcmd.stdout
and "fatal: could not read Username" in gitcmd.stdout
):
# User needs to be authenticated, and Git wants to prompt for
# username and password.
print(
"git requires authentication, but repo cannot perform "
"interactive authentication. Check git credentials.",
file=output_redir,
)
break
elif (
ret == 128
and gitcmd.stdout
and "remote helper 'sso' aborted session" in gitcmd.stdout
):
# User needs to be authenticated, and Git wants to prompt for
# username and password.
print(
"git requires authentication, but repo cannot perform "
"interactive authentication.",
file=output_redir,
)
raise GitAuthError(gitcmd.stdout)
break
elif current_branch_only and is_sha1 and ret == 128:
# Exit code 128 means "couldn't find the ref you asked for"; if
# we're in sha1 mode, we just tried sync'ing from the upstream
# field; it doesn't exist, thus abort the optimization attempt
# and do a full sync.
break
elif depth and is_sha1 and ret == 1:
# In sha1 mode, when depth is enabled, syncing the revision
# from upstream may not work because some servers only allow
# fetching named refs. Fetching a specific sha1 may result
# in an error like 'server does not allow request for
# unadvertised object'. In this case, attempt a full sync
# without depth.
break
elif ret < 0:
# Git died with a signal, exit immediately.
break
# Figure out how long to sleep before the next attempt, if there is
# one.
if not verbose and gitcmd.stdout:
print(
f"\n{self.name}:\n{gitcmd.stdout}",
end="",
file=output_redir,
)
if try_n < retry_fetches - 1:
print(
"%s: sleeping %s seconds before retrying"
% (self.name, retry_cur_sleep),
file=output_redir,
)
time.sleep(retry_cur_sleep)
retry_cur_sleep = min(
retry_exp_factor * retry_cur_sleep, MAXIMUM_RETRY_SLEEP_SEC
)
retry_cur_sleep *= 1 - random.uniform(
-RETRY_JITTER_PERCENT, RETRY_JITTER_PERCENT
)
# Retry later due to HTTP 429 Too Many Requests.
elif (
gitcmd.stdout
and "error:" in gitcmd.stdout
and "HTTP 429" in gitcmd.stdout
):
# Fallthru to sleep+retry logic at the bottom.
pass
if initial:
if alt_dir:
if old_packed != "":
_lwrite(packed_refs, old_packed)
else:
platform_utils.remove(packed_refs)
self.bare_git.pack_refs("--all", "--prune")
# TODO(b/360889369#comment24): git may gc commits incorrectly.
# Until the root cause is fixed, retry fetch with --refetch
# which will bring the repository into a good state.
elif gitcmd.stdout and (
"could not parse commit" in gitcmd.stdout
or "unable to parse commit" in gitcmd.stdout
):
cmd.insert(1, "--refetch")
print(
"could not parse commit error, retrying with refetch",
file=output_redir,
)
continue
# Try to prune remote branches once in case there are conflicts.
# For example, if the remote had refs/heads/upstream, but
# deleted that and now has refs/heads/upstream/foo.
elif (
gitcmd.stdout
and "error:" in gitcmd.stdout
and "git remote prune" in gitcmd.stdout
and not prune_tried
):
prune_tried = True
prunecmd = GitCommand(
self,
["remote", "prune", name],
bare=True,
ssh_proxy=ssh_proxy,
)
ret = prunecmd.Wait()
if ret:
break
print(
"retrying fetch after pruning remote branches",
file=output_redir,
)
# Continue right away so we don't sleep as we shouldn't
# need to.
continue
elif (
ret == 128
and gitcmd.stdout
and "fatal: could not read Username" in gitcmd.stdout
):
# User needs to be authenticated, and Git wants to prompt
# for username and password.
print(
"git requires authentication, but repo cannot perform "
"interactive authentication. Check git credentials.",
file=output_redir,
)
break
elif (
ret == 128
and gitcmd.stdout
and "remote helper 'sso' aborted session" in gitcmd.stdout
):
# User needs to be authenticated, and Git wants to prompt
# for username and password.
print(
"git requires authentication, but repo cannot perform "
"interactive authentication.",
file=output_redir,
)
raise GitAuthError(gitcmd.stdout)
break
elif current_branch_only and is_sha1 and ret == 128:
# Exit code 128 means "couldn't find the ref you asked for";
# if we're in sha1 mode, we just tried sync'ing from the
# upstream field; it doesn't exist, thus abort the
# optimization attempt and do a full sync.
break
elif depth and is_sha1 and ret == 1:
# In sha1 mode, when depth is enabled, syncing the revision
# from upstream may not work because some servers only allow
# fetching named refs. Fetching a specific sha1 may result
# in an error like 'server does not allow request for
# unadvertised object'. In this case, attempt a full sync
# without depth.
break
elif ret < 0:
# Git died with a signal, exit immediately.
break
# Figure out how long to sleep before the next attempt, if
# there is one.
if not verbose and gitcmd.stdout:
print(
f"\n{self.name}:\n{gitcmd.stdout}",
end="",
file=output_redir,
)
if try_n < retry_fetches - 1:
print(
"%s: sleeping %s seconds before retrying"
% (self.name, retry_cur_sleep),
file=output_redir,
)
time.sleep(retry_cur_sleep)
retry_cur_sleep = min(
retry_exp_factor * retry_cur_sleep,
MAXIMUM_RETRY_SLEEP_SEC,
)
retry_cur_sleep *= 1 - random.uniform(
-RETRY_JITTER_PERCENT, RETRY_JITTER_PERCENT
)
finally:
if initial:
if alt_tmp_refs:
delete_cmds = "".join(
f"delete {ref}\n" for ref in alt_tmp_refs
)
GitCommand(
self,
["update-ref", "--stdin"],
bare=True,
input=delete_cmds,
log_as_error=False,
).Wait()
for ref in alt_tmp_refs:
self.bare_ref.deleted(ref)
self.bare_git.pack_refs("--all", "--prune")
if is_sha1 and current_branch_only:
# We just synced the upstream given branch; verify we
# got what we wanted, else trigger a second run of all
# refs.
if not self._CheckForImmutableRevision():
if not self._CheckForImmutableRevision(
use_superproject=use_superproject
):
# Sync the current branch only with depth set to None.
# We always pass depth=None down to avoid infinite recursion.
return self._RemoteFetch(
@@ -2806,6 +2957,7 @@ class Project:
quiet=quiet,
verbose=verbose,
output_redir=output_redir,
use_superproject=use_superproject,
current_branch_only=current_branch_only and depth,
initial=False,
alt_dir=alt_dir,
@@ -3316,6 +3468,15 @@ class Project:
remote.ResetFetch(mirror=True)
remote.Save()
# Disable auto-gc for depth=1 to prevent hangs during lazy fetches
# inside git checkout for partial clones.
effective_depth = (
self.clone_depth or self.manifest.manifestProject.depth
)
if effective_depth == 1:
self.config.SetBoolean("maintenance.auto", False)
self.config.SetInt("gc.auto", 0)
def _InitMRef(self):
"""Initialize the pseudo m/<manifest branch> ref."""
if self.manifest.branch:
@@ -3612,14 +3773,20 @@ class Project:
here. The path updates will happen independently.
"""
# Figure out where in .repo/projects/ it's pointing to.
if not os.path.islink(os.path.join(dotgit, "refs")):
gitdir = None
for name in ("refs", "reftable", "objects"):
path = os.path.join(dotgit, name)
if os.path.islink(path):
gitdir = os.path.dirname(os.path.realpath(path))
break
else:
raise GitError(
f"{dotgit}: unsupported checkout state", project=project
)
gitdir = os.path.dirname(os.path.realpath(os.path.join(dotgit, "refs")))
# Remove known symlink paths that exist in .repo/projects/.
KNOWN_LINKS = {
# go/keep-sorted start
"config",
"description",
"hooks",
@@ -3628,9 +3795,11 @@ class Project:
"objects",
"packed-refs",
"refs",
"reftable",
"rr-cache",
"shallow",
"svn",
# go/keep-sorted end
}
# Paths that we know will be in both, but are safe to clobber in
# .repo/projects/.
@@ -3655,7 +3824,16 @@ class Project:
dotgit_path = os.path.join(dotgit, name)
if name in KNOWN_LINKS:
if not platform_utils.islink(dotgit_path):
unknown_paths.append(f"{dotgit_path}: should be a symlink")
# In reftable format, refs and reftable can be directories.
if name in ("refs", "reftable") and platform_utils.isdir(
dotgit_path
):
pass
else:
unknown_paths.append(
f"{dotgit_path}: should be a symlink"
)
else:
gitdir_path = os.path.join(gitdir, name)
if name not in SAFE_TO_CLOBBER and os.path.exists(gitdir_path):
@@ -3677,7 +3855,14 @@ class Project:
if name.endswith("~") or (name[0] == "#" and name[-1] == "#"):
platform_utils.remove(dotgit_path)
elif name in KNOWN_LINKS:
platform_utils.remove(dotgit_path)
if (
name in ("refs", "reftable")
and platform_utils.isdir(dotgit_path)
and not platform_utils.islink(dotgit_path)
):
platform_utils.rmtree(dotgit_path)
else:
platform_utils.remove(dotgit_path)
else:
gitdir_path = os.path.join(gitdir, name)
platform_utils.remove(gitdir_path, missing_ok=True)
@@ -3895,11 +4080,13 @@ class Project:
def GetHead(self):
"""Return the ref that HEAD points to."""
try:
symbolic_head = self.rev_parse("--symbolic-full-name", HEAD)
if symbolic_head == HEAD:
# Detached HEAD. Return the commit SHA instead.
return self.rev_parse(HEAD)
return symbolic_head
return self.symbolic_ref("-q", HEAD, log_as_error=False)
except GitError:
pass
try:
# If symbolic-ref fails, try to treat as detached HEAD.
return self.rev_parse(HEAD)
except GitError as e:
logger.warning(
"project %s: unparseable HEAD; trying to recover.\n"
@@ -4765,6 +4952,7 @@ class ManifestProject(MetaProject):
quiet=not verbose,
verbose=verbose,
clone_bundle=clone_bundle,
use_superproject=use_superproject,
current_branch_only=current_branch_only,
tags=tags,
submodules=submodules,
+24 -1
View File
@@ -14,9 +14,32 @@
[tool.black]
line-length = 80
# NB: Keep in sync with tox.ini.
target-version = ['py36', 'py37', 'py38', 'py39', 'py310', 'py311'] #, 'py312'
# Config file for the isort python module.
# This is used to enforce import sorting standards.
#
# https://pycqa.github.io/isort/docs/configuration/options.html
[tool.isort]
# Be compatible with `black` since it also matches what we want.
profile = 'black'
line_length = 80
length_sort = false
force_single_line = true
lines_after_imports = 2
from_first = false
case_sensitive = false
force_sort_within_sections = true
order_by_type = false
# Ignore generated files.
extend_skip_glob = '*_pb2.py'
# Allow importing multiple classes on a single line from these modules.
# https://google.github.io/styleguide/pyguide#s2.2-imports
single_line_exclusions = ['abc', 'collections.abc', 'typing']
[tool.pytest.ini_options]
markers = """
skip_cq: Skip tests in the CQ. Should be rarely used!
+5 -2
View File
@@ -90,7 +90,10 @@ def check_license(path: Path, lines: list[str]) -> bool:
def check_path(opts: argparse.Namespace, path: Path) -> bool:
"""Check a single path."""
data = path.read_text(encoding="utf-8")
try:
data = path.read_text(encoding="utf-8")
except FileNotFoundError:
return True
lines = data.splitlines()
# NB: Use list comprehension and not a generator so we run all the checks.
return all(
@@ -103,7 +106,7 @@ def check_path(opts: argparse.Namespace, path: Path) -> bool:
def check_paths(opts: argparse.Namespace, paths: list[Path]) -> bool:
"""Check all the paths."""
# NB: Use list comprehension and not a generator so we check all paths.
return all([check_path(opts, x) for x in paths])
return all([check_path(opts, x) for x in paths]) # noqa: C419
def find_files(opts: argparse.Namespace) -> list[Path]:
+10 -7
View File
@@ -27,6 +27,9 @@ import sys
import util
assert sys.version_info >= (3, 9), "Release framework requires Python 3.9+"
def sign(opts):
"""Sign the launcher!"""
output = ""
@@ -82,18 +85,18 @@ Repo launcher bucket:
gs://git-repo-downloads/
You should first upload it with a specific version:
gsutil cp -a public-read {opts.launcher} gs://git-repo-downloads/repo-{version}
gsutil cp -a public-read {opts.launcher}.asc gs://git-repo-downloads/repo-{version}.asc
gcloud storage cp --predefined-acl=publicRead {opts.launcher} gs://git-repo-downloads/repo-{version}
gcloud storage cp --predefined-acl=publicRead {opts.launcher}.asc gs://git-repo-downloads/repo-{version}.asc
Then to make it the public default:
gsutil cp -a public-read gs://git-repo-downloads/repo-{version} gs://git-repo-downloads/repo
gsutil cp -a public-read gs://git-repo-downloads/repo-{version}.asc gs://git-repo-downloads/repo.asc
gcloud storage cp --predefined-acl=publicRead gs://git-repo-downloads/repo-{version} gs://git-repo-downloads/repo
gcloud storage cp --predefined-acl=publicRead gs://git-repo-downloads/repo-{version}.asc gs://git-repo-downloads/repo.asc
NB: If a rollback is necessary, the GS bucket archives old versions, and may be
accessed by specifying their unique id number.
gsutil ls -la gs://git-repo-downloads/repo gs://git-repo-downloads/repo.asc
gsutil cp -a public-read gs://git-repo-downloads/repo#<unique id> gs://git-repo-downloads/repo
gsutil cp -a public-read gs://git-repo-downloads/repo.asc#<unique id> gs://git-repo-downloads/repo.asc
gcloud storage ls --long --all-versions gs://git-repo-downloads/repo gs://git-repo-downloads/repo.asc
gcloud storage cp --predefined-acl=publicRead gs://git-repo-downloads/repo#<unique id> gs://git-repo-downloads/repo
gcloud storage cp --predefined-acl=publicRead gs://git-repo-downloads/repo.asc#<unique id> gs://git-repo-downloads/repo.asc
""" # noqa: E501
)
+3
View File
@@ -30,6 +30,9 @@ import sys
import util
assert sys.version_info >= (3, 9), "Release framework requires Python 3.9+"
# We currently sign with the old DSA key as it's been around the longest.
# We should transition to RSA by Jun 2020, and ECC by Jun 2021.
KEYID = util.KEYID_DSA
+1 -1
View File
@@ -24,7 +24,7 @@ from typing import List, Optional
import urllib.request
assert sys.version_info >= (3, 8), "Python 3.8+ required"
assert sys.version_info >= (3, 9), "Release framework requires Python 3.9+"
TOPDIR = Path(__file__).resolve().parent.parent
+10 -4
View File
@@ -30,6 +30,9 @@ import tempfile
from typing import List
assert sys.version_info >= (3, 9), "Release framework requires Python 3.9+"
THIS_FILE = Path(__file__).resolve()
TOPDIR = THIS_FILE.parent.parent
MANDIR = TOPDIR.joinpath("man")
@@ -62,7 +65,11 @@ def main(argv: List[str]) -> int:
parser = get_parser()
opts = parser.parse_args(argv)
if not shutil.which("help2man"):
help2man = ["help2man"]
cipd_help2man = TOPDIR / ".cipd_bin/bin/help2man"
if cipd_help2man.exists():
help2man = [cipd_help2man]
elif not shutil.which("help2man"):
sys.exit("Please install help2man to continue.")
# Let repo know we're generating man pages so it can avoid some dynamic
@@ -77,7 +84,7 @@ def main(argv: List[str]) -> int:
version = RepoSourceVersion()
cmdlist = [
[
"help2man",
*help2man,
"-N",
"-n",
f"repo {cmd} - manual page for repo {cmd}",
@@ -96,7 +103,7 @@ def main(argv: List[str]) -> int:
]
cmdlist.append(
[
"help2man",
*help2man,
"-N",
"-n",
"repository management tool built on top of git",
@@ -174,7 +181,6 @@ def replace_regex(data):
"""
regex = (
(r"(It was generated by help2man) [0-9.]+", r"\g<1>."),
(r"^\033\[[0-9;]*m([^\033]*)\033\[m", r"\g<1>"),
(r"^\.IP\n(.*:)\n", r".SS \g<1>\n"),
(r"^\.PP\nDescription", r".SH DETAILS"),
)
+47 -8
View File
@@ -21,7 +21,11 @@ import shlex
import shutil
import subprocess
import sys
from typing import List
# NB: While tests/* support Python >=3.6 to match requirements.json for `repo`,
# the higher level runner logic does not need to be held back.
assert sys.version_info >= (3, 9), "Test/release framework requires Python 3.9+"
ROOT_DIR = os.path.dirname(os.path.realpath(__file__))
@@ -38,7 +42,7 @@ def is_ci() -> bool:
return os.getenv("LUCI_CQ") == "yes"
def run_pytest(argv: List[str]) -> int:
def run_pytest(argv: list[str]) -> int:
"""Returns the exit code from pytest."""
if is_ci():
argv = ["-m", "not skip_cq"] + argv
@@ -51,7 +55,7 @@ def run_pytest(argv: List[str]) -> int:
).returncode
def run_pytest_py38(argv: List[str]) -> int:
def run_pytest_py38(argv: list[str]) -> int:
"""Returns the exit code from pytest under Python 3.8."""
if is_ci():
argv = ["-m", "not skip_cq"] + argv
@@ -77,6 +81,14 @@ def run_pytest_py38(argv: List[str]) -> int:
def run_black():
"""Returns the exit code from black."""
argv = ["--version"]
log_cmd("black", argv)
subprocess.run(
[sys.executable, "-m", "black"] + argv,
check=True,
cwd=ROOT_DIR,
)
# Black by default only matches .py files. We have to list standalone
# scripts manually.
extra_programs = [
@@ -96,6 +108,14 @@ def run_black():
def run_flake8():
"""Returns the exit code from flake8."""
argv = ["--version"]
log_cmd("flake8", argv)
subprocess.run(
[sys.executable, "-m", "flake8"] + argv,
check=True,
cwd=ROOT_DIR,
)
argv = [ROOT_DIR]
log_cmd("flake8", argv)
return subprocess.run(
@@ -107,6 +127,14 @@ def run_flake8():
def run_isort():
"""Returns the exit code from isort."""
argv = ["--version-number"]
log_cmd("isort", argv)
subprocess.run(
[sys.executable, "-m", "isort"] + argv,
check=True,
cwd=ROOT_DIR,
)
argv = ["--check", ROOT_DIR]
log_cmd("isort", argv)
return subprocess.run(
@@ -129,11 +157,6 @@ def run_check_metadata():
def run_update_manpages() -> int:
"""Returns the exit code from release/update-manpages."""
# Allow this to fail on CI, but not local devs.
if is_ci() and not shutil.which("help2man"):
print("update-manpages: help2man not found; skipping test")
return 0
argv = ["--check"]
log_cmd("release/update-manpages", argv)
return subprocess.run(
@@ -143,8 +166,24 @@ def run_update_manpages() -> int:
).returncode
def cipd_bootstrap() -> None:
"""Install packages via cipd if available."""
argv = ["ensure", "-root", ".cipd_bin", "-ensure-file", "cipd_manifest.txt"]
log_cmd("cipd", argv)
try:
subprocess.run(
["cipd"] + argv,
check=True,
cwd=ROOT_DIR,
)
except FileNotFoundError:
# Skip if the user doesn't have cipd from depot_tools.
return
def main(argv):
"""The main entry."""
cipd_bootstrap()
checks = (
functools.partial(run_pytest, argv),
functools.partial(run_pytest_py38, argv),
+13 -3
View File
@@ -48,10 +48,10 @@ wheel: <
version: "version:3.0.7"
>
# Required by pytest==8.3.4
# Required by pytest==8.3.4 and flake8-bugbear==24.12.12
wheel: <
name: "infra/python/wheels/attrs-py2_py3"
version: "version:21.4.0"
name: "infra/python/wheels/attrs-py3"
version: "version:24.2.0"
>
# NB: Keep in sync with constraints.txt.
@@ -119,6 +119,16 @@ wheel: <
version: "version:2.10.0"
>
wheel: <
name: "infra/python/wheels/flake8-bugbear-py3"
version: "version:24.12.12"
>
wheel: <
name: "infra/python/wheels/flake8-comprehensions-py3"
version: "version:3.16.0"
>
wheel: <
name: "infra/python/wheels/isort-py3"
version: "version:5.10.1"
+20 -5
View File
@@ -52,12 +52,12 @@ def _parse_ssh_version(ver_str=None):
@functools.lru_cache(maxsize=None)
def version():
"""return ssh version as a tuple"""
"""Return ssh version as a tuple.
If ssh is not available, a FileNotFoundError will be raised.
"""
try:
return _parse_ssh_version()
except FileNotFoundError:
print("fatal: ssh not installed", file=sys.stderr)
sys.exit(1)
except subprocess.CalledProcessError as e:
print(
"fatal: unable to detect ssh version"
@@ -102,9 +102,18 @@ class ProxyManager:
self._clients = manager.list()
# Path to directory for holding master sockets.
self._sock_path = None
# See if ssh is usable.
self._ssh_installed = False
def __enter__(self):
"""Enter a new context."""
# Check which version of ssh is available.
try:
version()
self._ssh_installed = True
except FileNotFoundError:
self._ssh_installed = False
return self
def __exit__(self, exc_type, exc_value, traceback):
@@ -140,7 +149,7 @@ class ProxyManager:
while True:
try:
procs.pop(0)
except: # noqa: E722
except IndexError:
break
def close(self):
@@ -282,6 +291,9 @@ class ProxyManager:
def preconnect(self, url):
"""If |uri| will create a ssh connection, setup the ssh master for it.""" # noqa: E501
if not self._ssh_installed:
return False
m = URI_ALL.match(url)
if m:
scheme = m.group(1)
@@ -306,6 +318,9 @@ class ProxyManager:
This has all the master sockets so clients can talk to them.
"""
if not self._ssh_installed:
return None
if self._sock_path is None:
if not create:
return None
+1
View File
@@ -94,6 +94,7 @@ It is equivalent to "git branch -D <branchname>".
def Execute(self, opt, args):
nb = args[0].split()
self.TryOverrideManifestWithSmartSync()
err = collections.defaultdict(list)
success = collections.defaultdict(list)
aggregate_errors = []
+7 -7
View File
@@ -102,6 +102,12 @@ revision to a locally executed git command, use REPO_LREV.
REPO_RREV is the name of the revision from the manifest, exactly
as written in the manifest.
REPO_UPSTREAM is the name of the upstream branch as specified in the
manifest.
REPO_DEST_BRANCH is the name of the destination branch for code review,
as specified in the manifest.
REPO_COUNT is the total number of projects being iterated.
REPO_I is the current (1-based) iteration count. Can be used in
@@ -236,13 +242,7 @@ without iterating through the remaining projects.
mirror = self.manifest.IsMirror
smart_sync_manifest_name = "smart_sync_override.xml"
smart_sync_manifest_path = os.path.join(
self.manifest.manifestProject.worktree, smart_sync_manifest_name
)
if os.path.isfile(smart_sync_manifest_path):
self.manifest.Override(smart_sync_manifest_path)
self.TryOverrideManifestWithSmartSync()
if opt.regex:
projects = self.FindProjects(args, all_manifests=all_trees)
+23 -5
View File
@@ -16,6 +16,7 @@ import os
from typing import List, Set
from command import Command
from git_command import git_require
from git_command import GitCommand
import platform_utils
from progress import Progress
@@ -204,6 +205,7 @@ class Gc(Command):
[
"rev-list",
"--objects",
"--missing=allow-promisor",
f"--remotes={project.remote.name}",
"--filter=blob:none",
"--tags",
@@ -215,7 +217,12 @@ class Gc(Command):
# Get all local objects and pack them.
local_head_objects_cmd = GitCommand(
project,
["rev-list", "--objects", "HEAD^{tree}"],
[
"rev-list",
"--objects",
"--missing=allow-promisor",
"HEAD^{tree}",
],
capture_stdout=True,
verify_command=True,
)
@@ -224,6 +231,7 @@ class Gc(Command):
[
"rev-list",
"--objects",
"--missing=allow-promisor",
"--all",
"--reflog",
"--indexed-objects",
@@ -284,11 +292,21 @@ class Gc(Command):
args, all_manifests=not opt.this_manifest_only
)
ret = self.delete_unused_projects(projects, opt)
# If the user specified projects, fetch the global list separately
# to avoid deleting untargeted projects.
if args:
all_projects = self.GetProjects(
[], all_manifests=not opt.this_manifest_only
)
else:
all_projects = projects
ret = self.delete_unused_projects(all_projects, opt)
if ret != 0:
return ret
if not opt.repack:
return
if opt.repack:
git_require((2, 17, 0), fail=True, msg="--repack")
ret = self.repack_projects(projects, opt)
return self.repack_projects(projects, opt)
return ret
+1 -1
View File
@@ -93,7 +93,7 @@ contain a line that matches both expressions:
pt = getattr(parser.values, "cmd_argv", None)
if pt is None:
pt = []
setattr(parser.values, "cmd_argv", pt)
parser.values.cmd_argv = pt
if opt_str == "-(":
pt.append("(")
+3 -5
View File
@@ -59,7 +59,7 @@ Displays detailed usage information about a command.
def PrintAllCommandsBody(self):
print("The complete list of recognized repo commands is:")
commandNames = list(sorted(all_commands))
commandNames = sorted(all_commands)
self._PrintCommands(commandNames)
print(
"See 'repo help <command>' for more information on a "
@@ -74,10 +74,8 @@ Displays detailed usage information about a command.
def PrintCommonCommandsBody(self):
print("The most commonly used repo commands are:")
commandNames = list(
sorted(
name for name, command in all_commands.items() if command.COMMON
)
commandNames = sorted(
name for name, command in all_commands.items() if command.COMMON
)
self._PrintCommands(commandNames)
+322 -124
View File
@@ -12,14 +12,41 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import enum
import functools
import io
import json
import optparse
import sys
from typing import Any, Dict, List, NamedTuple
from color import Coloring
from command import DEFAULT_LOCAL_JOBS
from command import PagedCommand
from git_refs import R_HEADS
from git_refs import R_M
class BranchInfo(NamedTuple):
"""Holds information about a branch in a project."""
relpath: str
name: str
commits: Any
date: str
is_current: bool
class OutputFormat(enum.Enum):
"""Type for the requested output format."""
# Human-readable text output.
TEXT = enum.auto()
# Machine-readable JSON output.
JSON = enum.auto()
class _Coloring(Coloring):
def __init__(self, config):
Coloring.__init__(self, config, "status")
@@ -27,10 +54,11 @@ class _Coloring(Coloring):
class Info(PagedCommand):
COMMON = True
PARALLEL_JOBS = DEFAULT_LOCAL_JOBS
helpSummary = (
"Get info on the manifest branch, current branch or unmerged branches"
)
helpUsage = "%prog [-dl] [-o [-c]] [<project>...]"
helpUsage = "%prog [-dl] [-o [-c]] [--format=<format>] [<project>...]"
def _Options(self, p):
p.add_option(
@@ -46,6 +74,30 @@ class Info(PagedCommand):
action="store_true",
help="show overview of all local commits",
)
p.add_option(
"--include-summary",
action="store_true",
default=True,
help="include manifest summary (default: true)",
)
p.add_option(
"--no-include-summary",
dest="include_summary",
action="store_false",
help="exclude manifest summary",
)
p.add_option(
"--include-projects",
action="store_true",
default=True,
help="include project details (default: true)",
)
p.add_option(
"--no-include-projects",
dest="include_projects",
action="store_false",
help="exclude project details",
)
p.add_option(
"-c",
"--current-branch",
@@ -72,8 +124,39 @@ class Info(PagedCommand):
action="store_true",
help="disable all remote operations",
)
formats = tuple(x.lower() for x in OutputFormat.__members__.keys())
p.add_option(
"--format",
default=OutputFormat.TEXT.name.lower(),
choices=formats,
help=f"output format: {', '.join(formats)} (default: %default)",
)
def WantPager(self, opt):
return OutputFormat[opt.format.upper()] == OutputFormat.TEXT
def ValidateOptions(self, opt, args):
output_format = OutputFormat[opt.format.upper()]
if output_format == OutputFormat.JSON:
if opt.all:
self.OptionParser.error("--diff is not supported with JSON")
if opt.overview:
self.OptionParser.error("--overview is not supported with JSON")
def Execute(self, opt, args):
if not opt.this_manifest_only:
self.manifest = self.manifest.outer_client
self.TryOverrideManifestWithSmartSync()
output_format = OutputFormat[opt.format.upper()]
if output_format == OutputFormat.JSON:
self._ExecuteJson(opt, args)
else:
self._ExecuteText(opt, args)
def _ExecuteText(self, opt, args) -> None:
"""Output info as human-readable text."""
self.out = _Coloring(self.client.globalConfig)
self.heading = self.out.printer("heading", attr="bold")
self.headtext = self.out.nofmt_printer("headtext", fg="yellow")
@@ -84,157 +167,272 @@ class Info(PagedCommand):
self.opt = opt
if not opt.this_manifest_only:
self.manifest = self.manifest.outer_client
manifestConfig = self.manifest.manifestProject.config
mergeBranch = manifestConfig.GetBranch("default").merge
manifestGroups = self.manifest.GetManifestGroupsStr()
if opt.include_summary:
self._printSummary()
self.heading("Manifest branch: ")
if self.manifest.default.revisionExpr:
self.headtext(self.manifest.default.revisionExpr)
self.out.nl()
self.heading("Manifest merge branch: ")
# The manifest might not have a merge branch if it isn't in a git repo,
# e.g. if `repo init --standalone-manifest` is used.
self.headtext(mergeBranch or "")
self.out.nl()
self.heading("Manifest groups: ")
self.headtext(manifestGroups)
self.out.nl()
sp = self.manifest.superproject
srev = sp.commit_id if sp and sp.commit_id else "None"
self.heading("Superproject revision: ")
self.headtext(srev)
self.out.nl()
self.printSeparator()
if not opt.overview:
if not opt.include_projects:
return
elif not opt.overview:
self._printDiffInfo(opt, args)
else:
self._printCommitOverview(opt, args)
def _getSummaryData(self) -> Dict[str, Any]:
"""Gather manifest summary data as a dict."""
manifestConfig = self.manifest.manifestProject.config
mergeBranch = manifestConfig.GetBranch("default").merge
manifestGroups = self.manifest.GetManifestGroupsStr()
sp = self.manifest.superproject
srev = sp.commit_id if sp and sp.commit_id else None
return {
"manifest_branch": self.manifest.default.revisionExpr or "",
"manifest_merge_branch": mergeBranch or "",
"manifest_groups": manifestGroups,
"superproject_revision": srev,
}
def _getProjectData(self, project) -> Dict[str, Any]:
"""Gather project data as a dict."""
data = {
"name": project.name,
"mount_path": project.worktree,
"current_revision": project.GetRevisionId(),
"manifest_revision": project.revisionExpr,
"local_branches": list(project.GetBranches()),
}
currentBranch = project.CurrentBranch
if currentBranch:
data["current_branch"] = currentBranch
return data
def _ExecuteJson(self, opt, args) -> None:
"""Output info as JSON."""
result = {}
if opt.include_summary:
result["summary"] = self._getSummaryData()
if opt.include_projects:
projs = self.GetProjects(
args, all_manifests=not opt.this_manifest_only
)
result["projects"] = [self._getProjectData(p) for p in projs]
json_settings = {
# JSON style guide says Unicode characters are fully allowed.
"ensure_ascii": False,
# We use 2 space indent to match JSON style guide.
"indent": 2,
"separators": (",", ": "),
"sort_keys": True,
}
sys.stdout.write(json.dumps(result, **json_settings) + "\n")
def _printSummary(self) -> None:
"""Print manifest summary in text format."""
data = self._getSummaryData()
self.heading("Manifest branch: ")
self.headtext(data["manifest_branch"])
self.out.nl()
self.heading("Manifest merge branch: ")
self.headtext(data["manifest_merge_branch"])
self.out.nl()
self.heading("Manifest groups: ")
self.headtext(data["manifest_groups"])
self.out.nl()
self.heading("Superproject revision: ")
self.headtext(data["superproject_revision"] or "None")
self.out.nl()
self.printSeparator()
def printSeparator(self):
self.text("----------------------------")
self.out.nl()
@classmethod
def _DiffHelper(cls, project_idx: int, opt: Any) -> str:
"""Helper for ParallelContext to get diff info for a project."""
buf = io.StringIO()
project = cls.get_parallel_context()["projects"][project_idx]
config = cls.get_parallel_context()["config"]
out = _Coloring(config)
out.redirect(buf)
heading = out.printer("heading", attr="bold")
headtext = out.nofmt_printer("headtext", fg="yellow")
redtext = out.printer("redtext", fg="red")
sha = out.printer("sha", fg="yellow")
text = out.nofmt_printer("text")
dimtext = out.printer("dimtext", attr="dim")
heading("Project: ")
headtext(project.name)
out.nl()
heading("Mount path: ")
headtext(project.worktree)
out.nl()
heading("Current revision: ")
headtext(project.GetRevisionId())
out.nl()
currentBranch = project.CurrentBranch
if currentBranch:
heading("Current branch: ")
headtext(currentBranch)
out.nl()
heading("Manifest revision: ")
headtext(project.revisionExpr)
out.nl()
localBranches = list(project.GetBranches().keys())
heading("Local Branches: ")
redtext(str(len(localBranches)))
if localBranches:
text(" [")
text(", ".join(localBranches))
text("]")
out.nl()
if opt.all:
if not opt.local:
project.Sync_NetworkHalf(quiet=True, current_branch_only=True)
branch = project.manifest.manifestProject.config.GetBranch(
"default"
).merge
if branch.startswith(R_HEADS):
branch = branch[len(R_HEADS) :]
logTarget = R_M + branch
bareTmp = project.bare_git._bare
project.bare_git._bare = False
localCommits = project.bare_git.rev_list(
"--abbrev=8",
"--abbrev-commit",
"--pretty=oneline",
logTarget + "..",
"--",
)
originCommits = project.bare_git.rev_list(
"--abbrev=8",
"--abbrev-commit",
"--pretty=oneline",
".." + logTarget,
"--",
)
project.bare_git._bare = bareTmp
heading("Local Commits: ")
redtext(str(len(localCommits)))
dimtext(" (on current branch)")
out.nl()
for c in localCommits:
split = c.split()
sha(split[0] + " ")
text(" ".join(split[1:]))
out.nl()
text("----------------------------")
out.nl()
heading("Remote Commits: ")
redtext(str(len(originCommits)))
out.nl()
for c in originCommits:
split = c.split()
sha(split[0] + " ")
text(" ".join(split[1:]))
out.nl()
text("----------------------------")
out.nl()
return buf.getvalue()
def _printDiffInfo(self, opt, args):
# We let exceptions bubble up to main as they'll be well structured.
projs = self.GetProjects(args, all_manifests=not opt.this_manifest_only)
for p in projs:
self.heading("Project: ")
self.headtext(p.name)
self.out.nl()
def _ProcessResults(_pool, _output, results):
for output in results:
if output:
print(output, end="")
self.heading("Mount path: ")
self.headtext(p.worktree)
self.out.nl()
with self.ParallelContext():
self.get_parallel_context()["projects"] = projs
self.get_parallel_context()[
"config"
] = self.manifest.manifestProject.config
self.heading("Current revision: ")
self.headtext(p.GetRevisionId())
self.out.nl()
self.ExecuteInParallel(
opt.jobs,
functools.partial(self._DiffHelper, opt=opt),
range(len(projs)),
callback=_ProcessResults,
ordered=True,
chunksize=1,
)
currentBranch = p.CurrentBranch
if currentBranch:
self.heading("Current branch: ")
self.headtext(currentBranch)
self.out.nl()
@classmethod
def _OverviewHelper(cls, project_idx: int, opt: Any) -> List[BranchInfo]:
"""Helper to get overview of uploadable branches."""
project = cls.get_parallel_context()["projects"][project_idx]
self.heading("Manifest revision: ")
self.headtext(p.revisionExpr)
self.out.nl()
branches = []
br = [project.GetUploadableBranch(x) for x in project.GetBranches()]
br = [x for x in br if x]
if opt.current_branch:
br = [x for x in br if x.name == project.CurrentBranch]
localBranches = list(p.GetBranches().keys())
self.heading("Local Branches: ")
self.redtext(str(len(localBranches)))
if localBranches:
self.text(" [")
self.text(", ".join(localBranches))
self.text("]")
self.out.nl()
if self.opt.all:
self.findRemoteLocalDiff(p)
self.printSeparator()
def findRemoteLocalDiff(self, project):
# Fetch all the latest commits.
if not self.opt.local:
project.Sync_NetworkHalf(quiet=True, current_branch_only=True)
branch = self.manifest.manifestProject.config.GetBranch("default").merge
if branch.startswith(R_HEADS):
branch = branch[len(R_HEADS) :]
logTarget = R_M + branch
bareTmp = project.bare_git._bare
project.bare_git._bare = False
localCommits = project.bare_git.rev_list(
"--abbrev=8",
"--abbrev-commit",
"--pretty=oneline",
logTarget + "..",
"--",
)
originCommits = project.bare_git.rev_list(
"--abbrev=8",
"--abbrev-commit",
"--pretty=oneline",
".." + logTarget,
"--",
)
project.bare_git._bare = bareTmp
self.heading("Local Commits: ")
self.redtext(str(len(localCommits)))
self.dimtext(" (on current branch)")
self.out.nl()
for c in localCommits:
split = c.split()
self.sha(split[0] + " ")
self.text(" ".join(split[1:]))
self.out.nl()
self.printSeparator()
self.heading("Remote Commits: ")
self.redtext(str(len(originCommits)))
self.out.nl()
for c in originCommits:
split = c.split()
self.sha(split[0] + " ")
self.text(" ".join(split[1:]))
self.out.nl()
for b in br:
branches.append(
BranchInfo(
relpath=project.RelPath(local=opt.this_manifest_only),
name=b.name,
commits=b.commits,
date=b.date,
is_current=b.name == project.CurrentBranch,
)
)
return branches
def _printCommitOverview(self, opt, args):
projs = self.GetProjects(args, all_manifests=not opt.this_manifest_only)
all_branches = []
for project in self.GetProjects(
args, all_manifests=not opt.this_manifest_only
):
br = [project.GetUploadableBranch(x) for x in project.GetBranches()]
br = [x for x in br if x]
if self.opt.current_branch:
br = [x for x in br if x.name == project.CurrentBranch]
all_branches.extend(br)
def _ProcessResults(_pool, _output, results):
for branches in results:
all_branches.extend(branches)
with self.ParallelContext():
self.get_parallel_context()["projects"] = projs
self.ExecuteInParallel(
opt.jobs,
functools.partial(self._OverviewHelper, opt=opt),
range(len(projs)),
callback=_ProcessResults,
ordered=True,
chunksize=1,
)
if not all_branches:
return
self.out.nl()
self.heading("Projects Overview")
project = None
current_relpath = None
for branch in all_branches:
if project != branch.project:
project = branch.project
if current_relpath != branch.relpath:
current_relpath = branch.relpath
self.out.nl()
self.headtext(project.RelPath(local=opt.this_manifest_only))
self.headtext(current_relpath)
self.out.nl()
commits = branch.commits
@@ -242,7 +440,7 @@ class Info(PagedCommand):
self.text(
"%s %-33s (%2d commit%s, %s)"
% (
branch.name == project.CurrentBranch and "*" or " ",
branch.is_current and "*" or " ",
branch.name,
len(commits),
len(commits) != 1 and "s" or "",
+1
View File
@@ -104,6 +104,7 @@ revision specified in the manifest.
def Execute(self, opt, args):
nb = args[0]
self.TryOverrideManifestWithSmartSync()
err_projects = []
err = []
projects = []
+204 -51
View File
@@ -64,6 +64,7 @@ from error import SyncError
from error import UpdateManifestError
import event_log
from git_command import git_require
from git_command import GitCommand
from git_config import GetUrlCookieFile
from git_refs import HEAD
from git_refs import R_HEADS
@@ -565,6 +566,11 @@ later is required to fix a server side protocol bug.
dest="use_superproject",
help="disable use of manifest superprojects",
)
p.add_option(
"--superproject-revision",
action="store",
help="sync to superproject revision (applies to outer manifest)",
)
p.add_option("--tags", action="store_true", help="fetch tags")
p.add_option(
"--no-tags",
@@ -668,6 +674,24 @@ later is required to fix a server side protocol bug.
or opt.current_branch_only
)
def _ConfigureSuperproject(
self,
opt: optparse.Values,
manifest,
revision: Optional[str] = None,
) -> bool:
"""Configure superproject with options."""
if not manifest.superproject:
return False
manifest.superproject.SetQuiet(not opt.verbose)
print_messages = git_superproject.PrintMessages(
opt.use_superproject, manifest
)
manifest.superproject.SetPrintMessages(print_messages)
if revision:
manifest.superproject.SetRevisionId(revision)
return print_messages
def _UpdateProjectsRevisionId(
self, opt, args, superproject_logging_data, manifest
):
@@ -741,11 +765,7 @@ later is required to fix a server side protocol bug.
if not use_super:
continue
m.superproject.SetQuiet(not opt.verbose)
print_messages = git_superproject.PrintMessages(
opt.use_superproject, m
)
m.superproject.SetPrintMessages(print_messages)
print_messages = self._ConfigureSuperproject(opt, m)
update_result = m.superproject.UpdateProjectsRevisionId(
per_manifest[m.path_prefix], git_event_log=self.git_event_log
)
@@ -808,6 +828,7 @@ later is required to fix a server side protocol bug.
quiet=opt.quiet,
verbose=opt.verbose,
output_redir=buf,
use_superproject=opt.use_superproject,
current_branch_only=cls._GetCurrentBranchOnly(
opt, project.manifest
),
@@ -840,6 +861,8 @@ later is required to fix a server side protocol bug.
)
except KeyboardInterrupt:
logger.error("Keyboard interrupt while processing %s", project.name)
if not cls.is_multiprocessing_active():
raise
except GitError as e:
logger.error("error.GitError: Cannot fetch %s", e)
errors.append(e)
@@ -946,7 +969,7 @@ later is required to fix a server side protocol bug.
"sync_dict"
] = multiprocessing.Manager().dict()
objdir_project_map = dict()
objdir_project_map = {}
for index, project in enumerate(projects):
objdir_project_map.setdefault(project.objdir, []).append(index)
projects_list = list(objdir_project_map.values())
@@ -1103,6 +1126,8 @@ later is required to fix a server side protocol bug.
errors.extend(syncbuf.errors)
except KeyboardInterrupt:
logger.error("Keyboard interrupt while processing %s", project.name)
if not cls.is_multiprocessing_active():
raise
except GitError as e:
logger.error(
"error.GitError: Cannot checkout %s: %s", project.name, e
@@ -1243,14 +1268,15 @@ later is required to fix a server side protocol bug.
return False
def _SetPreciousObjectsState(self, project: Project, opt):
@classmethod
def _SetPreciousObjectsState(cls, project: Project, opt):
"""Correct the preciousObjects state for the project.
Args:
project: the project to examine, and possibly correct.
opt: options given to sync.
"""
expected = self._GetPreciousObjectsState(project, opt)
expected = cls._GetPreciousObjectsState(project, opt)
actual = (
project.config.GetBoolean("extensions.preciousObjects") or False
)
@@ -1284,7 +1310,22 @@ later is required to fix a server side protocol bug.
project.config.SetString("extensions.preciousObjects", None)
project.config.SetString("gc.pruneExpire", None)
def _GCProjects(self, projects, opt, err_event):
@staticmethod
def _RunOneGC(project: Project, config: Optional[dict] = None) -> None:
"""Run auto GC on a single project."""
local_config = {}
if config:
local_config.update(config)
local_config["gc.autoDetach"] = "false"
project.bare_git.gc("--auto", config=local_config)
@classmethod
def _GCProjects(
cls,
projects: List[Project],
opt: optparse.Values,
err_event: _threading.Event,
) -> None:
"""Perform garbage collection.
If We are skipping garbage collection (opt.auto_gc not set), we still
@@ -1294,7 +1335,7 @@ later is required to fix a server side protocol bug.
if not opt.auto_gc:
# Just repair preciousObjects state, and return.
for project in projects:
self._SetPreciousObjectsState(project, opt)
cls._SetPreciousObjectsState(project, opt)
return
pm = Progress(
@@ -1304,9 +1345,8 @@ later is required to fix a server side protocol bug.
tidy_dirs = {}
for project in projects:
self._SetPreciousObjectsState(project, opt)
cls._SetPreciousObjectsState(project, opt)
project.config.SetString("gc.autoDetach", "false")
# Only call git gc once per objdir, but call pack-refs for the
# remainder.
if project.objdir not in tidy_dirs:
@@ -1327,7 +1367,7 @@ later is required to fix a server side protocol bug.
pm.update(msg=bare_git._project.name)
if run_gc:
bare_git.gc("--auto")
cls._RunOneGC(bare_git._project)
else:
bare_git.pack_refs()
pm.end()
@@ -1344,7 +1384,7 @@ later is required to fix a server side protocol bug.
try:
try:
if run_gc:
bare_git.gc("--auto", config=config)
cls._RunOneGC(bare_git._project, config=config)
else:
bare_git.pack_refs(config=config)
except GitError:
@@ -1437,11 +1477,20 @@ later is required to fix a server side protocol bug.
run 'git count-objects -v' and warn if the repository is accumulating
excessive pack files or garbage.
"""
projects = [p for p in projects if p.clone_depth]
# We only care about bloated projects if we have a git version that
# supports --no-auto-gc (2.23.0+) since what we use to disable auto-gc
# in Project._RemoteFetch.
if not git_require((2, 23, 0)):
return
projects = [
p
for p in projects
if p.clone_depth and not p.stateless_prune_needed
]
if not projects:
return
bloated_projects = []
pm = Progress(
"Checking for bloat", len(projects), delay=False, quiet=opt.quiet
)
@@ -1449,7 +1498,7 @@ later is required to fix a server side protocol bug.
def _ProcessResults(pool, pm, results):
for result in results:
if result:
bloated_projects.append(result)
self._bloated_projects.append(result)
pm.update(msg="")
with self.ParallelContext():
@@ -1464,15 +1513,6 @@ later is required to fix a server side protocol bug.
)
pm.end()
for project_name in bloated_projects:
warn_msg = (
f'warning: Project "{project_name}" is accumulating '
'unoptimized data. Please run "repo sync --auto-gc" or '
'"repo gc --repack" to clean up.'
)
self.git_event_log.ErrorEvent(warn_msg)
logger.warning(warn_msg)
def _UpdateRepoProject(self, opt, manifest, errors):
"""Fetch the repo project and check for updates."""
if opt.local_only:
@@ -1493,6 +1533,7 @@ later is required to fix a server side protocol bug.
quiet=opt.quiet,
verbose=opt.verbose,
output_redir=buf,
use_superproject=opt.use_superproject,
current_branch_only=self._GetCurrentBranchOnly(
opt, manifest
),
@@ -1610,9 +1651,10 @@ later is required to fix a server side protocol bug.
new_paths = {}
new_linkfile_paths = []
new_copyfile_paths = []
for project in self.GetProjects(
projects = self.GetProjects(
None, missing_ok=True, manifest=manifest, all_manifests=False
):
)
for project in projects:
new_linkfile_paths.extend(x.dest for x in project.linkfiles)
new_copyfile_paths.extend(x.dest for x in project.copyfiles)
@@ -1648,17 +1690,36 @@ later is required to fix a server side protocol bug.
)
for need_remove_file in need_remove_files:
# Try to remove the updated copyfile or linkfile.
# So, if the file is not exist, nothing need to do.
platform_utils.remove(
os.path.join(self.client.topdir, need_remove_file),
missing_ok=True,
need_remove_path = os.path.join(
self.client.topdir, need_remove_file
)
if os.path.isfile(need_remove_path):
platform_utils.remove(need_remove_path)
else:
platform_utils.removedirs(need_remove_path)
# Also try to remove empty parent directories.
parent = os.path.dirname(need_remove_path)
while parent != self.client.topdir:
try:
os.rmdir(parent)
except OSError:
break
parent = os.path.dirname(parent)
# Create copy-link-files.json, save dest path of "copyfile" and
# "linkfile".
with open(copylinkfile_path, "w", encoding="utf-8") as fp:
json.dump(new_paths, fp)
# Retry linkfile/copyfile creation for all projects. In
# interleaved sync mode, _CopyAndLinkFiles runs before this
# cleanup, so linkfiles whose dest was blocked by an old
# directory may have failed. _CopyAndLinkFiles is idempotent
# and skips dests that are already correct.
for project in projects:
project._CopyAndLinkFiles()
return True
def _SmartSyncSetup(self, opt, smart_sync_manifest_path, manifest):
@@ -1791,7 +1852,11 @@ later is required to fix a server side protocol bug.
mp: the manifestProject to query.
manifest_name: Manifest file to be reloaded.
"""
if not mp.standalone_manifest_url:
if opt.superproject_revision and mp.manifest == self.outer_manifest:
self._SyncToSuperprojectRev(
opt, mp.manifest, mp, manifest_name, errors
)
elif not mp.standalone_manifest_url:
self._UpdateManifestProject(opt, mp, manifest_name, errors)
if mp.manifest.submanifests:
@@ -1824,6 +1889,7 @@ later is required to fix a server side protocol bug.
quiet=not opt.verbose,
output_redir=buf,
verbose=opt.verbose,
use_superproject=opt.use_superproject,
current_branch_only=self._GetCurrentBranchOnly(
opt, mp.manifest
),
@@ -1934,15 +2000,33 @@ later is required to fix a server side protocol bug.
opt.jobs_network = min(opt.jobs_network, jobs_soft_limit)
opt.jobs_checkout = min(opt.jobs_checkout, jobs_soft_limit)
# Warn once if effective job counts seem excessively high.
sync_j_max = mp.manifest.default.sync_j_max or None
# Check for shared options.
# Prioritize --jobs, then --jobs-network, then --jobs-checkout.
job_options_to_check = (
("--jobs", opt.jobs),
("--jobs-network", opt.jobs_network),
("--jobs-checkout", opt.jobs_checkout),
job_attributes = (
("--jobs", "jobs"),
("--jobs-network", "jobs_network"),
("--jobs-checkout", "jobs_checkout"),
)
for name, value in job_options_to_check:
if value > self._JOBS_WARN_THRESHOLD:
warned = False
limit_warned = False
for name, attr in job_attributes:
value = getattr(opt, attr)
if sync_j_max and value > sync_j_max:
if not limit_warned:
logger.warning(
"warning: manifest limits %s to %d",
name,
sync_j_max,
)
limit_warned = True
setattr(opt, attr, sync_j_max)
value = sync_j_max
if not warned and value > self._JOBS_WARN_THRESHOLD:
logger.warning(
"High job count (%d > %d) specified for %s; this may "
"lead to excessive resource usage or diminishing returns.",
@@ -1950,21 +2034,23 @@ later is required to fix a server side protocol bug.
self._JOBS_WARN_THRESHOLD,
name,
)
break
warned = True
def Execute(self, opt, args):
errors = []
start_time = time.time()
try:
self._ExecuteHelper(opt, args, errors)
sync_duration_seconds = time.time() - start_time
except (RepoExitError, RepoChangedException):
raise
except (KeyboardInterrupt, Exception) as e:
raise RepoUnhandledExceptionError(e, aggregate_errors=errors)
# Run post-sync hook only after successful sync
self._RunPostSyncHook(opt)
self._RunPostSyncHook(opt, sync_duration_seconds=sync_duration_seconds)
def _RunPostSyncHook(self, opt):
def _RunPostSyncHook(self, opt, sync_duration_seconds=None):
"""Run post-sync hook if configured in manifest <repo-hooks>."""
hook = RepoHook.FromSubcmd(
hook_type="post-sync",
@@ -1972,10 +2058,60 @@ later is required to fix a server side protocol bug.
opt=opt,
abort_if_user_denies=False,
)
success = hook.Run(repo_topdir=self.client.topdir)
success = hook.Run(
repo_topdir=self.client.topdir,
sync_duration_seconds=sync_duration_seconds,
)
if not success:
print("Warning: post-sync hook reported failure.")
def _SyncToSuperprojectRev(
self,
opt: optparse.Values,
manifest,
mp: Project,
manifest_name: Optional[str],
errors: List[Exception],
) -> None:
"""Sync to a specific superproject commit."""
if not manifest.superproject:
raise SyncError("superproject not defined in manifest")
self._ConfigureSuperproject(
opt, manifest, revision=opt.superproject_revision
)
sync_result = manifest.superproject.Sync(self.git_event_log)
if not sync_result.success:
raise SyncError("failed to sync superproject")
cmd = ["show", f"{opt.superproject_revision}:.supermanifest"]
p = GitCommand(
None,
cmd,
gitdir=manifest.superproject._work_git,
bare=True,
capture_stdout=True,
capture_stderr=True,
)
if p.Wait() != 0:
raise SyncError(
f"failed to read .supermanifest from superproject: {p.stderr}"
)
try:
_, _, manifest_commit = p.stdout.strip().split()
except ValueError:
raise SyncError("could not parse .supermanifest")
mp.SetRevision(manifest_commit)
try:
self._UpdateManifestProject(opt, mp, manifest_name, errors)
except UpdateManifestError as e:
raise SyncError(
"failed to sync manifest project", aggregate_errors=[e]
)
def _ExecuteHelper(self, opt, args, errors):
manifest = self.outer_manifest
if not opt.outer_manifest:
@@ -1985,9 +2121,7 @@ later is required to fix a server side protocol bug.
manifest.Override(opt.manifest_name)
manifest_name = opt.manifest_name
smart_sync_manifest_path = os.path.join(
manifest.manifestProject.worktree, "smart_sync_override.xml"
)
smart_sync_manifest_path = self.GetSmartSyncOverridePath(manifest)
if opt.clone_bundle is None:
opt.clone_bundle = manifest.CloneBundle
@@ -2036,7 +2170,7 @@ later is required to fix a server side protocol bug.
):
mp.ConfigureCloneFilterForDepth("blob:none")
if opt.mp_update:
if opt.mp_update or opt.superproject_revision:
self._UpdateAllManifestProjects(opt, mp, manifest_name, errors)
else:
print("Skipping update of local manifest project.")
@@ -2070,6 +2204,7 @@ later is required to fix a server side protocol bug.
self._fetch_times = _FetchTimes(manifest)
self._local_sync_state = LocalSyncState(manifest)
self._bloated_projects = []
if opt.interleaved:
sync_method = self._SyncInterleaved
@@ -2086,6 +2221,9 @@ later is required to fix a server side protocol bug.
superproject_logging_data,
)
if not opt.quiet:
print("Finalizing sync state...")
# Log the previous sync analysis state from the config.
self.git_event_log.LogDataConfigEvents(
mp.config.GetSyncAnalysisStateData(), "previous_sync_state"
@@ -2104,7 +2242,17 @@ later is required to fix a server side protocol bug.
"experience, sync the entire tree."
)
self._CheckForBloatedProjects(all_projects, opt)
if existing:
self._CheckForBloatedProjects(all_projects, opt)
for project_name in sorted(self._bloated_projects):
warn_msg = (
f'warning: Project "{project_name}" is accumulating '
'unoptimized data. Please run "git repack -a -d" in the '
'project directory or "repo gc --repack" to clean up.'
)
self.git_event_log.ErrorEvent(warn_msg)
logger.warning(warn_msg)
if not opt.quiet:
print("repo sync has finished successfully.")
@@ -2331,6 +2479,7 @@ later is required to fix a server side protocol bug.
quiet=opt.quiet,
verbose=opt.verbose,
output_redir=network_output_capture,
use_superproject=opt.use_superproject,
current_branch_only=cls._GetCurrentBranchOnly(
opt, project.manifest
),
@@ -2354,6 +2503,8 @@ later is required to fix a server side protocol bug.
logger.error(
"Keyboard interrupt while processing %s", project.name
)
if not cls.is_multiprocessing_active():
raise
except GitError as e:
fetch_errors.append(e)
logger.error("error.GitError: Cannot fetch %s", e)
@@ -2404,6 +2555,8 @@ later is required to fix a server side protocol bug.
logger.error(
"Keyboard interrupt while processing %s", project.name
)
if not cls.is_multiprocessing_active():
raise
except GitError as e:
checkout_errors.append(e)
logger.error(
@@ -2628,7 +2781,7 @@ later is required to fix a server side protocol bug.
if previously_pending_relpaths == pending_relpaths:
stalled_projects_str = "\n".join(
f" - {path}"
for path in sorted(list(pending_relpaths))
for path in sorted(pending_relpaths)
)
logger.error(
"The following projects failed and could "
+6 -4
View File
@@ -27,6 +27,7 @@ from error import SilentRepoExitError
from error import UploadError
from git_command import GitCommand
from git_refs import R_HEADS
import git_superproject
from hooks import RepoHook
from project import ReviewableBranch
from repo_logging import RepoLogger
@@ -627,7 +628,7 @@ Gerrit Code Review: https://www.gerritcodereview.com/
# If using superproject, add the root repo as a push option.
manifest = branch.project.manifest
push_options = list(opt.push_options)
if manifest.manifestProject.use_superproject:
if git_superproject.UseSuperproject(None, manifest):
sp = manifest.superproject
if sp:
r_id = sp.repo_id
@@ -802,9 +803,10 @@ Gerrit Code Review: https://www.gerritcodereview.com/
project_list=pending_proj_names, worktree_list=pending_worktrees
):
if LocalSyncState(manifest).IsPartiallySynced():
logger.error(
"Partially synced tree detected. Syncing all projects "
"may resolve issues you're seeing."
logger.info(
"Tip: A partially synced tree was detected. "
"If this failure involves cross-project dependencies, "
"a full `repo sync` might help."
)
ret = 1
if ret:
+20
View File
@@ -0,0 +1,20 @@
# Repo Tests
There is a mixture of [pytest] & [Python unittest] in here. We adopted [pytest]
later on but didn't migrate existing tests (since they still work). New tests
should be written using [pytest] only.
## File layout
* `test_xxx.py`: Unittests for the `xxx` module in the main repo codebase.
Modules that are in subdirs normalize the `/` into `_`.
For example, [test_error.py](./test_error.py) is for the
[error.py](../error.py) module, and
[test_subcmds_forall.py](./test_subcmds_forall.py) is for the
[subcmds/forall.py](../subcmds/forall.py) module.
* [conftest.py](./conftest.py): Custom pytest fixtures for sharing.
* [utils_for_test.py](./utils_for_test.py): Helpers for sharing in tests.
[pytest]: https://pytest.org/
[Python unittest]: https://docs.python.org/3/library/unittest.html#unittest.TestCase
+11
View File
@@ -18,6 +18,7 @@ import pathlib
import pytest
import color
import platform_utils
import repo_trace
@@ -28,6 +29,16 @@ def disable_repo_trace(tmp_path):
repo_trace._TRACE_FILE = str(tmp_path / "TRACE_FILE_from_test")
@pytest.fixture(autouse=True)
def restore_default_coloring() -> None:
"""Force disable color for reproducible test behavior.
The few tests that cover color behavior can still reset/change the color as
needed.
"""
color.SetDefaultColoring("never")
# adapted from pytest-home 0.5.1
def _set_home(monkeypatch, path: pathlib.Path):
"""
+43 -43
View File
@@ -14,61 +14,61 @@
"""Unittests for the color.py module."""
import os
import unittest
import pytest
import utils_for_test
import color
import git_config
def fixture(*paths):
"""Return a path relative to test/fixtures."""
return os.path.join(os.path.dirname(__file__), "fixtures", *paths)
@pytest.fixture
def coloring() -> color.Coloring:
"""Create a Coloring object for testing."""
config_fixture = utils_for_test.FIXTURES_DIR / "test.gitconfig"
config = git_config.GitConfig(config_fixture)
color.SetDefaultColoring("true")
return color.Coloring(config, "status")
class ColoringTests(unittest.TestCase):
"""tests of the Coloring class."""
def test_Color_Parse_all_params_none(coloring: color.Coloring) -> None:
"""all params are None"""
val = coloring._parse(None, None, None, None)
assert val == ""
def setUp(self):
"""Create a GitConfig object using the test.gitconfig fixture."""
config_fixture = fixture("test.gitconfig")
self.config = git_config.GitConfig(config_fixture)
color.SetDefaultColoring("true")
self.color = color.Coloring(self.config, "status")
def test_Color_Parse_all_params_none(self):
"""all params are None"""
val = self.color._parse(None, None, None, None)
self.assertEqual("", val)
def test_Color_Parse_first_parameter_none(coloring: color.Coloring) -> None:
"""check fg & bg & attr"""
val = coloring._parse(None, "black", "red", "ul")
assert val == "\x1b[4;30;41m"
def test_Color_Parse_first_parameter_none(self):
"""check fg & bg & attr"""
val = self.color._parse(None, "black", "red", "ul")
self.assertEqual("\x1b[4;30;41m", val)
def test_Color_Parse_one_entry(self):
"""check fg"""
val = self.color._parse("one", None, None, None)
self.assertEqual("\033[33m", val)
def test_Color_Parse_one_entry(coloring: color.Coloring) -> None:
"""check fg"""
val = coloring._parse("one", None, None, None)
assert val == "\033[33m"
def test_Color_Parse_two_entry(self):
"""check fg & bg"""
val = self.color._parse("two", None, None, None)
self.assertEqual("\033[35;46m", val)
def test_Color_Parse_three_entry(self):
"""check fg & bg & attr"""
val = self.color._parse("three", None, None, None)
self.assertEqual("\033[4;30;41m", val)
def test_Color_Parse_two_entry(coloring: color.Coloring) -> None:
"""check fg & bg"""
val = coloring._parse("two", None, None, None)
assert val == "\033[35;46m"
def test_Color_Parse_reset_entry(self):
"""check reset entry"""
val = self.color._parse("reset", None, None, None)
self.assertEqual("\033[m", val)
def test_Color_Parse_empty_entry(self):
"""check empty entry"""
val = self.color._parse("none", "blue", "white", "dim")
self.assertEqual("\033[2;34;47m", val)
val = self.color._parse("empty", "green", "white", "bold")
self.assertEqual("\033[1;32;47m", val)
def test_Color_Parse_three_entry(coloring: color.Coloring) -> None:
"""check fg & bg & attr"""
val = coloring._parse("three", None, None, None)
assert val == "\033[4;30;41m"
def test_Color_Parse_reset_entry(coloring: color.Coloring) -> None:
"""check reset entry"""
val = coloring._parse("reset", None, None, None)
assert val == "\033[m"
def test_Color_Parse_empty_entry(coloring: color.Coloring) -> None:
"""check empty entry"""
val = coloring._parse("none", "blue", "white", "dim")
assert val == "\033[2;34;47m"
val = coloring._parse("empty", "green", "white", "bold")
assert val == "\033[1;32;47m"
+88
View File
@@ -0,0 +1,88 @@
# Copyright (C) 2026 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Unittests for the command.py module."""
from command import Command
class FakeProject:
"""Minimal project double for Command.GetProjects tests."""
def __init__(
self,
name,
relpath,
*,
gitdir=None,
derived_subprojects=None,
sync_s=False,
):
self.name = name
self.relpath = relpath
self.gitdir = gitdir or f"/git/{relpath}"
self.sync_s = sync_s
self.Exists = True
self._derived_subprojects = derived_subprojects or []
def GetDerivedSubprojects(self):
return list(self._derived_subprojects)
def MatchesGroups(self, _groups):
return True
def RelPath(self, local=True):
return self.relpath
class FakeManifest:
"""Minimal manifest double for Command.GetProjects tests."""
def __init__(self, projects):
self.projects = projects
def GetManifestGroupsStr(self):
return "default"
def test_get_projects_keeps_derived_subprojects_for_repeated_repo():
"""Derived subprojects are keyed by checkout path, not repo identity."""
submodule_a = FakeProject(
"submodule",
"src/one/submodule",
gitdir="/shared/modules/submodule.git",
)
submodule_b = FakeProject(
"submodule",
"src/two/submodule",
gitdir="/shared/modules/submodule.git",
)
project_a = FakeProject(
"project",
"src/one",
derived_subprojects=[submodule_a],
sync_s=True,
)
project_b = FakeProject(
"project",
"src/two",
derived_subprojects=[submodule_b],
sync_s=True,
)
manifest = FakeManifest([project_a, project_b])
cmd = Command(manifest=manifest)
projects = cmd.GetProjects([])
assert set(projects) == {project_a, project_b, submodule_a, submodule_b}
+18 -29
View File
@@ -14,43 +14,32 @@
"""Unittests for the editor.py module."""
import unittest
import pytest
from editor import Editor
class EditorTestCase(unittest.TestCase):
@pytest.fixture(autouse=True)
def reset_editor() -> None:
"""Take care of resetting Editor state across tests."""
def setUp(self):
self.setEditor(None)
def tearDown(self):
self.setEditor(None)
@staticmethod
def setEditor(editor):
Editor._editor = editor
Editor._editor = None
yield
Editor._editor = None
class GetEditor(EditorTestCase):
"""Check GetEditor behavior."""
def test_basic(self):
"""Basic checking of _GetEditor."""
self.setEditor(":")
self.assertEqual(":", Editor._GetEditor())
def test_basic() -> None:
"""Basic checking of _GetEditor."""
Editor._editor = ":"
assert Editor._GetEditor() == ":"
class EditString(EditorTestCase):
"""Check EditString behavior."""
def test_no_editor() -> None:
"""Check behavior when no editor is available."""
Editor._editor = ":"
assert Editor.EditString("foo") == "foo"
def test_no_editor(self):
"""Check behavior when no editor is available."""
self.setEditor(":")
self.assertEqual("foo", Editor.EditString("foo"))
def test_cat_editor(self):
"""Check behavior when editor is `cat`."""
self.setEditor("cat")
self.assertEqual("foo", Editor.EditString("foo"))
def test_cat_editor() -> None:
"""Check behavior when editor is `cat`."""
Editor._editor = "cat"
assert Editor.EditString("foo") == "foo"
+33 -32
View File
@@ -16,7 +16,9 @@
import inspect
import pickle
import unittest
from typing import Iterator, Type
import pytest
import command
import error
@@ -26,7 +28,7 @@ import project
from subcmds import all_modules
imports = all_modules + [
_IMPORTS = all_modules + [
error,
project,
git_command,
@@ -35,36 +37,35 @@ imports = all_modules + [
]
class PickleTests(unittest.TestCase):
"""Make sure all our custom exceptions can be pickled."""
def get_exceptions() -> Iterator[Type[Exception]]:
"""Return all our custom exceptions."""
for entry in _IMPORTS:
for name in dir(entry):
cls = getattr(entry, name)
if isinstance(cls, type) and issubclass(cls, Exception):
yield cls
def getExceptions(self):
"""Return all our custom exceptions."""
for entry in imports:
for name in dir(entry):
cls = getattr(entry, name)
if isinstance(cls, type) and issubclass(cls, Exception):
yield cls
def testExceptionLookup(self):
"""Make sure our introspection logic works."""
classes = list(self.getExceptions())
self.assertIn(error.HookError, classes)
# Don't assert the exact number to avoid being a change-detector test.
self.assertGreater(len(classes), 10)
def test_exception_lookup() -> None:
"""Make sure our introspection logic works."""
classes = list(get_exceptions())
assert error.HookError in classes
# Don't assert the exact number to avoid being a change-detector test.
assert len(classes) > 10
def testPickle(self):
"""Try to pickle all the exceptions."""
for cls in self.getExceptions():
args = inspect.getfullargspec(cls.__init__).args[1:]
obj = cls(*args)
p = pickle.dumps(obj)
try:
newobj = pickle.loads(p)
except Exception as e: # pylint: disable=broad-except
self.fail(
"Class %s is unable to be pickled: %s\n"
"Incomplete super().__init__(...) call?" % (cls, e)
)
self.assertIsInstance(newobj, cls)
self.assertEqual(str(obj), str(newobj))
@pytest.mark.parametrize("cls", get_exceptions())
def test_pickle(cls: Type[Exception]) -> None:
"""Try to pickle all the exceptions."""
args = inspect.getfullargspec(cls.__init__).args[1:]
obj = cls(*args)
p = pickle.dumps(obj)
try:
newobj = pickle.loads(p)
except Exception as e:
pytest.fail(
f"Class {cls} is unable to be pickled: {e}\n"
"Incomplete super().__init__(...) call?"
)
assert isinstance(newobj, cls)
assert str(obj) == str(newobj)
+208 -152
View File
@@ -14,177 +14,233 @@
"""Unittests for the git_config.py module."""
import os
import tempfile
import unittest
from pathlib import Path
from typing import Any
import pytest
import utils_for_test
import git_config
def fixture(*paths):
"""Return a path relative to test/fixtures."""
return os.path.join(os.path.dirname(__file__), "fixtures", *paths)
@pytest.fixture
def readonly_config() -> git_config.GitConfig:
"""Create a GitConfig object using the test.gitconfig fixture."""
config_fixture = utils_for_test.FIXTURES_DIR / "test.gitconfig"
return git_config.GitConfig(config_fixture)
class GitConfigReadOnlyTests(unittest.TestCase):
"""Read-only tests of the GitConfig class."""
def test_get_string_with_empty_config_values(
readonly_config: git_config.GitConfig,
) -> None:
"""Test config entries with no value.
def setUp(self):
"""Create a GitConfig object using the test.gitconfig fixture."""
config_fixture = fixture("test.gitconfig")
self.config = git_config.GitConfig(config_fixture)
[section]
empty
def test_GetString_with_empty_config_values(self):
"""
Test config entries with no value.
[section]
empty
"""
val = self.config.GetString("section.empty")
self.assertEqual(val, None)
def test_GetString_with_true_value(self):
"""
Test config entries with a string value.
[section]
nonempty = true
"""
val = self.config.GetString("section.nonempty")
self.assertEqual(val, "true")
def test_GetString_from_missing_file(self):
"""
Test missing config file
"""
config_fixture = fixture("not.present.gitconfig")
config = git_config.GitConfig(config_fixture)
val = config.GetString("empty")
self.assertEqual(val, None)
def test_GetBoolean_undefined(self):
"""Test GetBoolean on key that doesn't exist."""
self.assertIsNone(self.config.GetBoolean("section.missing"))
def test_GetBoolean_invalid(self):
"""Test GetBoolean on invalid boolean value."""
self.assertIsNone(self.config.GetBoolean("section.boolinvalid"))
def test_GetBoolean_true(self):
"""Test GetBoolean on valid true boolean."""
self.assertTrue(self.config.GetBoolean("section.booltrue"))
def test_GetBoolean_false(self):
"""Test GetBoolean on valid false boolean."""
self.assertFalse(self.config.GetBoolean("section.boolfalse"))
def test_GetInt_undefined(self):
"""Test GetInt on key that doesn't exist."""
self.assertIsNone(self.config.GetInt("section.missing"))
def test_GetInt_invalid(self):
"""Test GetInt on invalid integer value."""
self.assertIsNone(self.config.GetBoolean("section.intinvalid"))
def test_GetInt_valid(self):
"""Test GetInt on valid integers."""
TESTS = (
("inthex", 16),
("inthexk", 16384),
("int", 10),
("intk", 10240),
("intm", 10485760),
("intg", 10737418240),
)
for key, value in TESTS:
self.assertEqual(value, self.config.GetInt(f"section.{key}"))
"""
val = readonly_config.GetString("section.empty")
assert val is None
class GitConfigReadWriteTests(unittest.TestCase):
"""Read/write tests of the GitConfig class."""
def test_get_string_with_true_value(
readonly_config: git_config.GitConfig,
) -> None:
"""Test config entries with a string value.
def setUp(self):
self.tmpfile = tempfile.NamedTemporaryFile()
self.config = self.get_config()
[section]
nonempty = true
def get_config(self):
"""Get a new GitConfig instance."""
return git_config.GitConfig(self.tmpfile.name)
"""
val = readonly_config.GetString("section.nonempty")
assert val == "true"
def test_SetString(self):
"""Test SetString behavior."""
# Set a value.
self.assertIsNone(self.config.GetString("foo.bar"))
self.config.SetString("foo.bar", "val")
self.assertEqual("val", self.config.GetString("foo.bar"))
# Make sure the value was actually written out.
config = self.get_config()
self.assertEqual("val", config.GetString("foo.bar"))
def test_get_string_from_missing_file() -> None:
"""Test missing config file."""
config_fixture = utils_for_test.FIXTURES_DIR / "not.present.gitconfig"
config = git_config.GitConfig(config_fixture)
val = config.GetString("empty")
assert val is None
# Update the value.
self.config.SetString("foo.bar", "valll")
self.assertEqual("valll", self.config.GetString("foo.bar"))
config = self.get_config()
self.assertEqual("valll", config.GetString("foo.bar"))
# Delete the value.
self.config.SetString("foo.bar", None)
self.assertIsNone(self.config.GetString("foo.bar"))
config = self.get_config()
self.assertIsNone(config.GetString("foo.bar"))
def test_get_boolean_undefined(readonly_config: git_config.GitConfig) -> None:
"""Test GetBoolean on key that doesn't exist."""
assert readonly_config.GetBoolean("section.missing") is None
def test_SetBoolean(self):
"""Test SetBoolean behavior."""
# Set a true value.
self.assertIsNone(self.config.GetBoolean("foo.bar"))
for val in (True, 1):
self.config.SetBoolean("foo.bar", val)
self.assertTrue(self.config.GetBoolean("foo.bar"))
# Make sure the value was actually written out.
config = self.get_config()
self.assertTrue(config.GetBoolean("foo.bar"))
self.assertEqual("true", config.GetString("foo.bar"))
def test_get_boolean_invalid(readonly_config: git_config.GitConfig) -> None:
"""Test GetBoolean on invalid boolean value."""
assert readonly_config.GetBoolean("section.boolinvalid") is None
# Set a false value.
for val in (False, 0):
self.config.SetBoolean("foo.bar", val)
self.assertFalse(self.config.GetBoolean("foo.bar"))
# Make sure the value was actually written out.
config = self.get_config()
self.assertFalse(config.GetBoolean("foo.bar"))
self.assertEqual("false", config.GetString("foo.bar"))
def test_get_boolean_true(readonly_config: git_config.GitConfig) -> None:
"""Test GetBoolean on valid true boolean."""
assert readonly_config.GetBoolean("section.booltrue") is True
# Delete the value.
self.config.SetBoolean("foo.bar", None)
self.assertIsNone(self.config.GetBoolean("foo.bar"))
config = self.get_config()
self.assertIsNone(config.GetBoolean("foo.bar"))
def test_GetSyncAnalysisStateData(self):
"""Test config entries with a sync state analysis data."""
superproject_logging_data = {}
superproject_logging_data["test"] = False
options = type("options", (object,), {})()
options.verbose = "true"
options.mp_update = "false"
TESTS = (
("superproject.test", "false"),
("options.verbose", "true"),
("options.mpupdate", "false"),
("main.version", "1"),
)
self.config.UpdateSyncAnalysisState(options, superproject_logging_data)
sync_data = self.config.GetSyncAnalysisStateData()
for key, value in TESTS:
self.assertEqual(
sync_data[f"{git_config.SYNC_STATE_PREFIX}{key}"], value
)
self.assertTrue(
sync_data[f"{git_config.SYNC_STATE_PREFIX}main.synctime"]
)
def test_get_boolean_false(readonly_config: git_config.GitConfig) -> None:
"""Test GetBoolean on valid false boolean."""
assert readonly_config.GetBoolean("section.boolfalse") is False
def test_get_int_undefined(readonly_config: git_config.GitConfig) -> None:
"""Test GetInt on key that doesn't exist."""
assert readonly_config.GetInt("section.missing") is None
def test_get_int_invalid(readonly_config: git_config.GitConfig) -> None:
"""Test GetInt on invalid integer value."""
assert readonly_config.GetInt("section.intinvalid") is None
@pytest.mark.parametrize(
"key, expected",
(
("inthex", 16),
("inthexk", 16384),
("int", 10),
("intk", 10240),
("intm", 10485760),
("intg", 10737418240),
),
)
def test_get_int_valid(
readonly_config: git_config.GitConfig, key: str, expected: int
) -> None:
"""Test GetInt on valid integers."""
assert readonly_config.GetInt(f"section.{key}") == expected
@pytest.fixture
def rw_config_file(tmp_path: Path) -> Path:
"""Return a path to a temporary config file."""
return tmp_path / "config"
def test_set_string(rw_config_file: Path) -> None:
"""Test SetString behavior."""
config = git_config.GitConfig(str(rw_config_file))
# Set a value.
assert config.GetString("foo.bar") is None
config.SetString("foo.bar", "val")
assert config.GetString("foo.bar") == "val"
# Make sure the value was actually written out.
config2 = git_config.GitConfig(str(rw_config_file))
assert config2.GetString("foo.bar") == "val"
# Update the value.
config.SetString("foo.bar", "valll")
assert config.GetString("foo.bar") == "valll"
config3 = git_config.GitConfig(str(rw_config_file))
assert config3.GetString("foo.bar") == "valll"
# Delete the value.
config.SetString("foo.bar", None)
assert config.GetString("foo.bar") is None
config4 = git_config.GitConfig(str(rw_config_file))
assert config4.GetString("foo.bar") is None
def test_set_boolean(rw_config_file: Path) -> None:
"""Test SetBoolean behavior."""
config = git_config.GitConfig(str(rw_config_file))
# Set a true value.
assert config.GetBoolean("foo.bar") is None
for val in (True, 1):
config.SetBoolean("foo.bar", val)
assert config.GetBoolean("foo.bar") is True
# Make sure the value was actually written out.
config2 = git_config.GitConfig(str(rw_config_file))
assert config2.GetBoolean("foo.bar") is True
assert config2.GetString("foo.bar") == "true"
# Set a false value.
for val in (False, 0):
config.SetBoolean("foo.bar", val)
assert config.GetBoolean("foo.bar") is False
# Make sure the value was actually written out.
config3 = git_config.GitConfig(str(rw_config_file))
assert config3.GetBoolean("foo.bar") is False
assert config3.GetString("foo.bar") == "false"
# Delete the value.
config.SetBoolean("foo.bar", None)
assert config.GetBoolean("foo.bar") is None
config4 = git_config.GitConfig(str(rw_config_file))
assert config4.GetBoolean("foo.bar") is None
def test_set_int(rw_config_file: Path) -> None:
"""Test SetInt behavior."""
config = git_config.GitConfig(str(rw_config_file))
# Set a value.
assert config.GetInt("foo.bar") is None
config.SetInt("foo.bar", 10)
assert config.GetInt("foo.bar") == 10
# Make sure the value was actually written out.
config2 = git_config.GitConfig(str(rw_config_file))
assert config2.GetInt("foo.bar") == 10
assert config2.GetString("foo.bar") == "10"
# Update the value.
config.SetInt("foo.bar", 20)
assert config.GetInt("foo.bar") == 20
config3 = git_config.GitConfig(str(rw_config_file))
assert config3.GetInt("foo.bar") == 20
# Delete the value.
config.SetInt("foo.bar", None)
assert config.GetInt("foo.bar") is None
config4 = git_config.GitConfig(str(rw_config_file))
assert config4.GetInt("foo.bar") is None
def test_get_sync_analysis_state_data(rw_config_file: Path) -> None:
"""Test config entries with a sync state analysis data."""
config = git_config.GitConfig(str(rw_config_file))
superproject_logging_data: dict[str, Any] = {"test": False}
class Options:
"""Container for testing."""
options = Options()
options.verbose = "true"
options.mp_update = "false"
TESTS = (
("superproject.test", "false"),
("options.verbose", "true"),
("options.mpupdate", "false"),
("main.version", "1"),
)
config.UpdateSyncAnalysisState(options, superproject_logging_data)
sync_data = config.GetSyncAnalysisStateData()
for key, value in TESTS:
assert sync_data[f"{git_config.SYNC_STATE_PREFIX}{key}"] == value
assert sync_data[f"{git_config.SYNC_STATE_PREFIX}main.synctime"]
def test_remote_save_with_push_url_without_projectname(
rw_config_file: Path,
) -> None:
"""Test saving a remote pushUrl when projectname is unset."""
config = git_config.GitConfig(str(rw_config_file))
remote = config.GetRemote("origin")
remote.url = "https://example.com/repo"
remote.pushUrl = "ssh://example.com"
remote.projectname = None
remote.Save()
written_config = git_config.GitConfig(str(rw_config_file))
assert (
written_config.GetString("remote.origin.pushurl") == "ssh://example.com"
)
+99
View File
@@ -0,0 +1,99 @@
# Copyright (C) 2026 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Unittests for the git_refs.py module."""
import os
from pathlib import Path
import subprocess
import pytest
import utils_for_test
import git_refs
def _run(repo, *args):
return subprocess.run(
["git", "-C", repo, *args],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
encoding="utf-8",
check=True,
).stdout.strip()
def _init_repo(tmp_path, reftable=False):
repo = os.path.join(tmp_path, "repo")
ref_format = "reftable" if reftable else "files"
utils_for_test.init_git_tree(repo, ref_format=ref_format)
Path(os.path.join(repo, "a")).write_text("1")
_run(repo, "add", "a")
_run(repo, "commit", "-q", "-m", "init")
return repo
@pytest.mark.parametrize("reftable", [False, True])
def test_reads_refs(tmp_path, reftable):
if reftable and not utils_for_test.supports_reftable():
pytest.skip("reftable not supported")
repo = _init_repo(tmp_path, reftable=reftable)
gitdir = os.path.join(repo, ".git")
refs = git_refs.GitRefs(gitdir)
branch = _run(repo, "symbolic-ref", "--short", "HEAD")
head = _run(repo, "rev-parse", "HEAD")
assert refs.symref("HEAD") == f"refs/heads/{branch}"
assert refs.get("HEAD") == head
assert refs.get(f"refs/heads/{branch}") == head
@pytest.mark.parametrize("reftable", [False, True])
def test_updates_when_refs_change(tmp_path, reftable):
if reftable and not utils_for_test.supports_reftable():
pytest.skip("reftable not supported")
repo = _init_repo(tmp_path, reftable=reftable)
gitdir = os.path.join(repo, ".git")
refs = git_refs.GitRefs(gitdir)
head = _run(repo, "rev-parse", "HEAD")
assert refs.get("refs/heads/topic") == ""
_run(repo, "branch", "topic")
assert refs.get("refs/heads/topic") == head
_run(repo, "branch", "-D", "topic")
assert refs.get("refs/heads/topic") == ""
@pytest.mark.skipif(
not utils_for_test.supports_refs_migrate(),
reason="git refs migrate reftable support is required for this test",
)
def test_updates_when_storage_backend_toggles(tmp_path):
repo = _init_repo(tmp_path, reftable=False)
gitdir = os.path.join(repo, ".git")
refs = git_refs.GitRefs(gitdir)
head = _run(repo, "rev-parse", "HEAD")
assert refs.get("refs/heads/reftable-branch") == ""
_run(repo, "refs", "migrate", "--ref-format=reftable")
_run(repo, "branch", "reftable-branch")
assert refs.get("refs/heads/reftable-branch") == head
assert refs.get("refs/heads/files-branch") == ""
_run(repo, "refs", "migrate", "--ref-format=files")
_run(repo, "branch", "files-branch")
assert refs.get("refs/heads/files-branch") == head
+57 -1
View File
@@ -190,7 +190,7 @@ class SuperprojectTestCase(unittest.TestCase):
),
revision="refs/heads/main",
)
with mock.patch.object(self._superproject, "_branch", "junk"):
with mock.patch.object(self._superproject, "revision", "junk"):
sync_result = self._superproject.Sync(self.git_event_log)
self.assertFalse(sync_result.success)
self.assertTrue(sync_result.fatal)
@@ -461,6 +461,62 @@ class SuperprojectTestCase(unittest.TestCase):
"</manifest>",
)
def test_Init_success(self):
"""Test _Init succeeds and creates the work git dir."""
self.assertFalse(os.path.exists(self._superproject._work_git))
with mock.patch(
"git_superproject.GitCommand", autospec=True
) as mock_git_command:
instance = mock_git_command.return_value
instance.Wait.return_value = 0
self.assertTrue(self._superproject._Init())
mock_git_command.assert_called_once()
args, kwargs = mock_git_command.call_args
self.assertEqual(args[1][:2], ["init", "--bare"])
tmp_git_name = args[1][2]
self.assertTrue(
tmp_git_name.startswith(".tmp-superproject-initgitdir-")
)
self.assertTrue(os.path.exists(self._superproject._work_git))
tmp_git_path = os.path.join(
self._superproject._superproject_path, tmp_git_name
)
self.assertFalse(os.path.exists(tmp_git_path))
def test_Init_already_exists(self):
"""Test _Init returns early if the work git dir already exists."""
os.mkdir(self._superproject._superproject_path)
os.mkdir(self._superproject._work_git)
with mock.patch(
"git_superproject.GitCommand", autospec=True
) as mock_git_command:
self.assertTrue(self._superproject._Init())
mock_git_command.assert_not_called()
def test_Init_failure_cleans_up(self):
"""Test _Init cleans up the temporary directory if 'git init' fails."""
with mock.patch(
"git_superproject.GitCommand", autospec=True
) as mock_git_command:
instance = mock_git_command.return_value
instance.Wait.return_value = 1
instance.stderr = "mock git init failure"
self.assertFalse(self._superproject._Init())
mock_git_command.assert_called_once()
args, kwargs = mock_git_command.call_args
tmp_git_name = args[1][2]
tmp_git_path = os.path.join(
self._superproject._superproject_path, tmp_git_name
)
self.assertFalse(os.path.exists(tmp_git_path))
self.assertFalse(os.path.exists(self._superproject._work_git))
def test_Fetch(self):
manifest = self.getXmlManifest(
"""
+416 -334
View File
@@ -14,19 +14,28 @@
"""Unittests for the git_trace2_event_log.py module."""
import contextlib
import io
import json
import os
import re
import socket
import tempfile
import threading
import unittest
from typing import Any, Dict, List, Optional
from unittest import mock
import pytest
import git_trace2_event_log
import platform_utils
def serverLoggingThread(socket_path, server_ready, received_traces):
def server_logging_thread(
socket_path: str,
server_ready: threading.Condition,
received_traces: List[str],
) -> None:
"""Helper function to receive logs over a Unix domain socket.
Appends received messages on the provided socket and appends to
@@ -55,352 +64,425 @@ def serverLoggingThread(socket_path, server_ready, received_traces):
received_traces.extend(data.decode("utf-8").splitlines())
class EventLogTestCase(unittest.TestCase):
"""TestCase for the EventLog module."""
PARENT_SID_KEY = "GIT_TRACE2_PARENT_SID"
PARENT_SID_VALUE = "parent_sid"
SELF_SID_REGEX = r"repo-\d+T\d+Z-.*"
FULL_SID_REGEX = rf"^{PARENT_SID_VALUE}/{SELF_SID_REGEX}"
PARENT_SID_KEY = "GIT_TRACE2_PARENT_SID"
PARENT_SID_VALUE = "parent_sid"
SELF_SID_REGEX = r"repo-\d+T\d+Z-.*"
FULL_SID_REGEX = rf"^{PARENT_SID_VALUE}/{SELF_SID_REGEX}"
def setUp(self):
"""Load the event_log module every time."""
self._event_log_module = None
# By default we initialize with the expected case where
# repo launches us (so GIT_TRACE2_PARENT_SID is set).
env = {
self.PARENT_SID_KEY: self.PARENT_SID_VALUE,
}
self._event_log_module = git_trace2_event_log.EventLog(env=env)
self._log_data = None
@pytest.fixture
def event_log() -> git_trace2_event_log.EventLog:
"""Fixture for the EventLog module."""
# By default we initialize with the expected case where
# repo launches us (so GIT_TRACE2_PARENT_SID is set).
env = {PARENT_SID_KEY: PARENT_SID_VALUE}
return git_trace2_event_log.EventLog(env=env)
def verifyCommonKeys(
self, log_entry, expected_event_name=None, full_sid=True
def verify_common_keys(
log_entry: Dict[str, Any],
expected_event_name: Optional[str] = None,
full_sid: bool = True,
) -> None:
"""Helper function to verify common event log keys."""
assert "event" in log_entry
assert "sid" in log_entry
assert "thread" in log_entry
assert "time" in log_entry
# Do basic data format validation.
if expected_event_name:
assert expected_event_name == log_entry["event"]
if full_sid:
assert re.match(FULL_SID_REGEX, log_entry["sid"])
else:
assert re.match(SELF_SID_REGEX, log_entry["sid"])
assert re.match(r"^\d+-\d+-\d+T\d+:\d+:\d+\.\d+\+00:00$", log_entry["time"])
def read_log(log_path: str) -> List[Dict[str, Any]]:
"""Helper function to read log data into a list."""
log_data = []
with open(log_path, mode="rb") as f:
for line in f:
log_data.append(json.loads(line))
return log_data
def remove_prefix(s: str, prefix: str) -> str:
"""Return a copy string after removing |prefix| from |s|, if present or
the original string."""
if s.startswith(prefix):
return s[len(prefix) :]
else:
return s
def test_initial_state_with_parent_sid(
event_log: git_trace2_event_log.EventLog,
) -> None:
"""Test initial state when 'GIT_TRACE2_PARENT_SID' is set by parent."""
assert re.match(FULL_SID_REGEX, event_log.full_sid)
def test_initial_state_no_parent_sid() -> None:
"""Test initial state when 'GIT_TRACE2_PARENT_SID' is not set."""
# Setup an empty environment dict (no parent sid).
event_log = git_trace2_event_log.EventLog(env={})
assert re.match(SELF_SID_REGEX, event_log.full_sid)
def test_version_event(event_log: git_trace2_event_log.EventLog) -> None:
"""Test 'version' event data is valid.
Verify that the 'version' event is written even when no other
events are added.
Expected event log:
<version event>
"""
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = event_log.Write(path=tempdir)
log_data = read_log(log_path)
# A log with no added events should only have the version entry.
assert len(log_data) == 1
version_event = log_data[0]
verify_common_keys(version_event, expected_event_name="version")
# Check for 'version' event specific fields.
assert "evt" in version_event
assert "exe" in version_event
# Verify "evt" version field is a string.
assert isinstance(version_event["evt"], str)
def test_start_event(event_log: git_trace2_event_log.EventLog) -> None:
"""Test and validate 'start' event data is valid.
Expected event log:
<version event>
<start event>
"""
event_log.StartEvent([])
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = event_log.Write(path=tempdir)
log_data = read_log(log_path)
assert len(log_data) == 2
start_event = log_data[1]
verify_common_keys(log_data[0], expected_event_name="version")
verify_common_keys(start_event, expected_event_name="start")
# Check for 'start' event specific fields.
assert "argv" in start_event
assert isinstance(start_event["argv"], list)
def test_exit_event_result_none(
event_log: git_trace2_event_log.EventLog,
) -> None:
"""Test 'exit' event data is valid when result is None.
We expect None result to be converted to 0 in the exit event data.
Expected event log:
<version event>
<exit event>
"""
event_log.ExitEvent(None)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = event_log.Write(path=tempdir)
log_data = read_log(log_path)
assert len(log_data) == 2
exit_event = log_data[1]
verify_common_keys(log_data[0], expected_event_name="version")
verify_common_keys(exit_event, expected_event_name="exit")
# Check for 'exit' event specific fields.
assert "code" in exit_event
# 'None' result should convert to 0 (successful) return code.
assert exit_event["code"] == 0
def test_exit_event_result_integer(
event_log: git_trace2_event_log.EventLog,
) -> None:
"""Test 'exit' event data is valid when result is an integer.
Expected event log:
<version event>
<exit event>
"""
event_log.ExitEvent(2)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = event_log.Write(path=tempdir)
log_data = read_log(log_path)
assert len(log_data) == 2
exit_event = log_data[1]
verify_common_keys(log_data[0], expected_event_name="version")
verify_common_keys(exit_event, expected_event_name="exit")
# Check for 'exit' event specific fields.
assert "code" in exit_event
assert exit_event["code"] == 2
def test_command_event(event_log: git_trace2_event_log.EventLog) -> None:
"""Test and validate 'command' event data is valid.
Expected event log:
<version event>
<command event>
"""
event_log.CommandEvent(name="repo", subcommands=["init", "this"])
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = event_log.Write(path=tempdir)
log_data = read_log(log_path)
assert len(log_data) == 2
command_event = log_data[1]
verify_common_keys(log_data[0], expected_event_name="version")
verify_common_keys(command_event, expected_event_name="cmd_name")
# Check for 'command' event specific fields.
assert "name" in command_event
assert command_event["name"] == "repo-init-this"
def test_def_params_event_repo_config(
event_log: git_trace2_event_log.EventLog,
) -> None:
"""Test 'def_params' event data outputs only repo config keys.
Expected event log:
<version event>
<def_param event>
<def_param event>
"""
config = {
"git.foo": "bar",
"repo.partialclone": "true",
"repo.partialclonefilter": "blob:none",
}
event_log.DefParamRepoEvents(config)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = event_log.Write(path=tempdir)
log_data = read_log(log_path)
assert len(log_data) == 3
def_param_events = log_data[1:]
verify_common_keys(log_data[0], expected_event_name="version")
for event in def_param_events:
verify_common_keys(event, expected_event_name="def_param")
# Check for 'def_param' event specific fields.
assert "param" in event
assert "value" in event
assert event["param"].startswith("repo.")
def test_def_params_event_no_repo_config(
event_log: git_trace2_event_log.EventLog,
) -> None:
"""Test 'def_params' event data won't output non-repo config keys.
Expected event log:
<version event>
"""
config = {
"git.foo": "bar",
"git.core.foo2": "baz",
}
event_log.DefParamRepoEvents(config)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = event_log.Write(path=tempdir)
log_data = read_log(log_path)
assert len(log_data) == 1
verify_common_keys(log_data[0], expected_event_name="version")
def test_data_event_config(event_log: git_trace2_event_log.EventLog) -> None:
"""Test 'data' event data outputs all config keys.
Expected event log:
<version event>
<data event>
<data event>
"""
config = {
"git.foo": "bar",
"repo.partialclone": "false",
"repo.syncstate.superproject.hassuperprojecttag": "true",
"repo.syncstate.superproject.sys.argv": ["--", "sync", "protobuf"],
}
prefix_value = "prefix"
event_log.LogDataConfigEvents(config, prefix_value)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = event_log.Write(path=tempdir)
log_data = read_log(log_path)
assert len(log_data) == 5
data_events = log_data[1:]
verify_common_keys(log_data[0], expected_event_name="version")
for event in data_events:
verify_common_keys(event)
# Check for 'data' event specific fields.
assert "key" in event
assert "value" in event
key = event["key"]
key = remove_prefix(key, f"{prefix_value}/")
value = event["value"]
assert event_log.GetDataEventName(value) == event["event"]
assert key in config
assert value == config[key]
def test_error_event(event_log: git_trace2_event_log.EventLog) -> None:
"""Test and validate 'error' event data is valid.
Expected event log:
<version event>
<error event>
"""
msg = "invalid option: --cahced"
fmt = "invalid option: %s"
event_log.ErrorEvent(msg, fmt)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = event_log.Write(path=tempdir)
log_data = read_log(log_path)
assert len(log_data) == 2
error_event = log_data[1]
verify_common_keys(log_data[0], expected_event_name="version")
verify_common_keys(error_event, expected_event_name="error")
# Check for 'error' event specific fields.
assert "msg" in error_event
assert "fmt" in error_event
assert error_event["msg"] == f"RepoErrorEvent:{msg}"
assert error_event["fmt"] == f"RepoErrorEvent:{fmt}"
def test_write_with_filename(event_log: git_trace2_event_log.EventLog) -> None:
"""Test Write() with a path to a file exits with None."""
assert event_log.Write(path="path/to/file") is None
def test_write_with_git_config(
tmp_path,
event_log: git_trace2_event_log.EventLog,
) -> None:
"""Test Write() uses the git config path when 'git config' call succeeds."""
with mock.patch.object(
event_log,
"_GetEventTargetPath",
return_value=str(tmp_path),
):
"""Helper function to verify common event log keys."""
self.assertIn("event", log_entry)
self.assertIn("sid", log_entry)
self.assertIn("thread", log_entry)
self.assertIn("time", log_entry)
assert os.path.dirname(event_log.Write()) == str(tmp_path)
# Do basic data format validation.
if expected_event_name:
self.assertEqual(expected_event_name, log_entry["event"])
if full_sid:
self.assertRegex(log_entry["sid"], self.FULL_SID_REGEX)
else:
self.assertRegex(log_entry["sid"], self.SELF_SID_REGEX)
self.assertRegex(
log_entry["time"], r"^\d+-\d+-\d+T\d+:\d+:\d+\.\d+\+00:00$"
def test_write_no_git_config(event_log: git_trace2_event_log.EventLog) -> None:
"""Test Write() with no git config variable present exits with None."""
with mock.patch.object(event_log, "_GetEventTargetPath", return_value=None):
assert event_log.Write() is None
def test_write_non_string(event_log: git_trace2_event_log.EventLog) -> None:
"""Test Write() with non-string type for |path| throws TypeError."""
with pytest.raises(TypeError):
event_log.Write(path=1234)
@pytest.mark.skipif(
not hasattr(socket, "AF_UNIX"), reason="Requires AF_UNIX sockets"
)
def test_write_socket(event_log: git_trace2_event_log.EventLog) -> None:
"""Test Write() with Unix domain socket and validate received traces."""
received_traces: List[str] = []
with tempfile.TemporaryDirectory(prefix="test_server_sockets") as tempdir:
socket_path = os.path.join(tempdir, "server.sock")
server_ready = threading.Condition()
# Start "server" listening on Unix domain socket at socket_path.
server_thread = threading.Thread(
target=server_logging_thread,
args=(socket_path, server_ready, received_traces),
)
try:
server_thread.start()
def readLog(self, log_path):
"""Helper function to read log data into a list."""
log_data = []
with open(log_path, mode="rb") as f:
for line in f:
log_data.append(json.loads(line))
return log_data
with server_ready:
server_ready.wait(timeout=120)
def remove_prefix(self, s, prefix):
"""Return a copy string after removing |prefix| from |s|, if present or
the original string."""
if s.startswith(prefix):
return s[len(prefix) :]
else:
return s
event_log.StartEvent([])
path = event_log.Write(path=f"af_unix:{socket_path}")
finally:
server_thread.join(timeout=5)
def test_initial_state_with_parent_sid(self):
"""Test initial state when 'GIT_TRACE2_PARENT_SID' is set by parent."""
self.assertRegex(self._event_log_module.full_sid, self.FULL_SID_REGEX)
assert path == f"af_unix:stream:{socket_path}"
assert len(received_traces) == 2
version_event = json.loads(received_traces[0])
start_event = json.loads(received_traces[1])
verify_common_keys(version_event, expected_event_name="version")
verify_common_keys(start_event, expected_event_name="start")
# Check for 'start' event specific fields.
assert "argv" in start_event
assert isinstance(start_event["argv"], list)
def test_initial_state_no_parent_sid(self):
"""Test initial state when 'GIT_TRACE2_PARENT_SID' is not set."""
# Setup an empty environment dict (no parent sid).
self._event_log_module = git_trace2_event_log.EventLog(env={})
self.assertRegex(self._event_log_module.full_sid, self.SELF_SID_REGEX)
def test_version_event(self):
"""Test 'version' event data is valid.
class TestEventLogVerbose:
"""TestCase for the EventLog module verbose logging."""
Verify that the 'version' event is written even when no other
events are addded.
def test_write_socket_error_no_verbose(self) -> None:
"""Test Write() suppression of socket errors when not verbose."""
event_log = git_trace2_event_log.EventLog(env={})
event_log.verbose = False
with contextlib.redirect_stderr(
io.StringIO()
) as mock_stderr, mock.patch("socket.socket", side_effect=OSError):
event_log.Write(path="af_unix:stream:/tmp/test_sock")
assert mock_stderr.getvalue() == ""
Expected event log:
<version event>
"""
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log_module.Write(path=tempdir)
self._log_data = self.readLog(log_path)
# A log with no added events should only have the version entry.
self.assertEqual(len(self._log_data), 1)
version_event = self._log_data[0]
self.verifyCommonKeys(version_event, expected_event_name="version")
# Check for 'version' event specific fields.
self.assertIn("evt", version_event)
self.assertIn("exe", version_event)
# Verify "evt" version field is a string.
self.assertIsInstance(version_event["evt"], str)
def test_start_event(self):
"""Test and validate 'start' event data is valid.
Expected event log:
<version event>
<start event>
"""
self._event_log_module.StartEvent([])
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log_module.Write(path=tempdir)
self._log_data = self.readLog(log_path)
self.assertEqual(len(self._log_data), 2)
start_event = self._log_data[1]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version")
self.verifyCommonKeys(start_event, expected_event_name="start")
# Check for 'start' event specific fields.
self.assertIn("argv", start_event)
self.assertTrue(isinstance(start_event["argv"], list))
def test_exit_event_result_none(self):
"""Test 'exit' event data is valid when result is None.
We expect None result to be converted to 0 in the exit event data.
Expected event log:
<version event>
<exit event>
"""
self._event_log_module.ExitEvent(None)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log_module.Write(path=tempdir)
self._log_data = self.readLog(log_path)
self.assertEqual(len(self._log_data), 2)
exit_event = self._log_data[1]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version")
self.verifyCommonKeys(exit_event, expected_event_name="exit")
# Check for 'exit' event specific fields.
self.assertIn("code", exit_event)
# 'None' result should convert to 0 (successful) return code.
self.assertEqual(exit_event["code"], 0)
def test_exit_event_result_integer(self):
"""Test 'exit' event data is valid when result is an integer.
Expected event log:
<version event>
<exit event>
"""
self._event_log_module.ExitEvent(2)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log_module.Write(path=tempdir)
self._log_data = self.readLog(log_path)
self.assertEqual(len(self._log_data), 2)
exit_event = self._log_data[1]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version")
self.verifyCommonKeys(exit_event, expected_event_name="exit")
# Check for 'exit' event specific fields.
self.assertIn("code", exit_event)
self.assertEqual(exit_event["code"], 2)
def test_command_event(self):
"""Test and validate 'command' event data is valid.
Expected event log:
<version event>
<command event>
"""
self._event_log_module.CommandEvent(
name="repo", subcommands=["init", "this"]
)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log_module.Write(path=tempdir)
self._log_data = self.readLog(log_path)
self.assertEqual(len(self._log_data), 2)
command_event = self._log_data[1]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version")
self.verifyCommonKeys(command_event, expected_event_name="cmd_name")
# Check for 'command' event specific fields.
self.assertIn("name", command_event)
self.assertEqual(command_event["name"], "repo-init-this")
def test_def_params_event_repo_config(self):
"""Test 'def_params' event data outputs only repo config keys.
Expected event log:
<version event>
<def_param event>
<def_param event>
"""
config = {
"git.foo": "bar",
"repo.partialclone": "true",
"repo.partialclonefilter": "blob:none",
}
self._event_log_module.DefParamRepoEvents(config)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log_module.Write(path=tempdir)
self._log_data = self.readLog(log_path)
self.assertEqual(len(self._log_data), 3)
def_param_events = self._log_data[1:]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version")
for event in def_param_events:
self.verifyCommonKeys(event, expected_event_name="def_param")
# Check for 'def_param' event specific fields.
self.assertIn("param", event)
self.assertIn("value", event)
self.assertTrue(event["param"].startswith("repo."))
def test_def_params_event_no_repo_config(self):
"""Test 'def_params' event data won't output non-repo config keys.
Expected event log:
<version event>
"""
config = {
"git.foo": "bar",
"git.core.foo2": "baz",
}
self._event_log_module.DefParamRepoEvents(config)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log_module.Write(path=tempdir)
self._log_data = self.readLog(log_path)
self.assertEqual(len(self._log_data), 1)
self.verifyCommonKeys(self._log_data[0], expected_event_name="version")
def test_data_event_config(self):
"""Test 'data' event data outputs all config keys.
Expected event log:
<version event>
<data event>
<data event>
"""
config = {
"git.foo": "bar",
"repo.partialclone": "false",
"repo.syncstate.superproject.hassuperprojecttag": "true",
"repo.syncstate.superproject.sys.argv": ["--", "sync", "protobuf"],
}
prefix_value = "prefix"
self._event_log_module.LogDataConfigEvents(config, prefix_value)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log_module.Write(path=tempdir)
self._log_data = self.readLog(log_path)
self.assertEqual(len(self._log_data), 5)
data_events = self._log_data[1:]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version")
for event in data_events:
self.verifyCommonKeys(event)
# Check for 'data' event specific fields.
self.assertIn("key", event)
self.assertIn("value", event)
key = event["key"]
key = self.remove_prefix(key, f"{prefix_value}/")
value = event["value"]
self.assertEqual(
self._event_log_module.GetDataEventName(value), event["event"]
)
self.assertTrue(key in config and value == config[key])
def test_error_event(self):
"""Test and validate 'error' event data is valid.
Expected event log:
<version event>
<error event>
"""
msg = "invalid option: --cahced"
fmt = "invalid option: %s"
self._event_log_module.ErrorEvent(msg, fmt)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log_module.Write(path=tempdir)
self._log_data = self.readLog(log_path)
self.assertEqual(len(self._log_data), 2)
error_event = self._log_data[1]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version")
self.verifyCommonKeys(error_event, expected_event_name="error")
# Check for 'error' event specific fields.
self.assertIn("msg", error_event)
self.assertIn("fmt", error_event)
self.assertEqual(error_event["msg"], f"RepoErrorEvent:{msg}")
self.assertEqual(error_event["fmt"], f"RepoErrorEvent:{fmt}")
def test_write_with_filename(self):
"""Test Write() with a path to a file exits with None."""
self.assertIsNone(self._event_log_module.Write(path="path/to/file"))
def test_write_with_git_config(self):
"""Test Write() uses the git config path when 'git config' call
succeeds."""
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
with mock.patch.object(
self._event_log_module,
"_GetEventTargetPath",
return_value=tempdir,
):
self.assertEqual(
os.path.dirname(self._event_log_module.Write()), tempdir
)
def test_write_no_git_config(self):
"""Test Write() with no git config variable present exits with None."""
with mock.patch.object(
self._event_log_module, "_GetEventTargetPath", return_value=None
def test_write_socket_error_verbose(self) -> None:
"""Test Write() printing of socket errors when verbose."""
event_log = git_trace2_event_log.EventLog(env={})
event_log.verbose = True
with contextlib.redirect_stderr(
io.StringIO()
) as mock_stderr, mock.patch(
"socket.socket", side_effect=OSError("Mock error")
):
self.assertIsNone(self._event_log_module.Write())
def test_write_non_string(self):
"""Test Write() with non-string type for |path| throws TypeError."""
with self.assertRaises(TypeError):
self._event_log_module.Write(path=1234)
def test_write_socket(self):
"""Test Write() with Unix domain socket for |path| and validate received
traces."""
received_traces = []
with tempfile.TemporaryDirectory(
prefix="test_server_sockets"
) as tempdir:
socket_path = os.path.join(tempdir, "server.sock")
server_ready = threading.Condition()
# Start "server" listening on Unix domain socket at socket_path.
server_thread = threading.Thread(
target=serverLoggingThread,
args=(socket_path, server_ready, received_traces),
event_log.Write(path="af_unix:stream:/tmp/test_sock")
assert (
"git trace2 logging failed: Mock error"
in mock_stderr.getvalue()
)
try:
server_thread.start()
with server_ready:
server_ready.wait(timeout=120)
def test_write_file_error_no_verbose(self) -> None:
"""Test Write() suppression of file errors when not verbose."""
event_log = git_trace2_event_log.EventLog(env={})
event_log.verbose = False
with contextlib.redirect_stderr(
io.StringIO()
) as mock_stderr, mock.patch(
"tempfile.NamedTemporaryFile", side_effect=FileExistsError
):
event_log.Write(path="/tmp")
assert mock_stderr.getvalue() == ""
self._event_log_module.StartEvent([])
path = self._event_log_module.Write(
path=f"af_unix:{socket_path}"
)
finally:
server_thread.join(timeout=5)
self.assertEqual(path, f"af_unix:stream:{socket_path}")
self.assertEqual(len(received_traces), 2)
version_event = json.loads(received_traces[0])
start_event = json.loads(received_traces[1])
self.verifyCommonKeys(version_event, expected_event_name="version")
self.verifyCommonKeys(start_event, expected_event_name="start")
# Check for 'start' event specific fields.
self.assertIn("argv", start_event)
self.assertIsInstance(start_event["argv"], list)
def test_write_file_error_verbose(self) -> None:
"""Test Write() printing of file errors when verbose."""
event_log = git_trace2_event_log.EventLog(env={})
event_log.verbose = True
with contextlib.redirect_stderr(
io.StringIO()
) as mock_stderr, mock.patch(
"tempfile.NamedTemporaryFile",
side_effect=FileExistsError("Mock error"),
):
event_log.Write(path="/tmp")
assert (
"git trace2 logging failed: FileExistsError"
in mock_stderr.getvalue()
)
+84 -32
View File
@@ -14,42 +14,94 @@
"""Unittests for the hooks.py module."""
import unittest
from io import StringIO
import sys
import pytest
import hooks
class RepoHookShebang(unittest.TestCase):
"""Check shebang parsing in RepoHook."""
@pytest.mark.parametrize(
"data",
(
"",
"#\n# foo\n",
"# Bad shebang in script\n#!/foo\n",
),
)
def test_no_shebang(data: str) -> None:
"""Lines w/out shebangs should be rejected."""
assert hooks.RepoHook._ExtractInterpFromShebang(data) is None
def test_no_shebang(self):
"""Lines w/out shebangs should be rejected."""
DATA = ("", "#\n# foo\n", "# Bad shebang in script\n#!/foo\n")
for data in DATA:
self.assertIsNone(hooks.RepoHook._ExtractInterpFromShebang(data))
def test_direct_interp(self):
"""Lines whose shebang points directly to the interpreter."""
DATA = (
("#!/foo", "/foo"),
("#! /foo", "/foo"),
("#!/bin/foo ", "/bin/foo"),
("#! /usr/foo ", "/usr/foo"),
("#! /usr/foo -args", "/usr/foo"),
)
for shebang, interp in DATA:
self.assertEqual(
hooks.RepoHook._ExtractInterpFromShebang(shebang), interp
)
@pytest.mark.parametrize(
"shebang, interp",
(
("#!/foo", "/foo"),
("#! /foo", "/foo"),
("#!/bin/foo ", "/bin/foo"),
("#! /usr/foo ", "/usr/foo"),
("#! /usr/foo -args", "/usr/foo"),
),
)
def test_direct_interp(shebang: str, interp: str) -> None:
"""Lines whose shebang points directly to the interpreter."""
assert hooks.RepoHook._ExtractInterpFromShebang(shebang) == interp
def test_env_interp(self):
"""Lines whose shebang launches through `env`."""
DATA = (
("#!/usr/bin/env foo", "foo"),
("#!/bin/env foo", "foo"),
("#! /bin/env /bin/foo ", "/bin/foo"),
)
for shebang, interp in DATA:
self.assertEqual(
hooks.RepoHook._ExtractInterpFromShebang(shebang), interp
)
@pytest.mark.parametrize(
"shebang, interp",
(
("#!/usr/bin/env foo", "foo"),
("#!/bin/env foo", "foo"),
("#! /bin/env /bin/foo ", "/bin/foo"),
),
)
def test_env_interp(shebang: str, interp: str) -> None:
"""Lines whose shebang launches through `env`."""
assert hooks.RepoHook._ExtractInterpFromShebang(shebang) == interp
def test_post_sync_argument_validation() -> None:
"""Test that post-sync hook requires exact API arguments."""
class FakeProject:
def __init__(self):
self.worktree = "/some/path"
self.enabled_repo_hooks = ["post-sync"]
hook = hooks.RepoHook(
hook_type="post-sync",
hooks_project=FakeProject(),
repo_topdir="/topdir",
manifest_url="https://gerrit",
allow_all_hooks=True,
)
old_stderr = sys.stderr
sys.stderr = StringIO()
try:
# Call with missing arg `sync_duration_seconds`
res = hook.Run(repo_topdir="/topdir")
assert res is False
assert "hook 'post-sync' called incorrectly" in sys.stderr.getvalue()
# Mock _CheckHook and _ExecuteHook to test success path
hook._CheckHook = lambda: None
executed_kwargs = {}
def fake_execute(**kw):
executed_kwargs.update(kw)
hook._ExecuteHook = fake_execute
res = hook.Run(repo_topdir="/topdir", sync_duration_seconds=12.345)
assert res is True
assert executed_kwargs.get("sync_duration_seconds") == 12.345
finally:
sys.stderr = old_stderr
+166
View File
@@ -0,0 +1,166 @@
# Copyright (C) 2026 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tests for the main repo script and subcommand routing."""
from unittest import mock
import pytest
from main import _Repo
@pytest.fixture(name="repo")
def fixture_repo():
repo = _Repo("repodir")
# Overriding the command list here ensures that we are only testing
# against a fixed set of commands, reducing fragility to new
# subcommands being added to the main repo tool.
repo.commands = {"start": None, "sync": None, "smart": None}
return repo
@pytest.fixture(name="mock_config")
def fixture_mock_config():
return mock.MagicMock()
@mock.patch("time.sleep")
def test_autocorrect_delay(mock_sleep, repo, mock_config):
"""Test autocorrect with positive delay."""
mock_config.GetString.return_value = "10"
res = repo._autocorrect_command_name("tart", mock_config)
mock_config.GetString.assert_called_with("help.autocorrect")
mock_sleep.assert_called_with(1.0)
assert res == "start"
@mock.patch("time.sleep")
def test_autocorrect_delay_one(mock_sleep, repo, mock_config):
"""Test autocorrect with '1' (0.1s delay, not immediate)."""
mock_config.GetString.return_value = "1"
res = repo._autocorrect_command_name("tart", mock_config)
mock_sleep.assert_called_with(0.1)
assert res == "start"
@mock.patch("time.sleep", side_effect=KeyboardInterrupt())
def test_autocorrect_delay_interrupt(mock_sleep, repo, mock_config):
"""Test autocorrect handles KeyboardInterrupt during delay."""
mock_config.GetString.return_value = "10"
res = repo._autocorrect_command_name("tart", mock_config)
mock_sleep.assert_called_with(1.0)
assert res is None
@mock.patch("time.sleep")
def test_autocorrect_immediate(mock_sleep, repo, mock_config):
"""Test autocorrect with immediate/negative delay."""
# Test numeric negative.
mock_config.GetString.return_value = "-1"
res = repo._autocorrect_command_name("tart", mock_config)
mock_sleep.assert_not_called()
assert res == "start"
# Test string boolean "true".
mock_config.GetString.return_value = "true"
res = repo._autocorrect_command_name("tart", mock_config)
mock_sleep.assert_not_called()
assert res == "start"
# Test string boolean "yes".
mock_config.GetString.return_value = "YES"
res = repo._autocorrect_command_name("tart", mock_config)
mock_sleep.assert_not_called()
assert res == "start"
# Test string boolean "immediate".
mock_config.GetString.return_value = "Immediate"
res = repo._autocorrect_command_name("tart", mock_config)
mock_sleep.assert_not_called()
assert res == "start"
def test_autocorrect_zero_or_show(repo, mock_config):
"""Test autocorrect with zero delay (suggestions only)."""
# Test numeric zero.
mock_config.GetString.return_value = "0"
res = repo._autocorrect_command_name("tart", mock_config)
assert res is None
# Test string boolean "false".
mock_config.GetString.return_value = "False"
res = repo._autocorrect_command_name("tart", mock_config)
assert res is None
# Test string boolean "show".
mock_config.GetString.return_value = "show"
res = repo._autocorrect_command_name("tart", mock_config)
assert res is None
def test_autocorrect_never(repo, mock_config):
"""Test autocorrect with 'never'."""
mock_config.GetString.return_value = "never"
res = repo._autocorrect_command_name("tart", mock_config)
assert res is None
@mock.patch("builtins.input", return_value="y")
def test_autocorrect_prompt_yes(mock_input, repo, mock_config):
"""Test autocorrect with prompt and user answers yes."""
mock_config.GetString.return_value = "prompt"
res = repo._autocorrect_command_name("tart", mock_config)
assert res == "start"
@mock.patch("builtins.input", return_value="n")
def test_autocorrect_prompt_no(mock_input, repo, mock_config):
"""Test autocorrect with prompt and user answers no."""
mock_config.GetString.return_value = "prompt"
res = repo._autocorrect_command_name("tart", mock_config)
assert res is None
@mock.patch("builtins.input", return_value="y")
def test_autocorrect_multiple_candidates(mock_input, repo, mock_config):
"""Test autocorrect with multiple matches forces a prompt."""
mock_config.GetString.return_value = "10" # Normally just delay
# 'snart' matches both 'start' and 'smart' with > 0.7 ratio
res = repo._autocorrect_command_name("snart", mock_config)
# Because there are multiple candidates, it should prompt
mock_input.assert_called_once()
assert res == "start"
@mock.patch("builtins.input", side_effect=KeyboardInterrupt())
def test_autocorrect_prompt_interrupt(mock_input, repo, mock_config):
"""Test autocorrect with prompt and user interrupts."""
mock_config.GetString.return_value = "prompt"
res = repo._autocorrect_command_name("tart", mock_config)
assert res is None
+619 -465
View File
File diff suppressed because it is too large Load Diff
+90 -26
View File
@@ -14,39 +14,103 @@
"""Unittests for the platform_utils.py module."""
import os
import tempfile
import unittest
from pathlib import Path
import pytest
import platform_utils
class RemoveTests(unittest.TestCase):
"""Check remove() helper."""
def test_remove_missing_ok(tmp_path: Path) -> None:
"""Check missing_ok handling."""
path = tmp_path / "test"
def testMissingOk(self):
"""Check missing_ok handling."""
with tempfile.TemporaryDirectory() as tmpdir:
path = os.path.join(tmpdir, "test")
# Should not fail.
platform_utils.remove(path, missing_ok=True)
# Should not fail.
platform_utils.remove(path, missing_ok=True)
# Should fail.
with pytest.raises(OSError):
platform_utils.remove(path)
with pytest.raises(OSError):
platform_utils.remove(path, missing_ok=False)
# Should fail.
self.assertRaises(OSError, platform_utils.remove, path)
self.assertRaises(
OSError, platform_utils.remove, path, missing_ok=False
)
# Should not fail if it exists.
path.touch()
platform_utils.remove(path, missing_ok=True)
assert not path.exists()
# Should not fail if it exists.
open(path, "w").close()
platform_utils.remove(path, missing_ok=True)
self.assertFalse(os.path.exists(path))
path.touch()
platform_utils.remove(path)
assert not path.exists()
open(path, "w").close()
platform_utils.remove(path)
self.assertFalse(os.path.exists(path))
path.touch()
platform_utils.remove(path, missing_ok=False)
assert not path.exists()
open(path, "w").close()
platform_utils.remove(path, missing_ok=False)
self.assertFalse(os.path.exists(path))
def test_removedirs_nonexistent(tmp_path: Path) -> None:
"""removedirs should silently succeed on nonexistent paths."""
platform_utils.removedirs(tmp_path / "does-not-exist")
def test_removedirs_symlink(tmp_path: Path) -> None:
"""removedirs should remove a symlink."""
link = tmp_path / "link"
link.symlink_to("target")
platform_utils.removedirs(link)
assert not link.exists()
def test_removedirs_empty_dir(tmp_path: Path) -> None:
"""removedirs should remove an empty directory."""
d = tmp_path / "empty"
d.mkdir()
platform_utils.removedirs(d)
assert not d.exists()
def test_removedirs_nested_empty_dirs(tmp_path: Path) -> None:
"""removedirs should remove nested empty directories."""
d = tmp_path / "a" / "b" / "c"
d.mkdir(parents=True)
platform_utils.removedirs(tmp_path / "a")
assert not (tmp_path / "a").exists()
def test_removedirs_symlinks_inside_dir(tmp_path: Path) -> None:
"""removedirs should remove symlinks inside a directory."""
d = tmp_path / "dir"
d.mkdir()
(d / "link1").symlink_to("target1")
(d / "link2").symlink_to("target2")
platform_utils.removedirs(d)
assert not d.exists()
def test_removedirs_preserves_user_files(tmp_path: Path) -> None:
"""removedirs should not delete regular files or their parent dirs."""
d = tmp_path / "dir"
d.mkdir()
(d / "link").symlink_to("target")
(d / "user-file.txt").write_text("keep me")
platform_utils.removedirs(d)
assert d.exists()
assert not (d / "link").exists()
assert (d / "user-file.txt").read_text() == "keep me"
def test_removedirs_deep_nested_with_symlinks(tmp_path: Path) -> None:
"""removedirs should handle deep nesting: sub/dir/target."""
d = tmp_path / "sub" / "dir"
d.mkdir(parents=True)
(d / "link").symlink_to("target")
platform_utils.removedirs(tmp_path / "sub")
assert not (tmp_path / "sub").exists()
def test_removedirs_regular_file_noop(tmp_path: Path) -> None:
"""removedirs should not delete a regular file."""
f = tmp_path / "file.txt"
f.write_text("data")
platform_utils.removedirs(f)
assert f.exists()
+435 -22
View File
@@ -17,37 +17,22 @@
import contextlib
import os
from pathlib import Path
import shutil
import subprocess
import tempfile
from typing import Optional
import unittest
from unittest import mock
import utils_for_test
import error
import git_command
import git_config
import manifest_xml
import platform_utils
import project
@contextlib.contextmanager
def TempGitTree():
"""Create a new empty git checkout for testing."""
with tempfile.TemporaryDirectory(prefix="repo-tests") as tempdir:
# Tests need to assume, that main is default branch at init,
# which is not supported in config until 2.28.
cmd = ["git", "init"]
if git_command.git_require((2, 28, 0)):
cmd += ["--initial-branch=main"]
else:
# Use template dir for init.
templatedir = tempfile.mkdtemp(prefix=".test-template")
with open(os.path.join(templatedir, "HEAD"), "w") as fp:
fp.write("ref: refs/heads/main\n")
cmd += ["--template", templatedir]
subprocess.check_call(cmd, cwd=tempdir)
yield tempdir
class FakeProject:
"""A fake for Project for basic functionality."""
@@ -63,13 +48,16 @@ class FakeProject:
)
self.config = git_config.GitConfig.ForRepository(gitdir=self.gitdir)
def RelPath(self, local: Optional[bool] = None) -> str:
return self.name
class ReviewableBranchTests(unittest.TestCase):
"""Check ReviewableBranch behavior."""
def test_smoke(self):
"""A quick run through everything."""
with TempGitTree() as tempdir:
with utils_for_test.TempGitTree() as tempdir:
fakeproj = FakeProject(tempdir)
# Generate some commits.
@@ -116,6 +104,111 @@ class ProjectTests(unittest.TestCase):
"abcd00%21%21_%2b",
)
@unittest.skipUnless(
utils_for_test.supports_reftable(),
"git reftable support is required for this test",
)
def test_get_head_unborn_reftable(self):
with tempfile.TemporaryDirectory(prefix="repo-tests") as tempdir:
subprocess.check_call(
[
"git",
"-c",
"init.defaultRefFormat=reftable",
"init",
"-q",
tempdir,
]
)
fakeproj = FakeProject(tempdir)
expected = subprocess.check_output(
["git", "-C", tempdir, "symbolic-ref", "-q", "HEAD"],
encoding="utf-8",
).strip()
self.assertEqual(expected, fakeproj.work_git.GetHead())
def _get_derived_subproject_url(self, submodule_url):
with tempfile.TemporaryDirectory(prefix="repo-tests") as tempdir:
class FakeManifest:
def __init__(self, topdir):
self.topdir = topdir
self.globalConfig = None
self.is_multimanifest = False
self.path_prefix = ""
self.paths = {}
def GetSubprojectName(self, parent, path):
return path
def GetSubprojectPaths(self, parent, name, path):
relpath = path
worktree = os.path.join(self.topdir, path)
gitdir = os.path.join(self.topdir, f"{path}.git")
objdir = os.path.join(self.topdir, f"{path}.obj")
os.makedirs(worktree, exist_ok=True)
os.makedirs(gitdir, exist_ok=True)
os.makedirs(objdir, exist_ok=True)
return relpath, worktree, gitdir, objdir
manifest = FakeManifest(tempdir)
worktree = os.path.join(tempdir, "parent")
gitdir = os.path.join(tempdir, "parent.git")
objdir = os.path.join(tempdir, "parent.obj")
os.makedirs(worktree)
os.makedirs(gitdir)
os.makedirs(objdir)
parent = project.Project(
manifest=manifest,
name="parent",
remote=project.RemoteSpec(
"origin", url="https://example.com/platform/superproject"
),
gitdir=gitdir,
objdir=objdir,
worktree=worktree,
relpath="parent",
revisionExpr="refs/heads/main",
revisionId=None,
)
def fake_get_submodules(current):
if current is parent:
return [("subrev", "child", submodule_url, "false")]
return []
with mock.patch.object(
project.Project, "_GetSubmodules", autospec=True
) as get_submodules:
get_submodules.side_effect = fake_get_submodules
result = parent.GetDerivedSubprojects()
self.assertEqual(1, len(result))
return result[0].remote.url
def test_derived_subproject_joins_only_git_relative_urls(self):
tests = (
(
"./submodule",
"https://example.com/platform/superproject/submodule",
),
("../sibling", "https://example.com/platform/sibling"),
)
for submodule_url, expected in tests:
with self.subTest(submodule_url=submodule_url):
self.assertEqual(
expected, self._get_derived_subproject_url(submodule_url)
)
def test_derived_subproject_leaves_dot_prefixed_names_unchanged(self):
for submodule_url in (".foo", "..bar"):
with self.subTest(submodule_url=submodule_url):
self.assertEqual(
submodule_url,
self._get_derived_subproject_url(submodule_url),
)
class CopyLinkTestCase(unittest.TestCase):
"""TestCase for stub repo client checkouts.
@@ -354,11 +447,53 @@ class LinkFile(CopyLinkTestCase):
os.path.join("git-project", "foo.txt"), os.readlink(dest)
)
def test_replace_empty_dir_with_symlink(self):
"""A linkfile should replace an empty real directory at the dest path.
This is the common case: the old linkfiles inside the directory were
already cleaned up by UpdateCopyLinkfileList, leaving an empty parent
directory behind.
"""
src_dir = os.path.join(self.worktree, "dot-llms")
os.makedirs(src_dir)
dest = os.path.join(self.topdir, "mydir")
os.makedirs(dest)
lf = self.LinkFile("dot-llms", "mydir")
lf._Link()
self.assertTrue(os.path.islink(dest))
self.assertEqual(
os.path.join("git-project", "dot-llms"), os.readlink(dest)
)
def test_nonempty_dir_not_clobbered(self):
"""A linkfile must not delete a non-empty directory.
If the user created files in a directory that a new linkfile wants
to replace, __linkIt should fail safely rather than deleting content.
"""
src_dir = os.path.join(self.worktree, "dot-llms")
os.makedirs(src_dir)
dest = os.path.join(self.topdir, "mydir")
os.makedirs(dest)
user_file = os.path.join(dest, "user-notes.txt")
self.touch(user_file)
lf = self.LinkFile("dot-llms", "mydir")
lf._Link()
# The directory should NOT be replaced — user content is preserved.
self.assertFalse(os.path.islink(dest))
self.assertTrue(os.path.isdir(dest))
self.assertTrue(os.path.exists(user_file))
class MigrateWorkTreeTests(unittest.TestCase):
"""Check _MigrateOldWorkTreeGitDir handling."""
_SYMLINKS = {
# go/keep-sorted start
"config",
"description",
"hooks",
@@ -367,9 +502,11 @@ class MigrateWorkTreeTests(unittest.TestCase):
"objects",
"packed-refs",
"refs",
"reftable",
"rr-cache",
"shallow",
"svn",
# go/keep-sorted end
}
_FILES = {
"COMMIT_EDITMSG",
@@ -448,6 +585,25 @@ class MigrateWorkTreeTests(unittest.TestCase):
for name in self._SYMLINKS:
self.assertTrue((dotgit / name).is_symlink())
def test_reftable_anchor_with_refs_dir(self):
"""Migrate when reftable/ and refs/ are directories."""
with self._simple_layout() as tempdir:
dotgit = tempdir / "src/test/.git"
(dotgit / "refs").unlink()
(dotgit / "refs").mkdir()
(dotgit / "refs" / "heads").write_text("dummy")
(dotgit / "reftable").unlink()
(dotgit / "reftable").mkdir()
(dotgit / "reftable" / "tables.list").write_text("dummy")
project.Project._MigrateOldWorkTreeGitDir(str(dotgit))
self.assertTrue(dotgit.is_symlink())
self.assertEqual(
os.readlink(dotgit),
os.path.normpath("../../.repo/projects/src/test.git"),
)
class ManifestPropertiesFetchedCorrectly(unittest.TestCase):
"""Ensure properties are fetched properly."""
@@ -467,7 +623,7 @@ class ManifestPropertiesFetchedCorrectly(unittest.TestCase):
def test_manifest_config_properties(self):
"""Test we are fetching the manifest config properties correctly."""
with TempGitTree() as tempdir:
with utils_for_test.TempGitTree() as tempdir:
fakeproj = self.setUpManifest(tempdir)
# Set property using the expected Set method, then ensure
@@ -534,3 +690,260 @@ class ManifestPropertiesFetchedCorrectly(unittest.TestCase):
fakeproj.config.SetString("manifest.platform", "auto")
self.assertEqual(fakeproj.manifest_platform, "auto")
class StatelessSyncTests(unittest.TestCase):
"""Tests for stateless sync strategy."""
def _get_project(self, tempdir):
manifest = mock.MagicMock()
manifest.manifestProject.depth = None
manifest.manifestProject.dissociate = False
manifest.manifestProject.clone_filter = None
manifest.is_multimanifest = False
manifest.manifestProject.config.GetBoolean.return_value = False
remote = mock.MagicMock()
remote.name = "origin"
remote.url = "http://"
proj = project.Project(
manifest=manifest,
name="test-project",
remote=remote,
gitdir=os.path.join(tempdir, ".git"),
objdir=os.path.join(tempdir, ".git"),
worktree=tempdir,
relpath="test-project",
revisionExpr="1234abcd",
revisionId=None,
sync_strategy="stateless",
)
proj._CheckForImmutableRevision = mock.MagicMock(return_value=False)
proj._LsRemote = mock.MagicMock(
return_value="1234abcd\trefs/heads/main\n"
)
proj.bare_git = mock.MagicMock()
proj.bare_git.rev_parse.return_value = "5678abcd"
proj.bare_git.rev_list.return_value = ["0"]
proj.IsDirty = mock.MagicMock(return_value=False)
proj.GetBranches = mock.MagicMock(return_value=[])
proj.DeleteWorktree = mock.MagicMock()
proj._InitGitDir = mock.MagicMock()
proj._RemoteFetch = mock.MagicMock(return_value=True)
proj._InitRemote = mock.MagicMock()
proj._InitMRef = mock.MagicMock()
return proj
def test_sync_network_half_stateless_prune_needed(self):
"""Test stateless sync queues prune when needed."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir)
res = proj.Sync_NetworkHalf()
self.assertTrue(res.success)
proj.DeleteWorktree.assert_not_called()
self.assertTrue(proj.stateless_prune_needed)
proj._RemoteFetch.assert_called_once()
def test_sync_local_half_stateless_prune(self):
"""Test stateless GC pruning is queued in Sync_LocalHalf."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir)
proj.stateless_prune_needed = True
proj._Checkout = mock.MagicMock()
proj._InitWorkTree = mock.MagicMock()
proj.IsRebaseInProgress = mock.MagicMock(return_value=False)
proj.IsCherryPickInProgress = mock.MagicMock(return_value=False)
proj.bare_ref = mock.MagicMock()
proj.bare_ref.all = {}
proj.GetRevisionId = mock.MagicMock(return_value="1234abcd")
proj._CopyAndLinkFiles = mock.MagicMock()
proj.work_git = mock.MagicMock()
proj.work_git.GetHead.return_value = "5678abcd"
syncbuf = project.SyncBuffer(proj.config)
with mock.patch("project.GitCommand") as mock_git_cmd:
mock_cmd_instance = mock.MagicMock()
mock_cmd_instance.Wait.return_value = 0
mock_git_cmd.return_value = mock_cmd_instance
proj.Sync_LocalHalf(syncbuf)
syncbuf.Finish()
self.assertEqual(mock_git_cmd.call_count, 2)
mock_git_cmd.assert_any_call(
proj, ["reflog", "expire", "--expire=all", "--all"], bare=True
)
mock_git_cmd.assert_any_call(
proj,
["gc", "--prune=now"],
bare=True,
capture_stdout=True,
capture_stderr=True,
)
def test_sync_local_half_no_upstream_propagates_force_checkout(self):
"""Test Sync_LocalHalf forwards force_checkout when detaching."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir)
proj._InitWorkTree = mock.MagicMock()
proj.CleanPublishedCache = mock.MagicMock()
proj.GetRevisionId = mock.MagicMock(return_value="1234abcd")
proj._Checkout = mock.MagicMock()
proj._CopyAndLinkFiles = mock.MagicMock()
proj.work_git = mock.MagicMock()
proj.work_git.GetHead.return_value = "refs/heads/topic"
proj.bare_ref = mock.MagicMock()
proj.bare_ref.all = {"refs/heads/topic": "5678abcd"}
branch = mock.MagicMock()
branch.name = "topic"
branch.LocalMerge = False
proj.GetBranch = mock.MagicMock(return_value=branch)
syncbuf = project.SyncBuffer(proj.config)
proj.Sync_LocalHalf(syncbuf, force_checkout=True)
proj._Checkout.assert_called_once_with(
"1234abcd", force_checkout=True, quiet=True
)
proj._CopyAndLinkFiles.assert_called_once_with()
def test_sync_network_half_stateless_skips_if_stash(self):
"""Test stateless sync skips if stash exists."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir)
proj.HasStash = mock.MagicMock(return_value=True)
res = proj.Sync_NetworkHalf()
self.assertTrue(res.success)
self.assertFalse(getattr(proj, "stateless_prune_needed", False))
def test_sync_network_half_stateless_skips_if_local_commits(self):
"""Test stateless sync skips if there are local-only commits."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir)
proj.bare_git.rev_list.return_value = ["1"]
res = proj.Sync_NetworkHalf()
self.assertTrue(res.success)
self.assertFalse(getattr(proj, "stateless_prune_needed", False))
class SyncOptimizationTests(unittest.TestCase):
"""Tests for sync optimization logic involving shallow clones."""
def _get_project(self, tempdir, depth=None):
manifest = mock.MagicMock()
manifest.manifestProject.depth = depth
manifest.manifestProject.dissociate = False
manifest.manifestProject.clone_filter = None
manifest.is_multimanifest = False
manifest.manifestProject.config.GetBoolean.return_value = False
manifest.IsMirror = False
remote = mock.MagicMock()
remote.name = "origin"
remote.url = "http://"
proj = project.Project(
manifest=manifest,
name="test-project",
remote=remote,
gitdir=os.path.join(tempdir, "gitdir"),
objdir=os.path.join(tempdir, "objdir"),
worktree=tempdir,
relpath="test-project",
revisionExpr="0123456789abcdef0123456789abcdef01234567",
revisionId=None,
)
proj._CheckForImmutableRevision = mock.MagicMock(return_value=True)
proj.DeleteWorktree = mock.MagicMock()
proj._InitGitDir = mock.MagicMock()
proj._InitRemote = mock.MagicMock()
proj._InitMRef = mock.MagicMock()
return proj
def test_sync_network_half_shallow_missing_fetches(self):
"""Test Sync_NetworkHalf fetches if shallow file is missing."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir, depth=1)
# Ensure gitdir does not exist to simulate new project
if os.path.exists(proj.gitdir):
shutil.rmtree(proj.gitdir)
shallow_path = os.path.join(proj.gitdir, "shallow")
if os.path.exists(shallow_path):
os.unlink(shallow_path)
proj._RemoteFetch = mock.MagicMock(return_value=True)
res = proj.Sync_NetworkHalf(optimized_fetch=True)
self.assertTrue(res.success)
proj._RemoteFetch.assert_called_once()
def test_sync_network_half_shallow_exists_skips(self):
"""Test Sync_NetworkHalf skips fetch if shallow file exists."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir, depth=1)
os.makedirs(proj.gitdir, exist_ok=True)
os.makedirs(proj.objdir, exist_ok=True)
with open(os.path.join(proj.gitdir, "shallow"), "w") as f:
f.write("")
proj._RemoteFetch = mock.MagicMock()
res = proj.Sync_NetworkHalf(optimized_fetch=True)
self.assertTrue(res.success)
proj._RemoteFetch.assert_not_called()
def test_remote_fetch_shallow_missing_fetches(self):
"""Test _RemoteFetch fetches if shallow file is missing."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir, depth=1)
shallow_path = os.path.join(proj.gitdir, "shallow")
if os.path.exists(shallow_path):
os.unlink(shallow_path)
with mock.patch("project.GitCommand") as mock_git_cmd:
mock_cmd_instance = mock.MagicMock()
mock_cmd_instance.Wait.return_value = 0
mock_git_cmd.return_value = mock_cmd_instance
res = proj._RemoteFetch(
current_branch_only=True,
depth=1,
use_superproject=False,
)
self.assertTrue(res)
mock_git_cmd.assert_called()
def test_remote_fetch_shallow_exists_skips(self):
"""Test _RemoteFetch skips fetch if shallow file exists."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir, depth=1)
os.makedirs(proj.gitdir, exist_ok=True)
os.makedirs(proj.objdir, exist_ok=True)
with open(os.path.join(proj.gitdir, "shallow"), "w") as f:
f.write("")
with mock.patch("project.GitCommand") as mock_git_cmd:
res = proj._RemoteFetch(
current_branch_only=True,
depth=1,
use_superproject=False,
)
self.assertTrue(res)
mock_git_cmd.assert_not_called()
+77 -71
View File
@@ -12,90 +12,96 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Unit test for repo_logging module."""
"""Unittests for the repo_logging.py module."""
import contextlib
import io
import logging
import unittest
import re
from unittest import mock
import pytest
from color import SetDefaultColoring
from error import RepoExitError
from repo_logging import RepoLogger
class TestRepoLogger(unittest.TestCase):
@mock.patch.object(RepoLogger, "error")
def test_log_aggregated_errors_logs_aggregated_errors(self, mock_error):
"""Test if log_aggregated_errors logs a list of aggregated errors."""
logger = RepoLogger(__name__)
logger.log_aggregated_errors(
RepoExitError(
aggregate_errors=[
Exception("foo"),
Exception("bar"),
Exception("baz"),
Exception("hello"),
Exception("world"),
Exception("test"),
]
)
)
mock_error.assert_has_calls(
[
mock.call("=" * 80),
mock.call(
"Repo command failed due to the following `%s` errors:",
"RepoExitError",
),
mock.call("foo\nbar\nbaz\nhello\nworld"),
mock.call("+%d additional errors...", 1),
@mock.patch.object(RepoLogger, "error")
def test_log_aggregated_errors_logs_aggregated_errors(mock_error) -> None:
"""Test if log_aggregated_errors logs a list of aggregated errors."""
logger = RepoLogger(__name__)
logger.log_aggregated_errors(
RepoExitError(
aggregate_errors=[
Exception("foo"),
Exception("bar"),
Exception("baz"),
Exception("hello"),
Exception("world"),
Exception("test"),
]
)
)
@mock.patch.object(RepoLogger, "error")
def test_log_aggregated_errors_logs_single_error(self, mock_error):
"""Test if log_aggregated_errors logs empty aggregated_errors."""
mock_error.assert_has_calls(
[
mock.call("=" * 80),
mock.call(
"Repo command failed due to the following `%s` errors:",
"RepoExitError",
),
mock.call("foo\nbar\nbaz\nhello\nworld"),
mock.call("+%d additional errors...", 1),
]
)
@mock.patch.object(RepoLogger, "error")
def test_log_aggregated_errors_logs_single_error(mock_error) -> None:
"""Test if log_aggregated_errors logs empty aggregated_errors."""
logger = RepoLogger(__name__)
logger.log_aggregated_errors(RepoExitError())
mock_error.assert_has_calls(
[
mock.call("=" * 80),
mock.call("Repo command failed: %s", "RepoExitError"),
]
)
@pytest.mark.parametrize(
"level",
(
logging.INFO,
logging.WARN,
logging.ERROR,
),
)
def test_log_with_format_string(level: int) -> None:
"""Test different log levels with format strings."""
name = logging.getLevelName(level)
# Set color output to "always" for consistent test results.
# This ensures the logger's behavior is uniform across different
# environments and git configurations.
SetDefaultColoring("always")
# Regex pattern to match optional ANSI color codes.
# \033 - Escape character
# \[ - Opening square bracket
# [0-9;]* - Zero or more digits or semicolons
# m - Ending 'm' character
# ? - Makes the entire group optional
opt_color = r"(\033\[[0-9;]*m)?"
output = io.StringIO()
with contextlib.redirect_stderr(output):
logger = RepoLogger(__name__)
logger.log_aggregated_errors(RepoExitError())
logger.log(level, "%s", "100% pass")
mock_error.assert_has_calls(
[
mock.call("=" * 80),
mock.call("Repo command failed: %s", "RepoExitError"),
]
)
def test_log_with_format_string(self):
"""Test different log levels with format strings."""
# Set color output to "always" for consistent test results.
# This ensures the logger's behavior is uniform across different
# environments and git configurations.
SetDefaultColoring("always")
# Regex pattern to match optional ANSI color codes.
# \033 - Escape character
# \[ - Opening square bracket
# [0-9;]* - Zero or more digits or semicolons
# m - Ending 'm' character
# ? - Makes the entire group optional
opt_color = r"(\033\[[0-9;]*m)?"
for level in (logging.INFO, logging.WARN, logging.ERROR):
name = logging.getLevelName(level)
with self.subTest(level=level, name=name):
output = io.StringIO()
with contextlib.redirect_stderr(output):
logger = RepoLogger(__name__)
logger.log(level, "%s", "100% pass")
self.assertRegex(
output.getvalue().strip(),
f"^{opt_color}100% pass{opt_color}$",
f"failed for level {name}",
)
assert re.search(
f"^{opt_color}100% pass{opt_color}$", output.getvalue().strip()
), f"failed for level {name}"
+24 -33
View File
@@ -15,46 +15,37 @@
"""Unittests for the repo_trace.py module."""
import os
import unittest
from unittest import mock
import pytest
import repo_trace
class TraceTests(unittest.TestCase):
def test_trace_max_size_enforced(monkeypatch: pytest.MonkeyPatch) -> None:
"""Check Trace behavior."""
content = "git chicken"
def testTrace_MaxSizeEnforced(self):
content = "git chicken"
with repo_trace.Trace(content, first_trace=True):
pass
first_trace_size = os.path.getsize(repo_trace._TRACE_FILE)
with repo_trace.Trace(content, first_trace=True):
pass
first_trace_size = os.path.getsize(repo_trace._TRACE_FILE)
with repo_trace.Trace(content):
pass
assert os.path.getsize(repo_trace._TRACE_FILE) > first_trace_size
with repo_trace.Trace(content):
pass
self.assertGreater(
os.path.getsize(repo_trace._TRACE_FILE), first_trace_size
)
# Check we clear everything if the last chunk is larger than _MAX_SIZE.
monkeypatch.setattr(repo_trace, "_MAX_SIZE", 0)
with repo_trace.Trace(content, first_trace=True):
pass
assert os.path.getsize(repo_trace._TRACE_FILE) == first_trace_size
# Check we clear everything is the last chunk is larger than _MAX_SIZE.
with mock.patch("repo_trace._MAX_SIZE", 0):
with repo_trace.Trace(content, first_trace=True):
pass
self.assertEqual(
first_trace_size, os.path.getsize(repo_trace._TRACE_FILE)
)
# Check we only clear the chunks we need to.
new_max = (first_trace_size + 1) / (1024 * 1024)
monkeypatch.setattr(repo_trace, "_MAX_SIZE", new_max)
with repo_trace.Trace(content, first_trace=True):
pass
assert os.path.getsize(repo_trace._TRACE_FILE) == first_trace_size * 2
# Check we only clear the chunks we need to.
repo_trace._MAX_SIZE = (first_trace_size + 1) / (1024 * 1024)
with repo_trace.Trace(content, first_trace=True):
pass
self.assertEqual(
first_trace_size * 2, os.path.getsize(repo_trace._TRACE_FILE)
)
with repo_trace.Trace(content, first_trace=True):
pass
self.assertEqual(
first_trace_size * 2, os.path.getsize(repo_trace._TRACE_FILE)
)
with repo_trace.Trace(content, first_trace=True):
pass
assert os.path.getsize(repo_trace._TRACE_FILE) == first_trace_size * 2
+64 -45
View File
@@ -16,65 +16,84 @@
import multiprocessing
import subprocess
import unittest
from typing import Tuple
from unittest import mock
import pytest
import ssh
class SshTests(unittest.TestCase):
"""Tests the ssh functions."""
@pytest.fixture(autouse=True)
def clear_ssh_version_cache() -> None:
"""Clear the ssh version cache before each test."""
ssh.version.cache_clear()
def test_parse_ssh_version(self):
"""Check _parse_ssh_version() handling."""
ver = ssh._parse_ssh_version("Unknown\n")
self.assertEqual(ver, ())
ver = ssh._parse_ssh_version("OpenSSH_1.0\n")
self.assertEqual(ver, (1, 0))
ver = ssh._parse_ssh_version(
"OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13, OpenSSL 1.0.1f 6 Jan 2014\n"
)
self.assertEqual(ver, (6, 6, 1))
ver = ssh._parse_ssh_version(
"OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\n"
)
self.assertEqual(ver, (7, 6))
ver = ssh._parse_ssh_version("OpenSSH_9.0p1, LibreSSL 3.3.6\n")
self.assertEqual(ver, (9, 0))
def test_version(self):
"""Check version() handling."""
with mock.patch("ssh._run_ssh_version", return_value="OpenSSH_1.2\n"):
self.assertEqual(ssh.version(), (1, 2))
@pytest.mark.parametrize(
"input_str, expected",
(
("Unknown\n", ()),
("OpenSSH_1.0\n", (1, 0)),
(
"OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13, OpenSSL 1.0.1f 6 Jan 2014\n",
(6, 6, 1),
),
(
"OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017\n",
(7, 6),
),
("OpenSSH_9.0p1, LibreSSL 3.3.6\n", (9, 0)),
),
)
def test_parse_ssh_version(input_str: str, expected: Tuple[int, ...]) -> None:
"""Check _parse_ssh_version() handling."""
assert ssh._parse_ssh_version(input_str) == expected
def test_context_manager_empty(self):
"""Verify context manager with no clients works correctly."""
with multiprocessing.Manager() as manager:
with ssh.ProxyManager(manager):
pass
def test_context_manager_child_cleanup(self):
"""Verify orphaned clients & masters get cleaned up."""
with multiprocessing.Manager() as manager:
def test_version() -> None:
"""Check version() handling."""
with mock.patch("ssh._run_ssh_version", return_value="OpenSSH_1.2\n"):
assert ssh.version() == (1, 2)
def test_context_manager_empty() -> None:
"""Verify context manager with no clients works correctly."""
with multiprocessing.Manager() as manager:
with ssh.ProxyManager(manager):
pass
def test_context_manager_child_cleanup() -> None:
"""Verify orphaned clients & masters get cleaned up."""
with multiprocessing.Manager() as manager:
with mock.patch("ssh.version", return_value=(1, 2)):
with ssh.ProxyManager(manager) as ssh_proxy:
client = subprocess.Popen(["sleep", "964853320"])
ssh_proxy.add_client(client)
master = subprocess.Popen(["sleep", "964853321"])
ssh_proxy.add_master(master)
# If the process still exists, these will throw timeout errors.
client.wait(0)
master.wait(0)
# If the process still exists, these will throw timeout errors.
client.wait(0)
master.wait(0)
def test_ssh_sock(self):
"""Check sock() function."""
manager = multiprocessing.Manager()
def test_ssh_sock(monkeypatch: pytest.MonkeyPatch) -> None:
"""Check sock() function."""
with multiprocessing.Manager() as manager:
proxy = ssh.ProxyManager(manager)
with mock.patch("tempfile.mkdtemp", return_value="/tmp/foo"):
# Old ssh version uses port.
with mock.patch("ssh.version", return_value=(6, 6)):
self.assertTrue(proxy.sock().endswith("%p"))
monkeypatch.setattr(
"tempfile.mkdtemp", lambda *args, **kwargs: "/tmp/foo"
)
proxy._sock_path = None
# New ssh version uses hash.
with mock.patch("ssh.version", return_value=(6, 7)):
self.assertTrue(proxy.sock().endswith("%C"))
# Old ssh version uses port.
with mock.patch("ssh.version", return_value=(6, 6)):
with proxy as ssh_proxy:
assert ssh_proxy.sock().endswith("%p")
proxy._sock_path = None
# New ssh version uses hash.
with mock.patch("ssh.version", return_value=(6, 7)):
with proxy as ssh_proxy:
assert ssh_proxy.sock().endswith("%C")
proxy._sock_path = None
+128 -87
View File
@@ -15,123 +15,164 @@
"""Unittests for the subcmds module (mostly __init__.py than subcommands)."""
import optparse
import unittest
from typing import Type
import pytest
from command import Command
import subcmds
class AllCommands(unittest.TestCase):
"""Check registered all_commands."""
# NB: We don't test all subcommands as we want to avoid "change detection"
# tests, so we just look for the most common/important ones here that are
# unlikely to ever change.
@pytest.mark.parametrize(
"cmd", ("cherry-pick", "help", "init", "start", "sync", "upload")
)
def test_required_basic(cmd: str) -> None:
"""Basic checking of registered commands."""
assert cmd in subcmds.all_commands
def test_required_basic(self):
"""Basic checking of registered commands."""
# NB: We don't test all subcommands as we want to avoid "change
# detection" tests, so we just look for the most common/important ones
# here that are unlikely to ever change.
for cmd in {"cherry-pick", "help", "init", "start", "sync", "upload"}:
self.assertIn(cmd, subcmds.all_commands)
def test_naming(self):
"""Verify we don't add things that we shouldn't."""
for cmd in subcmds.all_commands:
# Reject filename suffixes like "help.py".
self.assertNotIn(".", cmd)
@pytest.mark.parametrize("name", subcmds.all_commands.keys())
def test_naming(name: str) -> None:
"""Verify we don't add things that we shouldn't."""
# Reject filename suffixes like "help.py".
assert "." not in name
# Make sure all '_' were converted to '-'.
self.assertNotIn("_", cmd)
# Make sure all '_' were converted to '-'.
assert "_" not in name
# Reject internal python paths like "__init__".
self.assertFalse(cmd.startswith("__"))
# Reject internal python paths like "__init__".
assert not name.startswith("__")
def test_help_desc_style(self):
"""Force some consistency in option descriptions.
Python's optparse & argparse has a few default options like --help.
Their option description text uses lowercase sentence fragments, so
enforce our options follow the same style so UI is consistent.
@pytest.mark.parametrize("name, cls", subcmds.all_commands.items())
def test_help_desc_style(name: str, cls: Type[Command]) -> None:
"""Force some consistency in option descriptions.
We enforce:
* Text starts with lowercase.
* Text doesn't end with period.
"""
for name, cls in subcmds.all_commands.items():
cmd = cls()
parser = cmd.OptionParser
for option in parser.option_list:
if option.help == optparse.SUPPRESS_HELP:
continue
Python's optparse & argparse has a few default options like --help.
Their option description text uses lowercase sentence fragments, so
enforce our options follow the same style so UI is consistent.
c = option.help[0]
self.assertEqual(
c.lower(),
c,
msg=f"subcmds/{name}.py: {option.get_opt_string()}: "
f'help text should start with lowercase: "{option.help}"',
)
We enforce:
* Text starts with lowercase.
* Text doesn't end with period.
"""
cmd = cls()
parser = cmd.OptionParser
for option in parser.option_list:
if option.help == optparse.SUPPRESS_HELP or not option.help:
continue
self.assertNotEqual(
option.help[-1],
".",
msg=f"subcmds/{name}.py: {option.get_opt_string()}: "
f'help text should not end in a period: "{option.help}"',
)
c = option.help[0]
assert c.lower() == c, (
f"subcmds/{name}.py: {option.get_opt_string()}: "
f'help text should start with lowercase: "{option.help}"'
)
def test_cli_option_style(self):
"""Force some consistency in option flags."""
for name, cls in subcmds.all_commands.items():
cmd = cls()
parser = cmd.OptionParser
for option in parser.option_list:
for opt in option._long_opts:
self.assertNotIn(
"_",
opt,
msg=f"subcmds/{name}.py: {opt}: only use dashes in "
"options, not underscores",
)
assert option.help[-1] != ".", (
f"subcmds/{name}.py: {option.get_opt_string()}: "
f'help text should not end in a period: "{option.help}"'
)
def test_cli_option_dest(self):
"""Block redundant dest= arguments."""
def _check_dest(opt):
"""Check the dest= setting."""
# If the destination is not set, nothing to check.
# If long options are not set, then there's no implicit destination.
# If callback is used, then a destination might be needed because
# optparse cannot assume a value is always stored.
if opt.dest is None or not opt._long_opts or opt.callback:
return
@pytest.mark.parametrize("name, cls", subcmds.all_commands.items())
def test_cli_option_style(name: str, cls: Type[Command]) -> None:
"""Force some consistency in option flags."""
cmd = cls()
parser = cmd.OptionParser
for option in parser.option_list:
for opt in option._long_opts:
assert "_" not in opt, (
f"subcmds/{name}.py: {opt}: only use dashes in "
"options, not underscores"
)
long = opt._long_opts[0]
assert long.startswith("--")
# This matches optparse's behavior.
implicit_dest = long[2:].replace("-", "_")
if implicit_dest == opt.dest:
bad_opts.append((str(opt), opt.dest))
# Hook the option check list.
optparse.Option.CHECK_METHODS.insert(0, _check_dest)
def test_cli_option_dest() -> None:
"""Block redundant dest= arguments."""
bad_opts: list[tuple[str, str]] = []
def _check_dest(opt: optparse.Option) -> None:
"""Check the dest= setting."""
# If the destination is not set, nothing to check.
# If long options are not set, then there's no implicit destination.
# If callback is used, then a destination might be needed because
# optparse cannot assume a value is always stored.
if opt.dest is None or not opt._long_opts or opt.callback:
return
long = opt._long_opts[0]
assert long.startswith("--")
# This matches optparse's behavior.
implicit_dest = long[2:].replace("-", "_")
if implicit_dest == opt.dest:
bad_opts.append((str(opt), opt.dest))
# Hook the option check list.
optparse.Option.CHECK_METHODS.insert(0, _check_dest)
try:
# Gather all the bad options up front so people can see all bad options
# instead of failing at the first one.
all_bad_opts = {}
all_bad_opts: dict[str, list[tuple[str, str]]] = {}
for name, cls in subcmds.all_commands.items():
bad_opts = all_bad_opts[name] = []
bad_opts = []
cmd = cls()
# Trigger construction of parser.
cmd.OptionParser
_ = cmd.OptionParser
all_bad_opts[name] = bad_opts
errmsg = None
for name, bad_opts in sorted(all_bad_opts.items()):
if bad_opts:
errmsg = ""
for name, bad_opts_list in sorted(all_bad_opts.items()):
if bad_opts_list:
if not errmsg:
errmsg = "Omit redundant dest= when defining options.\n"
errmsg += f"\nSubcommand {name} (subcmds/{name}.py):\n"
errmsg += "".join(
f" {opt}: dest='{dest}'\n" for opt, dest in bad_opts
f" {opt}: dest='{dest}'\n" for opt, dest in bad_opts_list
)
if errmsg:
self.fail(errmsg)
pytest.fail(errmsg)
finally:
# Make sure we aren't popping the wrong stuff.
assert optparse.Option.CHECK_METHODS.pop(0) is _check_dest
@pytest.mark.parametrize("name, cls", subcmds.all_commands.items())
def test_common_validate_options(name: str, cls: Type[Command]) -> None:
"""Verify CommonValidateOptions sets up expected fields."""
cmd = cls()
opts, args = cmd.OptionParser.parse_args([])
# Verify the fields don't exist yet.
assert not hasattr(
opts, "verbose"
), f"{name}: has verbose before validation"
assert not hasattr(opts, "quiet"), f"{name}: has quiet before validation"
cmd.CommonValidateOptions(opts, args)
# Verify the fields exist now.
assert hasattr(opts, "verbose"), f"{name}: missing verbose after validation"
assert hasattr(opts, "quiet"), f"{name}: missing quiet after validation"
assert hasattr(
opts, "outer_manifest"
), f"{name}: missing outer_manifest after validation"
def test_attribute_error_repro() -> None:
"""Confirm that accessing verbose before CommonValidateOptions fails."""
from subcmds.sync import Sync
cmd = Sync()
opts, args = cmd.OptionParser.parse_args([])
# This confirms that without the fix in main.py, an AttributeError
# would be raised because CommonValidateOptions hasn't been called yet.
with pytest.raises(AttributeError):
_ = opts.verbose
cmd.CommonValidateOptions(opts, args)
assert hasattr(opts, "verbose")
+67 -117
View File
@@ -14,143 +14,93 @@
"""Unittests for the forall subcmd."""
from io import StringIO
import os
from shutil import rmtree
import subprocess
import tempfile
import unittest
import contextlib
import io
from pathlib import Path
from unittest import mock
import git_command
import utils_for_test
import manifest_xml
import project
import subcmds
class AllCommands(unittest.TestCase):
"""Check registered all_commands."""
def _create_manifest_with_8_projects(
topdir: Path,
) -> manifest_xml.XmlManifest:
"""Create a setup of 8 projects to execute forall."""
repodir = topdir / ".repo"
manifest_dir = repodir / "manifests"
manifest_file = repodir / manifest_xml.MANIFEST_FILE_NAME
def setUp(self):
"""Common setup."""
self.tempdirobj = tempfile.TemporaryDirectory(prefix="forall_tests")
self.tempdir = self.tempdirobj.name
self.repodir = os.path.join(self.tempdir, ".repo")
self.manifest_dir = os.path.join(self.repodir, "manifests")
self.manifest_file = os.path.join(
self.repodir, manifest_xml.MANIFEST_FILE_NAME
)
self.local_manifest_dir = os.path.join(
self.repodir, manifest_xml.LOCAL_MANIFESTS_DIR_NAME
)
os.mkdir(self.repodir)
os.mkdir(self.manifest_dir)
repodir.mkdir()
manifest_dir.mkdir()
def tearDown(self):
"""Common teardown."""
rmtree(self.tempdir, ignore_errors=True)
# Set up a manifest git dir for parsing to work.
gitdir = repodir / "manifests.git"
gitdir.mkdir()
(gitdir / "config").write_text(
"""[remote "origin"]
url = https://localhost:0/manifest
verbose = false
"""
)
def initTempGitTree(self, git_dir):
"""Create a new empty git checkout for testing."""
# Add the manifest data.
manifest_file.write_text(
"""
<manifest>
<remote name="origin" fetch="http://localhost" />
<default remote="origin" revision="refs/heads/main" />
<project name="project1" path="tests/path1" />
<project name="project2" path="tests/path2" />
<project name="project3" path="tests/path3" />
<project name="project4" path="tests/path4" />
<project name="project5" path="tests/path5" />
<project name="project6" path="tests/path6" />
<project name="project7" path="tests/path7" />
<project name="project8" path="tests/path8" />
</manifest>
""",
encoding="utf-8",
)
# Tests need to assume, that main is default branch at init,
# which is not supported in config until 2.28.
cmd = ["git", "init", "-q"]
if git_command.git_require((2, 28, 0)):
cmd += ["--initial-branch=main"]
else:
# Use template dir for init
templatedir = os.path.join(self.tempdirobj.name, ".test-template")
os.makedirs(templatedir)
with open(os.path.join(templatedir, "HEAD"), "w") as fp:
fp.write("ref: refs/heads/main\n")
cmd += ["--template", templatedir]
cmd += [git_dir]
subprocess.check_call(cmd)
# Set up 8 empty projects to match the manifest.
for x in range(1, 9):
(repodir / "projects" / "tests" / f"path{x}.git").mkdir(parents=True)
(repodir / "project-objects" / f"project{x}.git").mkdir(parents=True)
git_path = topdir / "tests" / f"path{x}"
utils_for_test.init_git_tree(git_path)
def getXmlManifestWith8Projects(self):
"""Create and return a setup of 8 projects with enough dummy
files and setup to execute forall."""
return manifest_xml.XmlManifest(str(repodir), str(manifest_file))
# Set up a manifest git dir for parsing to work
gitdir = os.path.join(self.repodir, "manifests.git")
os.mkdir(gitdir)
with open(os.path.join(gitdir, "config"), "w") as fp:
fp.write(
"""[remote "origin"]
url = https://localhost:0/manifest
verbose = false
"""
)
# Add the manifest data
manifest_data = """
<manifest>
<remote name="origin" fetch="http://localhost" />
<default remote="origin" revision="refs/heads/main" />
<project name="project1" path="tests/path1" />
<project name="project2" path="tests/path2" />
<project name="project3" path="tests/path3" />
<project name="project4" path="tests/path4" />
<project name="project5" path="tests/path5" />
<project name="project6" path="tests/path6" />
<project name="project7" path="tests/path7" />
<project name="project8" path="tests/path8" />
</manifest>
"""
with open(self.manifest_file, "w", encoding="utf-8") as fp:
fp.write(manifest_data)
def test_forall_all_projects_called_once(tmp_path: Path) -> None:
"""Test that all projects get a command run once each."""
manifest = _create_manifest_with_8_projects(tmp_path)
# Set up 8 empty projects to match the manifest
for x in range(1, 9):
os.makedirs(
os.path.join(
self.repodir, "projects/tests/path" + str(x) + ".git"
)
)
os.makedirs(
os.path.join(
self.repodir, "project-objects/project" + str(x) + ".git"
)
)
git_path = os.path.join(self.tempdir, "tests/path" + str(x))
self.initTempGitTree(git_path)
cmd = subcmds.forall.Forall()
cmd.manifest = manifest
return manifest_xml.XmlManifest(self.repodir, self.manifest_file)
# Use echo project names as the test of forall.
opts, args = cmd.OptionParser.parse_args(["-c", "echo $REPO_PROJECT"])
opts.verbose = False
# Use mock to capture stdout from the forall run
@unittest.mock.patch("sys.stdout", new_callable=StringIO)
def test_forall_all_projects_called_once(self, mock_stdout):
"""Test that all projects get a command run once each."""
manifest_with_8_projects = self.getXmlManifestWith8Projects()
cmd = subcmds.forall.Forall()
cmd.manifest = manifest_with_8_projects
# Use echo project names as the test of forall
opts, args = cmd.OptionParser.parse_args(["-c", "echo $REPO_PROJECT"])
opts.verbose = False
# Mock to not have the Execute fail on remote check
with contextlib.redirect_stdout(io.StringIO()) as stdout:
# Mock to not have the Execute fail on remote check.
with mock.patch.object(
project.Project, "GetRevisionId", return_value="refs/heads/main"
):
# Run the forall command
# Run the forall command.
cmd.Execute(opts, args)
# Verify that we got every project name in the prints
for x in range(1, 9):
self.assertIn("project" + str(x), mock_stdout.getvalue())
output = stdout.getvalue()
# Verify that we got every project name in the output.
for x in range(1, 9):
assert f"project{x}" in output
# Split the captured output into lines to count them
line_count = 0
for line in mock_stdout.getvalue().split("\n"):
# A commented out print to stderr as a reminder
# that stdout is mocked, include sys and uncomment if needed
# print(line, file=sys.stderr)
if len(line) > 0:
line_count += 1
# Verify that we didn't get more lines than expected
assert line_count == 8
# Split the captured output into lines to count them.
line_count = sum(1 for x in output.splitlines() if x)
# Verify that we didn't get more lines than expected.
assert line_count == 8
+82
View File
@@ -0,0 +1,82 @@
# Copyright (C) 2026 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Unittests for the subcmds/gc.py module."""
import unittest
from unittest import mock
from subcmds import gc
class GcCommand(unittest.TestCase):
"""Tests for gc command."""
def setUp(self):
self.cmd = gc.Gc()
self.opt, self.args = self.cmd.OptionParser.parse_args([])
self.opt.this_manifest_only = False
self.opt.repack = False
self.mock_get_projects = mock.patch.object(
self.cmd, "GetProjects"
).start()
self.mock_delete = mock.patch.object(
self.cmd, "delete_unused_projects", return_value=0
).start()
self.mock_repack = mock.patch.object(
self.cmd, "repack_projects", return_value=0
).start()
def tearDown(self):
mock.patch.stopall()
def test_gc_no_args(self):
"""Test gc without specific projects."""
self.mock_get_projects.return_value = ["all_projects"]
self.cmd.Execute(self.opt, [])
self.mock_get_projects.assert_called_once_with([], all_manifests=True)
self.mock_delete.assert_called_once_with(["all_projects"], self.opt)
self.mock_repack.assert_not_called()
def test_gc_with_args(self):
"""Test gc with specific projects uses all_projects for delete."""
self.mock_get_projects.side_effect = [["projA"], ["all_projects"]]
self.opt.repack = True
self.cmd.Execute(self.opt, ["projA"])
self.mock_get_projects.assert_has_calls(
[
mock.call(["projA"], all_manifests=True),
mock.call([], all_manifests=True),
]
)
self.mock_delete.assert_called_once_with(["all_projects"], self.opt)
self.mock_repack.assert_called_once_with(["projA"], self.opt)
def test_gc_exit_on_delete_failure(self):
"""Test gc exits if delete_unused_projects fails."""
self.mock_get_projects.return_value = ["all_projects"]
self.mock_delete.return_value = 1
self.opt.repack = True
ret = self.cmd.Execute(self.opt, [])
self.assertEqual(ret, 1)
self.mock_repack.assert_not_called()
+201
View File
@@ -0,0 +1,201 @@
# Copyright (C) 2026 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Unittests for the subcmds/info.py module."""
import json
from unittest import mock
import pytest
from subcmds import info
def _get_cmd() -> info.Info:
"""Build a mock-backed Info command for testing."""
manifest = mock.MagicMock()
manifest.default.revisionExpr = "refs/heads/main"
manifest.manifestProject.config.GetBranch.return_value.merge = (
"refs/heads/main"
)
manifest.GetManifestGroupsStr.return_value = "all"
manifest.superproject = None
manifest.outer_client = manifest
client = mock.MagicMock()
git_event_log = mock.MagicMock()
return info.Info(
manifest=manifest,
client=client,
git_event_log=git_event_log,
)
def test_include_options_default_true() -> None:
"""Both include options should default to True."""
opts, _ = _get_cmd().OptionParser.parse_args([])
assert opts.include_summary
assert opts.include_projects
def test_no_include_summary_parses() -> None:
"""--no-include-summary should set include_summary to False."""
opts, _ = _get_cmd().OptionParser.parse_args(["--no-include-summary"])
assert not opts.include_summary
def test_no_include_projects_parses() -> None:
"""--no-include-projects should set include_projects to False."""
opts, _ = _get_cmd().OptionParser.parse_args(["--no-include-projects"])
assert not opts.include_projects
def test_format_default_text() -> None:
"""Default format should be text."""
opts, _ = _get_cmd().OptionParser.parse_args([])
assert opts.format == "text"
def test_format_json_parses() -> None:
"""--format=json should be accepted."""
opts, _ = _get_cmd().OptionParser.parse_args(["--format=json"])
assert opts.format == "json"
def test_no_include_projects_skips_projects() -> None:
"""--no-include-projects should skip project iteration."""
cmd = _get_cmd()
opts, args = cmd.OptionParser.parse_args(["--no-include-projects"])
with mock.patch.object(
cmd, "_printDiffInfo"
) as mock_diff, mock.patch.object(
cmd, "_printCommitOverview"
) as mock_overview:
cmd.Execute(opts, args)
mock_diff.assert_not_called()
mock_overview.assert_not_called()
def test_no_include_summary_skips_summary() -> None:
"""--no-include-summary should not query or print manifest metadata."""
cmd = _get_cmd()
opts, args = cmd.OptionParser.parse_args(["--no-include-summary"])
with mock.patch.object(cmd, "_printDiffInfo"):
cmd.Execute(opts, args)
cmd.manifest.GetManifestGroupsStr.assert_not_called()
def test_default_calls_diff_info() -> None:
"""Default options should call _printDiffInfo."""
cmd = _get_cmd()
opts, args = cmd.OptionParser.parse_args([])
with mock.patch.object(
cmd, "_printDiffInfo"
) as mock_diff, mock.patch.object(
cmd, "_printCommitOverview"
) as mock_overview:
cmd.Execute(opts, args)
mock_diff.assert_called_once_with(opts, args)
mock_overview.assert_not_called()
def test_overview_calls_commit_overview() -> None:
"""--overview should call _printCommitOverview, not _printDiffInfo."""
cmd = _get_cmd()
opts, args = cmd.OptionParser.parse_args(["--overview"])
with mock.patch.object(
cmd, "_printDiffInfo"
) as mock_diff, mock.patch.object(
cmd, "_printCommitOverview"
) as mock_overview:
cmd.Execute(opts, args)
mock_diff.assert_not_called()
mock_overview.assert_called_once_with(opts, args)
def test_no_include_projects_with_overview() -> None:
"""--no-include-projects should take priority over --overview."""
cmd = _get_cmd()
opts, args = cmd.OptionParser.parse_args(
["--no-include-projects", "--overview"]
)
with mock.patch.object(
cmd, "_printDiffInfo"
) as mock_diff, mock.patch.object(
cmd, "_printCommitOverview"
) as mock_overview:
cmd.Execute(opts, args)
mock_diff.assert_not_called()
mock_overview.assert_not_called()
def test_json_summary_only(capsys) -> None:
"""--format=json --no-include-projects should emit only summary."""
cmd = _get_cmd()
opts, args = cmd.OptionParser.parse_args(
["--format=json", "--no-include-projects"]
)
cmd.Execute(opts, args)
data = json.loads(capsys.readouterr().out)
assert "summary" in data
assert "projects" not in data
assert data["summary"]["manifest_branch"] == "refs/heads/main"
assert data["summary"]["manifest_groups"] == "all"
def test_json_no_summary(capsys) -> None:
"""--format=json --no-include-summary should omit summary."""
cmd = _get_cmd()
opts, args = cmd.OptionParser.parse_args(
["--format=json", "--no-include-summary", "--no-include-projects"]
)
cmd.Execute(opts, args)
data = json.loads(capsys.readouterr().out)
assert "summary" not in data
def test_json_rejects_diff() -> None:
"""--format=json --diff should be rejected."""
cmd = _get_cmd()
opts, args = cmd.OptionParser.parse_args(["--format=json", "--diff"])
with pytest.raises(SystemExit):
cmd.ValidateOptions(opts, args)
def test_json_rejects_overview() -> None:
"""--format=json --overview should be rejected."""
cmd = _get_cmd()
opts, args = cmd.OptionParser.parse_args(["--format=json", "--overview"])
with pytest.raises(SystemExit):
cmd.ValidateOptions(opts, args)
def test_json_disables_pager() -> None:
"""--format=json should disable the pager."""
cmd = _get_cmd()
opts, _ = cmd.OptionParser.parse_args(["--format=json"])
assert not cmd.WantPager(opts)
def test_text_enables_pager() -> None:
"""Default text format should enable the pager."""
cmd = _get_cmd()
opts, _ = cmd.OptionParser.parse_args([])
assert cmd.WantPager(opts)
+27 -24
View File
@@ -14,33 +14,36 @@
"""Unittests for the subcmds/init.py module."""
import unittest
from typing import List
import pytest
from subcmds import init
class InitCommand(unittest.TestCase):
"""Check registered all_commands."""
@pytest.mark.parametrize(
"argv",
([],),
)
def test_cli_parser_good(argv: List[str]) -> None:
"""Check valid command line options."""
cmd = init.Init()
opts, args = cmd.OptionParser.parse_args(argv)
cmd.ValidateOptions(opts, args)
def setUp(self):
self.cmd = init.Init()
def test_cli_parser_good(self):
"""Check valid command line options."""
ARGV = ([],)
for argv in ARGV:
opts, args = self.cmd.OptionParser.parse_args(argv)
self.cmd.ValidateOptions(opts, args)
def test_cli_parser_bad(self):
"""Check invalid command line options."""
ARGV = (
# Too many arguments.
["url", "asdf"],
# Conflicting options.
["--mirror", "--archive"],
)
for argv in ARGV:
opts, args = self.cmd.OptionParser.parse_args(argv)
with self.assertRaises(SystemExit):
self.cmd.ValidateOptions(opts, args)
@pytest.mark.parametrize(
"argv",
(
# Too many arguments.
["url", "asdf"],
# Conflicting options.
["--mirror", "--archive"],
),
)
def test_cli_parser_bad(argv: List[str]) -> None:
"""Check invalid command line options."""
cmd = init.Init()
opts, args = cmd.OptionParser.parse_args(argv)
with pytest.raises(SystemExit):
cmd.ValidateOptions(opts, args)
+220
View File
@@ -0,0 +1,220 @@
# Copyright (C) 2026 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Unittests for the status subcmd."""
import contextlib
import io
import os
from pathlib import Path
import subprocess
from typing import List, Tuple
from unittest import mock
import pytest
import utils_for_test
import manifest_xml
import subcmds
@pytest.fixture
def repo_client_checkout(
tmp_path: Path,
) -> Tuple[Path, manifest_xml.XmlManifest]:
"""Create a basic repo client checkout for status tests."""
# Create in a subdir to avoid noise (like the repo_trace file).
topdir = tmp_path / "client_checkout"
repodir = topdir / ".repo"
manifest_dir = repodir / "manifests"
manifest_file = repodir / manifest_xml.MANIFEST_FILE_NAME
repodir.mkdir(parents=True)
manifest_dir.mkdir()
gitdir = repodir / "manifests.git"
gitdir.mkdir()
(gitdir / "config").write_text(
"""[remote "origin"]
url = https://localhost:0/manifest
verbose = false
"""
)
_init_temp_git_tree(manifest_dir)
manifest_file.write_text(
"""
<manifest>
<remote name="origin" fetch="http://localhost" />
<default remote="origin" revision="refs/heads/main" />
<project name="proj" path="src/proj" />
</manifest>
""",
encoding="utf-8",
)
(repodir / "projects" / "src" / "proj.git").mkdir(parents=True)
(repodir / "project-objects" / "proj.git").mkdir(parents=True)
worktree = topdir / "src" / "proj"
worktree.parent.mkdir(parents=True, exist_ok=True)
_init_temp_git_tree(worktree)
manifest = manifest_xml.XmlManifest(str(repodir), str(manifest_file))
return topdir, manifest
def _init_temp_git_tree(git_dir: Path) -> None:
"""Create a new git checkout with an initial commit for testing."""
utils_for_test.init_git_tree(git_dir)
(git_dir / "README").write_text("init")
subprocess.check_call(["git", "add", "README"], cwd=git_dir)
subprocess.check_call(["git", "commit", "-q", "-m", "init"], cwd=git_dir)
def _run_status(manifest: manifest_xml.XmlManifest, argv: List[str]) -> None:
"""Run the status subcommand with parsed options against a test manifest."""
cmd = subcmds.status.Status()
cmd.manifest = manifest
cmd.client = mock.MagicMock(globalConfig=manifest.globalConfig)
opts, args = cmd.OptionParser.parse_args(argv + ["--jobs=1"])
cmd.CommonValidateOptions(opts, args)
cmd.Execute(opts, args)
def _status_lines(output: str) -> List[str]:
"""Normalize path separators and split command output into lines."""
return output.replace(os.sep, "/").splitlines()
def _assert_project_header(line: str, project_path: str, branch: str) -> None:
"""Assert a status project header line for a project and branch."""
expected = f"project {(project_path + '/ '):<40}branch {branch}"
assert line == expected
def _assert_orphan_block(lines: List[str], expected: List[str]) -> None:
"""Assert orphan block header and entries, independent of entry ordering."""
assert lines
assert lines[0] == ("Objects not within a project (orphans)")
orphan_lines = lines[1:]
assert len(orphan_lines) == len(expected)
assert sorted(orphan_lines) == sorted(expected)
def test_orphans_basic(
repo_client_checkout: Tuple[Path, manifest_xml.XmlManifest],
) -> None:
"""Verify -o output includes project header and orphan block."""
topdir, manifest = repo_client_checkout
project_path = next(iter(manifest.paths.keys()))
(topdir / "src" / "orphan_dir").mkdir(parents=True)
(topdir / "orphan.txt").write_text("data")
with contextlib.redirect_stdout(io.StringIO()) as stdout:
_run_status(manifest, ["-o"])
lines = _status_lines(stdout.getvalue())
_assert_project_header(lines[0], project_path, "main")
_assert_orphan_block(
lines[1:],
[
" --\torphan.txt",
" --\tsrc/orphan_dir/",
],
)
def test_empty_status_without_orphans(
repo_client_checkout: Tuple[Path, manifest_xml.XmlManifest],
) -> None:
"""Verify clean status without -o prints only the project header line."""
_, manifest = repo_client_checkout
project_path = next(iter(manifest.paths.keys()))
with contextlib.redirect_stdout(io.StringIO()) as stdout:
_run_status(manifest, [])
lines = _status_lines(stdout.getvalue())
assert len(lines) == 1
_assert_project_header(lines[0], project_path, "main")
def test_status_without_orphans(
repo_client_checkout: Tuple[Path, manifest_xml.XmlManifest],
) -> None:
"""Verify modified tracked file appears in status output without -o."""
topdir, manifest = repo_client_checkout
project_path = next(iter(manifest.paths.keys()))
(topdir / project_path / "README").write_text("updated")
with contextlib.redirect_stdout(io.StringIO()) as stdout:
_run_status(manifest, [])
lines = _status_lines(stdout.getvalue())
assert len(lines) == 2
_assert_project_header(lines[0], project_path, "main")
assert lines[1] == " -m\tREADME"
def test_status_with_orphans_and_modified_file(
repo_client_checkout: Tuple[Path, manifest_xml.XmlManifest],
) -> None:
"""Verify modified-file status plus orphan block."""
topdir, manifest = repo_client_checkout
project_path = next(iter(manifest.paths.keys()))
(topdir / project_path / "README").write_text("updated")
(topdir / "src" / "orphan_dir").mkdir(parents=True)
(topdir / "orphan.txt").write_text("data")
with contextlib.redirect_stdout(io.StringIO()) as stdout:
_run_status(manifest, ["-o"])
lines = _status_lines(stdout.getvalue())
_assert_project_header(lines[0], project_path, "main")
assert lines[1] == " -m\tREADME"
_assert_orphan_block(
lines[2:],
[
" --\torphan.txt",
" --\tsrc/orphan_dir/",
],
)
def test_empty_status_after_start_shows_started_branch(
repo_client_checkout: Tuple[Path, manifest_xml.XmlManifest],
) -> None:
"""Verify status shows the started branch name when the tree is clean."""
topdir, manifest = repo_client_checkout
project_path = next(iter(manifest.paths.keys()))
project_worktree = topdir / project_path
started_branch = "topic/test-status-branch"
subprocess.check_call(
["git", "checkout", "-q", "-b", started_branch], cwd=project_worktree
)
with contextlib.redirect_stdout(io.StringIO()) as stdout:
_run_status(manifest, [])
lines = _status_lines(stdout.getvalue())
assert len(lines) == 1
_assert_project_header(lines[0], project_path, started_branch)
+513
View File
@@ -13,6 +13,7 @@
# limitations under the License.
"""Unittests for the subcmds/sync.py module."""
import json
import os
import shutil
import tempfile
@@ -97,6 +98,35 @@ def test_cli_jobs(argv, jobs_manifest, jobs, jobs_net, jobs_check):
"""Tests --jobs option behavior."""
mp = mock.MagicMock()
mp.manifest.default.sync_j = jobs_manifest
mp.manifest.default.sync_j_max = None
cmd = sync.Sync()
opts, args = cmd.OptionParser.parse_args(argv)
cmd.ValidateOptions(opts, args)
with mock.patch.object(sync, "_rlimit_nofile", return_value=(256, 256)):
with mock.patch.object(os, "cpu_count", return_value=OS_CPU_COUNT):
cmd._ValidateOptionsWithManifest(opts, mp)
assert opts.jobs == jobs
assert opts.jobs_network == jobs_net
assert opts.jobs_checkout == jobs_check
@pytest.mark.parametrize(
"argv, jobs_manifest, jobs_manifest_max, jobs, jobs_net, jobs_check",
[
(["--jobs=10"], None, 5, 5, 5, 5),
(["--jobs=10", "--jobs-network=10"], None, 5, 5, 5, 5),
(["--jobs=10", "--jobs-checkout=10"], None, 5, 5, 5, 5),
],
)
def test_cli_jobs_sync_j_max(
argv, jobs_manifest, jobs_manifest_max, jobs, jobs_net, jobs_check
):
"""Tests --jobs option behavior with sync-j-max."""
mp = mock.MagicMock()
mp.manifest.default.sync_j = jobs_manifest
mp.manifest.default.sync_j_max = jobs_manifest_max
cmd = sync.Sync()
opts, args = cmd.OptionParser.parse_args(argv)
@@ -448,6 +478,181 @@ class GetPreciousObjectsState(unittest.TestCase):
)
class KeyboardInterruptTest(unittest.TestCase):
"""Tests for KeyboardInterrupt handling in Sync operations."""
def setUp(self):
self.project = mock.MagicMock(name="project")
self.project.name = "project"
self.project.relpath = "proj"
self.project.manifest.IsArchive = False
self.opt = mock.Mock()
self.opt.quiet = True
self.opt.verbose = False
self.opt.tags = False
self.sync_dict = {}
self.get_parallel_context_mock = {
"projects": [self.project],
"sync_dict": self.sync_dict,
"ssh_proxy": None,
}
@mock.patch("subcmds.sync.Sync.is_multiprocessing_active")
def test_fetch_one_keyboard_interrupt_main_process(self, mock_is_active):
"""Test that _FetchOne re-raises KeyboardInterrupt if not worker."""
mock_is_active.return_value = False
self.project.Sync_NetworkHalf.side_effect = KeyboardInterrupt()
with mock.patch.object(
sync.Sync,
"get_parallel_context",
return_value=self.get_parallel_context_mock,
):
with self.assertRaises(KeyboardInterrupt):
sync.Sync._FetchOne(self.opt, 0)
@mock.patch("subcmds.sync.Sync.is_multiprocessing_active")
def test_fetch_one_keyboard_interrupt_worker_process(self, mock_is_active):
"""Test that _FetchOne suppresses KeyboardInterrupt in workers."""
mock_is_active.return_value = True
self.project.Sync_NetworkHalf.side_effect = KeyboardInterrupt()
with mock.patch.object(
sync.Sync,
"get_parallel_context",
return_value=self.get_parallel_context_mock,
):
result = sync.Sync._FetchOne(self.opt, 0)
self.assertFalse(result.success)
class CheckForBloatedProjects(unittest.TestCase):
"""Tests for Sync._CheckForBloatedProjects."""
def setUp(self):
self.cmd = sync.Sync()
self.opt = mock.Mock()
self.opt.quiet = True
self.opt.jobs = 1
self.project = mock.MagicMock(clone_depth="1")
self.project.name = "project"
self.project.Exists = True
self.project.worktree = "worktree"
self.project.stateless_prune_needed = False
self.cmd.git_event_log = mock.MagicMock()
self.cmd._bloated_projects = []
@mock.patch("subcmds.sync.git_require")
def test_git_version_unsupported(self, mock_git_require):
"""Test that it returns early if git version is unsupported."""
mock_git_require.return_value = False
self.cmd._CheckForBloatedProjects([self.project], self.opt)
self.assertFalse(self.cmd.git_event_log.ErrorEvent.called)
@mock.patch("subcmds.sync.git_require")
def test_no_projects(self, mock_git_require):
"""Test that it returns early if no projects have clone_depth."""
mock_git_require.return_value = True
self.project.clone_depth = None
self.cmd._CheckForBloatedProjects([self.project], self.opt)
self.assertFalse(self.cmd.git_event_log.ErrorEvent.called)
@mock.patch("subcmds.sync.git_require")
@mock.patch("subcmds.sync.Progress")
def test_bloated_project_found(self, mock_progress, mock_git_require):
"""Test that it adds project to _bloated_projects."""
mock_git_require.return_value = True
self.cmd.get_parallel_context = mock.Mock(
return_value={"projects": [self.project]}
)
def mock_execute_in_parallel(
jobs, func, work_items, callback, **kwargs
):
callback(None, mock.Mock(), ["project"])
return True
self.cmd.ExecuteInParallel = mock_execute_in_parallel
with mock.patch.object(self.cmd, "ParallelContext"):
self.cmd._CheckForBloatedProjects([self.project], self.opt)
self.assertEqual(self.cmd._bloated_projects, ["project"])
@mock.patch("subcmds.sync.git_require")
@mock.patch("subcmds.sync.Progress")
def test_stateless_prune_excluded(self, mock_progress, mock_git_require):
"""Test that projects pruned for stateless sync are excluded."""
mock_git_require.return_value = True
self.project.stateless_prune_needed = True
self.cmd.ExecuteInParallel = mock.Mock()
with mock.patch.object(self.cmd, "ParallelContext"):
self.cmd._CheckForBloatedProjects([self.project], self.opt)
self.assertFalse(self.cmd.ExecuteInParallel.called)
self.assertEqual(self.cmd._bloated_projects, [])
class GCProjectsTest(unittest.TestCase):
"""Tests for Sync._GCProjects."""
def setUp(self):
self.cmd = sync.Sync()
self.opt = mock.Mock()
self.opt.quiet = True
self.opt.auto_gc = True
self.opt.jobs = 1
self.project = mock.MagicMock()
self.project.name = "project"
self.project.objdir = "objdir"
self.project.gitdir = "gitdir"
self.project.bare_git = mock.MagicMock()
self.project.bare_git._project = self.project
self.cmd.git_event_log = mock.MagicMock()
@mock.patch("subcmds.sync.Progress")
def test_GCProjects_skip_gc(self, mock_progress):
"""Test that it skips GC if opt.auto_gc is False."""
self.opt.auto_gc = False
with mock.patch.object(
sync.Sync, "_SetPreciousObjectsState"
) as mock_set_state:
self.cmd._GCProjects([self.project], self.opt, None)
mock_set_state.assert_called_once_with(self.project, self.opt)
self.assertFalse(self.project.bare_git.gc.called)
@mock.patch("subcmds.sync.Progress")
def test_GCProjects_sequential(self, mock_progress):
"""Test sequential GC (jobs < 2)."""
with mock.patch.object(sync.Sync, "_SetPreciousObjectsState"):
self.cmd._GCProjects([self.project], self.opt, None)
self.project.bare_git.gc.assert_called_once_with(
"--auto", config={"gc.autoDetach": "false"}
)
# Verify that gc.autoDetach was not permanently set in config.
for call in self.project.config.SetString.call_args_list:
self.assertNotEqual(call.args[0], "gc.autoDetach")
@mock.patch("subcmds.sync.Progress")
def test_GCProjects_parallel(self, mock_progress):
"""Test parallel GC (jobs >= 2)."""
self.opt.jobs = 2
with mock.patch.object(sync.Sync, "_SetPreciousObjectsState"):
with mock.patch("subcmds.sync._threading.Thread") as mock_thread:
mock_t = mock.MagicMock()
mock_thread.return_value = mock_t
err_event = mock.Mock()
err_event.is_set.return_value = False
self.cmd._GCProjects([self.project], self.opt, err_event)
self.assertTrue(mock_thread.called)
class SyncCommand(unittest.TestCase):
"""Tests for cmd.Execute."""
@@ -941,3 +1146,311 @@ class InterleavedSyncTest(unittest.TestCase):
self.assertTrue(result.checkout_success)
project.Sync_NetworkHalf.assert_called_once()
project.Sync_LocalHalf.assert_not_called()
class UpdateCopyLinkfileListTest(unittest.TestCase):
"""Tests for Sync.UpdateCopyLinkfileList."""
def setUp(self):
self.tempdirobj = tempfile.TemporaryDirectory(prefix="repo_tests")
self.topdir = self.tempdirobj.name
self.repodir = os.path.join(self.topdir, ".repo")
os.makedirs(self.repodir)
manifest = mock.MagicMock()
manifest.subdir = self.repodir
self.manifest = manifest
git_event_log = mock.MagicMock(ErrorEvent=mock.Mock(return_value=None))
self.cmd = sync.Sync(
manifest=manifest,
outer_client=mock.MagicMock(),
git_event_log=git_event_log,
)
self.cmd.client = mock.MagicMock(topdir=self.topdir)
def tearDown(self):
self.tempdirobj.cleanup()
def _write_copylinkfile_json(self, data: dict) -> None:
path = os.path.join(self.repodir, "copy-link-files.json")
with open(path, "w") as f:
json.dump(data, f)
def _setup_projects(self, linkfile_dests: list) -> None:
project = mock.MagicMock()
project.linkfiles = [mock.MagicMock(dest=d) for d in linkfile_dests]
project.copyfiles = []
mock.patch.object(
self.cmd, "GetProjects", return_value=[project]
).start()
def test_removes_old_symlink_dest(self):
"""Old linkfile dests that are symlinks should be removed."""
old_dest = os.path.join(self.topdir, "old-link")
os.symlink("target", old_dest)
self._write_copylinkfile_json(
{"linkfile": ["old-link"], "copyfile": []}
)
self._setup_projects([])
self.cmd.UpdateCopyLinkfileList(self.manifest)
self.assertFalse(os.path.lexists(old_dest))
def test_does_not_delete_through_new_symlink(self):
"""Old dests that resolve through a new symlink must not delete files.
When the manifest changes from individual linkfiles inside a directory
to a single directory linkfile, and _CopyAndLinkFiles has already
created the symlink (interleaved mode), cleanup must not follow the
symlink and delete real project files.
"""
project_dir = os.path.join(self.topdir, "vendor", "tools", "llms")
os.makedirs(os.path.join(project_dir, "dot-llms", "rules"))
os.makedirs(os.path.join(project_dir, "dot-llms", "skills"))
with open(
os.path.join(project_dir, "dot-llms", "rules", "basics.md"), "w"
) as f:
f.write("# basics")
with open(
os.path.join(project_dir, "dot-llms", "skills", "repo.md"), "w"
) as f:
f.write("# repo")
# Simulate interleaved mode: .llms -> vendor/tools/llms/dot-llms.
llms_link = os.path.join(self.topdir, ".llms")
os.symlink("vendor/tools/llms/dot-llms", llms_link)
self._write_copylinkfile_json(
{"linkfile": [".llms/rules", ".llms/skills"], "copyfile": []}
)
self._setup_projects([".llms"])
self.cmd.UpdateCopyLinkfileList(self.manifest)
# Real project files must still exist.
self.assertTrue(
os.path.exists(
os.path.join(project_dir, "dot-llms", "rules", "basics.md")
)
)
self.assertTrue(
os.path.exists(
os.path.join(project_dir, "dot-llms", "skills", "repo.md")
),
)
self.assertTrue(os.path.islink(llms_link))
def test_cleans_up_empty_parent_dirs(self):
"""After removing old dests, empty parent directories are removed."""
llms_dir = os.path.join(self.topdir, ".llms")
os.makedirs(llms_dir)
os.symlink(
"../vendor/tools/llms/rules", os.path.join(llms_dir, "rules")
)
os.symlink(
"../vendor/tools/llms/skills",
os.path.join(llms_dir, "skills"),
)
self._write_copylinkfile_json(
{"linkfile": [".llms/rules", ".llms/skills"], "copyfile": []}
)
self._setup_projects([".llms"])
self.cmd.UpdateCopyLinkfileList(self.manifest)
self.assertFalse(os.path.lexists(os.path.join(llms_dir, "rules")))
self.assertFalse(os.path.lexists(os.path.join(llms_dir, "skills")))
# Parent directory should be removed since it's now empty.
self.assertFalse(os.path.exists(llms_dir))
def test_preserves_nonempty_parent_dirs(self):
"""Non-empty parent directories are preserved after old dest removal."""
llms_dir = os.path.join(self.topdir, ".llms")
os.makedirs(llms_dir)
os.symlink(
"../vendor/tools/llms/rules", os.path.join(llms_dir, "rules")
)
with open(os.path.join(llms_dir, "my-notes.txt"), "w") as f:
f.write("user content")
self._write_copylinkfile_json(
{"linkfile": [".llms/rules"], "copyfile": []}
)
self._setup_projects([".llms"])
self.cmd.UpdateCopyLinkfileList(self.manifest)
self.assertFalse(os.path.lexists(os.path.join(llms_dir, "rules")))
self.assertTrue(os.path.exists(os.path.join(llms_dir, "my-notes.txt")))
self.assertTrue(os.path.isdir(llms_dir))
class SyncToSuperprojectRevTests(unittest.TestCase):
"""Tests for Sync._SyncToSuperprojectRev."""
def setUp(self):
self.repodir = tempfile.mkdtemp(".repo")
self.manifest = mock.MagicMock(repodir=self.repodir)
self.manifest.superproject = mock.MagicMock()
self.manifest.path_prefix = ""
self.mp = mock.MagicMock()
self.cmd = sync.Sync(manifest=self.manifest)
self.cmd.outer_manifest = self.manifest
self.opt = mock.Mock()
self.opt.verbose = False
self.opt.superproject_revision = "deadbeef"
self.opt.mp_update = True
self.errors = []
def tearDown(self):
shutil.rmtree(self.repodir)
@mock.patch("subcmds.sync.GitCommand")
def test_successful_sync(self, mock_git_command):
"""Test successful sync to superproject rev."""
mock_superproject = self.manifest.superproject
mock_superproject.Sync.return_value = mock.Mock(success=True)
mock_git = mock.Mock()
mock_git.Wait.return_value = 0
mock_git.stdout = "proj branch manifest_commit_hash\n"
mock_git_command.return_value = mock_git
with mock.patch.object(
self.cmd, "_UpdateManifestProject"
) as mock_update:
self.cmd._SyncToSuperprojectRev(
self.opt, self.manifest, self.mp, "name", self.errors
)
mock_superproject.SetRevisionId.assert_called_with("deadbeef")
mock_superproject.Sync.assert_called_once()
mock_git_command.assert_called_once()
self.mp.SetRevision.assert_called_with("manifest_commit_hash")
mock_update.assert_called_once()
self.assertEqual(self.errors, [])
@mock.patch("subcmds.sync.GitCommand")
def test_parse_error(self, mock_git_command):
"""Test error when .supermanifest cannot be parsed."""
mock_superproject = self.manifest.superproject
mock_superproject.Sync.return_value = mock.Mock(success=True)
mock_git = mock.Mock()
mock_git.Wait.return_value = 0
# Invalid format (not 3 parts)
mock_git.stdout = "invalid_content\n"
mock_git_command.return_value = mock_git
with self.assertRaises(sync.SyncError) as e:
self.cmd._SyncToSuperprojectRev(
self.opt, self.manifest, self.mp, "name", self.errors
)
self.assertIn("could not parse .supermanifest", str(e.exception))
@mock.patch("subcmds.sync.GitCommand")
def test_read_error(self, mock_git_command):
"""Test error when reading .supermanifest fails."""
mock_superproject = self.manifest.superproject
mock_superproject.Sync.return_value = mock.Mock(success=True)
mock_git = mock.Mock()
mock_git.Wait.return_value = 1
mock_git.stderr = "git error"
mock_git_command.return_value = mock_git
with self.assertRaises(sync.SyncError) as e:
self.cmd._SyncToSuperprojectRev(
self.opt, self.manifest, self.mp, "name", self.errors
)
self.assertIn("failed to read .supermanifest", str(e.exception))
def test_no_superproject(self):
"""Test error when superproject is not defined."""
self.manifest.superproject = None
with self.assertRaises(sync.SyncError) as e:
self.cmd._SyncToSuperprojectRev(
self.opt, self.manifest, self.mp, "name", self.errors
)
self.assertIn("superproject not defined", str(e.exception))
@mock.patch("subcmds.sync.GitCommand")
def test_sync_failure(self, mock_git_command):
"""Test error when superproject sync fails."""
mock_superproject = self.manifest.superproject
mock_superproject.Sync.return_value = mock.Mock(success=False)
with self.assertRaises(sync.SyncError) as e:
self.cmd._SyncToSuperprojectRev(
self.opt, self.manifest, self.mp, "name", self.errors
)
self.assertIn("failed to sync superproject", str(e.exception))
class UpdateAllManifestProjectsTests(unittest.TestCase):
"""Tests for Sync._UpdateAllManifestProjects."""
def setUp(self):
self.repodir = tempfile.mkdtemp(".repo")
self.manifest = mock.MagicMock(repodir=self.repodir)
self.manifest.superproject = mock.MagicMock()
self.manifest.path_prefix = ""
self.manifest.standalone_manifest_url = None
self.manifest.submanifests = {}
self.mp = mock.MagicMock()
self.mp.manifest = self.manifest
self.mp.standalone_manifest_url = None
self.cmd = sync.Sync(manifest=self.manifest)
self.cmd.outer_manifest = self.manifest
self.opt = mock.Mock()
self.opt.verbose = False
self.opt.superproject_revision = None
self.opt.mp_update = True
self.errors = []
def tearDown(self):
shutil.rmtree(self.repodir)
def test_superproject_revision_outer_manifest(self):
"""Test that _SyncToSuperprojectRev is called for outer manifest."""
self.opt.superproject_revision = "deadbeef"
with mock.patch.object(
self.cmd, "_SyncToSuperprojectRev"
) as mock_sync_to_rev:
self.cmd._UpdateAllManifestProjects(
self.opt, self.mp, "name", self.errors
)
mock_sync_to_rev.assert_called_once_with(
self.opt, self.manifest, self.mp, "name", self.errors
)
def test_superproject_revision_submanifest(self):
"""Test that _SyncToSuperprojectRev is NOT called for submanifest."""
self.opt.superproject_revision = "deadbeef"
submanifest = mock.MagicMock()
submanifest.path_prefix = "sub/"
submanifest.standalone_manifest_url = None
self.mp.manifest = submanifest
with mock.patch.object(
self.cmd, "_SyncToSuperprojectRev"
) as mock_sync_to_rev:
with mock.patch.object(
self.cmd, "_UpdateManifestProject"
) as mock_update_manifest:
self.cmd._UpdateAllManifestProjects(
self.opt, self.mp, "name", self.errors
)
mock_sync_to_rev.assert_not_called()
mock_update_manifest.assert_called_once()
+33 -38
View File
@@ -14,9 +14,10 @@
"""Unittests for the subcmds/upload.py module."""
import unittest
from unittest import mock
import pytest
from error import GitError
from error import UploadError
from subcmds import upload
@@ -26,45 +27,39 @@ class UnexpectedError(Exception):
"""An exception not expected by upload command."""
class UploadCommand(unittest.TestCase):
"""Check registered all_commands."""
# A stub people list (reviewers, cc).
_STUB_PEOPLE = ([], [])
def setUp(self):
self.cmd = upload.Upload()
self.branch = mock.MagicMock()
self.people = mock.MagicMock()
self.opt, _ = self.cmd.OptionParser.parse_args([])
mock.patch.object(
self.cmd, "_AppendAutoList", return_value=None
).start()
mock.patch.object(self.cmd, "git_event_log").start()
def tearDown(self):
mock.patch.stopall()
@pytest.fixture
def cmd() -> upload.Upload:
"""Fixture to provide an Upload command instance with mocked methods."""
cmd = upload.Upload()
with mock.patch.object(
cmd, "_AppendAutoList", return_value=None
), mock.patch.object(cmd, "git_event_log"):
yield cmd
def test_UploadAndReport_UploadError(self):
"""Check UploadExitError raised when UploadError encountered."""
side_effect = UploadError("upload error")
with mock.patch.object(
self.cmd, "_UploadBranch", side_effect=side_effect
):
with self.assertRaises(upload.UploadExitError):
self.cmd._UploadAndReport(self.opt, [self.branch], self.people)
def test_UploadAndReport_GitError(self):
"""Check UploadExitError raised when GitError encountered."""
side_effect = GitError("some git error")
with mock.patch.object(
self.cmd, "_UploadBranch", side_effect=side_effect
):
with self.assertRaises(upload.UploadExitError):
self.cmd._UploadAndReport(self.opt, [self.branch], self.people)
def test_UploadAndReport_UploadError(cmd: upload.Upload) -> None:
"""Check UploadExitError raised when UploadError encountered."""
opt, _ = cmd.OptionParser.parse_args([])
with mock.patch.object(cmd, "_UploadBranch", side_effect=UploadError("")):
with pytest.raises(upload.UploadExitError):
cmd._UploadAndReport(opt, [mock.MagicMock()], _STUB_PEOPLE)
def test_UploadAndReport_UnhandledError(self):
"""Check UnexpectedError passed through."""
side_effect = UnexpectedError("some os error")
with mock.patch.object(
self.cmd, "_UploadBranch", side_effect=side_effect
):
with self.assertRaises(type(side_effect)):
self.cmd._UploadAndReport(self.opt, [self.branch], self.people)
def test_UploadAndReport_GitError(cmd: upload.Upload) -> None:
"""Check UploadExitError raised when GitError encountered."""
opt, _ = cmd.OptionParser.parse_args([])
with mock.patch.object(cmd, "_UploadBranch", side_effect=GitError("")):
with pytest.raises(upload.UploadExitError):
cmd._UploadAndReport(opt, [mock.MagicMock()], _STUB_PEOPLE)
def test_UploadAndReport_UnhandledError(cmd: upload.Upload) -> None:
"""Check UnexpectedError passed through."""
opt, _ = cmd.OptionParser.parse_args([])
with mock.patch.object(cmd, "_UploadBranch", side_effect=UnexpectedError):
with pytest.raises(UnexpectedError):
cmd._UploadAndReport(opt, [mock.MagicMock()], _STUB_PEOPLE)
-28
View File
@@ -1,28 +0,0 @@
# Copyright (C) 2022 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Unittests for the update_manpages module."""
import unittest
from release import update_manpages
class UpdateManpagesTest(unittest.TestCase):
"""Tests the update-manpages code."""
def test_replace_regex(self):
"""Check that replace_regex works."""
data = "\n\033[1mSummary\033[m\n"
self.assertEqual(update_manpages.replace_regex(data), "\nSummary\n")
+345 -314
View File
@@ -19,266 +19,303 @@ import os
import re
import subprocess
import sys
import tempfile
import unittest
from unittest import mock
import git_command
import pytest
import utils_for_test
import main
import wrapper
def fixture(*paths):
"""Return a path relative to tests/fixtures."""
return os.path.join(os.path.dirname(__file__), "fixtures", *paths)
@pytest.fixture(autouse=True)
def reset_wrapper() -> None:
"""Reset the wrapper module every time."""
wrapper.Wrapper.cache_clear()
class RepoWrapperTestCase(unittest.TestCase):
"""TestCase for the wrapper module."""
def setUp(self):
"""Load the wrapper module every time."""
wrapper.Wrapper.cache_clear()
self.wrapper = wrapper.Wrapper()
@pytest.fixture
def repo_wrapper() -> wrapper.Wrapper:
"""Fixture for the wrapper module."""
return wrapper.Wrapper()
class RepoWrapperUnitTest(RepoWrapperTestCase):
class GitCheckout:
"""Class to hold git checkout info for tests."""
def __init__(self, git_dir, rev_list):
self.git_dir = git_dir
self.rev_list = rev_list
@pytest.fixture(scope="module")
def git_checkout(tmp_path_factory) -> GitCheckout:
"""Fixture for tests that use a real/small git checkout.
Create a repo to operate on, but do it once per-test-run.
"""
tempdir = tmp_path_factory.mktemp("repo-rev-tests")
run_git = wrapper.Wrapper().run_git
remote = os.path.join(tempdir, "remote")
os.mkdir(remote)
utils_for_test.init_git_tree(remote)
run_git("commit", "--allow-empty", "-minit", cwd=remote)
run_git("branch", "stable", cwd=remote)
run_git("tag", "v1.0", cwd=remote)
run_git("commit", "--allow-empty", "-m2nd commit", cwd=remote)
rev_list = run_git("rev-list", "HEAD", cwd=remote).stdout.splitlines()
run_git("init", cwd=tempdir)
run_git(
"fetch",
remote,
"+refs/heads/*:refs/remotes/origin/*",
cwd=tempdir,
)
yield GitCheckout(tempdir, rev_list)
class TestRepoWrapper:
"""Tests helper functions in the repo wrapper"""
def test_version(self):
def test_version(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Make sure _Version works."""
with self.assertRaises(SystemExit) as e:
with pytest.raises(SystemExit) as e:
with mock.patch("sys.stdout", new_callable=io.StringIO) as stdout:
with mock.patch(
"sys.stderr", new_callable=io.StringIO
) as stderr:
self.wrapper._Version()
self.assertEqual(0, e.exception.code)
self.assertEqual("", stderr.getvalue())
self.assertIn("repo launcher version", stdout.getvalue())
repo_wrapper._Version()
assert e.value.code == 0
assert stderr.getvalue() == ""
assert "repo launcher version" in stdout.getvalue()
def test_python_constraints(self):
def test_python_constraints(self, repo_wrapper: wrapper.Wrapper) -> None:
"""The launcher should never require newer than main.py."""
self.assertGreaterEqual(
main.MIN_PYTHON_VERSION_HARD, self.wrapper.MIN_PYTHON_VERSION_HARD
assert (
main.MIN_PYTHON_VERSION_HARD >= repo_wrapper.MIN_PYTHON_VERSION_HARD
)
self.assertGreaterEqual(
main.MIN_PYTHON_VERSION_SOFT, self.wrapper.MIN_PYTHON_VERSION_SOFT
assert (
main.MIN_PYTHON_VERSION_SOFT >= repo_wrapper.MIN_PYTHON_VERSION_SOFT
)
# Make sure the versions are themselves in sync.
self.assertGreaterEqual(
self.wrapper.MIN_PYTHON_VERSION_SOFT,
self.wrapper.MIN_PYTHON_VERSION_HARD,
assert (
repo_wrapper.MIN_PYTHON_VERSION_SOFT
>= repo_wrapper.MIN_PYTHON_VERSION_HARD
)
def test_init_parser(self):
def test_repo_script_is_executable(self) -> None:
"""The repo launcher script should be executable."""
repo_path = utils_for_test.THIS_DIR.parent / "repo"
assert os.access(repo_path, os.X_OK), f"{repo_path} is not executable"
def test_init_parser(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Make sure 'init' GetParser works."""
parser = self.wrapper.GetParser()
parser = repo_wrapper.GetParser()
opts, args = parser.parse_args([])
self.assertEqual([], args)
self.assertIsNone(opts.manifest_url)
assert args == []
assert opts.manifest_url is None
class SetGitTrace2ParentSid(RepoWrapperTestCase):
class TestSetGitTrace2ParentSid:
"""Check SetGitTrace2ParentSid behavior."""
KEY = "GIT_TRACE2_PARENT_SID"
VALID_FORMAT = re.compile(r"^repo-[0-9]{8}T[0-9]{6}Z-P[0-9a-f]{8}$")
def test_first_set(self):
def test_first_set(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Test env var not yet set."""
env = {}
self.wrapper.SetGitTrace2ParentSid(env)
self.assertIn(self.KEY, env)
repo_wrapper.SetGitTrace2ParentSid(env)
assert self.KEY in env
value = env[self.KEY]
self.assertRegex(value, self.VALID_FORMAT)
assert self.VALID_FORMAT.match(value)
def test_append(self):
def test_append(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Test env var is appended."""
env = {self.KEY: "pfx"}
self.wrapper.SetGitTrace2ParentSid(env)
self.assertIn(self.KEY, env)
repo_wrapper.SetGitTrace2ParentSid(env)
assert self.KEY in env
value = env[self.KEY]
self.assertTrue(value.startswith("pfx/"))
self.assertRegex(value[4:], self.VALID_FORMAT)
assert value.startswith("pfx/")
assert self.VALID_FORMAT.match(value[4:])
def test_global_context(self):
def test_global_context(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check os.environ gets updated by default."""
os.environ.pop(self.KEY, None)
self.wrapper.SetGitTrace2ParentSid()
self.assertIn(self.KEY, os.environ)
repo_wrapper.SetGitTrace2ParentSid()
assert self.KEY in os.environ
value = os.environ[self.KEY]
self.assertRegex(value, self.VALID_FORMAT)
assert self.VALID_FORMAT.match(value)
class RunCommand(RepoWrapperTestCase):
class TestRunCommand:
"""Check run_command behavior."""
def test_capture(self):
def test_capture(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check capture_output handling."""
ret = self.wrapper.run_command(["echo", "hi"], capture_output=True)
ret = repo_wrapper.run_command(["echo", "hi"], capture_output=True)
# echo command appends OS specific linesep, but on Windows + Git Bash
# we get UNIX ending, so we allow both.
self.assertIn(ret.stdout, ["hi" + os.linesep, "hi\n"])
assert ret.stdout in ["hi" + os.linesep, "hi\n"]
def test_check(self):
def test_check(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check check handling."""
self.wrapper.run_command(["true"], check=False)
self.wrapper.run_command(["true"], check=True)
self.wrapper.run_command(["false"], check=False)
with self.assertRaises(subprocess.CalledProcessError):
self.wrapper.run_command(["false"], check=True)
repo_wrapper.run_command(["true"], check=False)
repo_wrapper.run_command(["true"], check=True)
repo_wrapper.run_command(["false"], check=False)
with pytest.raises(subprocess.CalledProcessError):
repo_wrapper.run_command(["false"], check=True)
class RunGit(RepoWrapperTestCase):
class TestRunGit:
"""Check run_git behavior."""
def test_capture(self):
def test_capture(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check capture_output handling."""
ret = self.wrapper.run_git("--version")
self.assertIn("git", ret.stdout)
ret = repo_wrapper.run_git("--version")
assert "git" in ret.stdout
def test_check(self):
def test_check(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check check handling."""
with self.assertRaises(self.wrapper.CloneFailure):
self.wrapper.run_git("--version-asdfasdf")
self.wrapper.run_git("--version-asdfasdf", check=False)
with pytest.raises(repo_wrapper.CloneFailure):
repo_wrapper.run_git("--version-asdfasdf")
repo_wrapper.run_git("--version-asdfasdf", check=False)
class ParseGitVersion(RepoWrapperTestCase):
class TestParseGitVersion:
"""Check ParseGitVersion behavior."""
def test_autoload(self):
def test_autoload(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check we can load the version from the live git."""
ret = self.wrapper.ParseGitVersion()
self.assertIsNotNone(ret)
assert repo_wrapper.ParseGitVersion() is not None
def test_bad_ver(self):
def test_bad_ver(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check handling of bad git versions."""
ret = self.wrapper.ParseGitVersion(ver_str="asdf")
self.assertIsNone(ret)
assert repo_wrapper.ParseGitVersion(ver_str="asdf") is None
def test_normal_ver(self):
def test_normal_ver(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check handling of normal git versions."""
ret = self.wrapper.ParseGitVersion(ver_str="git version 2.25.1")
self.assertEqual(2, ret.major)
self.assertEqual(25, ret.minor)
self.assertEqual(1, ret.micro)
self.assertEqual("2.25.1", ret.full)
ret = repo_wrapper.ParseGitVersion(ver_str="git version 2.25.1")
assert ret.major == 2
assert ret.minor == 25
assert ret.micro == 1
assert ret.full == "2.25.1"
def test_extended_ver(self):
def test_extended_ver(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check handling of extended distro git versions."""
ret = self.wrapper.ParseGitVersion(
ret = repo_wrapper.ParseGitVersion(
ver_str="git version 1.30.50.696.g5e7596f4ac-goog"
)
self.assertEqual(1, ret.major)
self.assertEqual(30, ret.minor)
self.assertEqual(50, ret.micro)
self.assertEqual("1.30.50.696.g5e7596f4ac-goog", ret.full)
assert ret.major == 1
assert ret.minor == 30
assert ret.micro == 50
assert ret.full == "1.30.50.696.g5e7596f4ac-goog"
class CheckGitVersion(RepoWrapperTestCase):
class TestCheckGitVersion:
"""Check _CheckGitVersion behavior."""
def test_unknown(self):
def test_unknown(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Unknown versions should abort."""
with mock.patch.object(
self.wrapper, "ParseGitVersion", return_value=None
repo_wrapper, "ParseGitVersion", return_value=None
):
with self.assertRaises(self.wrapper.CloneFailure):
self.wrapper._CheckGitVersion()
with pytest.raises(repo_wrapper.CloneFailure):
repo_wrapper._CheckGitVersion()
def test_old(self):
def test_old(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Old versions should abort."""
with mock.patch.object(
self.wrapper,
repo_wrapper,
"ParseGitVersion",
return_value=self.wrapper.GitVersion(1, 0, 0, "1.0.0"),
return_value=repo_wrapper.GitVersion(1, 0, 0, "1.0.0"),
):
with self.assertRaises(self.wrapper.CloneFailure):
self.wrapper._CheckGitVersion()
with pytest.raises(repo_wrapper.CloneFailure):
repo_wrapper._CheckGitVersion()
def test_new(self):
def test_new(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Newer versions should run fine."""
with mock.patch.object(
self.wrapper,
repo_wrapper,
"ParseGitVersion",
return_value=self.wrapper.GitVersion(100, 0, 0, "100.0.0"),
return_value=repo_wrapper.GitVersion(100, 0, 0, "100.0.0"),
):
self.wrapper._CheckGitVersion()
repo_wrapper._CheckGitVersion()
class Requirements(RepoWrapperTestCase):
class TestRequirements:
"""Check Requirements handling."""
def test_missing_file(self):
def test_missing_file(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Don't crash if the file is missing (old version)."""
testdir = os.path.dirname(os.path.realpath(__file__))
self.assertIsNone(self.wrapper.Requirements.from_dir(testdir))
self.assertIsNone(
self.wrapper.Requirements.from_file(
os.path.join(testdir, "xxxxxxxxxxxxxxxxxxxxxxxx")
assert (
repo_wrapper.Requirements.from_dir(utils_for_test.THIS_DIR) is None
)
assert (
repo_wrapper.Requirements.from_file(
utils_for_test.THIS_DIR / "xxxxxxxxxxxxxxxxxxxxxxxx"
)
is None
)
def test_corrupt_data(self):
def test_corrupt_data(self, repo_wrapper: wrapper.Wrapper) -> None:
"""If the file can't be parsed, don't blow up."""
self.assertIsNone(self.wrapper.Requirements.from_file(__file__))
self.assertIsNone(self.wrapper.Requirements.from_data(b"x"))
assert repo_wrapper.Requirements.from_file(__file__) is None
assert repo_wrapper.Requirements.from_data(b"x") is None
def test_valid_data(self):
def test_valid_data(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Make sure we can parse the file we ship."""
self.assertIsNotNone(self.wrapper.Requirements.from_data(b"{}"))
rootdir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
self.assertIsNotNone(self.wrapper.Requirements.from_dir(rootdir))
self.assertIsNotNone(
self.wrapper.Requirements.from_file(
os.path.join(rootdir, "requirements.json")
)
assert repo_wrapper.Requirements.from_data(b"{}") is not None
rootdir = utils_for_test.THIS_DIR.parent
assert repo_wrapper.Requirements.from_dir(rootdir) is not None
assert (
repo_wrapper.Requirements.from_file(rootdir / "requirements.json")
is not None
)
def test_format_ver(self):
def test_format_ver(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check format_ver can format."""
self.assertEqual(
"1.2.3", self.wrapper.Requirements._format_ver((1, 2, 3))
)
self.assertEqual("1", self.wrapper.Requirements._format_ver([1]))
assert repo_wrapper.Requirements._format_ver((1, 2, 3)) == "1.2.3"
assert repo_wrapper.Requirements._format_ver([1]) == "1"
def test_assert_all_unknown(self):
def test_assert_all_unknown(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check assert_all works with incompatible file."""
reqs = self.wrapper.Requirements({})
reqs = repo_wrapper.Requirements({})
reqs.assert_all()
def test_assert_all_new_repo(self):
def test_assert_all_new_repo(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check assert_all accepts new enough repo."""
reqs = self.wrapper.Requirements({"repo": {"hard": [1, 0]}})
reqs = repo_wrapper.Requirements({"repo": {"hard": [1, 0]}})
reqs.assert_all()
def test_assert_all_old_repo(self):
def test_assert_all_old_repo(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check assert_all rejects old repo."""
reqs = self.wrapper.Requirements({"repo": {"hard": [99999, 0]}})
with self.assertRaises(SystemExit):
reqs = repo_wrapper.Requirements({"repo": {"hard": [99999, 0]}})
with pytest.raises(SystemExit):
reqs.assert_all()
def test_assert_all_new_python(self):
def test_assert_all_new_python(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check assert_all accepts new enough python."""
reqs = self.wrapper.Requirements({"python": {"hard": sys.version_info}})
reqs = repo_wrapper.Requirements({"python": {"hard": sys.version_info}})
reqs.assert_all()
def test_assert_all_old_python(self):
def test_assert_all_old_python(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check assert_all rejects old python."""
reqs = self.wrapper.Requirements({"python": {"hard": [99999, 0]}})
with self.assertRaises(SystemExit):
reqs = repo_wrapper.Requirements({"python": {"hard": [99999, 0]}})
with pytest.raises(SystemExit):
reqs.assert_all()
def test_assert_ver_unknown(self):
def test_assert_ver_unknown(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check assert_ver works with incompatible file."""
reqs = self.wrapper.Requirements({})
reqs = repo_wrapper.Requirements({})
reqs.assert_ver("xxx", (1, 0))
def test_assert_ver_new(self):
def test_assert_ver_new(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check assert_ver allows new enough versions."""
reqs = self.wrapper.Requirements(
reqs = repo_wrapper.Requirements(
{"git": {"hard": [1, 0], "soft": [2, 0]}}
)
reqs.assert_ver("git", (1, 0))
@@ -286,285 +323,279 @@ class Requirements(RepoWrapperTestCase):
reqs.assert_ver("git", (2, 0))
reqs.assert_ver("git", (2, 5))
def test_assert_ver_old(self):
def test_assert_ver_old(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check assert_ver rejects old versions."""
reqs = self.wrapper.Requirements(
reqs = repo_wrapper.Requirements(
{"git": {"hard": [1, 0], "soft": [2, 0]}}
)
with self.assertRaises(SystemExit):
with pytest.raises(SystemExit):
reqs.assert_ver("git", (0, 5))
class NeedSetupGnuPG(RepoWrapperTestCase):
class TestNeedSetupGnuPG:
"""Check NeedSetupGnuPG behavior."""
def test_missing_dir(self):
def test_missing_dir(self, tmp_path, repo_wrapper: wrapper.Wrapper) -> None:
"""The ~/.repoconfig tree doesn't exist yet."""
with tempfile.TemporaryDirectory(prefix="repo-tests") as tempdir:
self.wrapper.home_dot_repo = os.path.join(tempdir, "foo")
self.assertTrue(self.wrapper.NeedSetupGnuPG())
repo_wrapper.home_dot_repo = str(tmp_path / "foo")
assert repo_wrapper.NeedSetupGnuPG()
def test_missing_keyring(self):
def test_missing_keyring(
self, tmp_path, repo_wrapper: wrapper.Wrapper
) -> None:
"""The keyring-version file doesn't exist yet."""
with tempfile.TemporaryDirectory(prefix="repo-tests") as tempdir:
self.wrapper.home_dot_repo = tempdir
self.assertTrue(self.wrapper.NeedSetupGnuPG())
repo_wrapper.home_dot_repo = str(tmp_path)
assert repo_wrapper.NeedSetupGnuPG()
def test_empty_keyring(self):
def test_empty_keyring(
self, tmp_path, repo_wrapper: wrapper.Wrapper
) -> None:
"""The keyring-version file exists, but is empty."""
with tempfile.TemporaryDirectory(prefix="repo-tests") as tempdir:
self.wrapper.home_dot_repo = tempdir
with open(os.path.join(tempdir, "keyring-version"), "w"):
pass
self.assertTrue(self.wrapper.NeedSetupGnuPG())
repo_wrapper.home_dot_repo = str(tmp_path)
(tmp_path / "keyring-version").write_text("")
assert repo_wrapper.NeedSetupGnuPG()
def test_old_keyring(self):
def test_old_keyring(self, tmp_path, repo_wrapper: wrapper.Wrapper) -> None:
"""The keyring-version file exists, but it's old."""
with tempfile.TemporaryDirectory(prefix="repo-tests") as tempdir:
self.wrapper.home_dot_repo = tempdir
with open(os.path.join(tempdir, "keyring-version"), "w") as fp:
fp.write("1.0\n")
self.assertTrue(self.wrapper.NeedSetupGnuPG())
repo_wrapper.home_dot_repo = str(tmp_path)
(tmp_path / "keyring-version").write_text("1.0\n")
assert repo_wrapper.NeedSetupGnuPG()
def test_new_keyring(self):
def test_new_keyring(self, tmp_path, repo_wrapper: wrapper.Wrapper) -> None:
"""The keyring-version file exists, and is up-to-date."""
with tempfile.TemporaryDirectory(prefix="repo-tests") as tempdir:
self.wrapper.home_dot_repo = tempdir
with open(os.path.join(tempdir, "keyring-version"), "w") as fp:
fp.write("1000.0\n")
self.assertFalse(self.wrapper.NeedSetupGnuPG())
repo_wrapper.home_dot_repo = str(tmp_path)
(tmp_path / "keyring-version").write_text("1000.0\n")
assert not repo_wrapper.NeedSetupGnuPG()
class SetupGnuPG(RepoWrapperTestCase):
class TestSetupGnuPG:
"""Check SetupGnuPG behavior."""
def test_full(self):
def test_full(self, tmp_path, repo_wrapper: wrapper.Wrapper) -> None:
"""Make sure it works completely."""
with tempfile.TemporaryDirectory(prefix="repo-tests") as tempdir:
self.wrapper.home_dot_repo = tempdir
self.wrapper.gpg_dir = os.path.join(
self.wrapper.home_dot_repo, "gnupg"
)
self.assertTrue(self.wrapper.SetupGnuPG(True))
with open(os.path.join(tempdir, "keyring-version")) as fp:
data = fp.read()
self.assertEqual(
".".join(str(x) for x in self.wrapper.KEYRING_VERSION),
data.strip(),
)
repo_wrapper.home_dot_repo = str(tmp_path)
repo_wrapper.gpg_dir = str(tmp_path / "gnupg")
assert repo_wrapper.SetupGnuPG(True)
data = (tmp_path / "keyring-version").read_text()
assert (
".".join(str(x) for x in repo_wrapper.KEYRING_VERSION)
== data.strip()
)
class VerifyRev(RepoWrapperTestCase):
class TestVerifyRev:
"""Check verify_rev behavior."""
def test_verify_passes(self):
def test_verify_passes(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check when we have a valid signed tag."""
desc_result = subprocess.CompletedProcess([], 0, "v1.0\n", "")
gpg_result = subprocess.CompletedProcess([], 0, "", "")
with mock.patch.object(
self.wrapper, "run_git", side_effect=(desc_result, gpg_result)
repo_wrapper, "run_git", side_effect=(desc_result, gpg_result)
):
ret = self.wrapper.verify_rev(
ret = repo_wrapper.verify_rev(
"/", "refs/heads/stable", "1234", True
)
self.assertEqual("v1.0^0", ret)
assert ret == "v1.0^0"
def test_unsigned_commit(self):
def test_unsigned_commit(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check we fall back to signed tag when we have an unsigned commit."""
desc_result = subprocess.CompletedProcess([], 0, "v1.0-10-g1234\n", "")
gpg_result = subprocess.CompletedProcess([], 0, "", "")
with mock.patch.object(
self.wrapper, "run_git", side_effect=(desc_result, gpg_result)
repo_wrapper, "run_git", side_effect=(desc_result, gpg_result)
):
ret = self.wrapper.verify_rev(
ret = repo_wrapper.verify_rev(
"/", "refs/heads/stable", "1234", True
)
self.assertEqual("v1.0^0", ret)
assert ret == "v1.0^0"
def test_verify_fails(self):
def test_verify_fails(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Check we fall back to signed tag when we have an unsigned commit."""
desc_result = subprocess.CompletedProcess([], 0, "v1.0-10-g1234\n", "")
gpg_result = Exception
gpg_result = RuntimeError
with mock.patch.object(
self.wrapper, "run_git", side_effect=(desc_result, gpg_result)
repo_wrapper, "run_git", side_effect=(desc_result, gpg_result)
):
with self.assertRaises(Exception):
self.wrapper.verify_rev("/", "refs/heads/stable", "1234", True)
with pytest.raises(RuntimeError):
repo_wrapper.verify_rev("/", "refs/heads/stable", "1234", True)
class GitCheckoutTestCase(RepoWrapperTestCase):
"""Tests that use a real/small git checkout."""
GIT_DIR = None
REV_LIST = None
@classmethod
def setUpClass(cls):
# Create a repo to operate on, but do it once per-class.
cls.tempdirobj = tempfile.TemporaryDirectory(prefix="repo-rev-tests")
cls.GIT_DIR = cls.tempdirobj.name
run_git = wrapper.Wrapper().run_git
remote = os.path.join(cls.GIT_DIR, "remote")
os.mkdir(remote)
# Tests need to assume, that main is default branch at init,
# which is not supported in config until 2.28.
if git_command.git_require((2, 28, 0)):
initstr = "--initial-branch=main"
else:
# Use template dir for init.
templatedir = tempfile.mkdtemp(prefix=".test-template")
with open(os.path.join(templatedir, "HEAD"), "w") as fp:
fp.write("ref: refs/heads/main\n")
initstr = "--template=" + templatedir
run_git("init", initstr, cwd=remote)
run_git("commit", "--allow-empty", "-minit", cwd=remote)
run_git("branch", "stable", cwd=remote)
run_git("tag", "v1.0", cwd=remote)
run_git("commit", "--allow-empty", "-m2nd commit", cwd=remote)
cls.REV_LIST = run_git(
"rev-list", "HEAD", cwd=remote
).stdout.splitlines()
run_git("init", cwd=cls.GIT_DIR)
run_git(
"fetch",
remote,
"+refs/heads/*:refs/remotes/origin/*",
cwd=cls.GIT_DIR,
)
@classmethod
def tearDownClass(cls):
if not cls.tempdirobj:
return
cls.tempdirobj.cleanup()
class ResolveRepoRev(GitCheckoutTestCase):
class TestResolveRepoRev:
"""Check resolve_repo_rev behavior."""
def test_explicit_branch(self):
def test_explicit_branch(
self,
repo_wrapper: wrapper.Wrapper,
git_checkout: GitCheckout,
) -> None:
"""Check refs/heads/branch argument."""
rrev, lrev = self.wrapper.resolve_repo_rev(
self.GIT_DIR, "refs/heads/stable"
rrev, lrev = repo_wrapper.resolve_repo_rev(
git_checkout.git_dir, "refs/heads/stable"
)
self.assertEqual("refs/heads/stable", rrev)
self.assertEqual(self.REV_LIST[1], lrev)
assert rrev == "refs/heads/stable"
assert lrev == git_checkout.rev_list[1]
with self.assertRaises(self.wrapper.CloneFailure):
self.wrapper.resolve_repo_rev(self.GIT_DIR, "refs/heads/unknown")
with pytest.raises(repo_wrapper.CloneFailure):
repo_wrapper.resolve_repo_rev(
git_checkout.git_dir, "refs/heads/unknown"
)
def test_explicit_tag(self):
def test_explicit_tag(
self,
repo_wrapper: wrapper.Wrapper,
git_checkout: GitCheckout,
) -> None:
"""Check refs/tags/tag argument."""
rrev, lrev = self.wrapper.resolve_repo_rev(
self.GIT_DIR, "refs/tags/v1.0"
rrev, lrev = repo_wrapper.resolve_repo_rev(
git_checkout.git_dir, "refs/tags/v1.0"
)
self.assertEqual("refs/tags/v1.0", rrev)
self.assertEqual(self.REV_LIST[1], lrev)
assert rrev == "refs/tags/v1.0"
assert lrev == git_checkout.rev_list[1]
with self.assertRaises(self.wrapper.CloneFailure):
self.wrapper.resolve_repo_rev(self.GIT_DIR, "refs/tags/unknown")
with pytest.raises(repo_wrapper.CloneFailure):
repo_wrapper.resolve_repo_rev(
git_checkout.git_dir, "refs/tags/unknown"
)
def test_branch_name(self):
def test_branch_name(
self,
repo_wrapper: wrapper.Wrapper,
git_checkout: GitCheckout,
) -> None:
"""Check branch argument."""
rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, "stable")
self.assertEqual("refs/heads/stable", rrev)
self.assertEqual(self.REV_LIST[1], lrev)
rrev, lrev = repo_wrapper.resolve_repo_rev(
git_checkout.git_dir, "stable"
)
assert rrev == "refs/heads/stable"
assert lrev == git_checkout.rev_list[1]
rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, "main")
self.assertEqual("refs/heads/main", rrev)
self.assertEqual(self.REV_LIST[0], lrev)
rrev, lrev = repo_wrapper.resolve_repo_rev(git_checkout.git_dir, "main")
assert rrev == "refs/heads/main"
assert lrev == git_checkout.rev_list[0]
def test_tag_name(self):
def test_tag_name(
self,
repo_wrapper: wrapper.Wrapper,
git_checkout: GitCheckout,
) -> None:
"""Check tag argument."""
rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, "v1.0")
self.assertEqual("refs/tags/v1.0", rrev)
self.assertEqual(self.REV_LIST[1], lrev)
rrev, lrev = repo_wrapper.resolve_repo_rev(git_checkout.git_dir, "v1.0")
assert rrev == "refs/tags/v1.0"
assert lrev == git_checkout.rev_list[1]
def test_full_commit(self):
def test_full_commit(
self,
repo_wrapper: wrapper.Wrapper,
git_checkout: GitCheckout,
) -> None:
"""Check specific commit argument."""
commit = self.REV_LIST[0]
rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, commit)
self.assertEqual(commit, rrev)
self.assertEqual(commit, lrev)
commit = git_checkout.rev_list[0]
rrev, lrev = repo_wrapper.resolve_repo_rev(git_checkout.git_dir, commit)
assert rrev == commit
assert lrev == commit
def test_partial_commit(self):
def test_partial_commit(
self,
repo_wrapper: wrapper.Wrapper,
git_checkout: GitCheckout,
) -> None:
"""Check specific (partial) commit argument."""
commit = self.REV_LIST[0][0:20]
rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, commit)
self.assertEqual(self.REV_LIST[0], rrev)
self.assertEqual(self.REV_LIST[0], lrev)
commit = git_checkout.rev_list[0][0:20]
rrev, lrev = repo_wrapper.resolve_repo_rev(git_checkout.git_dir, commit)
assert rrev == git_checkout.rev_list[0]
assert lrev == git_checkout.rev_list[0]
def test_unknown(self):
def test_unknown(
self,
repo_wrapper: wrapper.Wrapper,
git_checkout: GitCheckout,
) -> None:
"""Check unknown ref/commit argument."""
with self.assertRaises(self.wrapper.CloneFailure):
self.wrapper.resolve_repo_rev(self.GIT_DIR, "boooooooya")
with pytest.raises(repo_wrapper.CloneFailure):
repo_wrapper.resolve_repo_rev(git_checkout.git_dir, "boooooooya")
class CheckRepoVerify(RepoWrapperTestCase):
class TestCheckRepoVerify:
"""Check check_repo_verify behavior."""
def test_no_verify(self):
def test_no_verify(self, repo_wrapper: wrapper.Wrapper) -> None:
"""Always fail with --no-repo-verify."""
self.assertFalse(self.wrapper.check_repo_verify(False))
assert not repo_wrapper.check_repo_verify(False)
def test_gpg_initialized(self):
def test_gpg_initialized(
self,
repo_wrapper: wrapper.Wrapper,
) -> None:
"""Should pass if gpg is setup already."""
with mock.patch.object(
self.wrapper, "NeedSetupGnuPG", return_value=False
repo_wrapper, "NeedSetupGnuPG", return_value=False
):
self.assertTrue(self.wrapper.check_repo_verify(True))
assert repo_wrapper.check_repo_verify(True)
def test_need_gpg_setup(self):
def test_need_gpg_setup(
self,
repo_wrapper: wrapper.Wrapper,
) -> None:
"""Should pass/fail based on gpg setup."""
with mock.patch.object(
self.wrapper, "NeedSetupGnuPG", return_value=True
repo_wrapper, "NeedSetupGnuPG", return_value=True
):
with mock.patch.object(self.wrapper, "SetupGnuPG") as m:
with mock.patch.object(repo_wrapper, "SetupGnuPG") as m:
m.return_value = True
self.assertTrue(self.wrapper.check_repo_verify(True))
assert repo_wrapper.check_repo_verify(True)
m.return_value = False
self.assertFalse(self.wrapper.check_repo_verify(True))
assert not repo_wrapper.check_repo_verify(True)
class CheckRepoRev(GitCheckoutTestCase):
class TestCheckRepoRev:
"""Check check_repo_rev behavior."""
def test_verify_works(self):
def test_verify_works(
self,
repo_wrapper: wrapper.Wrapper,
git_checkout: GitCheckout,
) -> None:
"""Should pass when verification passes."""
with mock.patch.object(
self.wrapper, "check_repo_verify", return_value=True
repo_wrapper, "check_repo_verify", return_value=True
):
with mock.patch.object(
self.wrapper, "verify_rev", return_value="12345"
repo_wrapper, "verify_rev", return_value="12345"
):
rrev, lrev = self.wrapper.check_repo_rev(self.GIT_DIR, "stable")
self.assertEqual("refs/heads/stable", rrev)
self.assertEqual("12345", lrev)
rrev, lrev = repo_wrapper.check_repo_rev(
git_checkout.git_dir, "stable"
)
assert rrev == "refs/heads/stable"
assert lrev == "12345"
def test_verify_fails(self):
def test_verify_fails(
self,
repo_wrapper: wrapper.Wrapper,
git_checkout: GitCheckout,
) -> None:
"""Should fail when verification fails."""
with mock.patch.object(
self.wrapper, "check_repo_verify", return_value=True
repo_wrapper, "check_repo_verify", return_value=True
):
with mock.patch.object(
self.wrapper, "verify_rev", side_effect=Exception
repo_wrapper, "verify_rev", side_effect=RuntimeError
):
with self.assertRaises(Exception):
self.wrapper.check_repo_rev(self.GIT_DIR, "stable")
with pytest.raises(RuntimeError):
repo_wrapper.check_repo_rev(git_checkout.git_dir, "stable")
def test_verify_ignore(self):
def test_verify_ignore(
self,
repo_wrapper: wrapper.Wrapper,
git_checkout: GitCheckout,
) -> None:
"""Should pass when verification is disabled."""
with mock.patch.object(
self.wrapper, "verify_rev", side_effect=Exception
repo_wrapper, "verify_rev", side_effect=RuntimeError
):
rrev, lrev = self.wrapper.check_repo_rev(
self.GIT_DIR, "stable", repo_verify=False
rrev, lrev = repo_wrapper.check_repo_rev(
git_checkout.git_dir, "stable", repo_verify=False
)
self.assertEqual("refs/heads/stable", rrev)
self.assertEqual(self.REV_LIST[1], lrev)
assert rrev == "refs/heads/stable"
assert lrev == git_checkout.rev_list[1]
+104
View File
@@ -0,0 +1,104 @@
# Copyright (C) 2026 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Various utility code used by tests.
If you want to write a per-test fixture, see conftest.py instead.
"""
import contextlib
import functools
from pathlib import Path
import subprocess
import tempfile
from typing import Optional, Union
import git_command
THIS_FILE = Path(__file__).resolve()
THIS_DIR = THIS_FILE.parent
FIXTURES_DIR = THIS_DIR / "fixtures"
def init_git_tree(
path: Union[str, Path],
ref_format: Optional[str] = None,
) -> None:
"""Initialize `path` as a new git repo."""
with contextlib.ExitStack() as stack:
# Tests need to assume, that main is default branch at init,
# which is not supported in config until 2.28.
cmd = ["git"]
if ref_format:
cmd += ["-c", f"init.defaultRefFormat={ref_format}"]
cmd += ["init"]
if git_command.git_require((2, 28, 0)):
cmd += ["--initial-branch=main"]
else:
# Use template dir for init.
templatedir = stack.enter_context(
tempfile.mkdtemp(prefix="git-template")
)
(Path(templatedir) / "HEAD").write_text("ref: refs/heads/main\n")
cmd += ["--template", templatedir]
cmd += [path]
subprocess.run(cmd, check=True)
@contextlib.contextmanager
def TempGitTree():
"""Create a new empty git checkout for testing."""
with tempfile.TemporaryDirectory(prefix="repo-tests") as tempdir:
init_git_tree(tempdir)
yield tempdir
@functools.lru_cache(maxsize=None)
def supports_reftable() -> bool:
"""Check if git supports reftable."""
with tempfile.TemporaryDirectory(prefix="repo-tests") as tempdir:
proc = subprocess.run(
["git", "-c", "init.defaultRefFormat=reftable", "init"],
cwd=tempdir,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
check=False,
)
return proc.returncode == 0
@functools.lru_cache(maxsize=None)
def supports_refs_migrate() -> bool:
"""Check if git supports refs migrate."""
with tempfile.TemporaryDirectory(prefix="repo-tests") as tempdir:
subprocess.check_call(
["git", "-c", "init.defaultRefFormat=files", "init"],
cwd=tempdir,
)
proc = subprocess.run(
[
"git",
"refs",
"migrate",
"--ref-format=reftable",
"--dry-run",
],
cwd=tempdir,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
check=False,
)
return proc.returncode == 0
-63
View File
@@ -1,63 +0,0 @@
# Copyright (C) 2019 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# https://tox.readthedocs.io/
[tox]
envlist = lint, py36, py37, py38, py39, py310, py311, py312
requires = virtualenv<20.22.0
[gh-actions]
python =
3.6: py36
3.7: py37
3.8: py38
3.9: py39
3.10: py310
3.11: py311
3.12: py312
[testenv]
deps =
-c constraints.txt
black
flake8
isort
pytest
pytest-timeout
commands = {envpython} run_tests {posargs}
setenv =
GIT_AUTHOR_NAME = Repo test author
GIT_COMMITTER_NAME = Repo test committer
EMAIL = repo@gerrit.nodomain
[testenv:lint]
skip_install = true
deps =
-c constraints.txt
black
flake8
commands =
black --check {posargs:. repo run_tests release/update-hooks release/update-manpages}
flake8
[testenv:format]
skip_install = true
deps =
-c constraints.txt
black
flake8
commands =
black {posargs:. repo run_tests release/update-hooks release/update-manpages}
flake8