Compare commits

...

19 Commits

Author SHA1 Message Date
Gavin Mak baa281d99e sync: Refactor to use _RunOneGC and fix config leakage
Extract _RunOneGC to handle GC on a single project. This refactoring
makes it easier to invoke GC from parallel worker tasks.

Also, avoid modifying the passed-in config dictionary in _RunOneGC by
creating a local copy, preventing unintended side effects on other
commands sharing the same config.

Bug: 498290329
Change-Id: I7b77ed6629b14b5ee3322870b9c6c8ce2bfd6ea2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574923
Reviewed-by: Becky Siegel <beckysiegel@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-04-20 15:34:07 -07:00
Gavin Mak 7e9079b7cf sync: Switch to using self._bloated_projects
Store bloated projects in self._bloated_projects and print warnings at
the end of execution. This sets up for moving the check to workers.

Bug: 498290329
Change-Id: I993f1fd741db2994d480994861588eb18f6c5503
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574922
Reviewed-by: Becky Siegel <beckysiegel@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-04-20 14:07:14 -07:00
Gavin Mak 61bd6b3ccb tests: Add tests for _CheckForBloatedProjects and _GCProjects
Change-Id: I6790a13929b8fc06a3197ca405dedf08d39d54c9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574926
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Becky Siegel <beckysiegel@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-04-20 13:31:41 -07:00
Marty Heavey 32e7327ca6 upload: Clarify partial sync message on hook failure
Demote the partial sync warning from error to info and rephrase it to
be a tip rather than an error. This prevents users from thinking that
a partial sync is the cause of their hook failures when it is often a linting failure.

The message now suggests that a full sync might help if there are
cross-project dependencies, instead of implying it will fix any issue.

Change-Id: I5d8c52b53ac315aa9f145ed069798bf201fa0815
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574262
Tested-by: Marty Heavey <mheavey@google.com>
Commit-Queue: Marty Heavey <mheavey@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-04-20 01:52:25 -07:00
Gavin Mak 4f707ff91e sync: Provide feedback during post-sync operations
After the main sync progress bar finishes, there's a pause while some
post-sync operations run. Print something to provide feedback so the
user doesn't think repo has hung.

Bug: 503869525
Change-Id: I695fd560e60dcb394e6844a56c8a336ca1f71c74
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574425
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Becky Siegel <beckysiegel@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2026-04-17 15:47:15 -07:00
Gavin Mak e2671c184e progress: Ignore updates after progress ends
This addresses the occasional "..working.." message at the end of sync.

Bug: 503869525
Change-Id: I489d29fa8ae588d77abb7fee5f157800d4abb368
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/574482
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Becky Siegel <beckysiegel@google.com>
2026-04-17 15:43:14 -07:00
Becky Siegel b43a20b859 project: Avoid skipping fetches for shallow clones without .git/shallow
When optimizing fetches for projects with immutable revisions, the fetch
should not be skipped if the project is configured for a shallow clone
(depth > 0) but the .git/shallow file is missing. The absence of the
.git/shallow file means the repository is not a shallow clone, or the
shallow clone is incomplete, so a fetch is necessary to ensure the
revision is present.

Bug: 503081454
Change-Id: Ic3549612bcd69050a926652ee4e522c79ad8124c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/573821
Tested-by: Becky Siegel <beckysiegel@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Becky Siegel <beckysiegel@google.com>
2026-04-17 09:16:36 -07:00
Miyako.Enei 8869a30283 project: Drop --no-deref from update-ref --stdin
repo calls `git update-ref --stdin` when updating multiple refs during
repo init and repo sync. Historically, `--no-deref` was also passed.

Older Git 2.17 which we still support rejects the combination of
`--stdin` and `--no-deref`, emitting a usage error even when the stdin
input is valid.

The `--no-deref` option is only meaningful when updating symbolic refs
such as HEAD. The stdin-based update-ref path only operates on explicit
refs (tags, remote refs, alternates) and never symbolic refs.

Remove the unnecessary option to restore compatibility with Git 2.17
while preserving identical behavior on newer Git versions.

Tested with:
  - Git 2.17.1
  - Git 2.34.1

Change-Id: I22001de03800f5699b26a40bc1fb1fec002ed048
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/571721
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Enei <miyako.enei@alpsalpine.com>
Tested-by: Enei <miyako.enei@alpsalpine.com>
2026-04-12 16:39:28 -07:00
Gavin Mak 3b0eebeccf project: implement stateless sync pruning logic
Implement in-situ shallow re-fetching and garbage collection logic.
Enables repositories with sync-strategy="stateless" to reclaim disk
space by running reflog expire and git gc --prune=now if the working
tree is clean and has no local commits.

Bug: 498730431
Change-Id: I940bdc9b74da29d3f7b13566667dcddea769ebd3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/568463
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-04-09 12:09:40 -07:00
Gavin Mak 00991bfb42 manifest: Add sync-strategy attribute to project elements
The only supported sync-strategy is "stateless". The intent is to keep
the local workspace as small as possible by not keeping history during
syncs. This prevents disk space waste for projects with large binaries
where we only care about the current version.

A follow up change will implement the logic.

Bug: 498730431
Change-Id: I84a436a9ca2492893163c6cfda6c28dc62a568f0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/568462
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-04-09 12:09:28 -07:00
Nasser Grainawi e8338b54bd tests: Convert forall subcmd test to pytest
Rewrite tests/test_subcmds_forall.py from unittest.TestCase to pytest
function-style tests to match the surrounding test suite conventions.

Replace setUp/tearDown and class-based helpers with tmp_path-based
setup, switch stdout capture to contextlib.redirect_stdout, and keep the
existing behavior checks intact (all eight projects are invoked exactly
once).

Change-Id: I9243f3461aa6850f867bdb864f4a34c442f817f6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/569821
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
Tested-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-04-09 11:51:47 -07:00
Gavin Mak 951666fb23 gc: Fix hang during repack in partial clones
Add `--missing=allow-promisor` to `git rev-list` calls in
`repack_projects`. This prevents Git from auto-fetching missing objects
from the promisor remote, which can cause stalls due to sequential
network requests.

Also add a Git version check to ensure Git is at least 2.17.0 before
running `--repack`, as `--missing=allow-promisor` was introduced in that
version.

Bug: 500133631
Change-Id: I2dcf9b46fac4c6a53a3c2a46f06f61d6aec40f2f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/570361
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Sam Saccone <samccone@google.com>
2026-04-08 12:54:39 -07:00
Carlos Fernandez 854b330967 test_wrapper: add test for repo script executable permission
Add a test to verify that the repo launcher script has the
executable bit set, guarding against accidental permission changes.

Change-Id: I314658b57ed174673188fbbc5962d9fdeefac97d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/569242
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Carlos Fernandez <carlosfsanz@meta.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Carlos Fernandez <carlosfsanz@meta.com>
2026-04-06 14:16:04 -07:00
Mike Frysinger 654690e1b8 tests: convert more tests to pytest
Change-Id: Id4d48b61dc435564c336385bbc4944eb475d1942
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/569443
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-04-06 11:36:39 -07:00
Mike Frysinger ac2be4c089 tests: convert __file__ usage to pathlib
Change-Id: I2408b0ac97629f0d5fc92779b78bf1ff159a6f83
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/569442
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
2026-04-06 11:15:58 -07:00
Mike Frysinger 3d819e8e3e tests: unify fixture() helper with Path constant
Change-Id: I63751042391f5cc3e06af7067bc83d67bd0716dc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/569441
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-04-06 11:15:14 -07:00
Carlos Fernandez 573983948a Fix all flake8 warnings from newer flake8-bugbear and flake8-comprehensions
Address warnings introduced by flake8-bugbear 24.12.12 and
flake8-comprehensions 3.16.0:

- C408: Replace dict()/list() calls with literal {} and []
- C413: Remove unnecessary list() around sorted()
- C414: Remove unnecessary list() inside sorted()
- C419: Suppress intentional list comprehension in all() (noqa)
- B001: Replace bare except with except Exception
- B006: Replace mutable default arguments with None
- B010: Replace setattr() with direct attribute assignment
- B017: Use RuntimeError instead of Exception in tests
- B019: Suppress lru_cache on methods for long-lived objects (noqa)
- B033: Remove duplicate item in set literal

Change-Id: If4693d3e946200bbc22f689f7b94da604addcb80
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/566321
Tested-by: Carlos Fernandez <carlosfsanz@meta.com>
Commit-Queue: Carlos Fernandez <carlosfsanz@meta.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2026-04-03 07:50:52 -07:00
Gavin Mak 3f3c681a02 project: Refactor GetHead to use symbolic-ref first
Simplify branch resolution and optimize unborn branch detection by
prioritizing symbolic-ref over rev-parse.

Change-Id: Ic62dcb87cd051dafb00d520b1157be2e32abc2ab
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/563222
Reviewed-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2026-04-02 13:57:05 -07:00
Sam Saccone 242e97d9dd Implement command forgiveness with autocorrect
Similar to `git`, when a user types an unknown command like `repo tart`,
we now use `difflib.get_close_matches` to suggest similar commands.

If `help.autocorrect` is set in the git config, it will optionally
prompt the user to automatically run the assumed command, or wait
for a configured delay before executing it.

Verification Steps:
1. Created a dummy repo project locally.
2. Verified `help.autocorrect=0|false|off|no|show` suggests
   command and exits.
3. Verified `help.autocorrect=1|true|on|yes|immediate`
   automatically runs suggestion.
4. Verified `help.autocorrect=<number>` runs after
   `<number>*0.1` seconds.
5. Verified `help.autocorrect=never` exits immediately without
   suggestions.
6. Verified `help.autocorrect=prompt` asks user to accept [y/n]
   and handles correctly.

BUG: b/489753302

Change-Id: I6dcd63229cbd7badf5404459b48690c68f5b4857
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/558021
Tested-by: Sam Saccone <samccone@google.com>
Commit-Queue: Sam Saccone <samccone@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2026-03-24 15:47:37 -07:00
27 changed files with 2303 additions and 1425 deletions
+17
View File
@@ -85,6 +85,7 @@ following DTD:
<!ATTLIST project upstream CDATA #IMPLIED> <!ATTLIST project upstream CDATA #IMPLIED>
<!ATTLIST project clone-depth CDATA #IMPLIED> <!ATTLIST project clone-depth CDATA #IMPLIED>
<!ATTLIST project force-path CDATA #IMPLIED> <!ATTLIST project force-path CDATA #IMPLIED>
<!ATTLIST project sync-strategy CDATA #IMPLIED>
<!ELEMENT annotation EMPTY> <!ELEMENT annotation EMPTY>
<!ATTLIST annotation name CDATA #REQUIRED> <!ATTLIST annotation name CDATA #REQUIRED>
@@ -389,6 +390,22 @@ rather than the `name` attribute. This attribute only applies to the
local mirrors syncing, it will be ignored when syncing the projects in a local mirrors syncing, it will be ignored when syncing the projects in a
client working directory. client working directory.
Attribute `sync-strategy`: Set the sync strategy used when fetching this
project. Currently the only supported value is `stateless`. When set to
`stateless`, repo will run a reflog expiration and aggressive garbage collection
at the end of the sync process. This is useful for projects that contain
large binary files and use `clone-depth="1"`, where garbage can accumulate
as binaries are added, deleted, or modified across successive syncs.
During a stateless sync, repo checks the following before cleaning up:
1. The project does not share an object directory with other projects.
2. The working tree is clean (no uncommitted changes, no untracked files).
3. There are no unpushed local commits.
4. There is no Git stash.
If any of these conditions are not met, repo falls back to a standard
sync without garbage collection.
### Element extend-project ### Element extend-project
Modify the attributes of the named project. Modify the attributes of the named project.
+3 -3
View File
@@ -47,7 +47,7 @@ logger = RepoLogger(__file__)
class _GitCall: class _GitCall:
@functools.lru_cache(maxsize=None) @functools.lru_cache(maxsize=None) # noqa: B019
def version_tuple(self): def version_tuple(self):
ret = Wrapper().ParseGitVersion() ret = Wrapper().ParseGitVersion()
if ret is None: if ret is None:
@@ -95,7 +95,7 @@ def RepoSourceVersion():
ver = ver[1:] ver = ver[1:]
else: else:
ver = "unknown" ver = "unknown"
setattr(RepoSourceVersion, "version", ver) RepoSourceVersion.version = ver
return ver return ver
@@ -611,7 +611,7 @@ class GitCommandError(GitError):
self.git_stderr = git_stderr self.git_stderr = git_stderr
@property @property
@functools.lru_cache(maxsize=None) @functools.lru_cache(maxsize=None) # noqa: B019
def suggestion(self): def suggestion(self):
"""Returns helpful next steps for the given stderr.""" """Returns helpful next steps for the given stderr."""
if not self.git_stderr: if not self.git_stderr:
+2 -2
View File
@@ -42,7 +42,7 @@ SYNC_STATE_PREFIX = "repo.syncstate."
ID_RE = re.compile(r"^[0-9a-f]{40}$") ID_RE = re.compile(r"^[0-9a-f]{40}$")
REVIEW_CACHE = dict() REVIEW_CACHE = {}
def IsChange(rev): def IsChange(rev):
@@ -111,7 +111,7 @@ class GitConfig:
return cls(configfile=os.path.join(gitdir, "config"), defaults=defaults) return cls(configfile=os.path.join(gitdir, "config"), defaults=defaults)
def __init__(self, configfile, defaults=None, jsonFile=None): def __init__(self, configfile, defaults=None, jsonFile=None):
self.file = configfile self.file = str(configfile)
self.defaults = defaults self.defaults = defaults
self._cache_dict = None self._cache_dict = None
self._section_dict = None self._section_dict = None
+106 -6
View File
@@ -19,6 +19,7 @@ People shouldn't run this directly; instead, they should use the `repo` wrapper
which takes care of execing this entry point. which takes care of execing this entry point.
""" """
import difflib
import getpass import getpass
import json import json
import netrc import netrc
@@ -29,6 +30,7 @@ import signal
import sys import sys
import textwrap import textwrap
import time import time
from typing import Optional
import urllib.request import urllib.request
from repo_logging import RepoLogger from repo_logging import RepoLogger
@@ -292,6 +294,102 @@ class _Repo:
result = run() result = run()
return result return result
def _autocorrect_command_name(
self, name: str, config: RepoConfig
) -> Optional[str]:
"""Autocorrect command name based on user's git config."""
close_commands = difflib.get_close_matches(
name, self.commands.keys(), n=5, cutoff=0.7
)
if not close_commands:
logger.error(
"repo: '%s' is not a repo command. See 'repo help'.", name
)
return None
assumed = close_commands[0]
autocorrect = config.GetString("help.autocorrect")
# If there are multiple close matches, git won't automatically run one.
# We'll always prompt instead of guessing.
if len(close_commands) > 1:
autocorrect = "prompt"
# Handle git configuration boolean values:
# 0, "false", "off", "no", "show": show suggestion (default)
# 1, "true", "on", "yes", "immediate": run suggestion immediately
# "never": don't run or show any suggested command
# "prompt": show the suggestion and prompt for confirmation
# positive number > 1: run suggestion after specified deciseconds
if autocorrect is None:
autocorrect = "0"
autocorrect = autocorrect.lower()
if autocorrect in ("0", "false", "off", "no", "show"):
autocorrect = 0
elif autocorrect in ("true", "on", "yes", "immediate"):
autocorrect = -1 # immediate
elif autocorrect == "never":
return None
elif autocorrect == "prompt":
logger.warning(
"You called a repo command named "
"'%s', which does not exist.",
name,
)
try:
resp = input(f"Run '{assumed}' instead [y/N]? ")
if resp.lower().startswith("y"):
return assumed
except (KeyboardInterrupt, EOFError):
pass
return None
else:
try:
autocorrect = int(autocorrect)
except ValueError:
autocorrect = 0
if autocorrect != 0:
if autocorrect < 0:
logger.warning(
"You called a repo command named "
"'%s', which does not exist.\n"
"Continuing assuming that "
"you meant '%s'.",
name,
assumed,
)
else:
delay = autocorrect * 0.1
logger.warning(
"You called a repo command named "
"'%s', which does not exist.\n"
"Continuing in %.1f seconds, assuming "
"that you meant '%s'.",
name,
delay,
assumed,
)
try:
time.sleep(delay)
except KeyboardInterrupt:
return None
return assumed
logger.error(
"repo: '%s' is not a repo command. See 'repo help'.", name
)
logger.warning(
"The most similar command%s\n\t%s",
"s are" if len(close_commands) > 1 else " is",
"\n\t".join(close_commands),
)
return None
def _RunLong(self, name, gopts, argv, git_trace2_event_log): def _RunLong(self, name, gopts, argv, git_trace2_event_log):
"""Execute the (longer running) requested subcommand.""" """Execute the (longer running) requested subcommand."""
result = 0 result = 0
@@ -306,7 +404,14 @@ class _Repo:
outer_client=outer_client, outer_client=outer_client,
) )
try: if name not in self.commands:
corrected_name = self._autocorrect_command_name(
name, outer_client.globalConfig
)
if not corrected_name:
return 1
name = corrected_name
cmd = self.commands[name]( cmd = self.commands[name](
repodir=self.repodir, repodir=self.repodir,
client=repo_client, client=repo_client,
@@ -315,11 +420,6 @@ class _Repo:
outer_manifest=outer_client.manifest, outer_manifest=outer_client.manifest,
git_event_log=git_trace2_event_log, git_event_log=git_trace2_event_log,
) )
except KeyError:
logger.error(
"repo: '%s' is not a repo command. See 'repo help'.", name
)
return 1
Editor.globalConfig = cmd.client.globalConfig Editor.globalConfig = cmd.client.globalConfig
+41 -9
View File
@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man. .\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "March 2026" "repo manifest" "Repo Manual" .TH REPO "1" "April 2026" "repo manifest" "Repo Manual"
.SH NAME .SH NAME
repo \- repo manifest - manual page for repo manifest repo \- repo manifest - manual page for repo manifest
.SH SYNOPSIS .SH SYNOPSIS
@@ -165,15 +165,32 @@ IDREF #IMPLIED>
.TP .TP
<!ATTLIST project revision <!ATTLIST project revision
CDATA #IMPLIED> CDATA #IMPLIED>
.TP
<!ATTLIST project dest\-branch
CDATA #IMPLIED>
.TP
<!ATTLIST project groups
CDATA #IMPLIED>
.TP
<!ATTLIST project sync\-c
CDATA #IMPLIED>
.TP
<!ATTLIST project sync\-s
CDATA #IMPLIED>
.TP
<!ATTLIST project sync\-tags
CDATA #IMPLIED>
.TP
<!ATTLIST project upstream
CDATA #IMPLIED>
.TP
<!ATTLIST project clone\-depth
CDATA #IMPLIED>
.TP
<!ATTLIST project force\-path
CDATA #IMPLIED>
.IP .IP
<!ATTLIST project dest\-branch CDATA #IMPLIED> <!ATTLIST project sync\-strategy CDATA #IMPLIED>
<!ATTLIST project groups CDATA #IMPLIED>
<!ATTLIST project sync\-c CDATA #IMPLIED>
<!ATTLIST project sync\-s CDATA #IMPLIED>
<!ATTLIST project sync\-tags CDATA #IMPLIED>
<!ATTLIST project upstream CDATA #IMPLIED>
<!ATTLIST project clone\-depth CDATA #IMPLIED>
<!ATTLIST project force\-path CDATA #IMPLIED>
.IP .IP
<!ELEMENT annotation EMPTY> <!ELEMENT annotation EMPTY>
<!ATTLIST annotation name CDATA #REQUIRED> <!ATTLIST annotation name CDATA #REQUIRED>
@@ -469,6 +486,21 @@ mirror repository according to its `path` attribute (if supplied) rather than
the `name` attribute. This attribute only applies to the local mirrors syncing, the `name` attribute. This attribute only applies to the local mirrors syncing,
it will be ignored when syncing the projects in a client working directory. it will be ignored when syncing the projects in a client working directory.
.PP .PP
Attribute `sync\-strategy`: Set the sync strategy used when fetching this
project. Currently the only supported value is `stateless`. When set to
`stateless`, repo will run a reflog expiration and aggressive garbage collection
at the end of the sync process. This is useful for projects that contain large
binary files and use `clone\-depth="1"`, where garbage can accumulate as binaries
are added, deleted, or modified across successive syncs.
.PP
During a stateless sync, repo checks the following before cleaning up: 1. The
project does not share an object directory with other projects. 2. The working
tree is clean (no uncommitted changes, no untracked files). 3. There are no
unpushed local commits. 4. There is no Git stash.
.PP
If any of these conditions are not met, repo falls back to a standard sync
without garbage collection.
.PP
Element extend\-project Element extend\-project
.PP .PP
Modify the attributes of the named project. Modify the attributes of the named project.
+8 -3
View File
@@ -759,14 +759,17 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
if p.clone_depth: if p.clone_depth:
e.setAttribute("clone-depth", str(p.clone_depth)) e.setAttribute("clone-depth", str(p.clone_depth))
if p.sync_strategy:
e.setAttribute("sync-strategy", str(p.sync_strategy))
self._output_manifest_project_extras(p, e) self._output_manifest_project_extras(p, e)
if p.subprojects: if p.subprojects:
subprojects = {subp.name for subp in p.subprojects} subprojects = {subp.name for subp in p.subprojects}
output_projects(p, e, list(sorted(subprojects))) output_projects(p, e, sorted(subprojects))
projects = {p.name for p in self._paths.values() if not p.parent} projects = {p.name for p in self._paths.values() if not p.parent}
output_projects(None, root, list(sorted(projects))) output_projects(None, root, sorted(projects))
if self._repo_hooks_project: if self._repo_hooks_project:
root.appendChild(doc.createTextNode("")) root.appendChild(doc.createTextNode(""))
@@ -823,7 +826,6 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
"submanifest", "submanifest",
# These are children of 'project' nodes. # These are children of 'project' nodes.
"annotation", "annotation",
"project",
"copyfile", "copyfile",
"linkfile", "linkfile",
} }
@@ -1939,6 +1941,8 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
% (self.manifestFile, clone_depth) % (self.manifestFile, clone_depth)
) )
sync_strategy = node.getAttribute("sync-strategy") or None
dest_branch = ( dest_branch = (
node.getAttribute("dest-branch") or self._default.destBranchExpr node.getAttribute("dest-branch") or self._default.destBranchExpr
) )
@@ -1985,6 +1989,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
sync_s=sync_s, sync_s=sync_s,
sync_tags=sync_tags, sync_tags=sync_tags,
clone_depth=clone_depth, clone_depth=clone_depth,
sync_strategy=sync_strategy,
upstream=upstream, upstream=upstream,
parent=parent, parent=parent,
dest_branch=dest_branch, dest_branch=dest_branch,
+2
View File
@@ -159,6 +159,8 @@ class Progress:
inc: The number of items completed. inc: The number of items completed.
msg: The message to display. If None, use the last message. msg: The message to display. If None, use the last message.
""" """
if self._ended:
return
self._done += inc self._done += inc
if msg is None: if msg is None:
msg = self._last_msg msg = self._last_msg
+124 -28
View File
@@ -225,7 +225,7 @@ class ReviewableBranch:
@property @property
def unabbrev_commits(self): def unabbrev_commits(self):
r = dict() r = {}
for commit in self.project.bare_git.rev_list( for commit in self.project.bare_git.rev_list(
not_rev(self.base), R_HEADS + self.name, "--" not_rev(self.base), R_HEADS + self.name, "--"
): ):
@@ -553,11 +553,12 @@ class Project:
revisionExpr, revisionExpr,
revisionId, revisionId,
rebase=True, rebase=True,
groups=set(), groups=None,
sync_c=False, sync_c=False,
sync_s=False, sync_s=False,
sync_tags=True, sync_tags=True,
clone_depth=None, clone_depth=None,
sync_strategy=None,
upstream=None, upstream=None,
parent=None, parent=None,
use_git_worktrees=False, use_git_worktrees=False,
@@ -605,11 +606,12 @@ class Project:
self.SetRevision(revisionExpr, revisionId=revisionId) self.SetRevision(revisionExpr, revisionId=revisionId)
self.rebase = rebase self.rebase = rebase
self.groups = groups self.groups = groups if groups is not None else set()
self.sync_c = sync_c self.sync_c = sync_c
self.sync_s = sync_s self.sync_s = sync_s
self.sync_tags = sync_tags self.sync_tags = sync_tags
self.clone_depth = clone_depth self.clone_depth = clone_depth
self.sync_strategy = sync_strategy
self.upstream = upstream self.upstream = upstream
self.parent = parent self.parent = parent
# NB: Do not use this setting in __init__ to change behavior so that the # NB: Do not use this setting in __init__ to change behavior so that the
@@ -627,6 +629,7 @@ class Project:
self.linkfiles = {} self.linkfiles = {}
self.annotations = [] self.annotations = []
self.dest_branch = dest_branch self.dest_branch = dest_branch
self.stateless_prune_needed = False
# This will be filled in if a project is later identified to be the # This will be filled in if a project is later identified to be the
# project containing repo hooks. # project containing repo hooks.
@@ -756,6 +759,18 @@ class Project:
return True return True
return False return False
def HasStash(self) -> bool:
"""Returns True if there is a stash in the repository."""
p = GitCommand(
self,
["rev-parse", "--verify", "refs/stash"],
bare=True,
capture_stdout=True,
capture_stderr=True,
log_as_error=False,
)
return p.Wait() == 0
_userident_name = None _userident_name = None
_userident_email = None _userident_email = None
@@ -943,7 +958,7 @@ class Project:
out.important("prior sync failed; rebase still in progress") out.important("prior sync failed; rebase still in progress")
out.nl() out.nl()
paths = list() paths = []
paths.extend(di.keys()) paths.extend(di.keys())
paths.extend(df.keys()) paths.extend(df.keys())
paths.extend(do) paths.extend(do)
@@ -1239,6 +1254,67 @@ class Project:
logger.error("error: Cannot extract archive %s: %s", tarpath, e) logger.error("error: Cannot extract archive %s: %s", tarpath, e)
return False return False
def _ShouldStatelessPrune(
self, use_superproject: Optional[bool] = None
) -> bool:
"""Determines if a stateless prune should be performed.
Stateless pruning reclaims space by running a reflog expiration and
garbage collection instead of an incremental fetch. It is only performed
if the repository is clean and has no local-only state.
"""
if not self.Exists:
return False
if self._CheckForImmutableRevision(use_superproject=use_superproject):
return False
# Query the target hash from remote to see if we are up-to-date.
target_hash = None
if IsId(self.revisionExpr):
target_hash = self.revisionExpr
else:
output = self._LsRemote(self.upstream or self.revisionExpr)
if output:
target_hash = output.splitlines()[0].split()[0]
if not target_hash:
return False
try:
local_head = self.bare_git.rev_parse("HEAD")
except GitError:
local_head = None
if target_hash == local_head:
return False
# Skip if sharing objects with other projects.
shares_objdir = self.UseAlternates or self.use_git_worktrees
if not shares_objdir:
for p in self.manifest.GetProjectsWithName(self.name):
if p != self and p.objdir == self.objdir:
shares_objdir = True
break
if shares_objdir:
return False
# Skip if HEAD contains any unpushed local commits.
try:
local_commits = self.bare_git.rev_list(
"--count", "HEAD", "--not", "--remotes", "--tags"
)
if int(local_commits[0]) > 0:
return False
except (GitError, IndexError, ValueError):
return False
if self.IsDirty(consider_untracked=True) or self.HasStash():
return False
return True
def Sync_NetworkHalf( def Sync_NetworkHalf(
self, self,
quiet=False, quiet=False,
@@ -1257,7 +1333,7 @@ class Project:
submodules=False, submodules=False,
ssh_proxy=None, ssh_proxy=None,
clone_filter=None, clone_filter=None,
partial_clone_exclude=set(), partial_clone_exclude=None,
clone_filter_for_depth=None, clone_filter_for_depth=None,
): ):
"""Perform only the network IO portion of the sync process. """Perform only the network IO portion of the sync process.
@@ -1310,10 +1386,17 @@ class Project:
if clone_bundle and os.path.exists(self.objdir): if clone_bundle and os.path.exists(self.objdir):
clone_bundle = False clone_bundle = False
if partial_clone_exclude is None:
partial_clone_exclude = set()
if self.name in partial_clone_exclude: if self.name in partial_clone_exclude:
clone_bundle = True clone_bundle = True
clone_filter = None clone_filter = None
if self.sync_strategy == "stateless" and self._ShouldStatelessPrune(
use_superproject
):
self.stateless_prune_needed = True
if is_new is None: if is_new is None:
is_new = not self.Exists is_new = not self.Exists
if is_new: if is_new:
@@ -1412,6 +1495,10 @@ class Project:
and self._CheckForImmutableRevision( and self._CheckForImmutableRevision(
use_superproject=use_superproject use_superproject=use_superproject
) )
and (
not depth
or os.path.exists(os.path.join(self.gitdir, "shallow"))
)
): ):
remote_fetched = True remote_fetched = True
try: try:
@@ -1598,6 +1685,23 @@ class Project:
def _dosubmodules(): def _dosubmodules():
self._SyncSubmodules(quiet=True) self._SyncSubmodules(quiet=True)
def _doprune() -> None:
"""Expire reflogs and run prune-now GC for stateless sync."""
GitCommand(
self,
["reflog", "expire", "--expire=all", "--all"],
bare=True,
).Wait()
p = GitCommand(
self,
["gc", "--prune=now"],
bare=True,
capture_stdout=True,
capture_stderr=True,
)
if p.Wait() != 0:
logger.warning("warn: %s: stateless gc failed", self.name)
head = self.work_git.GetHead() head = self.work_git.GetHead()
if head.startswith(R_HEADS): if head.startswith(R_HEADS):
branch = head[len(R_HEADS) :] branch = head[len(R_HEADS) :]
@@ -1643,6 +1747,8 @@ class Project:
fail(e) fail(e)
return return
self._CopyAndLinkFiles() self._CopyAndLinkFiles()
if self.stateless_prune_needed:
syncbuf.later2(self, _doprune, not verbose)
return return
if head == revid: if head == revid:
@@ -1789,6 +1895,9 @@ class Project:
if submodules: if submodules:
syncbuf.later1(self, _dosubmodules, not verbose) syncbuf.later1(self, _dosubmodules, not verbose)
if self.stateless_prune_needed:
syncbuf.later2(self, _doprune, not verbose)
def AddCopyFile(self, src, dest, topdir): def AddCopyFile(self, src, dest, topdir):
"""Mark |src| for copying to |dest| (relative to |topdir|). """Mark |src| for copying to |dest| (relative to |topdir|).
@@ -2511,6 +2620,9 @@ class Project:
if is_sha1 or tag_name is not None: if is_sha1 or tag_name is not None:
if self._CheckForImmutableRevision( if self._CheckForImmutableRevision(
use_superproject=use_superproject use_superproject=use_superproject
) and (
not depth
or os.path.exists(os.path.join(self.gitdir, "shallow"))
): ):
if verbose: if verbose:
print( print(
@@ -2568,7 +2680,7 @@ class Project:
if update_ref_cmds: if update_ref_cmds:
GitCommand( GitCommand(
self, self,
["update-ref", "--no-deref", "--stdin"], ["update-ref", "--stdin"],
bare=True, bare=True,
input="".join(update_ref_cmds), input="".join(update_ref_cmds),
).Wait() ).Wait()
@@ -2816,7 +2928,7 @@ class Project:
) )
GitCommand( GitCommand(
self, self,
["update-ref", "--no-deref", "--stdin"], ["update-ref", "--stdin"],
bare=True, bare=True,
input=delete_cmds, input=delete_cmds,
log_as_error=False, log_as_error=False,
@@ -3964,30 +4076,14 @@ class Project:
def GetHead(self): def GetHead(self):
"""Return the ref that HEAD points to.""" """Return the ref that HEAD points to."""
try: try:
symbolic_head = self.rev_parse("--symbolic-full-name", HEAD) return self.symbolic_ref("-q", HEAD, log_as_error=False)
if symbolic_head == HEAD:
# Detached HEAD. Return the commit SHA instead.
return self.rev_parse(HEAD)
return symbolic_head
except GitError as e:
# `git rev-parse --symbolic-full-name HEAD` will fail for unborn
# branches, so try symbolic-ref before falling back to raw file
# parsing.
try:
p = GitCommand(
self._project,
["symbolic-ref", "-q", HEAD],
bare=True,
gitdir=self._gitdir,
capture_stdout=True,
capture_stderr=True,
log_as_error=False,
)
if p.Wait() == 0:
return p.stdout.rstrip("\n")
except GitError: except GitError:
pass pass
try:
# If symbolic-ref fails, try to treat as detached HEAD.
return self.rev_parse(HEAD)
except GitError as e:
logger.warning( logger.warning(
"project %s: unparseable HEAD; trying to recover.\n" "project %s: unparseable HEAD; trying to recover.\n"
"Check that HEAD ref in .git/HEAD is valid. The error " "Check that HEAD ref in .git/HEAD is valid. The error "
+1 -1
View File
@@ -106,7 +106,7 @@ def check_path(opts: argparse.Namespace, path: Path) -> bool:
def check_paths(opts: argparse.Namespace, paths: list[Path]) -> bool: def check_paths(opts: argparse.Namespace, paths: list[Path]) -> bool:
"""Check all the paths.""" """Check all the paths."""
# NB: Use list comprehension and not a generator so we check all paths. # NB: Use list comprehension and not a generator so we check all paths.
return all([check_path(opts, x) for x in paths]) return all([check_path(opts, x) for x in paths]) # noqa: C419
def find_files(opts: argparse.Namespace) -> list[Path]: def find_files(opts: argparse.Namespace) -> list[Path]:
+13 -3
View File
@@ -48,10 +48,10 @@ wheel: <
version: "version:3.0.7" version: "version:3.0.7"
> >
# Required by pytest==8.3.4 # Required by pytest==8.3.4 and flake8-bugbear==24.12.12
wheel: < wheel: <
name: "infra/python/wheels/attrs-py2_py3" name: "infra/python/wheels/attrs-py3"
version: "version:21.4.0" version: "version:24.2.0"
> >
# NB: Keep in sync with constraints.txt. # NB: Keep in sync with constraints.txt.
@@ -119,6 +119,16 @@ wheel: <
version: "version:2.10.0" version: "version:2.10.0"
> >
wheel: <
name: "infra/python/wheels/flake8-bugbear-py3"
version: "version:24.12.12"
>
wheel: <
name: "infra/python/wheels/flake8-comprehensions-py3"
version: "version:3.16.0"
>
wheel: < wheel: <
name: "infra/python/wheels/isort-py3" name: "infra/python/wheels/isort-py3"
version: "version:5.10.1" version: "version:5.10.1"
+1 -1
View File
@@ -149,7 +149,7 @@ class ProxyManager:
while True: while True:
try: try:
procs.pop(0) procs.pop(0)
except: # noqa: E722 except IndexError:
break break
def close(self): def close(self):
+13 -4
View File
@@ -16,6 +16,7 @@ import os
from typing import List, Set from typing import List, Set
from command import Command from command import Command
from git_command import git_require
from git_command import GitCommand from git_command import GitCommand
import platform_utils import platform_utils
from progress import Progress from progress import Progress
@@ -204,6 +205,7 @@ class Gc(Command):
[ [
"rev-list", "rev-list",
"--objects", "--objects",
"--missing=allow-promisor",
f"--remotes={project.remote.name}", f"--remotes={project.remote.name}",
"--filter=blob:none", "--filter=blob:none",
"--tags", "--tags",
@@ -215,7 +217,12 @@ class Gc(Command):
# Get all local objects and pack them. # Get all local objects and pack them.
local_head_objects_cmd = GitCommand( local_head_objects_cmd = GitCommand(
project, project,
["rev-list", "--objects", "HEAD^{tree}"], [
"rev-list",
"--objects",
"--missing=allow-promisor",
"HEAD^{tree}",
],
capture_stdout=True, capture_stdout=True,
verify_command=True, verify_command=True,
) )
@@ -224,6 +231,7 @@ class Gc(Command):
[ [
"rev-list", "rev-list",
"--objects", "--objects",
"--missing=allow-promisor",
"--all", "--all",
"--reflog", "--reflog",
"--indexed-objects", "--indexed-objects",
@@ -297,7 +305,8 @@ class Gc(Command):
if ret != 0: if ret != 0:
return ret return ret
if not opt.repack: if opt.repack:
return git_require((2, 17, 0), fail=True, msg="--repack")
ret = self.repack_projects(projects, opt)
return self.repack_projects(projects, opt) return ret
+1 -1
View File
@@ -93,7 +93,7 @@ contain a line that matches both expressions:
pt = getattr(parser.values, "cmd_argv", None) pt = getattr(parser.values, "cmd_argv", None)
if pt is None: if pt is None:
pt = [] pt = []
setattr(parser.values, "cmd_argv", pt) parser.values.cmd_argv = pt
if opt_str == "-(": if opt_str == "-(":
pt.append("(") pt.append("(")
+2 -4
View File
@@ -59,7 +59,7 @@ Displays detailed usage information about a command.
def PrintAllCommandsBody(self): def PrintAllCommandsBody(self):
print("The complete list of recognized repo commands is:") print("The complete list of recognized repo commands is:")
commandNames = list(sorted(all_commands)) commandNames = sorted(all_commands)
self._PrintCommands(commandNames) self._PrintCommands(commandNames)
print( print(
"See 'repo help <command>' for more information on a " "See 'repo help <command>' for more information on a "
@@ -74,11 +74,9 @@ Displays detailed usage information about a command.
def PrintCommonCommandsBody(self): def PrintCommonCommandsBody(self):
print("The most commonly used repo commands are:") print("The most commonly used repo commands are:")
commandNames = list( commandNames = sorted(
sorted(
name for name, command in all_commands.items() if command.COMMON name for name, command in all_commands.items() if command.COMMON
) )
)
self._PrintCommands(commandNames) self._PrintCommands(commandNames)
print( print(
+39 -21
View File
@@ -947,7 +947,7 @@ later is required to fix a server side protocol bug.
"sync_dict" "sync_dict"
] = multiprocessing.Manager().dict() ] = multiprocessing.Manager().dict()
objdir_project_map = dict() objdir_project_map = {}
for index, project in enumerate(projects): for index, project in enumerate(projects):
objdir_project_map.setdefault(project.objdir, []).append(index) objdir_project_map.setdefault(project.objdir, []).append(index)
projects_list = list(objdir_project_map.values()) projects_list = list(objdir_project_map.values())
@@ -1244,14 +1244,15 @@ later is required to fix a server side protocol bug.
return False return False
def _SetPreciousObjectsState(self, project: Project, opt): @classmethod
def _SetPreciousObjectsState(cls, project: Project, opt):
"""Correct the preciousObjects state for the project. """Correct the preciousObjects state for the project.
Args: Args:
project: the project to examine, and possibly correct. project: the project to examine, and possibly correct.
opt: options given to sync. opt: options given to sync.
""" """
expected = self._GetPreciousObjectsState(project, opt) expected = cls._GetPreciousObjectsState(project, opt)
actual = ( actual = (
project.config.GetBoolean("extensions.preciousObjects") or False project.config.GetBoolean("extensions.preciousObjects") or False
) )
@@ -1285,7 +1286,22 @@ later is required to fix a server side protocol bug.
project.config.SetString("extensions.preciousObjects", None) project.config.SetString("extensions.preciousObjects", None)
project.config.SetString("gc.pruneExpire", None) project.config.SetString("gc.pruneExpire", None)
def _GCProjects(self, projects, opt, err_event): @staticmethod
def _RunOneGC(project: Project, config: Optional[dict] = None) -> None:
"""Run auto GC on a single project."""
local_config = {}
if config:
local_config.update(config)
local_config["gc.autoDetach"] = "false"
project.bare_git.gc("--auto", config=local_config)
@classmethod
def _GCProjects(
cls,
projects: List[Project],
opt: optparse.Values,
err_event: _threading.Event,
) -> None:
"""Perform garbage collection. """Perform garbage collection.
If We are skipping garbage collection (opt.auto_gc not set), we still If We are skipping garbage collection (opt.auto_gc not set), we still
@@ -1295,7 +1311,7 @@ later is required to fix a server side protocol bug.
if not opt.auto_gc: if not opt.auto_gc:
# Just repair preciousObjects state, and return. # Just repair preciousObjects state, and return.
for project in projects: for project in projects:
self._SetPreciousObjectsState(project, opt) cls._SetPreciousObjectsState(project, opt)
return return
pm = Progress( pm = Progress(
@@ -1305,9 +1321,8 @@ later is required to fix a server side protocol bug.
tidy_dirs = {} tidy_dirs = {}
for project in projects: for project in projects:
self._SetPreciousObjectsState(project, opt) cls._SetPreciousObjectsState(project, opt)
project.config.SetString("gc.autoDetach", "false")
# Only call git gc once per objdir, but call pack-refs for the # Only call git gc once per objdir, but call pack-refs for the
# remainder. # remainder.
if project.objdir not in tidy_dirs: if project.objdir not in tidy_dirs:
@@ -1328,7 +1343,7 @@ later is required to fix a server side protocol bug.
pm.update(msg=bare_git._project.name) pm.update(msg=bare_git._project.name)
if run_gc: if run_gc:
bare_git.gc("--auto") cls._RunOneGC(bare_git._project)
else: else:
bare_git.pack_refs() bare_git.pack_refs()
pm.end() pm.end()
@@ -1345,7 +1360,7 @@ later is required to fix a server side protocol bug.
try: try:
try: try:
if run_gc: if run_gc:
bare_git.gc("--auto", config=config) cls._RunOneGC(bare_git._project, config=config)
else: else:
bare_git.pack_refs(config=config) bare_git.pack_refs(config=config)
except GitError: except GitError:
@@ -1448,7 +1463,6 @@ later is required to fix a server side protocol bug.
if not projects: if not projects:
return return
bloated_projects = []
pm = Progress( pm = Progress(
"Checking for bloat", len(projects), delay=False, quiet=opt.quiet "Checking for bloat", len(projects), delay=False, quiet=opt.quiet
) )
@@ -1456,7 +1470,7 @@ later is required to fix a server side protocol bug.
def _ProcessResults(pool, pm, results): def _ProcessResults(pool, pm, results):
for result in results: for result in results:
if result: if result:
bloated_projects.append(result) self._bloated_projects.append(result)
pm.update(msg="") pm.update(msg="")
with self.ParallelContext(): with self.ParallelContext():
@@ -1471,15 +1485,6 @@ later is required to fix a server side protocol bug.
) )
pm.end() pm.end()
for project_name in bloated_projects:
warn_msg = (
f'warning: Project "{project_name}" is accumulating '
'unoptimized data. Please run "repo sync --auto-gc" or '
'"repo gc --repack" to clean up.'
)
self.git_event_log.ErrorEvent(warn_msg)
logger.warning(warn_msg)
def _UpdateRepoProject(self, opt, manifest, errors): def _UpdateRepoProject(self, opt, manifest, errors):
"""Fetch the repo project and check for updates.""" """Fetch the repo project and check for updates."""
if opt.local_only: if opt.local_only:
@@ -2097,6 +2102,7 @@ later is required to fix a server side protocol bug.
self._fetch_times = _FetchTimes(manifest) self._fetch_times = _FetchTimes(manifest)
self._local_sync_state = LocalSyncState(manifest) self._local_sync_state = LocalSyncState(manifest)
self._bloated_projects = []
if opt.interleaved: if opt.interleaved:
sync_method = self._SyncInterleaved sync_method = self._SyncInterleaved
@@ -2113,6 +2119,9 @@ later is required to fix a server side protocol bug.
superproject_logging_data, superproject_logging_data,
) )
if not opt.quiet:
print("Finalizing sync state...")
# Log the previous sync analysis state from the config. # Log the previous sync analysis state from the config.
self.git_event_log.LogDataConfigEvents( self.git_event_log.LogDataConfigEvents(
mp.config.GetSyncAnalysisStateData(), "previous_sync_state" mp.config.GetSyncAnalysisStateData(), "previous_sync_state"
@@ -2134,6 +2143,15 @@ later is required to fix a server side protocol bug.
if existing: if existing:
self._CheckForBloatedProjects(all_projects, opt) self._CheckForBloatedProjects(all_projects, opt)
for project_name in sorted(self._bloated_projects):
warn_msg = (
f'warning: Project "{project_name}" is accumulating '
'unoptimized data. Please run "repo sync --auto-gc" or '
'"repo gc --repack" to clean up.'
)
self.git_event_log.ErrorEvent(warn_msg)
logger.warning(warn_msg)
if not opt.quiet: if not opt.quiet:
print("repo sync has finished successfully.") print("repo sync has finished successfully.")
@@ -2657,7 +2675,7 @@ later is required to fix a server side protocol bug.
if previously_pending_relpaths == pending_relpaths: if previously_pending_relpaths == pending_relpaths:
stalled_projects_str = "\n".join( stalled_projects_str = "\n".join(
f" - {path}" f" - {path}"
for path in sorted(list(pending_relpaths)) for path in sorted(pending_relpaths)
) )
logger.error( logger.error(
"The following projects failed and could " "The following projects failed and could "
+4 -3
View File
@@ -803,9 +803,10 @@ Gerrit Code Review: https://www.gerritcodereview.com/
project_list=pending_proj_names, worktree_list=pending_worktrees project_list=pending_proj_names, worktree_list=pending_worktrees
): ):
if LocalSyncState(manifest).IsPartiallySynced(): if LocalSyncState(manifest).IsPartiallySynced():
logger.error( logger.info(
"Partially synced tree detected. Syncing all projects " "Tip: A partially synced tree was detected. "
"may resolve issues you're seeing." "If this failure involves cross-project dependencies, "
"a full `repo sync` might help."
) )
ret = 1 ret = 1
if ret: if ret:
+2 -8
View File
@@ -14,23 +14,17 @@
"""Unittests for the color.py module.""" """Unittests for the color.py module."""
import os
import pytest import pytest
import utils_for_test
import color import color
import git_config import git_config
def fixture(*paths: str) -> str:
"""Return a path relative to test/fixtures."""
return os.path.join(os.path.dirname(__file__), "fixtures", *paths)
@pytest.fixture @pytest.fixture
def coloring() -> color.Coloring: def coloring() -> color.Coloring:
"""Create a Coloring object for testing.""" """Create a Coloring object for testing."""
config_fixture = fixture("test.gitconfig") config_fixture = utils_for_test.FIXTURES_DIR / "test.gitconfig"
config = git_config.GitConfig(config_fixture) config = git_config.GitConfig(config_fixture)
color.SetDefaultColoring("true") color.SetDefaultColoring("true")
return color.Coloring(config, "status") return color.Coloring(config, "status")
+3 -8
View File
@@ -14,24 +14,19 @@
"""Unittests for the git_config.py module.""" """Unittests for the git_config.py module."""
import os
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
import pytest import pytest
import utils_for_test
import git_config import git_config
def fixture_path(*paths: str) -> str:
"""Return a path relative to test/fixtures."""
return os.path.join(os.path.dirname(__file__), "fixtures", *paths)
@pytest.fixture @pytest.fixture
def readonly_config() -> git_config.GitConfig: def readonly_config() -> git_config.GitConfig:
"""Create a GitConfig object using the test.gitconfig fixture.""" """Create a GitConfig object using the test.gitconfig fixture."""
config_fixture = fixture_path("test.gitconfig") config_fixture = utils_for_test.FIXTURES_DIR / "test.gitconfig"
return git_config.GitConfig(config_fixture) return git_config.GitConfig(config_fixture)
@@ -63,7 +58,7 @@ def test_get_string_with_true_value(
def test_get_string_from_missing_file() -> None: def test_get_string_from_missing_file() -> None:
"""Test missing config file.""" """Test missing config file."""
config_fixture = fixture_path("not.present.gitconfig") config_fixture = utils_for_test.FIXTURES_DIR / "not.present.gitconfig"
config = git_config.GitConfig(config_fixture) config = git_config.GitConfig(config_fixture)
val = config.GetString("empty") val = config.GetString("empty")
assert val is None assert val is None
+215 -188
View File
@@ -18,17 +18,24 @@ import contextlib
import io import io
import json import json
import os import os
import re
import socket import socket
import tempfile import tempfile
import threading import threading
import unittest from typing import Any, Dict, List, Optional
from unittest import mock from unittest import mock
import pytest
import git_trace2_event_log import git_trace2_event_log
import platform_utils import platform_utils
def serverLoggingThread(socket_path, server_ready, received_traces): def server_logging_thread(
socket_path: str,
server_ready: threading.Condition,
received_traces: List[str],
) -> None:
"""Helper function to receive logs over a Unix domain socket. """Helper function to receive logs over a Unix domain socket.
Appends received messages on the provided socket and appends to Appends received messages on the provided socket and appends to
@@ -57,46 +64,43 @@ def serverLoggingThread(socket_path, server_ready, received_traces):
received_traces.extend(data.decode("utf-8").splitlines()) received_traces.extend(data.decode("utf-8").splitlines())
class EventLogTestCase(unittest.TestCase):
"""TestCase for the EventLog module."""
PARENT_SID_KEY = "GIT_TRACE2_PARENT_SID" PARENT_SID_KEY = "GIT_TRACE2_PARENT_SID"
PARENT_SID_VALUE = "parent_sid" PARENT_SID_VALUE = "parent_sid"
SELF_SID_REGEX = r"repo-\d+T\d+Z-.*" SELF_SID_REGEX = r"repo-\d+T\d+Z-.*"
FULL_SID_REGEX = rf"^{PARENT_SID_VALUE}/{SELF_SID_REGEX}" FULL_SID_REGEX = rf"^{PARENT_SID_VALUE}/{SELF_SID_REGEX}"
def setUp(self):
"""Load the event_log module every time.""" @pytest.fixture
self._event_log = None def event_log() -> git_trace2_event_log.EventLog:
"""Fixture for the EventLog module."""
# By default we initialize with the expected case where # By default we initialize with the expected case where
# repo launches us (so GIT_TRACE2_PARENT_SID is set). # repo launches us (so GIT_TRACE2_PARENT_SID is set).
env = { env = {PARENT_SID_KEY: PARENT_SID_VALUE}
self.PARENT_SID_KEY: self.PARENT_SID_VALUE, return git_trace2_event_log.EventLog(env=env)
}
self._event_log = git_trace2_event_log.EventLog(env=env)
self._log_data = None
def verifyCommonKeys(
self, log_entry, expected_event_name=None, full_sid=True def verify_common_keys(
): log_entry: Dict[str, Any],
expected_event_name: Optional[str] = None,
full_sid: bool = True,
) -> None:
"""Helper function to verify common event log keys.""" """Helper function to verify common event log keys."""
self.assertIn("event", log_entry) assert "event" in log_entry
self.assertIn("sid", log_entry) assert "sid" in log_entry
self.assertIn("thread", log_entry) assert "thread" in log_entry
self.assertIn("time", log_entry) assert "time" in log_entry
# Do basic data format validation. # Do basic data format validation.
if expected_event_name: if expected_event_name:
self.assertEqual(expected_event_name, log_entry["event"]) assert expected_event_name == log_entry["event"]
if full_sid: if full_sid:
self.assertRegex(log_entry["sid"], self.FULL_SID_REGEX) assert re.match(FULL_SID_REGEX, log_entry["sid"])
else: else:
self.assertRegex(log_entry["sid"], self.SELF_SID_REGEX) assert re.match(SELF_SID_REGEX, log_entry["sid"])
self.assertRegex( assert re.match(r"^\d+-\d+-\d+T\d+:\d+:\d+\.\d+\+00:00$", log_entry["time"])
log_entry["time"], r"^\d+-\d+-\d+T\d+:\d+:\d+\.\d+\+00:00$"
)
def readLog(self, log_path):
def read_log(log_path: str) -> List[Dict[str, Any]]:
"""Helper function to read log data into a list.""" """Helper function to read log data into a list."""
log_data = [] log_data = []
with open(log_path, mode="rb") as f: with open(log_path, mode="rb") as f:
@@ -104,7 +108,8 @@ class EventLogTestCase(unittest.TestCase):
log_data.append(json.loads(line)) log_data.append(json.loads(line))
return log_data return log_data
def remove_prefix(self, s, prefix):
def remove_prefix(s: str, prefix: str) -> str:
"""Return a copy string after removing |prefix| from |s|, if present or """Return a copy string after removing |prefix| from |s|, if present or
the original string.""" the original string."""
if s.startswith(prefix): if s.startswith(prefix):
@@ -112,60 +117,69 @@ class EventLogTestCase(unittest.TestCase):
else: else:
return s return s
def test_initial_state_with_parent_sid(self):
"""Test initial state when 'GIT_TRACE2_PARENT_SID' is set by parent."""
self.assertRegex(self._event_log.full_sid, self.FULL_SID_REGEX)
def test_initial_state_no_parent_sid(self): def test_initial_state_with_parent_sid(
event_log: git_trace2_event_log.EventLog,
) -> None:
"""Test initial state when 'GIT_TRACE2_PARENT_SID' is set by parent."""
assert re.match(FULL_SID_REGEX, event_log.full_sid)
def test_initial_state_no_parent_sid() -> None:
"""Test initial state when 'GIT_TRACE2_PARENT_SID' is not set.""" """Test initial state when 'GIT_TRACE2_PARENT_SID' is not set."""
# Setup an empty environment dict (no parent sid). # Setup an empty environment dict (no parent sid).
self._event_log = git_trace2_event_log.EventLog(env={}) event_log = git_trace2_event_log.EventLog(env={})
self.assertRegex(self._event_log.full_sid, self.SELF_SID_REGEX) assert re.match(SELF_SID_REGEX, event_log.full_sid)
def test_version_event(self):
def test_version_event(event_log: git_trace2_event_log.EventLog) -> None:
"""Test 'version' event data is valid. """Test 'version' event data is valid.
Verify that the 'version' event is written even when no other Verify that the 'version' event is written even when no other
events are addded. events are added.
Expected event log: Expected event log:
<version event> <version event>
""" """
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir: with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log.Write(path=tempdir) log_path = event_log.Write(path=tempdir)
self._log_data = self.readLog(log_path) log_data = read_log(log_path)
# A log with no added events should only have the version entry. # A log with no added events should only have the version entry.
self.assertEqual(len(self._log_data), 1) assert len(log_data) == 1
version_event = self._log_data[0] version_event = log_data[0]
self.verifyCommonKeys(version_event, expected_event_name="version") verify_common_keys(version_event, expected_event_name="version")
# Check for 'version' event specific fields. # Check for 'version' event specific fields.
self.assertIn("evt", version_event) assert "evt" in version_event
self.assertIn("exe", version_event) assert "exe" in version_event
# Verify "evt" version field is a string. # Verify "evt" version field is a string.
self.assertIsInstance(version_event["evt"], str) assert isinstance(version_event["evt"], str)
def test_start_event(self):
def test_start_event(event_log: git_trace2_event_log.EventLog) -> None:
"""Test and validate 'start' event data is valid. """Test and validate 'start' event data is valid.
Expected event log: Expected event log:
<version event> <version event>
<start event> <start event>
""" """
self._event_log.StartEvent([]) event_log.StartEvent([])
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir: with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log.Write(path=tempdir) log_path = event_log.Write(path=tempdir)
self._log_data = self.readLog(log_path) log_data = read_log(log_path)
self.assertEqual(len(self._log_data), 2) assert len(log_data) == 2
start_event = self._log_data[1] start_event = log_data[1]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version") verify_common_keys(log_data[0], expected_event_name="version")
self.verifyCommonKeys(start_event, expected_event_name="start") verify_common_keys(start_event, expected_event_name="start")
# Check for 'start' event specific fields. # Check for 'start' event specific fields.
self.assertIn("argv", start_event) assert "argv" in start_event
self.assertTrue(isinstance(start_event["argv"], list)) assert isinstance(start_event["argv"], list)
def test_exit_event_result_none(self):
def test_exit_event_result_none(
event_log: git_trace2_event_log.EventLog,
) -> None:
"""Test 'exit' event data is valid when result is None. """Test 'exit' event data is valid when result is None.
We expect None result to be converted to 0 in the exit event data. We expect None result to be converted to 0 in the exit event data.
@@ -174,61 +188,68 @@ class EventLogTestCase(unittest.TestCase):
<version event> <version event>
<exit event> <exit event>
""" """
self._event_log.ExitEvent(None) event_log.ExitEvent(None)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir: with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log.Write(path=tempdir) log_path = event_log.Write(path=tempdir)
self._log_data = self.readLog(log_path) log_data = read_log(log_path)
self.assertEqual(len(self._log_data), 2) assert len(log_data) == 2
exit_event = self._log_data[1] exit_event = log_data[1]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version") verify_common_keys(log_data[0], expected_event_name="version")
self.verifyCommonKeys(exit_event, expected_event_name="exit") verify_common_keys(exit_event, expected_event_name="exit")
# Check for 'exit' event specific fields. # Check for 'exit' event specific fields.
self.assertIn("code", exit_event) assert "code" in exit_event
# 'None' result should convert to 0 (successful) return code. # 'None' result should convert to 0 (successful) return code.
self.assertEqual(exit_event["code"], 0) assert exit_event["code"] == 0
def test_exit_event_result_integer(self):
def test_exit_event_result_integer(
event_log: git_trace2_event_log.EventLog,
) -> None:
"""Test 'exit' event data is valid when result is an integer. """Test 'exit' event data is valid when result is an integer.
Expected event log: Expected event log:
<version event> <version event>
<exit event> <exit event>
""" """
self._event_log.ExitEvent(2) event_log.ExitEvent(2)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir: with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log.Write(path=tempdir) log_path = event_log.Write(path=tempdir)
self._log_data = self.readLog(log_path) log_data = read_log(log_path)
self.assertEqual(len(self._log_data), 2) assert len(log_data) == 2
exit_event = self._log_data[1] exit_event = log_data[1]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version") verify_common_keys(log_data[0], expected_event_name="version")
self.verifyCommonKeys(exit_event, expected_event_name="exit") verify_common_keys(exit_event, expected_event_name="exit")
# Check for 'exit' event specific fields. # Check for 'exit' event specific fields.
self.assertIn("code", exit_event) assert "code" in exit_event
self.assertEqual(exit_event["code"], 2) assert exit_event["code"] == 2
def test_command_event(self):
def test_command_event(event_log: git_trace2_event_log.EventLog) -> None:
"""Test and validate 'command' event data is valid. """Test and validate 'command' event data is valid.
Expected event log: Expected event log:
<version event> <version event>
<command event> <command event>
""" """
self._event_log.CommandEvent(name="repo", subcommands=["init", "this"]) event_log.CommandEvent(name="repo", subcommands=["init", "this"])
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir: with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log.Write(path=tempdir) log_path = event_log.Write(path=tempdir)
self._log_data = self.readLog(log_path) log_data = read_log(log_path)
self.assertEqual(len(self._log_data), 2) assert len(log_data) == 2
command_event = self._log_data[1] command_event = log_data[1]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version") verify_common_keys(log_data[0], expected_event_name="version")
self.verifyCommonKeys(command_event, expected_event_name="cmd_name") verify_common_keys(command_event, expected_event_name="cmd_name")
# Check for 'command' event specific fields. # Check for 'command' event specific fields.
self.assertIn("name", command_event) assert "name" in command_event
self.assertEqual(command_event["name"], "repo-init-this") assert command_event["name"] == "repo-init-this"
def test_def_params_event_repo_config(self):
def test_def_params_event_repo_config(
event_log: git_trace2_event_log.EventLog,
) -> None:
"""Test 'def_params' event data outputs only repo config keys. """Test 'def_params' event data outputs only repo config keys.
Expected event log: Expected event log:
@@ -241,24 +262,27 @@ class EventLogTestCase(unittest.TestCase):
"repo.partialclone": "true", "repo.partialclone": "true",
"repo.partialclonefilter": "blob:none", "repo.partialclonefilter": "blob:none",
} }
self._event_log.DefParamRepoEvents(config) event_log.DefParamRepoEvents(config)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir: with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log.Write(path=tempdir) log_path = event_log.Write(path=tempdir)
self._log_data = self.readLog(log_path) log_data = read_log(log_path)
self.assertEqual(len(self._log_data), 3) assert len(log_data) == 3
def_param_events = self._log_data[1:] def_param_events = log_data[1:]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version") verify_common_keys(log_data[0], expected_event_name="version")
for event in def_param_events: for event in def_param_events:
self.verifyCommonKeys(event, expected_event_name="def_param") verify_common_keys(event, expected_event_name="def_param")
# Check for 'def_param' event specific fields. # Check for 'def_param' event specific fields.
self.assertIn("param", event) assert "param" in event
self.assertIn("value", event) assert "value" in event
self.assertTrue(event["param"].startswith("repo.")) assert event["param"].startswith("repo.")
def test_def_params_event_no_repo_config(self):
def test_def_params_event_no_repo_config(
event_log: git_trace2_event_log.EventLog,
) -> None:
"""Test 'def_params' event data won't output non-repo config keys. """Test 'def_params' event data won't output non-repo config keys.
Expected event log: Expected event log:
@@ -268,16 +292,17 @@ class EventLogTestCase(unittest.TestCase):
"git.foo": "bar", "git.foo": "bar",
"git.core.foo2": "baz", "git.core.foo2": "baz",
} }
self._event_log.DefParamRepoEvents(config) event_log.DefParamRepoEvents(config)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir: with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log.Write(path=tempdir) log_path = event_log.Write(path=tempdir)
self._log_data = self.readLog(log_path) log_data = read_log(log_path)
self.assertEqual(len(self._log_data), 1) assert len(log_data) == 1
self.verifyCommonKeys(self._log_data[0], expected_event_name="version") verify_common_keys(log_data[0], expected_event_name="version")
def test_data_event_config(self):
def test_data_event_config(event_log: git_trace2_event_log.EventLog) -> None:
"""Test 'data' event data outputs all config keys. """Test 'data' event data outputs all config keys.
Expected event log: Expected event log:
@@ -292,30 +317,30 @@ class EventLogTestCase(unittest.TestCase):
"repo.syncstate.superproject.sys.argv": ["--", "sync", "protobuf"], "repo.syncstate.superproject.sys.argv": ["--", "sync", "protobuf"],
} }
prefix_value = "prefix" prefix_value = "prefix"
self._event_log.LogDataConfigEvents(config, prefix_value) event_log.LogDataConfigEvents(config, prefix_value)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir: with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log.Write(path=tempdir) log_path = event_log.Write(path=tempdir)
self._log_data = self.readLog(log_path) log_data = read_log(log_path)
self.assertEqual(len(self._log_data), 5) assert len(log_data) == 5
data_events = self._log_data[1:] data_events = log_data[1:]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version") verify_common_keys(log_data[0], expected_event_name="version")
for event in data_events: for event in data_events:
self.verifyCommonKeys(event) verify_common_keys(event)
# Check for 'data' event specific fields. # Check for 'data' event specific fields.
self.assertIn("key", event) assert "key" in event
self.assertIn("value", event) assert "value" in event
key = event["key"] key = event["key"]
key = self.remove_prefix(key, f"{prefix_value}/") key = remove_prefix(key, f"{prefix_value}/")
value = event["value"] value = event["value"]
self.assertEqual( assert event_log.GetDataEventName(value) == event["event"]
self._event_log.GetDataEventName(value), event["event"] assert key in config
) assert value == config[key]
self.assertTrue(key in config and value == config[key])
def test_error_event(self):
def test_error_event(event_log: git_trace2_event_log.EventLog) -> None:
"""Test and validate 'error' event data is valid. """Test and validate 'error' event data is valid.
Expected event log: Expected event log:
@@ -324,63 +349,64 @@ class EventLogTestCase(unittest.TestCase):
""" """
msg = "invalid option: --cahced" msg = "invalid option: --cahced"
fmt = "invalid option: %s" fmt = "invalid option: %s"
self._event_log.ErrorEvent(msg, fmt) event_log.ErrorEvent(msg, fmt)
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir: with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
log_path = self._event_log.Write(path=tempdir) log_path = event_log.Write(path=tempdir)
self._log_data = self.readLog(log_path) log_data = read_log(log_path)
self.assertEqual(len(self._log_data), 2) assert len(log_data) == 2
error_event = self._log_data[1] error_event = log_data[1]
self.verifyCommonKeys(self._log_data[0], expected_event_name="version") verify_common_keys(log_data[0], expected_event_name="version")
self.verifyCommonKeys(error_event, expected_event_name="error") verify_common_keys(error_event, expected_event_name="error")
# Check for 'error' event specific fields. # Check for 'error' event specific fields.
self.assertIn("msg", error_event) assert "msg" in error_event
self.assertIn("fmt", error_event) assert "fmt" in error_event
self.assertEqual(error_event["msg"], f"RepoErrorEvent:{msg}") assert error_event["msg"] == f"RepoErrorEvent:{msg}"
self.assertEqual(error_event["fmt"], f"RepoErrorEvent:{fmt}") assert error_event["fmt"] == f"RepoErrorEvent:{fmt}"
def test_write_with_filename(self):
def test_write_with_filename(event_log: git_trace2_event_log.EventLog) -> None:
"""Test Write() with a path to a file exits with None.""" """Test Write() with a path to a file exits with None."""
self.assertIsNone(self._event_log.Write(path="path/to/file")) assert event_log.Write(path="path/to/file") is None
def test_write_with_git_config(self):
"""Test Write() uses the git config path when 'git config' call def test_write_with_git_config(
succeeds.""" tmp_path,
with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir: event_log: git_trace2_event_log.EventLog,
) -> None:
"""Test Write() uses the git config path when 'git config' call succeeds."""
with mock.patch.object( with mock.patch.object(
self._event_log, event_log,
"_GetEventTargetPath", "_GetEventTargetPath",
return_value=tempdir, return_value=str(tmp_path),
): ):
self.assertEqual( assert os.path.dirname(event_log.Write()) == str(tmp_path)
os.path.dirname(self._event_log.Write()), tempdir
)
def test_write_no_git_config(self):
def test_write_no_git_config(event_log: git_trace2_event_log.EventLog) -> None:
"""Test Write() with no git config variable present exits with None.""" """Test Write() with no git config variable present exits with None."""
with mock.patch.object( with mock.patch.object(event_log, "_GetEventTargetPath", return_value=None):
self._event_log, "_GetEventTargetPath", return_value=None assert event_log.Write() is None
):
self.assertIsNone(self._event_log.Write())
def test_write_non_string(self):
def test_write_non_string(event_log: git_trace2_event_log.EventLog) -> None:
"""Test Write() with non-string type for |path| throws TypeError.""" """Test Write() with non-string type for |path| throws TypeError."""
with self.assertRaises(TypeError): with pytest.raises(TypeError):
self._event_log.Write(path=1234) event_log.Write(path=1234)
@unittest.skipIf(not hasattr(socket, "AF_UNIX"), "Requires AF_UNIX sockets")
def test_write_socket(self): @pytest.mark.skipif(
"""Test Write() with Unix domain socket for |path| and validate received not hasattr(socket, "AF_UNIX"), reason="Requires AF_UNIX sockets"
traces.""" )
received_traces = [] def test_write_socket(event_log: git_trace2_event_log.EventLog) -> None:
with tempfile.TemporaryDirectory( """Test Write() with Unix domain socket and validate received traces."""
prefix="test_server_sockets" received_traces: List[str] = []
) as tempdir: with tempfile.TemporaryDirectory(prefix="test_server_sockets") as tempdir:
socket_path = os.path.join(tempdir, "server.sock") socket_path = os.path.join(tempdir, "server.sock")
server_ready = threading.Condition() server_ready = threading.Condition()
# Start "server" listening on Unix domain socket at socket_path. # Start "server" listening on Unix domain socket at socket_path.
server_thread = threading.Thread( server_thread = threading.Thread(
target=serverLoggingThread, target=server_logging_thread,
args=(socket_path, server_ready, received_traces), args=(socket_path, server_ready, received_traces),
) )
try: try:
@@ -389,73 +415,74 @@ class EventLogTestCase(unittest.TestCase):
with server_ready: with server_ready:
server_ready.wait(timeout=120) server_ready.wait(timeout=120)
self._event_log.StartEvent([]) event_log.StartEvent([])
path = self._event_log.Write(path=f"af_unix:{socket_path}") path = event_log.Write(path=f"af_unix:{socket_path}")
finally: finally:
server_thread.join(timeout=5) server_thread.join(timeout=5)
self.assertEqual(path, f"af_unix:stream:{socket_path}") assert path == f"af_unix:stream:{socket_path}"
self.assertEqual(len(received_traces), 2) assert len(received_traces) == 2
version_event = json.loads(received_traces[0]) version_event = json.loads(received_traces[0])
start_event = json.loads(received_traces[1]) start_event = json.loads(received_traces[1])
self.verifyCommonKeys(version_event, expected_event_name="version") verify_common_keys(version_event, expected_event_name="version")
self.verifyCommonKeys(start_event, expected_event_name="start") verify_common_keys(start_event, expected_event_name="start")
# Check for 'start' event specific fields. # Check for 'start' event specific fields.
self.assertIn("argv", start_event) assert "argv" in start_event
self.assertIsInstance(start_event["argv"], list) assert isinstance(start_event["argv"], list)
class EventLogVerboseTestCase(unittest.TestCase): class TestEventLogVerbose:
"""TestCase for the EventLog module verbose logging.""" """TestCase for the EventLog module verbose logging."""
def setUp(self): def test_write_socket_error_no_verbose(self) -> None:
self._event_log = git_trace2_event_log.EventLog(env={})
def test_write_socket_error_no_verbose(self):
"""Test Write() suppression of socket errors when not verbose.""" """Test Write() suppression of socket errors when not verbose."""
self._event_log.verbose = False event_log = git_trace2_event_log.EventLog(env={})
event_log.verbose = False
with contextlib.redirect_stderr( with contextlib.redirect_stderr(
io.StringIO() io.StringIO()
) as mock_stderr, mock.patch("socket.socket", side_effect=OSError): ) as mock_stderr, mock.patch("socket.socket", side_effect=OSError):
self._event_log.Write(path="af_unix:stream:/tmp/test_sock") event_log.Write(path="af_unix:stream:/tmp/test_sock")
self.assertEqual(mock_stderr.getvalue(), "") assert mock_stderr.getvalue() == ""
def test_write_socket_error_verbose(self): def test_write_socket_error_verbose(self) -> None:
"""Test Write() printing of socket errors when verbose.""" """Test Write() printing of socket errors when verbose."""
self._event_log.verbose = True event_log = git_trace2_event_log.EventLog(env={})
event_log.verbose = True
with contextlib.redirect_stderr( with contextlib.redirect_stderr(
io.StringIO() io.StringIO()
) as mock_stderr, mock.patch( ) as mock_stderr, mock.patch(
"socket.socket", side_effect=OSError("Mock error") "socket.socket", side_effect=OSError("Mock error")
): ):
self._event_log.Write(path="af_unix:stream:/tmp/test_sock") event_log.Write(path="af_unix:stream:/tmp/test_sock")
self.assertIn( assert (
"git trace2 logging failed: Mock error", "git trace2 logging failed: Mock error"
mock_stderr.getvalue(), in mock_stderr.getvalue()
) )
def test_write_file_error_no_verbose(self): def test_write_file_error_no_verbose(self) -> None:
"""Test Write() suppression of file errors when not verbose.""" """Test Write() suppression of file errors when not verbose."""
self._event_log.verbose = False event_log = git_trace2_event_log.EventLog(env={})
event_log.verbose = False
with contextlib.redirect_stderr( with contextlib.redirect_stderr(
io.StringIO() io.StringIO()
) as mock_stderr, mock.patch( ) as mock_stderr, mock.patch(
"tempfile.NamedTemporaryFile", side_effect=FileExistsError "tempfile.NamedTemporaryFile", side_effect=FileExistsError
): ):
self._event_log.Write(path="/tmp") event_log.Write(path="/tmp")
self.assertEqual(mock_stderr.getvalue(), "") assert mock_stderr.getvalue() == ""
def test_write_file_error_verbose(self): def test_write_file_error_verbose(self) -> None:
"""Test Write() printing of file errors when verbose.""" """Test Write() printing of file errors when verbose."""
self._event_log.verbose = True event_log = git_trace2_event_log.EventLog(env={})
event_log.verbose = True
with contextlib.redirect_stderr( with contextlib.redirect_stderr(
io.StringIO() io.StringIO()
) as mock_stderr, mock.patch( ) as mock_stderr, mock.patch(
"tempfile.NamedTemporaryFile", "tempfile.NamedTemporaryFile",
side_effect=FileExistsError("Mock error"), side_effect=FileExistsError("Mock error"),
): ):
self._event_log.Write(path="/tmp") event_log.Write(path="/tmp")
self.assertIn( assert (
"git trace2 logging failed: FileExistsError", "git trace2 logging failed: FileExistsError"
mock_stderr.getvalue(), in mock_stderr.getvalue()
) )
+166
View File
@@ -0,0 +1,166 @@
# Copyright (C) 2026 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tests for the main repo script and subcommand routing."""
from unittest import mock
import pytest
from main import _Repo
@pytest.fixture(name="repo")
def fixture_repo():
repo = _Repo("repodir")
# Overriding the command list here ensures that we are only testing
# against a fixed set of commands, reducing fragility to new
# subcommands being added to the main repo tool.
repo.commands = {"start": None, "sync": None, "smart": None}
return repo
@pytest.fixture(name="mock_config")
def fixture_mock_config():
return mock.MagicMock()
@mock.patch("time.sleep")
def test_autocorrect_delay(mock_sleep, repo, mock_config):
"""Test autocorrect with positive delay."""
mock_config.GetString.return_value = "10"
res = repo._autocorrect_command_name("tart", mock_config)
mock_config.GetString.assert_called_with("help.autocorrect")
mock_sleep.assert_called_with(1.0)
assert res == "start"
@mock.patch("time.sleep")
def test_autocorrect_delay_one(mock_sleep, repo, mock_config):
"""Test autocorrect with '1' (0.1s delay, not immediate)."""
mock_config.GetString.return_value = "1"
res = repo._autocorrect_command_name("tart", mock_config)
mock_sleep.assert_called_with(0.1)
assert res == "start"
@mock.patch("time.sleep", side_effect=KeyboardInterrupt())
def test_autocorrect_delay_interrupt(mock_sleep, repo, mock_config):
"""Test autocorrect handles KeyboardInterrupt during delay."""
mock_config.GetString.return_value = "10"
res = repo._autocorrect_command_name("tart", mock_config)
mock_sleep.assert_called_with(1.0)
assert res is None
@mock.patch("time.sleep")
def test_autocorrect_immediate(mock_sleep, repo, mock_config):
"""Test autocorrect with immediate/negative delay."""
# Test numeric negative.
mock_config.GetString.return_value = "-1"
res = repo._autocorrect_command_name("tart", mock_config)
mock_sleep.assert_not_called()
assert res == "start"
# Test string boolean "true".
mock_config.GetString.return_value = "true"
res = repo._autocorrect_command_name("tart", mock_config)
mock_sleep.assert_not_called()
assert res == "start"
# Test string boolean "yes".
mock_config.GetString.return_value = "YES"
res = repo._autocorrect_command_name("tart", mock_config)
mock_sleep.assert_not_called()
assert res == "start"
# Test string boolean "immediate".
mock_config.GetString.return_value = "Immediate"
res = repo._autocorrect_command_name("tart", mock_config)
mock_sleep.assert_not_called()
assert res == "start"
def test_autocorrect_zero_or_show(repo, mock_config):
"""Test autocorrect with zero delay (suggestions only)."""
# Test numeric zero.
mock_config.GetString.return_value = "0"
res = repo._autocorrect_command_name("tart", mock_config)
assert res is None
# Test string boolean "false".
mock_config.GetString.return_value = "False"
res = repo._autocorrect_command_name("tart", mock_config)
assert res is None
# Test string boolean "show".
mock_config.GetString.return_value = "show"
res = repo._autocorrect_command_name("tart", mock_config)
assert res is None
def test_autocorrect_never(repo, mock_config):
"""Test autocorrect with 'never'."""
mock_config.GetString.return_value = "never"
res = repo._autocorrect_command_name("tart", mock_config)
assert res is None
@mock.patch("builtins.input", return_value="y")
def test_autocorrect_prompt_yes(mock_input, repo, mock_config):
"""Test autocorrect with prompt and user answers yes."""
mock_config.GetString.return_value = "prompt"
res = repo._autocorrect_command_name("tart", mock_config)
assert res == "start"
@mock.patch("builtins.input", return_value="n")
def test_autocorrect_prompt_no(mock_input, repo, mock_config):
"""Test autocorrect with prompt and user answers no."""
mock_config.GetString.return_value = "prompt"
res = repo._autocorrect_command_name("tart", mock_config)
assert res is None
@mock.patch("builtins.input", return_value="y")
def test_autocorrect_multiple_candidates(mock_input, repo, mock_config):
"""Test autocorrect with multiple matches forces a prompt."""
mock_config.GetString.return_value = "10" # Normally just delay
# 'snart' matches both 'start' and 'smart' with > 0.7 ratio
res = repo._autocorrect_command_name("snart", mock_config)
# Because there are multiple candidates, it should prompt
mock_input.assert_called_once()
assert res == "start"
@mock.patch("builtins.input", side_effect=KeyboardInterrupt())
def test_autocorrect_prompt_interrupt(mock_input, repo, mock_config):
"""Test autocorrect with prompt and user interrupts."""
mock_config.GetString.return_value = "prompt"
res = repo._autocorrect_command_name("tart", mock_config)
assert res is None
+551 -490
View File
File diff suppressed because it is too large Load Diff
+229
View File
@@ -17,10 +17,12 @@
import contextlib import contextlib
import os import os
from pathlib import Path from pathlib import Path
import shutil
import subprocess import subprocess
import tempfile import tempfile
from typing import Optional from typing import Optional
import unittest import unittest
from unittest import mock
import utils_for_test import utils_for_test
@@ -565,3 +567,230 @@ class ManifestPropertiesFetchedCorrectly(unittest.TestCase):
fakeproj.config.SetString("manifest.platform", "auto") fakeproj.config.SetString("manifest.platform", "auto")
self.assertEqual(fakeproj.manifest_platform, "auto") self.assertEqual(fakeproj.manifest_platform, "auto")
class StatelessSyncTests(unittest.TestCase):
"""Tests for stateless sync strategy."""
def _get_project(self, tempdir):
manifest = mock.MagicMock()
manifest.manifestProject.depth = None
manifest.manifestProject.dissociate = False
manifest.manifestProject.clone_filter = None
manifest.is_multimanifest = False
manifest.manifestProject.config.GetBoolean.return_value = False
remote = mock.MagicMock()
remote.name = "origin"
remote.url = "http://"
proj = project.Project(
manifest=manifest,
name="test-project",
remote=remote,
gitdir=os.path.join(tempdir, ".git"),
objdir=os.path.join(tempdir, ".git"),
worktree=tempdir,
relpath="test-project",
revisionExpr="1234abcd",
revisionId=None,
sync_strategy="stateless",
)
proj._CheckForImmutableRevision = mock.MagicMock(return_value=False)
proj._LsRemote = mock.MagicMock(
return_value="1234abcd\trefs/heads/main\n"
)
proj.bare_git = mock.MagicMock()
proj.bare_git.rev_parse.return_value = "5678abcd"
proj.bare_git.rev_list.return_value = ["0"]
proj.IsDirty = mock.MagicMock(return_value=False)
proj.GetBranches = mock.MagicMock(return_value=[])
proj.DeleteWorktree = mock.MagicMock()
proj._InitGitDir = mock.MagicMock()
proj._RemoteFetch = mock.MagicMock(return_value=True)
proj._InitRemote = mock.MagicMock()
proj._InitMRef = mock.MagicMock()
return proj
def test_sync_network_half_stateless_prune_needed(self):
"""Test stateless sync queues prune when needed."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir)
res = proj.Sync_NetworkHalf()
self.assertTrue(res.success)
proj.DeleteWorktree.assert_not_called()
self.assertTrue(proj.stateless_prune_needed)
proj._RemoteFetch.assert_called_once()
def test_sync_local_half_stateless_prune(self):
"""Test stateless GC pruning is queued in Sync_LocalHalf."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir)
proj.stateless_prune_needed = True
proj._Checkout = mock.MagicMock()
proj._InitWorkTree = mock.MagicMock()
proj.IsRebaseInProgress = mock.MagicMock(return_value=False)
proj.IsCherryPickInProgress = mock.MagicMock(return_value=False)
proj.bare_ref = mock.MagicMock()
proj.bare_ref.all = {}
proj.GetRevisionId = mock.MagicMock(return_value="1234abcd")
proj._CopyAndLinkFiles = mock.MagicMock()
proj.work_git = mock.MagicMock()
proj.work_git.GetHead.return_value = "5678abcd"
syncbuf = project.SyncBuffer(proj.config)
with mock.patch("project.GitCommand") as mock_git_cmd:
mock_cmd_instance = mock.MagicMock()
mock_cmd_instance.Wait.return_value = 0
mock_git_cmd.return_value = mock_cmd_instance
proj.Sync_LocalHalf(syncbuf)
syncbuf.Finish()
self.assertEqual(mock_git_cmd.call_count, 2)
mock_git_cmd.assert_any_call(
proj, ["reflog", "expire", "--expire=all", "--all"], bare=True
)
mock_git_cmd.assert_any_call(
proj,
["gc", "--prune=now"],
bare=True,
capture_stdout=True,
capture_stderr=True,
)
def test_sync_network_half_stateless_skips_if_stash(self):
"""Test stateless sync skips if stash exists."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir)
proj.HasStash = mock.MagicMock(return_value=True)
res = proj.Sync_NetworkHalf()
self.assertTrue(res.success)
self.assertFalse(getattr(proj, "stateless_prune_needed", False))
def test_sync_network_half_stateless_skips_if_local_commits(self):
"""Test stateless sync skips if there are local-only commits."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir)
proj.bare_git.rev_list.return_value = ["1"]
res = proj.Sync_NetworkHalf()
self.assertTrue(res.success)
self.assertFalse(getattr(proj, "stateless_prune_needed", False))
class SyncOptimizationTests(unittest.TestCase):
"""Tests for sync optimization logic involving shallow clones."""
def _get_project(self, tempdir, depth=None):
manifest = mock.MagicMock()
manifest.manifestProject.depth = depth
manifest.manifestProject.dissociate = False
manifest.manifestProject.clone_filter = None
manifest.is_multimanifest = False
manifest.manifestProject.config.GetBoolean.return_value = False
manifest.IsMirror = False
remote = mock.MagicMock()
remote.name = "origin"
remote.url = "http://"
proj = project.Project(
manifest=manifest,
name="test-project",
remote=remote,
gitdir=os.path.join(tempdir, "gitdir"),
objdir=os.path.join(tempdir, "objdir"),
worktree=tempdir,
relpath="test-project",
revisionExpr="0123456789abcdef0123456789abcdef01234567",
revisionId=None,
)
proj._CheckForImmutableRevision = mock.MagicMock(return_value=True)
proj.DeleteWorktree = mock.MagicMock()
proj._InitGitDir = mock.MagicMock()
proj._InitRemote = mock.MagicMock()
proj._InitMRef = mock.MagicMock()
return proj
def test_sync_network_half_shallow_missing_fetches(self):
"""Test Sync_NetworkHalf fetches if shallow file is missing."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir, depth=1)
# Ensure gitdir does not exist to simulate new project
if os.path.exists(proj.gitdir):
shutil.rmtree(proj.gitdir)
shallow_path = os.path.join(proj.gitdir, "shallow")
if os.path.exists(shallow_path):
os.unlink(shallow_path)
proj._RemoteFetch = mock.MagicMock(return_value=True)
res = proj.Sync_NetworkHalf(optimized_fetch=True)
self.assertTrue(res.success)
proj._RemoteFetch.assert_called_once()
def test_sync_network_half_shallow_exists_skips(self):
"""Test Sync_NetworkHalf skips fetch if shallow file exists."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir, depth=1)
os.makedirs(proj.gitdir, exist_ok=True)
os.makedirs(proj.objdir, exist_ok=True)
with open(os.path.join(proj.gitdir, "shallow"), "w") as f:
f.write("")
proj._RemoteFetch = mock.MagicMock()
res = proj.Sync_NetworkHalf(optimized_fetch=True)
self.assertTrue(res.success)
proj._RemoteFetch.assert_not_called()
def test_remote_fetch_shallow_missing_fetches(self):
"""Test _RemoteFetch fetches if shallow file is missing."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir, depth=1)
shallow_path = os.path.join(proj.gitdir, "shallow")
if os.path.exists(shallow_path):
os.unlink(shallow_path)
with mock.patch("project.GitCommand") as mock_git_cmd:
mock_cmd_instance = mock.MagicMock()
mock_cmd_instance.Wait.return_value = 0
mock_git_cmd.return_value = mock_cmd_instance
res = proj._RemoteFetch(
current_branch_only=True,
depth=1,
use_superproject=False,
)
self.assertTrue(res)
mock_git_cmd.assert_called()
def test_remote_fetch_shallow_exists_skips(self):
"""Test _RemoteFetch skips fetch if shallow file exists."""
with utils_for_test.TempGitTree() as tempdir:
proj = self._get_project(tempdir, depth=1)
os.makedirs(proj.gitdir, exist_ok=True)
os.makedirs(proj.objdir, exist_ok=True)
with open(os.path.join(proj.gitdir, "shallow"), "w") as f:
f.write("")
with mock.patch("project.GitCommand") as mock_git_cmd:
res = proj._RemoteFetch(
current_branch_only=True,
depth=1,
use_superproject=False,
)
self.assertTrue(res)
mock_git_cmd.assert_not_called()
+41 -73
View File
@@ -14,11 +14,9 @@
"""Unittests for the forall subcmd.""" """Unittests for the forall subcmd."""
from io import StringIO import contextlib
import os import io
from shutil import rmtree from pathlib import Path
import tempfile
import unittest
from unittest import mock from unittest import mock
import utils_for_test import utils_for_test
@@ -28,45 +26,30 @@ import project
import subcmds import subcmds
class AllCommands(unittest.TestCase): def _create_manifest_with_8_projects(
"""Check registered all_commands.""" topdir: Path,
) -> manifest_xml.XmlManifest:
"""Create a setup of 8 projects to execute forall."""
repodir = topdir / ".repo"
manifest_dir = repodir / "manifests"
manifest_file = repodir / manifest_xml.MANIFEST_FILE_NAME
def setUp(self): repodir.mkdir()
"""Common setup.""" manifest_dir.mkdir()
self.tempdirobj = tempfile.TemporaryDirectory(prefix="forall_tests")
self.tempdir = self.tempdirobj.name
self.repodir = os.path.join(self.tempdir, ".repo")
self.manifest_dir = os.path.join(self.repodir, "manifests")
self.manifest_file = os.path.join(
self.repodir, manifest_xml.MANIFEST_FILE_NAME
)
self.local_manifest_dir = os.path.join(
self.repodir, manifest_xml.LOCAL_MANIFESTS_DIR_NAME
)
os.mkdir(self.repodir)
os.mkdir(self.manifest_dir)
def tearDown(self): # Set up a manifest git dir for parsing to work.
"""Common teardown.""" gitdir = repodir / "manifests.git"
rmtree(self.tempdir, ignore_errors=True) gitdir.mkdir()
(gitdir / "config").write_text(
def getXmlManifestWith8Projects(self):
"""Create and return a setup of 8 projects with enough dummy
files and setup to execute forall."""
# Set up a manifest git dir for parsing to work
gitdir = os.path.join(self.repodir, "manifests.git")
os.mkdir(gitdir)
with open(os.path.join(gitdir, "config"), "w") as fp:
fp.write(
"""[remote "origin"] """[remote "origin"]
url = https://localhost:0/manifest url = https://localhost:0/manifest
verbose = false verbose = false
""" """
) )
# Add the manifest data # Add the manifest data.
manifest_data = """ manifest_file.write_text(
"""
<manifest> <manifest>
<remote name="origin" fetch="http://localhost" /> <remote name="origin" fetch="http://localhost" />
<default remote="origin" revision="refs/heads/main" /> <default remote="origin" revision="refs/heads/main" />
@@ -79,60 +62,45 @@ class AllCommands(unittest.TestCase):
<project name="project7" path="tests/path7" /> <project name="project7" path="tests/path7" />
<project name="project8" path="tests/path8" /> <project name="project8" path="tests/path8" />
</manifest> </manifest>
""" """,
with open(self.manifest_file, "w", encoding="utf-8") as fp: encoding="utf-8",
fp.write(manifest_data) )
# Set up 8 empty projects to match the manifest # Set up 8 empty projects to match the manifest.
for x in range(1, 9): for x in range(1, 9):
os.makedirs( (repodir / "projects" / "tests" / f"path{x}.git").mkdir(parents=True)
os.path.join( (repodir / "project-objects" / f"project{x}.git").mkdir(parents=True)
self.repodir, "projects/tests/path" + str(x) + ".git" git_path = topdir / "tests" / f"path{x}"
)
)
os.makedirs(
os.path.join(
self.repodir, "project-objects/project" + str(x) + ".git"
)
)
git_path = os.path.join(self.tempdir, "tests/path" + str(x))
utils_for_test.init_git_tree(git_path) utils_for_test.init_git_tree(git_path)
return manifest_xml.XmlManifest(self.repodir, self.manifest_file) return manifest_xml.XmlManifest(str(repodir), str(manifest_file))
# Use mock to capture stdout from the forall run
@unittest.mock.patch("sys.stdout", new_callable=StringIO) def test_forall_all_projects_called_once(tmp_path: Path) -> None:
def test_forall_all_projects_called_once(self, mock_stdout):
"""Test that all projects get a command run once each.""" """Test that all projects get a command run once each."""
manifest = _create_manifest_with_8_projects(tmp_path)
manifest_with_8_projects = self.getXmlManifestWith8Projects()
cmd = subcmds.forall.Forall() cmd = subcmds.forall.Forall()
cmd.manifest = manifest_with_8_projects cmd.manifest = manifest
# Use echo project names as the test of forall # Use echo project names as the test of forall.
opts, args = cmd.OptionParser.parse_args(["-c", "echo $REPO_PROJECT"]) opts, args = cmd.OptionParser.parse_args(["-c", "echo $REPO_PROJECT"])
opts.verbose = False opts.verbose = False
# Mock to not have the Execute fail on remote check with contextlib.redirect_stdout(io.StringIO()) as stdout:
# Mock to not have the Execute fail on remote check.
with mock.patch.object( with mock.patch.object(
project.Project, "GetRevisionId", return_value="refs/heads/main" project.Project, "GetRevisionId", return_value="refs/heads/main"
): ):
# Run the forall command # Run the forall command.
cmd.Execute(opts, args) cmd.Execute(opts, args)
# Verify that we got every project name in the prints output = stdout.getvalue()
# Verify that we got every project name in the output.
for x in range(1, 9): for x in range(1, 9):
self.assertIn("project" + str(x), mock_stdout.getvalue()) assert f"project{x}" in output
# Split the captured output into lines to count them # Split the captured output into lines to count them.
line_count = 0 line_count = sum(1 for x in output.splitlines() if x)
for line in mock_stdout.getvalue().split("\n"): # Verify that we didn't get more lines than expected.
# A commented out print to stderr as a reminder
# that stdout is mocked, include sys and uncomment if needed
# print(line, file=sys.stderr)
if len(line) > 0:
line_count += 1
# Verify that we didn't get more lines than expected
assert line_count == 8 assert line_count == 8
+109
View File
@@ -477,6 +477,115 @@ class GetPreciousObjectsState(unittest.TestCase):
) )
class CheckForBloatedProjects(unittest.TestCase):
"""Tests for Sync._CheckForBloatedProjects."""
def setUp(self):
self.cmd = sync.Sync()
self.opt = mock.Mock()
self.opt.quiet = True
self.opt.jobs = 1
self.project = mock.MagicMock(clone_depth="1")
self.project.name = "project"
self.project.Exists = True
self.project.worktree = "worktree"
self.cmd.git_event_log = mock.MagicMock()
self.cmd._bloated_projects = []
@mock.patch("subcmds.sync.git_require")
def test_git_version_unsupported(self, mock_git_require):
"""Test that it returns early if git version is unsupported."""
mock_git_require.return_value = False
self.cmd._CheckForBloatedProjects([self.project], self.opt)
self.assertFalse(self.cmd.git_event_log.ErrorEvent.called)
@mock.patch("subcmds.sync.git_require")
def test_no_projects(self, mock_git_require):
"""Test that it returns early if no projects have clone_depth."""
mock_git_require.return_value = True
self.project.clone_depth = None
self.cmd._CheckForBloatedProjects([self.project], self.opt)
self.assertFalse(self.cmd.git_event_log.ErrorEvent.called)
@mock.patch("subcmds.sync.git_require")
@mock.patch("subcmds.sync.Progress")
def test_bloated_project_found(self, mock_progress, mock_git_require):
"""Test that it adds project to _bloated_projects."""
mock_git_require.return_value = True
self.cmd.get_parallel_context = mock.Mock(
return_value={"projects": [self.project]}
)
def mock_execute_in_parallel(
jobs, func, work_items, callback, **kwargs
):
callback(None, mock.Mock(), ["project"])
return True
self.cmd.ExecuteInParallel = mock_execute_in_parallel
with mock.patch.object(self.cmd, "ParallelContext"):
self.cmd._CheckForBloatedProjects([self.project], self.opt)
self.assertEqual(self.cmd._bloated_projects, ["project"])
class GCProjectsTest(unittest.TestCase):
"""Tests for Sync._GCProjects."""
def setUp(self):
self.cmd = sync.Sync()
self.opt = mock.Mock()
self.opt.quiet = True
self.opt.auto_gc = True
self.opt.jobs = 1
self.project = mock.MagicMock()
self.project.name = "project"
self.project.objdir = "objdir"
self.project.gitdir = "gitdir"
self.project.bare_git = mock.MagicMock()
self.project.bare_git._project = self.project
self.cmd.git_event_log = mock.MagicMock()
@mock.patch("subcmds.sync.Progress")
def test_GCProjects_skip_gc(self, mock_progress):
"""Test that it skips GC if opt.auto_gc is False."""
self.opt.auto_gc = False
with mock.patch.object(
sync.Sync, "_SetPreciousObjectsState"
) as mock_set_state:
self.cmd._GCProjects([self.project], self.opt, None)
mock_set_state.assert_called_once_with(self.project, self.opt)
self.assertFalse(self.project.bare_git.gc.called)
@mock.patch("subcmds.sync.Progress")
def test_GCProjects_sequential(self, mock_progress):
"""Test sequential GC (jobs < 2)."""
with mock.patch.object(sync.Sync, "_SetPreciousObjectsState"):
self.cmd._GCProjects([self.project], self.opt, None)
self.project.bare_git.gc.assert_called_once_with(
"--auto", config={"gc.autoDetach": "false"}
)
# Verify that gc.autoDetach was not permanently set in config.
for call in self.project.config.SetString.call_args_list:
self.assertNotEqual(call.args[0], "gc.autoDetach")
@mock.patch("subcmds.sync.Progress")
def test_GCProjects_parallel(self, mock_progress):
"""Test parallel GC (jobs >= 2)."""
self.opt.jobs = 2
with mock.patch.object(sync.Sync, "_SetPreciousObjectsState"):
with mock.patch("subcmds.sync._threading.Thread") as mock_thread:
mock_t = mock.MagicMock()
mock_thread.return_value = mock_t
err_event = mock.Mock()
err_event.is_set.return_value = False
self.cmd._GCProjects([self.project], self.opt, err_event)
self.assertTrue(mock_thread.called)
class SyncCommand(unittest.TestCase): class SyncCommand(unittest.TestCase):
"""Tests for cmd.Execute.""" """Tests for cmd.Execute."""
+30 -35
View File
@@ -14,9 +14,10 @@
"""Unittests for the subcmds/upload.py module.""" """Unittests for the subcmds/upload.py module."""
import unittest
from unittest import mock from unittest import mock
import pytest
from error import GitError from error import GitError
from error import UploadError from error import UploadError
from subcmds import upload from subcmds import upload
@@ -26,45 +27,39 @@ class UnexpectedError(Exception):
"""An exception not expected by upload command.""" """An exception not expected by upload command."""
class UploadCommand(unittest.TestCase): # A stub people list (reviewers, cc).
"""Check registered all_commands.""" _STUB_PEOPLE = ([], [])
def setUp(self):
self.cmd = upload.Upload()
self.branch = mock.MagicMock()
self.people = mock.MagicMock()
self.opt, _ = self.cmd.OptionParser.parse_args([])
mock.patch.object(
self.cmd, "_AppendAutoList", return_value=None
).start()
mock.patch.object(self.cmd, "git_event_log").start()
def tearDown(self): @pytest.fixture
mock.patch.stopall() def cmd() -> upload.Upload:
"""Fixture to provide an Upload command instance with mocked methods."""
cmd = upload.Upload()
with mock.patch.object(
cmd, "_AppendAutoList", return_value=None
), mock.patch.object(cmd, "git_event_log"):
yield cmd
def test_UploadAndReport_UploadError(self):
def test_UploadAndReport_UploadError(cmd: upload.Upload) -> None:
"""Check UploadExitError raised when UploadError encountered.""" """Check UploadExitError raised when UploadError encountered."""
side_effect = UploadError("upload error") opt, _ = cmd.OptionParser.parse_args([])
with mock.patch.object( with mock.patch.object(cmd, "_UploadBranch", side_effect=UploadError("")):
self.cmd, "_UploadBranch", side_effect=side_effect with pytest.raises(upload.UploadExitError):
): cmd._UploadAndReport(opt, [mock.MagicMock()], _STUB_PEOPLE)
with self.assertRaises(upload.UploadExitError):
self.cmd._UploadAndReport(self.opt, [self.branch], self.people)
def test_UploadAndReport_GitError(self):
def test_UploadAndReport_GitError(cmd: upload.Upload) -> None:
"""Check UploadExitError raised when GitError encountered.""" """Check UploadExitError raised when GitError encountered."""
side_effect = GitError("some git error") opt, _ = cmd.OptionParser.parse_args([])
with mock.patch.object( with mock.patch.object(cmd, "_UploadBranch", side_effect=GitError("")):
self.cmd, "_UploadBranch", side_effect=side_effect with pytest.raises(upload.UploadExitError):
): cmd._UploadAndReport(opt, [mock.MagicMock()], _STUB_PEOPLE)
with self.assertRaises(upload.UploadExitError):
self.cmd._UploadAndReport(self.opt, [self.branch], self.people)
def test_UploadAndReport_UnhandledError(self):
def test_UploadAndReport_UnhandledError(cmd: upload.Upload) -> None:
"""Check UnexpectedError passed through.""" """Check UnexpectedError passed through."""
side_effect = UnexpectedError("some os error") opt, _ = cmd.OptionParser.parse_args([])
with mock.patch.object( with mock.patch.object(cmd, "_UploadBranch", side_effect=UnexpectedError):
self.cmd, "_UploadBranch", side_effect=side_effect with pytest.raises(UnexpectedError):
): cmd._UploadAndReport(opt, [mock.MagicMock()], _STUB_PEOPLE)
with self.assertRaises(type(side_effect)):
self.cmd._UploadAndReport(self.opt, [self.branch], self.people)
+488 -447
View File
File diff suppressed because it is too large Load Diff
+5
View File
@@ -27,6 +27,11 @@ from typing import Optional, Union
import git_command import git_command
THIS_FILE = Path(__file__).resolve()
THIS_DIR = THIS_FILE.parent
FIXTURES_DIR = THIS_DIR / "fixtures"
def init_git_tree( def init_git_tree(
path: Union[str, Path], path: Union[str, Path],
ref_format: Optional[str] = None, ref_format: Optional[str] = None,