Compare commits

..

19 Commits

Author SHA1 Message Date
Gavin Mak
239fad7146 hooks: verify hooks project has worktree before running
Skip hook if its project is not present on disk.

Bug: 434232630
Change-Id: I09a8b412d078af7a068d533f7be320d5b02327be
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/494441
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2025-07-28 08:37:08 -07:00
Kuang-che Wu
d3eec0acdd sync: fix connection error on macOS for interleaved sync
Bug: 377538810
Test: on macos, repo sync -j64
Change-Id: I6af4d4e6669dc882f165cbb9142ad4db9b346b73
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/494241
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Kuang-che Wu <kcwu@google.com>
Tested-by: Kuang-che Wu <kcwu@google.com>
2025-07-28 02:05:24 -07:00
Gavin Mak
7f7d70efe4 project: Fix GetHead to handle detached HEADs
The switch to git rev-parse caused GetHead() to return the literal
string 'HEAD' when in a detached state. This broke repo prune, which
expects a commit SHA.

Bug: 434077990
Change-Id: I80b7d5965749096b59e854f61e913aa74c857b99
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/494401
Reviewed-by: Scott Lee <ddoman@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2025-07-25 14:30:07 -07:00
Gavin Mak
720bd1e96b sync: Don't checkout if no worktree
Interleaved sync should not try checkout out a project if it's a mirror.

Change-Id: I2549faab197a3202d79a10e44b449b68d53e3fe7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/492942
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2025-07-23 15:57:49 -07:00
Gavin Mak
25858c8b16 sync: Default to interleaved mode
The previous default, "phased" sync (separate network and checkout
phases), can now be selected with `--no-interleaved`.

Bug: 421935613
Bug: 432082000
Change-Id: Ia8624daa609a28ea2f87f8ea4b42138d8b3e9269
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/489681
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2025-07-21 14:51:36 -07:00
Gavin Mak
52bab0ba27 project: Use git rev-parse to read HEAD
Don't directly read `.git/HEAD`, git already has a command for this.

Bug: 432200791
Change-Id: Iba030650224143eb07c44da1fa56341d9deb4288
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/492941
Reviewed-by: Scott Lee <ddoman@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2025-07-21 14:50:46 -07:00
Gavin Mak
2e6d0881d9 sync: Improve UI and error reporting for interleaved mode
This fixes two issues:
1. the progress bar could show a count greater than the total if new projects were discovered mid-sync. Update the progress bar total dynamically
2. Make "Stall detected" error message more actionable

Bug: 432206932
Change-Id: Ie2a4ada5b1770cae0302fb06590641c522cbb7e7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/491941
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2025-07-17 17:30:33 -07:00
Gavin Mak
74edacd8e5 project: Use plumbing commands to manage HEAD
Don't directly manipulate `.git/HEAD` since it bypasses Git's internal
state management.

Bug: 432200791
Change-Id: I1c9264bcf107d34574a82b60a22ea2c83792951b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/491841
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2025-07-17 15:41:59 -07:00
Gavin Mak
5d95ba8d85 progress: Make end() idempotent
This fixes the double "done" text on successful interleaved sync.

Bug: 421935613
Change-Id: I4f01418cb0340129a8f0a2a5835f7e3fa6a6b119
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/487081
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2025-07-02 13:11:23 -07:00
Kenny Cheng
82d500eb7a sync: support post-sync hook in <repo-hooks>
Add support for a new hook type "post-sync" declared in the manifest using
<repo-hooks>. This allows executing a script automatically after a successful
`repo sync`.

This is useful for initializing developer environments, installing project-wide
Git hooks, generating configs, and other post-sync automation tasks.

Example manifest usage:

  <project name="myorg/repo-hooks" path="hooks" revision="main" />
  <repo-hooks in-project="myorg/repo-hooks" enabled-list="post-sync">
    <hook name="post-sync" />
  </repo-hooks>

The hook script must be named `post-sync.py` and located at the root of the
hook project.

The post-sync hook does not block `repo sync`; if the script fails, the sync
still completes successfully with a warning.

Test: Added `post-sync.py` in hook project and verified it runs after `repo sync`

Bug: b/421694721
Change-Id: I69f3158f0fc319d73a85028d6e90fea02c1dc8c8
Signed-off-by: Kenny Cheng <chao.shun.cheng.tw@gmail.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/480581
Reviewed-by: Scott Lee <ddoman@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2025-07-01 16:11:50 -07:00
Matt Moeller
21269c3eed init: Add environment variable for git-lfs
Convenient way to always enable or disable git-lfs without having to
remember to put on the command line.

Useful if you want to ALWAYS have git-lfs enabled on your system when
you 'init' a new project.

Also useful if you are using the Jenkins repo plugin as it doesn't
provide an option for enabling git-lfs in its UI.

Change-Id: Ieb1bbe83de9c21523ab69b30fc5047c257d02731
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/437661
Commit-Queue: Scott Lee <ddoman@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Fatahillah Wk <fatahillahwkwk@gmail.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Matt Moeller <moeller.matt@gmail.com>
Reviewed-by: Yingchun Li <sword.l.dragon@gmail.com>
2025-06-30 15:27:26 -07:00
Gavin Mak
99b5a17f2c sync: Share final error handling logic between sync modes
Dedupe error reporting logic for phased and interleaved sync modes by
extracting it into _ReportErrors.

Error reporting will now distinguish between network and local failures
and lists the specific repos that failed in each phase.

Bug: 421935613
Change-Id: I4604a83943dbbd71d979158d7a1c4b8c243347d2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/484541
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2025-06-23 16:06:34 -07:00
Gavin Mak
df3c4017f9 sync: Share manifest list update logic between sync modes
Extract the manifest update loop from _SyncPhased into a new
_UpdateManifestLists method and use it in both sync types.

Bug: 421935613
Change-Id: If499a3ce4a0bbb3c4641dba52ca5c1c82b11f16f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/484341
Reviewed-by: Scott Lee <ddoman@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2025-06-23 09:17:05 -07:00
Gavin Mak
f7a3f99dc9 sync: Share self-update logic between sync modes
The logic for checking for repo self-updates lives in _FetchMain, which
is part of the "phased" sync path.

Extract this logic into a new _UpdateRepoProject helper method. Call
this common helper from _ExecuteHelper before either sync mode begins,
so the repo self-update check is always performed.

Bug: 421935613
Change-Id: I9a804f43fbf6239c4146be446040be531f12fc8a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/484041
Reviewed-by: Scott Lee <ddoman@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2025-06-23 09:12:41 -07:00
Gavin Mak
6b8e9fc8db sync: clarify job flags when using interleaved
--jobs-network and --jobs-checkout are ignored with --interleaved.

Bug: 421935613
Change-Id: Ib69413993c4f970b385bd09318972716e5ac3324
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/485021
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
2025-06-18 15:23:59 -07:00
Gavin Mak
7b6ffed4ae sync: Implement --interleaved sync worker
For each assigned project, the worker sequentially calls
Sync_NetworkHalf and Sync_LocalHalf, respecting --local-only and
--network-only flags. To prevent scrambled progress bars, all stderr
output from the checkout phase is captured (shown with --verbose).
Result objects now carry status and timing information from the worker
for state updates.

Bug: 421935613
Change-Id: I398602e08a375e974a8914e5fa48ffae673dda9b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/483301
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2025-06-18 10:26:27 -07:00
Gavin Mak
b4b323a8bd sync: Add orchestration logic for --interleaved
Introduce the parallel orchestration framework for `repo sync
--interleaved`.

The new logic respects project dependencies by processing them in
hierarchical levels. Projects sharing a git object directory are grouped
and processed serially. Also reuse the familiar fetch progress bar UX.

Bug: 421935613
Change-Id: Ia388a231fa96b3220e343f952f07021bc9817d19
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/483281
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
2025-06-17 16:13:36 -07:00
Gavin Mak
f91f4462e6 upload: fix FileNotFoundError when no superproject
Upload gets a FileNotFoundError if not using superproject because it
tries to access the superproject's repo_id before checking if
superproject was actually enabled.

Reorder the logic to check use_superproject first.

Change-Id: I65cd2adab481e799dd7bb75e1a83553ad6e34d8d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/484401
Tested-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2025-06-17 13:31:02 -07:00
Gavin Mak
85352825ff sync: Add scaffolding for interleaved sync
Prepare for an interleaved fetch and checkout mode for `repo sync`. The
goal of the new mode is to significantly speed up syncs by running fetch
and checkout operations in parallel for different projects, rather than
waiting for all fetches to complete before starting any checkouts.

Bug: 421935613
Change-Id: I8c66d1e790c7bba6280e409b95238c5e4e61a9c8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/482821
Reviewed-by: Scott Lee <ddoman@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>
2025-06-11 16:31:35 -07:00
10 changed files with 1289 additions and 140 deletions

View File

@@ -133,3 +133,43 @@ def main(project_list, worktree_list=None, **kwargs):
kwargs: Leave this here for forward-compatibility.
"""
```
### post-sync
This hook runs when `repo sync` completes without errors.
Note: This includes cases where no actual checkout may occur. The hook will still run.
For example:
- `repo sync -n` performs network fetches only and skips the checkout phase.
- `repo sync <project>` only updates the specified project(s).
- Partial failures may still result in a successful exit.
This hook is useful for post-processing tasks such as setting up git hooks,
bootstrapping configuration files, or running project initialization logic.
The hook is defined using the existing `<repo-hooks>` manifest block and is
optional. If the hook script fails or is missing, `repo sync` will still
complete successfully, and the error will be printed as a warning.
Example:
```xml
<project name="myorg/dev-tools" path="tools" revision="main" />
<repo-hooks in-project="myorg/dev-tools" enabled-list="post-sync">
<hook name="post-sync" />
</repo-hooks>
```
The `post-sync.py` file should be defined like:
```py
def main(repo_topdir=None, **kwargs):
"""Main function invoked directly by repo.
We must use the name "main" as that is what repo requires.
Args:
repo_topdir: The absolute path to the top-level directory of the repo workspace.
kwargs: Leave this here for forward-compatibility.
"""
```

View File

@@ -25,6 +25,7 @@ from git_refs import HEAD
# The API we've documented to hook authors. Keep in sync with repo-hooks.md.
_API_ARGS = {
"pre-upload": {"project_list", "worktree_list"},
"post-sync": {"repo_topdir"},
}
@@ -100,12 +101,11 @@ class RepoHook:
self._abort_if_user_denies = abort_if_user_denies
# Store the full path to the script for convenience.
if self._hooks_project:
self._script_fullpath = None
if self._hooks_project and self._hooks_project.worktree:
self._script_fullpath = os.path.join(
self._hooks_project.worktree, self._hook_type + ".py"
)
else:
self._script_fullpath = None
def _GetHash(self):
"""Return a hash of the contents of the hooks directory.
@@ -442,6 +442,7 @@ class RepoHook:
if (
self._bypass_hooks
or not self._hooks_project
or not self._script_fullpath
or self._hook_type not in self._hooks_project.enabled_repo_hooks
):
return True

View File

@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "September 2024" "repo smartsync" "Repo Manual"
.TH REPO "1" "June 2025" "repo smartsync" "Repo Manual"
.SH NAME
repo \- repo smartsync - manual page for repo smartsync
.SH SYNOPSIS
@@ -20,11 +20,12 @@ number of CPU cores)
.TP
\fB\-\-jobs\-network\fR=\fI\,JOBS\/\fR
number of network jobs to run in parallel (defaults to
\fB\-\-jobs\fR or 1)
\fB\-\-jobs\fR or 1). Ignored when \fB\-\-interleaved\fR is set
.TP
\fB\-\-jobs\-checkout\fR=\fI\,JOBS\/\fR
number of local checkout jobs to run in parallel
(defaults to \fB\-\-jobs\fR or 8)
(defaults to \fB\-\-jobs\fR or 8). Ignored when \fB\-\-interleaved\fR
is set
.TP
\fB\-f\fR, \fB\-\-force\-broken\fR
obsolete option (to be deleted in the future)
@@ -58,6 +59,9 @@ only update working tree, don't fetch
use the existing manifest checkout as\-is. (do not
update to the latest revision)
.TP
\fB\-\-interleaved\fR
fetch and checkout projects in parallel (experimental)
.TP
\fB\-n\fR, \fB\-\-network\-only\fR
fetch only, don't update working tree
.TP

View File

@@ -1,5 +1,5 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man.
.TH REPO "1" "September 2024" "repo sync" "Repo Manual"
.TH REPO "1" "June 2025" "repo sync" "Repo Manual"
.SH NAME
repo \- repo sync - manual page for repo sync
.SH SYNOPSIS
@@ -20,11 +20,12 @@ number of CPU cores)
.TP
\fB\-\-jobs\-network\fR=\fI\,JOBS\/\fR
number of network jobs to run in parallel (defaults to
\fB\-\-jobs\fR or 1)
\fB\-\-jobs\fR or 1). Ignored when \fB\-\-interleaved\fR is set
.TP
\fB\-\-jobs\-checkout\fR=\fI\,JOBS\/\fR
number of local checkout jobs to run in parallel
(defaults to \fB\-\-jobs\fR or 8)
(defaults to \fB\-\-jobs\fR or 8). Ignored when \fB\-\-interleaved\fR
is set
.TP
\fB\-f\fR, \fB\-\-force\-broken\fR
obsolete option (to be deleted in the future)
@@ -58,6 +59,9 @@ only update working tree, don't fetch
use the existing manifest checkout as\-is. (do not
update to the latest revision)
.TP
\fB\-\-interleaved\fR
fetch and checkout projects in parallel (experimental)
.TP
\fB\-n\fR, \fB\-\-network\-only\fR
fetch only, don't update working tree
.TP

View File

@@ -101,6 +101,7 @@ class Progress:
self._units = units
self._elide = elide and _TTY
self._quiet = quiet
self._ended = False
# Only show the active jobs section if we run more than one in parallel.
self._show_jobs = False
@@ -118,6 +119,11 @@ class Progress:
if not quiet and show_elapsed:
self._update_thread.start()
def update_total(self, new_total):
"""Updates the total if the new total is larger."""
if new_total > self._total:
self._total = new_total
def _update_loop(self):
while True:
self.update(inc=0)
@@ -195,7 +201,26 @@ class Progress:
)
)
def display_message(self, msg):
"""Clears the current progress line and prints a message above it.
The progress bar is then redrawn on the next line.
"""
if not _TTY or IsTraceToStderr() or self._quiet:
return
# Erase the current line, print the message with a newline,
# and then immediately redraw the progress bar on the new line.
sys.stderr.write("\r" + CSI_ERASE_LINE)
sys.stderr.write(msg + "\n")
sys.stderr.flush()
self.update(inc=0)
def end(self):
if self._ended:
return
self._ended = True
self._update_event.set()
if not _TTY or IsTraceToStderr() or self._quiet:
return

View File

@@ -2061,10 +2061,7 @@ class Project:
if head == revid:
# Same revision; just update HEAD to point to the new
# target branch, but otherwise take no other action.
_lwrite(
self.work_git.GetDotgitPath(subpath=HEAD),
f"ref: {R_HEADS}{name}\n",
)
self.work_git.SetHead(R_HEADS + name)
return True
GitCommand(
@@ -2100,9 +2097,7 @@ class Project:
revid = self.GetRevisionId(all_refs)
if head == revid:
_lwrite(
self.work_git.GetDotgitPath(subpath=HEAD), "%s\n" % revid
)
self.work_git.DetachHead(revid)
else:
self._Checkout(revid, quiet=True)
GitCommand(
@@ -3492,9 +3487,7 @@ class Project:
self._createDotGit(dotgit)
if init_dotgit:
_lwrite(
os.path.join(self.gitdir, HEAD), f"{self.GetRevisionId()}\n"
)
self.work_git.UpdateRef(HEAD, self.GetRevisionId(), detach=True)
# Finish checking out the worktree.
cmd = ["read-tree", "--reset", "-u", "-v", HEAD]
@@ -3841,19 +3834,15 @@ class Project:
def GetHead(self):
"""Return the ref that HEAD points to."""
path = self.GetDotgitPath(subpath=HEAD)
try:
with open(path) as fd:
line = fd.readline()
except OSError as e:
symbolic_head = self.rev_parse("--symbolic-full-name", HEAD)
if symbolic_head == HEAD:
# Detached HEAD. Return the commit SHA instead.
return self.rev_parse(HEAD)
return symbolic_head
except GitError as e:
path = self.GetDotgitPath(subpath=HEAD)
raise NoManifestException(path, str(e))
try:
line = line.decode()
except AttributeError:
pass
if line.startswith("ref: "):
return line[5:-1]
return line[:-1]
def SetHead(self, ref, message=None):
cmdv = []

View File

@@ -127,6 +127,7 @@ to update the working directory files.
return {
"REPO_MANIFEST_URL": "manifest_url",
"REPO_MIRROR_LOCATION": "reference",
"REPO_GIT_LFS": "git_lfs",
}
def _SyncManifest(self, opt):

File diff suppressed because it is too large Load Diff

View File

@@ -627,9 +627,12 @@ Gerrit Code Review: https://www.gerritcodereview.com/
# If using superproject, add the root repo as a push option.
manifest = branch.project.manifest
push_options = list(opt.push_options)
sp = manifest.superproject
if sp and sp.repo_id and manifest.manifestProject.use_superproject:
push_options.append(f"custom-keyed-value=rootRepo:{sp.repo_id}")
if manifest.manifestProject.use_superproject:
sp = manifest.superproject
if sp:
r_id = sp.repo_id
if r_id:
push_options.append(f"custom-keyed-value=rootRepo:{r_id}")
branch.UploadForReview(
people,

View File

@@ -305,8 +305,21 @@ class LocalSyncState(unittest.TestCase):
class FakeProject:
def __init__(self, relpath):
def __init__(self, relpath, name=None, objdir=None):
self.relpath = relpath
self.name = name or relpath
self.objdir = objdir or relpath
self.worktree = relpath
self.use_git_worktrees = False
self.UseAlternates = False
self.manifest = mock.MagicMock()
self.manifest.GetProjectsWithName.return_value = [self]
self.config = mock.MagicMock()
self.EnableRepositoryExtension = mock.MagicMock()
def RelPath(self, local=None):
return self.relpath
def __str__(self):
return f"project: {self.relpath}"
@@ -513,3 +526,413 @@ class SyncCommand(unittest.TestCase):
self.cmd.Execute(self.opt, [])
self.assertIn(self.sync_local_half_error, e.aggregate_errors)
self.assertIn(self.sync_network_half_error, e.aggregate_errors)
class SyncUpdateRepoProject(unittest.TestCase):
"""Tests for Sync._UpdateRepoProject."""
def setUp(self):
"""Common setup."""
self.repodir = tempfile.mkdtemp(".repo")
self.manifest = manifest = mock.MagicMock(repodir=self.repodir)
# Create a repoProject with a mock Sync_NetworkHalf.
repoProject = mock.MagicMock(name="repo")
repoProject.Sync_NetworkHalf = mock.Mock(
return_value=SyncNetworkHalfResult(True, None)
)
manifest.repoProject = repoProject
manifest.IsArchive = False
manifest.CloneFilter = None
manifest.PartialCloneExclude = None
manifest.CloneFilterForDepth = None
git_event_log = mock.MagicMock(ErrorEvent=mock.Mock(return_value=None))
self.cmd = sync.Sync(manifest=manifest, git_event_log=git_event_log)
opt, _ = self.cmd.OptionParser.parse_args([])
opt.local_only = False
opt.repo_verify = False
opt.verbose = False
opt.quiet = True
opt.force_sync = False
opt.clone_bundle = False
opt.tags = False
opt.optimized_fetch = False
opt.retry_fetches = 0
opt.prune = False
self.opt = opt
self.errors = []
mock.patch.object(sync.Sync, "_GetCurrentBranchOnly").start()
def tearDown(self):
shutil.rmtree(self.repodir)
mock.patch.stopall()
def test_fetches_when_stale(self):
"""Test it fetches when the repo project is stale."""
self.manifest.repoProject.LastFetch = time.time() - (
sync._ONE_DAY_S + 1
)
with mock.patch.object(sync, "_PostRepoFetch") as mock_post_fetch:
self.cmd._UpdateRepoProject(self.opt, self.manifest, self.errors)
self.manifest.repoProject.Sync_NetworkHalf.assert_called_once()
mock_post_fetch.assert_called_once()
self.assertEqual(self.errors, [])
def test_skips_when_fresh(self):
"""Test it skips fetch when repo project is fresh."""
self.manifest.repoProject.LastFetch = time.time()
with mock.patch.object(sync, "_PostRepoFetch") as mock_post_fetch:
self.cmd._UpdateRepoProject(self.opt, self.manifest, self.errors)
self.manifest.repoProject.Sync_NetworkHalf.assert_not_called()
mock_post_fetch.assert_not_called()
def test_skips_local_only(self):
"""Test it does nothing with --local-only."""
self.opt.local_only = True
self.manifest.repoProject.LastFetch = time.time() - (
sync._ONE_DAY_S + 1
)
with mock.patch.object(sync, "_PostRepoFetch") as mock_post_fetch:
self.cmd._UpdateRepoProject(self.opt, self.manifest, self.errors)
self.manifest.repoProject.Sync_NetworkHalf.assert_not_called()
mock_post_fetch.assert_not_called()
def test_post_repo_fetch_skipped_on_env_var(self):
"""Test _PostRepoFetch is skipped when REPO_SKIP_SELF_UPDATE is set."""
self.manifest.repoProject.LastFetch = time.time()
with mock.patch.dict(os.environ, {"REPO_SKIP_SELF_UPDATE": "1"}):
with mock.patch.object(sync, "_PostRepoFetch") as mock_post_fetch:
self.cmd._UpdateRepoProject(
self.opt, self.manifest, self.errors
)
mock_post_fetch.assert_not_called()
def test_fetch_failure_is_handled(self):
"""Test that a fetch failure is recorded and doesn't crash."""
self.manifest.repoProject.LastFetch = time.time() - (
sync._ONE_DAY_S + 1
)
fetch_error = GitError("Fetch failed")
self.manifest.repoProject.Sync_NetworkHalf.return_value = (
SyncNetworkHalfResult(False, fetch_error)
)
with mock.patch.object(sync, "_PostRepoFetch") as mock_post_fetch:
self.cmd._UpdateRepoProject(self.opt, self.manifest, self.errors)
self.manifest.repoProject.Sync_NetworkHalf.assert_called_once()
mock_post_fetch.assert_not_called()
self.assertEqual(self.errors, [fetch_error])
class InterleavedSyncTest(unittest.TestCase):
"""Tests for interleaved sync."""
def setUp(self):
"""Set up a sync command with mocks."""
self.repodir = tempfile.mkdtemp(".repo")
self.manifest = mock.MagicMock(repodir=self.repodir)
self.manifest.repoProject.LastFetch = time.time()
self.manifest.repoProject.worktree = self.repodir
self.manifest.manifestProject.worktree = self.repodir
self.manifest.IsArchive = False
self.manifest.CloneBundle = False
self.manifest.default.sync_j = 1
self.outer_client = mock.MagicMock()
self.outer_client.manifest.IsArchive = False
self.cmd = sync.Sync(
manifest=self.manifest, outer_client=self.outer_client
)
self.cmd.outer_manifest = self.manifest
# Mock projects.
self.projA = FakeProject("projA", objdir="objA")
self.projB = FakeProject("projB", objdir="objB")
self.projA_sub = FakeProject(
"projA/sub", name="projA_sub", objdir="objA_sub"
)
self.projC = FakeProject("projC", objdir="objC")
# Mock methods that are not part of the core interleaved sync logic.
mock.patch.object(self.cmd, "_UpdateAllManifestProjects").start()
mock.patch.object(self.cmd, "_UpdateProjectsRevisionId").start()
mock.patch.object(self.cmd, "_ValidateOptionsWithManifest").start()
mock.patch.object(sync, "_PostRepoUpgrade").start()
mock.patch.object(sync, "_PostRepoFetch").start()
# Mock parallel context for worker tests.
self.parallel_context_patcher = mock.patch(
"subcmds.sync.Sync.get_parallel_context"
)
self.mock_get_parallel_context = self.parallel_context_patcher.start()
self.sync_dict = {}
self.mock_context = {
"projects": [],
"sync_dict": self.sync_dict,
}
self.mock_get_parallel_context.return_value = self.mock_context
# Mock _GetCurrentBranchOnly for worker tests.
mock.patch.object(sync.Sync, "_GetCurrentBranchOnly").start()
def tearDown(self):
"""Clean up resources."""
shutil.rmtree(self.repodir)
mock.patch.stopall()
def test_interleaved_fail_fast(self):
"""Test that --fail-fast is respected in interleaved mode."""
opt, args = self.cmd.OptionParser.parse_args(
["--interleaved", "--fail-fast", "-j2"]
)
opt.quiet = True
# With projA/sub, _SafeCheckoutOrder creates two batches:
# 1. [projA, projB]
# 2. [projA/sub]
# We want to fail on the first batch and ensure the second isn't run.
all_projects = [self.projA, self.projB, self.projA_sub]
mock.patch.object(
self.cmd, "GetProjects", return_value=all_projects
).start()
# Mock ExecuteInParallel to simulate a failed run on the first batch of
# projects.
execute_mock = mock.patch.object(
self.cmd, "ExecuteInParallel", return_value=False
).start()
with self.assertRaises(sync.SyncFailFastError):
self.cmd._SyncInterleaved(
opt,
args,
[],
self.manifest,
self.manifest.manifestProject,
all_projects,
{},
)
execute_mock.assert_called_once()
def test_interleaved_shared_objdir_serial(self):
"""Test that projects with shared objdir are processed serially."""
opt, args = self.cmd.OptionParser.parse_args(["--interleaved", "-j4"])
opt.quiet = True
# Setup projects with a shared objdir.
self.projA.objdir = "common_objdir"
self.projC.objdir = "common_objdir"
all_projects = [self.projA, self.projB, self.projC]
mock.patch.object(
self.cmd, "GetProjects", return_value=all_projects
).start()
def execute_side_effect(jobs, target, work_items, **kwargs):
# The callback is a partial object. The first arg is the set we
# need to update to avoid the stall detection.
synced_relpaths_set = kwargs["callback"].args[0]
projects_in_pass = self.cmd.get_parallel_context()["projects"]
for item in work_items:
for project_idx in item:
synced_relpaths_set.add(
projects_in_pass[project_idx].relpath
)
return True
execute_mock = mock.patch.object(
self.cmd, "ExecuteInParallel", side_effect=execute_side_effect
).start()
self.cmd._SyncInterleaved(
opt,
args,
[],
self.manifest,
self.manifest.manifestProject,
all_projects,
{},
)
execute_mock.assert_called_once()
jobs_arg, _, work_items = execute_mock.call_args.args
self.assertEqual(jobs_arg, 2)
work_items_sets = {frozenset(item) for item in work_items}
expected_sets = {frozenset([0, 2]), frozenset([1])}
self.assertEqual(work_items_sets, expected_sets)
def _get_opts(self, args=None):
"""Helper to get default options for worker tests."""
if args is None:
args = ["--interleaved"]
opt, _ = self.cmd.OptionParser.parse_args(args)
# Set defaults for options used by the worker.
opt.quiet = True
opt.verbose = False
opt.force_sync = False
opt.clone_bundle = False
opt.tags = False
opt.optimized_fetch = False
opt.retry_fetches = 0
opt.prune = False
opt.detach_head = False
opt.force_checkout = False
opt.rebase = False
return opt
def test_worker_successful_sync(self):
"""Test _SyncProjectList with a successful fetch and checkout."""
opt = self._get_opts()
project = self.projA
project.Sync_NetworkHalf = mock.Mock(
return_value=SyncNetworkHalfResult(error=None, remote_fetched=True)
)
project.Sync_LocalHalf = mock.Mock()
project.manifest.manifestProject.config = mock.MagicMock()
self.mock_context["projects"] = [project]
with mock.patch("subcmds.sync.SyncBuffer") as mock_sync_buffer:
mock_sync_buf_instance = mock.MagicMock()
mock_sync_buf_instance.Finish.return_value = True
mock_sync_buffer.return_value = mock_sync_buf_instance
result_obj = self.cmd._SyncProjectList(opt, [0])
self.assertEqual(len(result_obj.results), 1)
result = result_obj.results[0]
self.assertTrue(result.fetch_success)
self.assertTrue(result.checkout_success)
self.assertIsNone(result.fetch_error)
self.assertIsNone(result.checkout_error)
project.Sync_NetworkHalf.assert_called_once()
project.Sync_LocalHalf.assert_called_once()
def test_worker_fetch_fails(self):
"""Test _SyncProjectList with a failed fetch."""
opt = self._get_opts()
project = self.projA
fetch_error = GitError("Fetch failed")
project.Sync_NetworkHalf = mock.Mock(
return_value=SyncNetworkHalfResult(
error=fetch_error, remote_fetched=False
)
)
project.Sync_LocalHalf = mock.Mock()
self.mock_context["projects"] = [project]
result_obj = self.cmd._SyncProjectList(opt, [0])
result = result_obj.results[0]
self.assertFalse(result.fetch_success)
self.assertFalse(result.checkout_success)
self.assertEqual(result.fetch_error, fetch_error)
self.assertIsNone(result.checkout_error)
project.Sync_NetworkHalf.assert_called_once()
project.Sync_LocalHalf.assert_not_called()
def test_worker_no_worktree(self):
"""Test interleaved sync does not checkout with no worktree."""
opt = self._get_opts()
project = self.projA
project.worktree = None
project.Sync_NetworkHalf = mock.Mock(
return_value=SyncNetworkHalfResult(error=None, remote_fetched=True)
)
project.Sync_LocalHalf = mock.Mock()
self.mock_context["projects"] = [project]
result_obj = self.cmd._SyncProjectList(opt, [0])
result = result_obj.results[0]
self.assertTrue(result.fetch_success)
self.assertTrue(result.checkout_success)
project.Sync_NetworkHalf.assert_called_once()
project.Sync_LocalHalf.assert_not_called()
def test_worker_fetch_fails_exception(self):
"""Test _SyncProjectList with an exception during fetch."""
opt = self._get_opts()
project = self.projA
fetch_error = GitError("Fetch failed")
project.Sync_NetworkHalf = mock.Mock(side_effect=fetch_error)
project.Sync_LocalHalf = mock.Mock()
self.mock_context["projects"] = [project]
result_obj = self.cmd._SyncProjectList(opt, [0])
result = result_obj.results[0]
self.assertFalse(result.fetch_success)
self.assertFalse(result.checkout_success)
self.assertEqual(result.fetch_error, fetch_error)
project.Sync_NetworkHalf.assert_called_once()
project.Sync_LocalHalf.assert_not_called()
def test_worker_checkout_fails(self):
"""Test _SyncProjectList with an exception during checkout."""
opt = self._get_opts()
project = self.projA
project.Sync_NetworkHalf = mock.Mock(
return_value=SyncNetworkHalfResult(error=None, remote_fetched=True)
)
checkout_error = GitError("Checkout failed")
project.Sync_LocalHalf = mock.Mock(side_effect=checkout_error)
project.manifest.manifestProject.config = mock.MagicMock()
self.mock_context["projects"] = [project]
with mock.patch("subcmds.sync.SyncBuffer"):
result_obj = self.cmd._SyncProjectList(opt, [0])
result = result_obj.results[0]
self.assertTrue(result.fetch_success)
self.assertFalse(result.checkout_success)
self.assertIsNone(result.fetch_error)
self.assertEqual(result.checkout_error, checkout_error)
project.Sync_NetworkHalf.assert_called_once()
project.Sync_LocalHalf.assert_called_once()
def test_worker_local_only(self):
"""Test _SyncProjectList with --local-only."""
opt = self._get_opts(["--interleaved", "--local-only"])
project = self.projA
project.Sync_NetworkHalf = mock.Mock()
project.Sync_LocalHalf = mock.Mock()
project.manifest.manifestProject.config = mock.MagicMock()
self.mock_context["projects"] = [project]
with mock.patch("subcmds.sync.SyncBuffer") as mock_sync_buffer:
mock_sync_buf_instance = mock.MagicMock()
mock_sync_buf_instance.Finish.return_value = True
mock_sync_buffer.return_value = mock_sync_buf_instance
result_obj = self.cmd._SyncProjectList(opt, [0])
result = result_obj.results[0]
self.assertTrue(result.fetch_success)
self.assertTrue(result.checkout_success)
project.Sync_NetworkHalf.assert_not_called()
project.Sync_LocalHalf.assert_called_once()
def test_worker_network_only(self):
"""Test _SyncProjectList with --network-only."""
opt = self._get_opts(["--interleaved", "--network-only"])
project = self.projA
project.Sync_NetworkHalf = mock.Mock(
return_value=SyncNetworkHalfResult(error=None, remote_fetched=True)
)
project.Sync_LocalHalf = mock.Mock()
self.mock_context["projects"] = [project]
result_obj = self.cmd._SyncProjectList(opt, [0])
result = result_obj.results[0]
self.assertTrue(result.fetch_success)
self.assertTrue(result.checkout_success)
project.Sync_NetworkHalf.assert_called_once()
project.Sync_LocalHalf.assert_not_called()