sync: Add heuristic warning for bloated shallow repositories

For clone-depth="1" repositories that are dirty or have local commits,
add a check at the end of sync to detect excessive git object
accumulation.

This prevents silent performance degradation and disk exhaustion in
large prebuilts repos where automatic GC is typically disabled from
https://gerrit.googlesource.com/git-repo/+/7f87c54043ce9a35a5bb60a09ee846f9d7070352

Bug: 379111283
Change-Id: I376f38e1555cc6e906d852f6e63dc1c8f6331b4f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/534701
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Gavin Mak <gavinmak@google.com>

.flake8

@@ -11,3 +11,7 @@ extend-ignore =
     E402,
     # E731: do not assign a lambda expression, use a def
     E731,
+exclude =
+    .venv,
+    venv,
+    .tox,

.github/workflows/close-pull-request.yml

@@ -0,0 +1,22 @@
+# GitHub actions workflow.
+# https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions
+
+# https://github.com/superbrothers/close-pull-request
+name: Close Pull Request
+
+on:
+  pull_request_target:
+    types: [opened]
+
+jobs:
+  run:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: superbrothers/close-pull-request@v3
+      with:
+        comment: >
+          Thanks for your contribution!
+          Unfortunately, we don't use GitHub pull requests to manage code
+          contributions to this repository.
+          Instead, please see [README.md](../blob/HEAD/CONTRIBUTING.md)
+          which provides full instructions on how to get involved.

.github/workflows/test-ci.yml

@@ -13,8 +13,9 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-latest, macos-latest, windows-latest]
-        python-version: ['3.6', '3.7', '3.8', '3.9', '3.10']
+        # ubuntu-20.04 is the last version that supports python 3.6
+        os: [ubuntu-20.04, macos-latest, windows-latest]
+        python-version: ['3.6', '3.7', '3.8', '3.9', '3.10', '3.11', '3.12']
     runs-on: ${{ matrix.os }}
 
     steps:

.isort.cfg

@@ -0,0 +1,41 @@
+# Copyright (C) 2023 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Config file for the isort python module.
+# This is used to enforce import sorting standards.
+#
+# https://pycqa.github.io/isort/docs/configuration/options.html
+
+[settings]
+# Be compatible with `black` since it also matches what we want.
+profile = black
+
+line_length = 80
+length_sort = false
+force_single_line = true
+lines_after_imports = 2
+from_first = false
+case_sensitive = false
+force_sort_within_sections = true
+order_by_type = false
+
+# Ignore generated files.
+extend_skip_glob = *_pb2.py
+
+# Allow importing multiple classes on a single line from these modules.
+# https://google.github.io/styleguide/pyguide#s2.2-imports
+single_line_exclusions =
+    abc,
+    collections.abc,
+    typing,

CONTRIBUTING.md

@@ -1,18 +1,19 @@
+# Submitting Changes
+
+Here's a short overview of the process.
+
+*   Make small logical changes.
+*   [Provide a meaningful commit message][commit-message-style].
+*   Make sure all code is under the Apache License, 2.0.
+*   Publish your changes for review.
+    *   `git push origin HEAD:refs/for/main`
+*   Make corrections if requested.
+*   [Verify your changes on Gerrit.](#verify)
+*   [Send to the commit queue for testing & merging.](#cq)
+
 [TOC]
 
-# Short Version
-
- - Make small logical changes.
- - [Provide a meaningful commit message][commit-message-style].
- - Make sure all code is under the Apache License, 2.0.
- - Publish your changes for review.
- - Make corrections if requested.
- - Verify your changes on gerrit so they can be submitted.
-
-   `git push https://gerrit-review.googlesource.com/git-repo HEAD:refs/for/main`
-
-
-# Long Version
+## Long Version
 
 I wanted a file describing how to submit patches for repo,
 so I started with the one found in the core Git distribution
@@ -174,12 +175,16 @@ commit. If you make the requested changes you will need to amend your commit
 and push it to the review server again.
 
 
-## Verify your changes on gerrit
+## Verify your changes on Gerrit {#verify}
 
 After you receive a Code-Review+2 from the maintainer, select the Verified
-button on the gerrit page for the change. This verifies that you have tested
+button on the Gerrit page for the change. This verifies that you have tested
 your changes and notifies the maintainer that they are ready to be submitted.
-The maintainer will then submit your changes to the repository.
 
+## Merge your changes via the commit queue {#cq}
+
+Once a change is ready to be merged, select the Commit-Queue+2 setting on the
+Gerrit page for it. This tells the CI system to test the change, and if it
+passes all the checks, automatically merges it.
 
 [commit-message-style]: https://chris.beams.io/posts/git-commit/

README.md

@@ -8,13 +8,13 @@ that you can put anywhere in your path.
 
 * Homepage: <https://gerrit.googlesource.com/git-repo/>
 * Mailing list: [repo-discuss on Google Groups][repo-discuss]
-* Bug reports: <https://bugs.chromium.org/p/gerrit/issues/list?q=component:Applications%3Erepo>
+* Bug reports: <https://issues.gerritcodereview.com/issues?q=is:open%20componentid:1370071>
 * Source: <https://gerrit.googlesource.com/git-repo/>
 * Overview: <https://source.android.com/source/developing.html>
 * Docs: <https://source.android.com/source/using-repo.html>
 * [repo Manifest Format](./docs/manifest-format.md)
 * [repo Hooks](./docs/repo-hooks.md)
-* [Submitting patches](./SUBMITTING_PATCHES.md)
+* [Contributing](./CONTRIBUTING.md)
 * Running Repo in [Microsoft Windows](./docs/windows.md)
 * GitHub mirror: <https://github.com/GerritCodeReview/git-repo>
 * Postsubmit tests: <https://github.com/GerritCodeReview/git-repo/actions>
@@ -50,6 +50,6 @@ $ chmod a+rx ~/.bin/repo
 ```
 
 
-[new-bug]: https://bugs.chromium.org/p/gerrit/issues/entry?template=Repo+tool+issue
-[issue tracker]: https://bugs.chromium.org/p/gerrit/issues/list?q=component:Applications%3Erepo
+[new-bug]: https://issues.gerritcodereview.com/issues/new?component=1370071
+[issue tracker]: https://issues.gerritcodereview.com/issues?q=is:open%20componentid:1370071
 [repo-discuss]: https://groups.google.com/forum/#!forum/repo-discuss

color.py

@@ -17,6 +17,7 @@ import sys
 
 import pager
 
+
 COLORS = {
     None: -1,
     "normal": -1,
@@ -102,7 +103,7 @@ def SetDefaultColoring(state):
         DEFAULT = "never"
 
 
-class Coloring(object):
+class Coloring:
     def __init__(self, config, section_type):
         self._section = "color.%s" % section_type
         self._config = config
@@ -193,7 +194,7 @@ class Coloring(object):
         if not opt:
             return _Color(fg, bg, attr)
 
-        v = self._config.GetString("%s.%s" % (self._section, opt))
+        v = self._config.GetString(f"{self._section}.{opt}")
         if v is None:
             return _Color(fg, bg, attr)
 
@@ -209,6 +210,7 @@ class Coloring(object):
                 if have_fg:
                     bg = a
                 else:
+                    have_fg = True
                     fg = a
             elif is_attr(a):
                 attr = a

command.py

@@ -12,15 +12,16 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import contextlib
 import multiprocessing
-import os
 import optparse
+import os
 import re
-import sys
 
-from event_log import EventLog
-from error import NoSuchProjectError
 from error import InvalidProjectGroupsError
+from error import NoSuchProjectError
+from error import RepoExitError
+from event_log import EventLog
 import progress
 
 
@@ -42,7 +43,11 @@ WORKER_BATCH_SIZE = 32
 DEFAULT_LOCAL_JOBS = min(os.cpu_count(), 8)
 
 
-class Command(object):
+class UsageError(RepoExitError):
+    """Exception thrown with invalid command usage."""
+
+
+class Command:
     """Base class for any command line action in repo."""
 
     # Singleton for all commands to track overall repo command execution and
@@ -66,12 +71,19 @@ class Command(object):
     # migrated subcommands can set it to False.
     MULTI_MANIFEST_SUPPORT = True
 
+    # Shared data across parallel execution workers.
+    _parallel_context = None
+
+    @classmethod
+    def get_parallel_context(cls):
+        assert cls._parallel_context is not None
+        return cls._parallel_context
+
     def __init__(
         self,
         repodir=None,
         client=None,
         manifest=None,
-        gitc_manifest=None,
         git_event_log=None,
         outer_client=None,
         outer_manifest=None,
@@ -80,7 +92,6 @@ class Command(object):
         self.client = client
         self.outer_client = outer_client or client
         self.manifest = manifest
-        self.gitc_manifest = gitc_manifest
         self.git_event_log = git_event_log
         self.outer_manifest = outer_manifest
 
@@ -215,7 +226,7 @@ class Command(object):
     def Usage(self):
         """Display usage and terminate."""
         self.OptionParser.print_usage()
-        sys.exit(1)
+        raise UsageError()
 
     def CommonValidateOptions(self, opt, args):
         """Validate common options."""
@@ -240,9 +251,39 @@ class Command(object):
         """Perform the action, after option parsing is complete."""
         raise NotImplementedError
 
-    @staticmethod
+    @classmethod
+    @contextlib.contextmanager
+    def ParallelContext(cls):
+        """Obtains the context, which is shared to ExecuteInParallel workers.
+
+        Callers can store data in the context dict before invocation of
+        ExecuteInParallel. The dict will then be shared to child workers of
+        ExecuteInParallel.
+        """
+        assert cls._parallel_context is None
+        cls._parallel_context = {}
+        try:
+            yield
+        finally:
+            cls._parallel_context = None
+
+    @classmethod
+    def _InitParallelWorker(cls, context, initializer):
+        cls._parallel_context = context
+        if initializer:
+            initializer()
+
+    @classmethod
     def ExecuteInParallel(
-        jobs, func, inputs, callback, output=None, ordered=False
+        cls,
+        jobs,
+        func,
+        inputs,
+        callback,
+        output=None,
+        ordered=False,
+        chunksize=WORKER_BATCH_SIZE,
+        initializer=None,
     ):
         """Helper for managing parallel execution boiler plate.
 
@@ -267,6 +308,9 @@ class Command(object):
             output: An output manager. May be progress.Progess or
                 color.Coloring.
             ordered: Whether the jobs should be processed in order.
+            chunksize: The number of jobs processed in batch by parallel
+                workers.
+            initializer: Worker initializer.
 
         Returns:
             The |callback| function's results are returned.
@@ -276,19 +320,23 @@ class Command(object):
             if len(inputs) == 1 or jobs == 1:
                 return callback(None, output, (func(x) for x in inputs))
             else:
-                with multiprocessing.Pool(jobs) as pool:
+                with multiprocessing.Pool(
+                    jobs,
+                    initializer=cls._InitParallelWorker,
+                    initargs=(cls._parallel_context, initializer),
+                ) as pool:
                     submit = pool.imap if ordered else pool.imap_unordered
                     return callback(
                         pool,
                         output,
-                        submit(func, inputs, chunksize=WORKER_BATCH_SIZE),
+                        submit(func, inputs, chunksize=chunksize),
                     )
         finally:
             if isinstance(output, progress.Progress):
                 output.end()
 
     def _ResetPathToProjectMap(self, projects):
-        self._by_path = dict((p.worktree, p) for p in projects)
+        self._by_path = {p.worktree: p for p in projects}
 
     def _UpdatePathToProjectMap(self, project):
         self._by_path[project.worktree] = project
@@ -351,7 +399,7 @@ class Command(object):
         result = []
 
         if not groups:
-            groups = manifest.GetGroupsStr()
+            groups = manifest.GetManifestGroupsStr()
         groups = [x for x in re.split(r"[,\s]+", groups) if x]
 
         if not args:
@@ -474,8 +522,7 @@ class Command(object):
             top = self.manifest
         yield top
         if not opt.this_manifest_only:
-            for child in top.all_children:
-                yield child
+            yield from top.all_children
 
 
 class InteractiveCommand(Command):
@@ -496,17 +543,7 @@ class PagedCommand(Command):
         return True
 
 
-class MirrorSafeCommand(object):
+class MirrorSafeCommand:
     """Command permits itself to run within a mirror, and does not require a
     working directory.
     """
-
-
-class GitcAvailableCommand(object):
-    """Command that requires GITC to be available, but does not require the
-    local client to be a GITC client.
-    """
-
-
-class GitcClientCommand(object):
-    """Command that requires the local client to be a GITC client."""

completion.bash

@@ -1,4 +1,4 @@
-# Copyright 2021 The Android Open Source Project
+# Copyright (C) 2021 The Android Open Source Project
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

constraints.txt

@@ -0,0 +1,2 @@
+# NB: Keep in sync with run_tests.vpython3.
+black<26

docs/internal-fs-layout.md

@@ -42,8 +42,12 @@ For example, if you want to change the manifest branch, you can simply run
     change the git URL/branch that this tracks, re-run `repo init` with the new
     settings.
 
-*   `.repo_fetchtimes.json`: Used by `repo sync` to record stats when syncing
-    the various projects.
+*   `.repo_fetchtimes.json`: Used by `repo sync` to record fetch times when
+    syncing the various projects.
+
+*   `.repo_localsyncstate.json`: Used by `repo sync` to detect and warn on
+    on partial tree syncs.  Partial syncs are allowed by `repo` itself, but are
+    unsupported by many projects where `repo` is used.
 
 ### Manifests
 
@@ -137,7 +141,7 @@ Instead, you should use standard Git workflows like [git worktree] or
     (e.g. a local mirror & a public review server) while avoiding duplicating
     the content.  However, this can run into problems if different remotes use
     the same path on their respective servers.  Best to avoid that.
-*   `subprojects/`: Like `projects/`, but for git submodules.
+*   `modules/`: Like `projects/`, but for git submodules.
 *   `subproject-objects/`: Like `project-objects/`, but for git submodules.
 *   `worktrees/`: Bare checkouts of every project synced by the manifest.  The
     filesystem layout matches the `<project name=...` setting in the manifest

docs/manifest-format.md

@@ -59,7 +59,7 @@ following DTD:
   <!ATTLIST manifest-server url CDATA #REQUIRED>
 
   <!ELEMENT submanifest EMPTY>
-  <!ATTLIST submanifest name           ID #REQUIRED>
+  <!ATTLIST submanifest name           ID    #REQUIRED>
   <!ATTLIST submanifest remote         IDREF #IMPLIED>
   <!ATTLIST submanifest project        CDATA #IMPLIED>
   <!ATTLIST submanifest manifest-name  CDATA #IMPLIED>
@@ -81,9 +81,9 @@ following DTD:
   <!ATTLIST project sync-c      CDATA #IMPLIED>
   <!ATTLIST project sync-s      CDATA #IMPLIED>
   <!ATTLIST project sync-tags   CDATA #IMPLIED>
-  <!ATTLIST project upstream CDATA #IMPLIED>
+  <!ATTLIST project upstream    CDATA #IMPLIED>
   <!ATTLIST project clone-depth CDATA #IMPLIED>
-  <!ATTLIST project force-path CDATA #IMPLIED>
+  <!ATTLIST project force-path  CDATA #IMPLIED>
 
   <!ELEMENT annotation EMPTY>
   <!ATTLIST annotation name  CDATA #REQUIRED>
@@ -95,25 +95,30 @@ following DTD:
   <!ATTLIST copyfile dest CDATA #REQUIRED>
 
   <!ELEMENT linkfile EMPTY>
-  <!ATTLIST linkfile src CDATA #REQUIRED>
+  <!ATTLIST linkfile src  CDATA #REQUIRED>
   <!ATTLIST linkfile dest CDATA #REQUIRED>
 
-  <!ELEMENT extend-project EMPTY>
-  <!ATTLIST extend-project name CDATA #REQUIRED>
-  <!ATTLIST extend-project path CDATA #IMPLIED>
-  <!ATTLIST extend-project dest-path CDATA #IMPLIED>
-  <!ATTLIST extend-project groups CDATA #IMPLIED>
-  <!ATTLIST extend-project revision CDATA #IMPLIED>
-  <!ATTLIST extend-project remote CDATA #IMPLIED>
+  <!ELEMENT extend-project (annotation*,
+                            copyfile*,
+                            linkfile*)>
+  <!ATTLIST extend-project name        CDATA #REQUIRED>
+  <!ATTLIST extend-project path        CDATA #IMPLIED>
+  <!ATTLIST extend-project dest-path   CDATA #IMPLIED>
+  <!ATTLIST extend-project groups      CDATA #IMPLIED>
+  <!ATTLIST extend-project revision    CDATA #IMPLIED>
+  <!ATTLIST extend-project remote      CDATA #IMPLIED>
   <!ATTLIST extend-project dest-branch CDATA #IMPLIED>
-  <!ATTLIST extend-project upstream CDATA #IMPLIED>
+  <!ATTLIST extend-project upstream    CDATA #IMPLIED>
+  <!ATTLIST extend-project base-rev    CDATA #IMPLIED>
 
   <!ELEMENT remove-project EMPTY>
-  <!ATTLIST remove-project name  CDATA #REQUIRED>
-  <!ATTLIST remove-project optional  CDATA #IMPLIED>
+  <!ATTLIST remove-project name     CDATA #IMPLIED>
+  <!ATTLIST remove-project path     CDATA #IMPLIED>
+  <!ATTLIST remove-project optional CDATA #IMPLIED>
+  <!ATTLIST remove-project base-rev CDATA #IMPLIED>
 
   <!ELEMENT repo-hooks EMPTY>
-  <!ATTLIST repo-hooks in-project CDATA #REQUIRED>
+  <!ATTLIST repo-hooks in-project   CDATA #REQUIRED>
   <!ATTLIST repo-hooks enabled-list CDATA #REQUIRED>
 
   <!ELEMENT superproject EMPTY>
@@ -122,7 +127,7 @@ following DTD:
   <!ATTLIST superproject revision CDATA #IMPLIED>
 
   <!ELEMENT contactinfo EMPTY>
-  <!ATTLIST contactinfo bugurl  CDATA #REQUIRED>
+  <!ATTLIST contactinfo bugurl CDATA #REQUIRED>
 
   <!ELEMENT include EMPTY>
   <!ATTLIST include name     CDATA #REQUIRED>
@@ -228,26 +233,7 @@ At most one manifest-server may be specified. The url attribute
 is used to specify the URL of a manifest server, which is an
 XML RPC service.
 
-The manifest server should implement the following RPC methods:
-
-    GetApprovedManifest(branch, target)
-
-Return a manifest in which each project is pegged to a known good revision
-for the current branch and target. This is used by repo sync when the
---smart-sync option is given.
-
-The target to use is defined by environment variables TARGET_PRODUCT
-and TARGET_BUILD_VARIANT. These variables are used to create a string
-of the form $TARGET_PRODUCT-$TARGET_BUILD_VARIANT, e.g. passion-userdebug.
-If one of those variables or both are not present, the program will call
-GetApprovedManifest without the target parameter and the manifest server
-should choose a reasonable default target.
-
-    GetManifest(tag)
-
-Return a manifest in which each project is pegged to the revision at
-the specified tag. This is used by repo sync when the --smart-tag option
-is given.
+See the [smart sync documentation](./smart-sync.md) for more details.
 
 
 ### Element submanifest
@@ -301,7 +287,7 @@ should be placed.  If not supplied, `revision` is used.
 
 `path` may not be an absolute path or use "." or ".." path components.
 
-Attribute `groups`: List of additional groups to which all projects
+Attribute `groups`: Set of additional groups to which all projects
 in the included submanifest belong. This appends and recurses, meaning
 all projects in submanifests carry all parent submanifest groups.
 Same syntax as the corresponding element of `project`.
@@ -369,7 +355,7 @@ When using `repo upload`, changes will be submitted for code
 review on this branch. If unspecified both here and in the
 default element, `revision` is used instead.
 
-Attribute `groups`: List of groups to which this project belongs,
+Attribute `groups`: Set of groups to which this project belongs,
 whitespace or comma separated.  All projects belong to the group
 "all", and each project automatically belongs to a group of
 its name:`name` and path:`path`.  E.g. for
@@ -417,7 +403,7 @@ of the repo client where the Git working directory for this project
 should be placed.  This is used to move a project in the checkout by
 overriding the existing `path` setting.
 
-Attribute `groups`: List of additional groups to which this project
+Attribute `groups`: Set of additional groups to which this project
 belongs.  Same syntax as the corresponding element of `project`.
 
 Attribute `revision`: If specified, overrides the revision of the original
@@ -432,22 +418,31 @@ project.  Same syntax as the corresponding element of `project`.
 Attribute `upstream`: If specified, overrides the upstream of the original
 project.  Same syntax as the corresponding element of `project`.
 
+Attribute `base-rev`: If specified, adds a check against the revision
+to be extended. Manifest parse will fail and give a list of mismatch extends
+if the revisions being extended have changed since base-rev was set.
+Intended for use with layered manifests using hash revisions to prevent
+patch branches hiding newer upstream revisions. Also compares named refs
+like branches or tags but is misleading if branches are used as base-rev.
+Same syntax as the corresponding element of `project`.
+
 ### Element annotation
 
 Zero or more annotation elements may be specified as children of a
-project or remote element. Each element describes a name-value pair.
-For projects, this name-value pair will be exported into each project's
-environment during a 'forall' command, prefixed with `REPO__`.  In addition,
-there is an optional attribute "keep" which accepts the case insensitive values
-"true" (default) or "false".  This attribute determines whether or not the
+project element, an extend-project element, or a remote element. Each
+element describes a name-value pair. For projects, this name-value pair
+will be exported into each project's environment during a 'forall'
+command, prefixed with `REPO__`.  In addition, there is an optional
+attribute "keep" which accepts the case insensitive values "true"
+(default) or "false". This attribute determines whether or not the
 annotation will be kept when exported with the manifest subcommand.
 
 ### Element copyfile
 
 Zero or more copyfile elements may be specified as children of a
-project element. Each element describes a src-dest pair of files;
-the "src" file will be copied to the "dest" place during `repo sync`
-command.
+project element, or an extend-project element. Each element describes a
+src-dest pair of files; the "src" file will be copied to the "dest"
+place during `repo sync` command.
 
 "src" is project relative, "dest" is relative to the top of the tree.
 Copying from paths outside of the project or to paths outside of the repo
@@ -458,10 +453,14 @@ Intermediate paths must not be symlinks either.
 
 Parent directories of "dest" will be automatically created if missing.
 
+The files are copied in the order they are specified in the manifests.
+If multiple elements specify the same source and destination, they will
+only be applied as one, based on the first occurence. Files are copied
+before any links specified via linkfile elements are created.
+
 ### Element linkfile
 
-It's just like copyfile and runs at the same time as copyfile but
-instead of copying it creates a symlink.
+It's just like copyfile, but instead of copying it creates a symlink.
 
 The symlink is created at "dest" (relative to the top of the tree) and
 points to the path specified by "src" which is a path in the project.
@@ -471,9 +470,14 @@ Parent directories of "dest" will be automatically created if missing.
 The symlink target may be a file or directory, but it may not point outside
 of the repo client.
 
+The links are created in the order they are specified in the manifests.
+If multiple elements specify the same source and destination, they will
+only be applied as one, based on the first occurence. Links are created
+after any files specified via copyfile elements are copied.
+
 ### Element remove-project
 
-Deletes the named project from the internal manifest table, possibly
+Deletes a project from the internal manifest table, possibly
 allowing a subsequent project element in the same manifest file to
 replace the project with a different source.
 
@@ -481,9 +485,28 @@ This element is mostly useful in a local manifest file, where
 the user can remove a project, and possibly replace it with their
 own definition.
 
+The project `name` or project `path` can be used to specify the remove target
+meaning one of them is required. If only name is specified, all
+projects with that name are removed.
+
+If both name and path are specified, only projects with the same name and
+path are removed, meaning projects with the same name but in other
+locations are kept.
+
+If only path is specified, a matching project is removed regardless of its
+name. Logic otherwise behaves like both are specified.
+
 Attribute `optional`: Set to true to ignore remove-project elements with no
 matching `project` element.
 
+Attribute `base-rev`: If specified, adds a check against the revision
+to be removed. Manifest parse will fail and give a list of mismatch removes
+if the revisions being removed have changed since base-rev was set.
+Intended for use with layered manifests using hash revisions to prevent
+patch branches hiding newer upstream revisions. Also compares named refs
+like branches or tags but is misleading if branches are used as base-rev.
+Same syntax as the corresponding element of `project`.
+
 ### Element repo-hooks
 
 NB: See the [practical documentation](./repo-hooks.md) for using repo hooks.
@@ -549,13 +572,16 @@ the manifest repository's root.
 "name" may not be an absolute path or use "." or ".." path components.
 These restrictions are not enforced for [Local Manifests].
 
-Attribute `groups`: List of additional groups to which all projects
+Attribute `groups`: Set of additional groups to which all projects
 in the included manifest belong. This appends and recurses, meaning
 all projects in included manifests carry all parent include groups.
+This also applies to all extend-project elements in the included manifests.
 Same syntax as the corresponding element of `project`.
 
 Attribute `revision`: Name of a Git branch (e.g. `main` or `refs/heads/main`)
-default to which all projects in the included manifest belong.
+default to which all projects in the included manifest belong. This recurses,
+meaning it will apply to all projects in all manifests included as a result of
+this element.
 
 ## Local Manifests {#local-manifests}
 

docs/python-support.md

@@ -1,47 +1,92 @@
 # Supported Python Versions
 
-With Python 2.7 officially going EOL on [01 Jan 2020](https://pythonclock.org/),
-we need a support plan for the repo project itself.
-Inevitably, there will be a long tail of users who still want to use Python 2 on
-their old LTS/corp systems and have little power to change the system.
+This documents the current supported Python versions, and tries to provide
+guidance for when we decide to drop support for older versions.
 
 ## Summary
 
-* Python 3.6 (released Dec 2016) is required by default starting with repo-2.x.
-* Older versions of Python (e.g. v2.7) may use the legacy feature-frozen branch
-  based on repo-1.x.
+*   Python 3.6 (released Dec 2016) is required starting with repo-2.0.
+*   Older versions of Python (e.g. v2.7) may use old releases via the repo-1.x
+    branch, but no support is provided.
 
-## Overview
-
-We provide a branch for Python 2 users that is feature-frozen.
-Bugfixes may be added on a best-effort basis or from the community, but largely
-no new features will be added, nor is support guaranteed.
-
-Users can select this during `repo init` time via the [repo launcher].
-Otherwise the default branches (e.g. stable & main) will be used which will
-require Python 3.
-
-This means the [repo launcher] needs to support both Python 2 & Python 3, but
-since it doesn't import any other repo code, this shouldn't be too problematic.
-
-The main branch will require Python 3.6 at a minimum.
-If the system has an older version of Python 3, then users will have to select
-the legacy Python 2 branch instead.
-
-### repo hooks
+## repo hooks
 
 Projects that use [repo hooks] run on independent schedules.
-They might migrate to Python 3 earlier or later than us.
-To support them, we'll probe the shebang of the hook script and if we find an
-interpreter in there that indicates a different version than repo is currently
-running under, we'll attempt to reexec ourselves under that.
+Since it's not possible to detect what version of Python the hooks were written
+or tested against, we always import & exec them with the active Python version.
 
-For example, a hook with a header like `#!/usr/bin/python2` will have repo
-execute `/usr/bin/python2` to execute the hook code specifically if repo is
-currently running Python 3.
+If the user's Python is too new for the [repo hooks], then it is up to the hooks
+maintainer to update.
 
-For more details, consult the [repo hooks] documentation.
+## Repo launcher
+
+The [repo launcher] is an independent script that can support older versions of
+Python without holding back the rest of the codebase.
+If it detects the current version of Python is too old, it will try to reexec
+via a newer version of Python via standard `pythonX.Y` interpreter names.
+
+However, this is provided as a nicety when it is not onerous, and there is no
+official support for older versions of Python than the rest of the codebase.
+
+If your default python interpreters are too old to run the launcher even though
+you have newer versions installed, your choices are:
+
+*   Modify the [repo launcher]'s shebang to suite your environment.
+*   Download an older version of the [repo launcher] and don't upgrade it.
+    Be aware that we do not guarantee old repo launchers will work with current
+    versions of repo.  Bug reports using old launchers will not be accepted.
+
+## When to drop support
+
+So far, Python 3.6 has provided most of the interesting features that we want
+(e.g. typing & f-strings), and there haven't been features in newer versions
+that are critical to us.
+
+That said, let's assume we need functionality that only exists in Python 3.7.
+How do we decide when it's acceptable to drop Python 3.6?
+
+1.  Review the [Project References](./release-process.md#project-references) to
+    see what major distros are using the previous version of Python, and when
+    they go EOL.  Generally we care about Ubuntu LTS & current/previous Debian
+    stable versions.
+    *   If they're all EOL already, then go for it, drop support.
+    *   If they aren't EOL, start a thread on [repo-discuss] to see how the user
+        base feels about the proposal.
+1.  Update the "soft" versions in the codebase.  This will start warning users
+    that the older version is deprecated.
+    *   Update [repo](/repo) if the launcher needs updating.
+        This only helps with people who download newer launchers.
+    *   Update [main.py](/main.py) for the main codebase.
+        This warns for everyone regardless of [repo launcher] version.
+    *   Update [requirements.json](/requirements.json).
+        This allows [repo launcher] to display warnings/errors without having
+        to execute the new codebase.  This helps in case of syntax or module
+        changes where older versions won't even be able to import the new code.
+1.  After some grace period (ideally at least 2 quarters after the first release
+    with the updated soft requirements), update the "hard" versions, and then
+    start using the new functionality.
+
+## Python 2.7 & 3.0-3.5
+
+> **There is no support for these versions.**
+> **Do not file bugs if you are using old Python versions.**
+> **Any such reports will be marked invalid and ignored.**
+> **Upgrade your distro and/or runtime instead.**
+
+Fetch an old version of the [repo launcher]:
+
+```sh
+$ curl https://storage.googleapis.com/git-repo-downloads/repo-2.32 > ~/.bin/repo-2.32
+$ chmod a+rx ~/.bin/repo-2.32
+```
+
+Then initialize an old version of repo:
+
+```sh
+$ repo-2.32 init --repo-rev=repo-1 ...
+```
 
 
+[repo-discuss]: https://groups.google.com/forum/#!forum/repo-discuss
 [repo hooks]: ./repo-hooks.md
 [repo launcher]: ../repo

docs/release-process.md

@@ -96,6 +96,9 @@ If that tag is valid, then repo will warn and use that commit instead.
 
 If that tag cannot be verified, it gives up and forces the user to resolve.
 
+If env variable `REPO_SKIP_SELF_UPDATE` is defined, this will
+bypass the self update algorithm.
+
 ### Force an update
 
 The `repo selfupdate` command can be used to force an immediate update.
@@ -202,7 +205,7 @@ still support them.
 Things in italics are things we used to care about but probably don't anymore.
 
 |   Date   |     EOL      | [Git][rel-g] | [Python][rel-p] | [SSH][rel-o] | [Ubuntu][rel-u] / [Debian][rel-d] | Git | Python | SSH |
-|:--------:|:------------:|:------------:|:---------------:|:------------:|-----------------------------------|-----|--------|-----|
+|:--------:|:------------:|:------------:|:---------------:|:------------:|-----------------------------------|:---:|:------:|:---:|
 | Apr 2008 |              |              |                 | 5.0          |
 | Jun 2008 |              |              |                 | 5.1          |
 | Oct 2008 | *Oct 2013*   |              | 2.6.0           |              | *10.04 Lucid* - 10.10 Maverick / *Squeeze* |
@@ -241,7 +244,7 @@ Things in italics are things we used to care about but probably don't anymore.
 | Feb 2014 | *Dec 2014*   | **1.9.0**    |                 |              | *14.04 Trusty* |
 | Mar 2014 | *Mar 2019*   |              | *3.4.0*         |              | *14.04 Trusty* - 15.10 Wily / *Jessie* |
 | Mar 2014 |              |              |                 | 6.6          | *14.04 Trusty* - 14.10 Utopic |
-| Apr 2014 | *Apr 2022*   |              |                 |              | *14.04 Trusty*       | 1.9.1    | 2.7.5 3.4.0  | 6.6 |
+| Apr 2014 | *Apr 2024*   |              |                 |              | *14.04 Trusty*       | 1.9.1    | 2.7.5 3.4.0  | 6.6 |
 | May 2014 | *Dec 2014*   | 2.0.0        |
 | Aug 2014 | *Dec 2014*   | *2.1.0*      |                 |              | 14.10 Utopic - 15.04 Vivid / *Jessie* |
 | Oct 2014 |              |              |                 | 6.7          | 15.04 Vivid |
@@ -262,7 +265,7 @@ Things in italics are things we used to care about but probably don't anymore.
 | Jan 2016 | *Jul 2017*   | *2.7.0*      |                 |              | *16.04 Xenial* |
 | Feb 2016 |              |              |                 | 7.2          | *16.04 Xenial* |
 | Mar 2016 | *Jul 2017*   | 2.8.0        |
-| Apr 2016 | *Apr 2024*   |              |                 |              | *16.04 Xenial*       | 2.7.4    | 2.7.11 3.5.1 | 7.2 |
+| Apr 2016 | *Apr 2026*   |              |                 |              | *16.04 Xenial*       | 2.7.4    | 2.7.11 3.5.1 | 7.2 |
 | Jun 2016 | *Jul 2017*   | 2.9.0        |                 |              | 16.10 Yakkety |
 | Jul 2016 |              |              |                 | 7.3          | 16.10 Yakkety |
 | Sep 2016 | *Sep 2017*   | 2.10.0       |
@@ -312,14 +315,33 @@ Things in italics are things we used to care about but probably don't anymore.
 | Oct 2020 |              |              |                 |              | 20.10 Groovy         | 2.27.0   | 2.7.18 3.8.6 | 8.3 |
 | Oct 2020 | **Oct 2025** |              | 3.9.0           |              | 21.04 Hirsute / **Bullseye** |
 | Dec 2020 | *Mar 2021*   | 2.30.0       |                 |              | 21.04 Hirsute / **Bullseye** |
-| Mar 2021 |              | 2.31.0       |
-| Mar 2021 |              |              |                 | 8.5          |
+| Mar 2021 |              | 2.31.0       |                 | 8.5          |
 | Apr 2021 |              |              |                 | 8.6          |
 | Apr 2021 | *Jan 2022*   |              |                 |              | 21.04 Hirsute        | 2.30.2   | 2.7.18 3.9.4 | 8.4 |
 | Jun 2021 |              | 2.32.0       |
-| Aug 2021 |              | 2.33.0       |
-| Aug 2021 |              |              |                 | 8.7          |
+| Aug 2021 |              | 2.33.0       |                 | 8.7          |
 | Aug 2021 | **Aug 2026** |              |                 |              | **Debian 11 Bullseye** | 2.30.2 | 2.7.18 3.9.2 | 8.4 |
+| Sep 2021 |              |              |                 | 8.8          |
+| Oct 2021 |              | 2.34.0       | 3.10.0          |              | **22.04 Jammy** |
+| Jan 2022 |              | 2.35.0       |
+| Feb 2022 |              |              |                 | 8.9          | **22.04 Jammy** |
+| Apr 2022 |              | 2.36.0       |                 | 9.0          |
+| Apr 2022 | **Apr 2032** |              |                 |              | **22.04 Jammy**      | 2.34.1 | 2.7.18 3.10.6 | 8.9 |
+| Jun 2022 |              | 2.37.0       |
+| Oct 2022 |              | 2.38.0       |                 | 9.1          |
+| Oct 2022 |              |              | 3.11.0          |              | **Bookworm** |
+| Dec 2022 |              | 2.39.0       |                 |              | **Bookworm** |
+| Feb 2023 |              |              |                 | 9.2          | **Bookworm** |
+| Mar 2023 |              | 2.40.0       |                 | 9.3          |
+| Jun 2023 |              | 2.41.0       |
+| Jun 2023 | **Jun 2028** |              |                 |              | **Debian 12 Bookworm** | 2.39.2 | 3.11.2 | 9.2 |
+| Aug 2023 |              | 2.42.0       |                 | 9.4          |
+| Oct 2023 |              |              | 3.12.0          | 9.5          |
+| Nov 2022 |              | 2.43.0       |
+| Dec 2023 |              |              |                 | 9.6          |
+| Feb 2024 |              | 2.44.0       |
+| Mar 2024 |              |              |                 | 9.7          |
+| Oct 2024 |              |              | 3.13.0          |
 | **Date** |   **EOL**    | **[Git][rel-g]** | **[Python][rel-p]** | **[SSH][rel-o]** | **[Ubuntu][rel-u] / [Debian][rel-d]** | **Git** | **Python** | **SSH** |
 
 
@@ -328,7 +350,7 @@ Things in italics are things we used to care about but probably don't anymore.
 [rel-g]: https://en.wikipedia.org/wiki/Git#Releases
 [rel-o]: https://www.openssh.com/releasenotes.html
 [rel-p]: https://en.wikipedia.org/wiki/History_of_Python#Table_of_versions
-[rel-u]: https://en.wikipedia.org/wiki/Ubuntu_version_history#Table_of_versions
+[rel-u]: https://wiki.ubuntu.com/Releases
 [example announcement]: https://groups.google.com/d/topic/repo-discuss/UGBNismWo1M/discussion
 [repo-discuss@googlegroups.com]: https://groups.google.com/forum/#!forum/repo-discuss
 [go/repo-release]: https://goto.google.com/repo-release

docs/repo-hooks.md

@@ -133,3 +133,43 @@ def main(project_list, worktree_list=None, **kwargs):
       kwargs: Leave this here for forward-compatibility.
     """
 ```
+
+### post-sync
+
+This hook runs when `repo sync` completes without errors.
+
+Note: This includes cases where no actual checkout may occur. The hook will still run.
+For example:
+- `repo sync -n` performs network fetches only and skips the checkout phase.
+- `repo sync <project>` only updates the specified project(s).
+- Partial failures may still result in a successful exit.
+
+This hook is useful for post-processing tasks such as setting up git hooks,
+bootstrapping configuration files, or running project initialization logic.
+
+The hook is defined using the existing `<repo-hooks>` manifest block and is
+optional. If the hook script fails or is missing, `repo sync` will still
+complete successfully, and the error will be printed as a warning.
+
+Example:
+
+```xml
+<project name="myorg/dev-tools" path="tools" revision="main" />
+<repo-hooks in-project="myorg/dev-tools" enabled-list="post-sync">
+  <hook name="post-sync" />
+</repo-hooks>
+```
+
+The `post-sync.py` file should be defined like:
+
+```py
+def main(repo_topdir=None, **kwargs):
+    """Main function invoked directly by repo.
+
+    We must use the name "main" as that is what repo requires.
+
+    Args:
+      repo_topdir: The absolute path to the top-level directory of the repo workspace.
+      kwargs: Leave this here for forward-compatibility.
+    """
+```

docs/smart-sync.md

@@ -0,0 +1,129 @@
+# repo Smart Syncing
+
+Repo normally fetches & syncs manifests from the same URL specified during
+`repo init`, and that often fetches the latest revisions of all projects in
+the manifest.  This flow works well for tracking and developing with the
+latest code, but often it's desirable to sync to other points.  For example,
+to get a local build matching a specific release or build to reproduce bugs
+reported by other people.
+
+Repo's sync subcommand has support for fetching manifests from a server over
+an XML-RPC connection.  The local configuration and network API are defined by
+repo, but individual projects have to host their own server for the client to
+communicate with.
+
+This process is called "smart syncing" -- instead of blindly fetching the latest
+revision of all projects and getting an unknown state to develop against, the
+client passes a request to the server and is given a matching manifest that
+typically specifies specific commits for every project to fetch a known source
+state.
+
+[TOC]
+
+## Manifest Configuration
+
+The manifest specifies the server to communicate with via the
+the [`<manifest-server>` element](manifest-format.md#Element-manifest_server)
+element.  This is how the client knows what service to talk to.
+
+```xml
+  <manifest-server url="https://example.com/your/manifest/server/url" />
+```
+
+If the URL starts with `persistent-`, then the
+[`git-remote-persistent-https` helper](https://github.com/git/git/blob/HEAD/contrib/persistent-https/README)
+is used to communicate with the server.
+
+## Credentials
+
+Credentials may be specified directly in typical `username:password`
+[URI syntax](https://en.wikipedia.org/wiki/URI#Syntax) in the
+`<manifest-server>` element directly in the manifest.
+
+If they are not specified, `repo sync` has `--manifest-server-username=USERNAME`
+and `--manifest-server-password=PASSWORD` options.
+
+If those are not used, then repo will look up the host in your
+[`~/.netrc`](https://docs.python.org/3/library/netrc.html) database.
+
+When making the connection, cookies matching the host are automatically loaded
+from the cookiejar specified in
+[Git's `http.cookiefile` setting](https://git-scm.com/docs/git-config#Documentation/git-config.txt-httpcookieFile).
+
+## Manifest Server
+
+Unfortunately, there are no public reference implementations.  Google has an
+internal one for Android, but it is written using Google's internal systems,
+so wouldn't be that helpful as a reference.
+
+That said, the XML-RPC API is pretty simple, so any standard XML-RPC server
+example would do.  Google's internal server uses Python's
+[xmlrpc.server.SimpleXMLRPCDispatcher](https://docs.python.org/3/library/xmlrpc.server.html).
+
+## Network API
+
+The manifest server should implement the following RPC methods.
+
+### GetApprovedManifest
+
+> `GetApprovedManifest(branch: str, target: Optional[str]) -> str`
+
+The meaning of `branch` and `target` is not strictly defined.  The server may
+interpret them however it wants.  The recommended interpretation is that the
+`branch` matches the manifest branch, and `target` is an identifier for your
+project that matches something users would build.
+
+See the client section below for how repo typically generates these values.
+
+The server will return a manifest or an error.  If it's an error, repo will
+show the output directly to the user to provide a limited feedback channel.
+
+If the user's request is ambiguous and could match multiple manifests, the
+server has to decide whether to pick one automatically (and silently such that
+the user won't know there were multiple matches), or return an error and force
+the user to be more specific.
+
+### GetManifest
+
+> `GetManifest(tag: str) -> str`
+
+The meaning of `tag` is not strictly defined.  Projects are encouraged to use
+a system where the tag matches a unique source state.
+
+See the client section below for how repo typically generates these values.
+
+The server will return a manifest or an error.  If it's an error, repo will
+show the output directly to the user to provide a limited feedback channel.
+
+If the user's request is ambiguous and could match multiple manifests, the
+server has to decide whether to pick one automatically (and silently such that
+the user won't know there were multiple matches), or return an error and force
+the user to be more specific.
+
+## Client Options
+
+Once repo has successfully downloaded the manifest from the server, it saves a
+copy into `.repo/manifests/smart_sync_override.xml` so users can examine it.
+The next time `repo sync` is run, this file is automatically replaced or removed
+based on the current set of options.
+
+### --smart-sync
+
+Repo will call `GetApprovedManifest(branch[, target])`.
+
+The `branch` is determined by the current manifest branch as specified by
+`--manifest-branch=BRANCH` when running `repo init`.
+
+The `target` is defined by environment variables in the order below.  If none
+of them match, then `target` is omitted.  These variables were decided as they
+match the settings Android build environments automatically setup.
+
+1.  `${SYNC_TARGET}`: If defined, the value is used directly.
+2.  `${TARGET_PRODUCT}-${TARGET_RELEASE}-${TARGET_BUILD_VARIANT}`: If these
+    variables are all defined, then they are merged with `-` and used.
+3.  `${TARGET_PRODUCT}-${TARGET_BUILD_VARIANT}`: If these variables are all
+    defined, then they are merged with `-` and used.
+
+### --smart-tag=TAG
+
+Repo will call `GetManifest(TAG)`.

editor.py

@@ -14,15 +14,15 @@
 
 import os
 import re
-import sys
 import subprocess
+import sys
 import tempfile
 
 from error import EditorError
 import platform_utils
 
 
-class Editor(object):
+class Editor:
     """Manages the user's preferred text editor."""
 
     _editor = None
@@ -104,9 +104,7 @@ least one of these before using this command.""",  # noqa: E501
             try:
                 rc = subprocess.Popen(args, shell=shell).wait()
             except OSError as e:
-                raise EditorError(
-                    "editor failed, %s: %s %s" % (str(e), editor, path)
-                )
+                raise EditorError(f"editor failed, {str(e)}: {editor} {path}")
             if rc != 0:
                 raise EditorError(
                     "editor failed with exit status %d: %s %s"

error.py

@@ -12,8 +12,55 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from typing import List
 
-class ManifestParseError(Exception):
+
+class BaseRepoError(Exception):
+    """All repo specific exceptions derive from BaseRepoError."""
+
+
+class RepoError(BaseRepoError):
+    """Exceptions thrown inside repo that can be handled."""
+
+    def __init__(self, *args, project: str = None) -> None:
+        super().__init__(*args)
+        self.project = project
+
+
+class RepoExitError(BaseRepoError):
+    """Exception thrown that result in termination of repo program.
+    - Should only be handled in main.py
+    """
+
+    def __init__(
+        self,
+        *args,
+        exit_code: int = 1,
+        aggregate_errors: List[Exception] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(*args, **kwargs)
+        self.exit_code = exit_code
+        self.aggregate_errors = aggregate_errors
+
+
+class RepoUnhandledExceptionError(RepoExitError):
+    """Exception that maintains error as reason for program exit."""
+
+    def __init__(
+        self,
+        error: BaseException,
+        **kwargs,
+    ) -> None:
+        super().__init__(error, **kwargs)
+        self.error = error
+
+
+class SilentRepoExitError(RepoExitError):
+    """RepoExitError that should no include CLI logging of issue/issues."""
+
+
+class ManifestParseError(RepoExitError):
     """Failed to parse the manifest file."""
 
 
@@ -25,11 +72,11 @@ class ManifestInvalidPathError(ManifestParseError):
     """A path used in <copyfile> or <linkfile> is incorrect."""
 
 
-class NoManifestException(Exception):
+class NoManifestException(RepoExitError):
     """The required manifest does not exist."""
 
-    def __init__(self, path, reason):
-        super().__init__(path, reason)
+    def __init__(self, path, reason, **kwargs):
+        super().__init__(path, reason, **kwargs)
         self.path = path
         self.reason = reason
 
@@ -37,55 +84,72 @@ class NoManifestException(Exception):
         return self.reason
 
 
-class EditorError(Exception):
+class EditorError(RepoError):
     """Unspecified error from the user's text editor."""
 
-    def __init__(self, reason):
-        super().__init__(reason)
+    def __init__(self, reason, **kwargs):
+        super().__init__(reason, **kwargs)
         self.reason = reason
 
     def __str__(self):
         return self.reason
 
 
-class GitError(Exception):
-    """Unspecified internal error from git."""
+class GitError(RepoError):
+    """Unspecified git related error."""
 
-    def __init__(self, command):
-        super().__init__(command)
-        self.command = command
+    def __init__(self, message, command_args=None, **kwargs):
+        super().__init__(message, **kwargs)
+        self.message = message
+        self.command_args = command_args
 
     def __str__(self):
-        return self.command
+        return self.message
 
 
-class UploadError(Exception):
+class GitAuthError(RepoExitError):
+    """Cannot talk to remote due to auth issue."""
+
+
+class UploadError(RepoError):
     """A bundle upload to Gerrit did not succeed."""
 
-    def __init__(self, reason):
-        super().__init__(reason)
+    def __init__(self, reason, **kwargs):
+        super().__init__(reason, **kwargs)
         self.reason = reason
 
     def __str__(self):
         return self.reason
 
 
-class DownloadError(Exception):
+class DownloadError(RepoExitError):
     """Cannot download a repository."""
 
-    def __init__(self, reason):
-        super().__init__(reason)
+    def __init__(self, reason, **kwargs):
+        super().__init__(reason, **kwargs)
         self.reason = reason
 
     def __str__(self):
         return self.reason
 
 
-class NoSuchProjectError(Exception):
+class InvalidArgumentsError(RepoExitError):
+    """Invalid command Arguments."""
+
+
+class SyncError(RepoExitError):
+    """Cannot sync repo."""
+
+
+class UpdateManifestError(RepoExitError):
+    """Cannot update manifest."""
+
+
+class NoSuchProjectError(RepoExitError):
     """A specified project does not exist in the work tree."""
 
-    def __init__(self, name=None):
-        super().__init__(name)
+    def __init__(self, name=None, **kwargs):
+        super().__init__(**kwargs)
         self.name = name
 
     def __str__(self):
@@ -94,11 +158,11 @@ class NoSuchProjectError(Exception):
         return self.name
 
 
-class InvalidProjectGroupsError(Exception):
+class InvalidProjectGroupsError(RepoExitError):
     """A specified project is not suitable for the specified groups"""
 
-    def __init__(self, name=None):
-        super().__init__(name)
+    def __init__(self, name=None, **kwargs):
+        super().__init__(**kwargs)
         self.name = name
 
     def __str__(self):
@@ -107,7 +171,7 @@ class InvalidProjectGroupsError(Exception):
         return self.name
 
 
-class RepoChangedException(Exception):
+class RepoChangedException(BaseRepoError):
     """Thrown if 'repo sync' results in repo updating its internal
     repo or manifest repositories.  In this special case we must
     use exec to re-execute repo with the new code and manifest.
@@ -118,7 +182,7 @@ class RepoChangedException(Exception):
         self.extra_args = extra_args or []
 
 
-class HookError(Exception):
+class HookError(RepoError):
     """Thrown if a 'repo-hook' could not be run.
 
     The common case is that the file wasn't present when we tried to run it.

event_log.py

@@ -15,12 +15,13 @@
 import json
 import multiprocessing
 
+
 TASK_COMMAND = "command"
 TASK_SYNC_NETWORK = "sync-network"
 TASK_SYNC_LOCAL = "sync-local"
 
 
-class EventLog(object):
+class EventLog:
     """Event log that records events that occurred during a repo invocation.
 
     Events are written to the log as a consecutive JSON entries, one per line.
@@ -167,8 +168,10 @@ class EventLog(object):
                 f.write("\n")
 
 
-# An integer id that is unique across this invocation of the program.
-_EVENT_ID = multiprocessing.Value("i", 1)
+# An integer id that is unique across this invocation of the program, to be set
+# by the first Add event. We can't set it here since it results in leaked
+# resources (see: https://issues.gerritcodereview.com/353656374).
+_EVENT_ID = None
 
 
 def _NextEventId():
@@ -177,6 +180,12 @@ def _NextEventId():
     Returns:
         A unique, to this invocation of the program, integer id.
     """
+    global _EVENT_ID
+    if _EVENT_ID is None:
+        # There is a small chance of race condition - two parallel processes
+        # setting up _EVENT_ID. However, we expect TASK_COMMAND to happen before
+        # mp kicks in.
+        _EVENT_ID = multiprocessing.Value("i", 1)
     with _EVENT_ID.get_lock():
         val = _EVENT_ID.value
         _EVENT_ID.value += 1

fetch.py

@@ -19,6 +19,12 @@ import sys
 from urllib.parse import urlparse
 from urllib.request import urlopen
 
+from error import RepoExitError
+
+
+class FetchFileError(RepoExitError):
+    """Exit error when fetch_file fails."""
+
 
 def fetch_file(url, verbose=False):
     """Fetch a file from the specified source using the appropriate protocol.
@@ -29,6 +35,7 @@ def fetch_file(url, verbose=False):
     scheme = urlparse(url).scheme
     if scheme == "gs":
         cmd = ["gsutil", "cat", url]
+        errors = []
         try:
             result = subprocess.run(
                 cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True
@@ -41,9 +48,10 @@ def fetch_file(url, verbose=False):
                 )
             return result.stdout
         except subprocess.CalledProcessError as e:
+            errors.append(e)
             print(
                 'fatal: error running "gsutil": %s' % e.stderr, file=sys.stderr
             )
-        sys.exit(1)
+        raise FetchFileError(aggregate_errors=errors)
     with urlopen(url) as f:
         return f.read()

git_command.py

@@ -13,42 +13,48 @@
 # limitations under the License.
 
 import functools
+import json
 import os
-import sys
+import re
 import subprocess
+import sys
 from typing import Any, Optional
 
 from error import GitError
+from error import RepoExitError
 from git_refs import HEAD
+from git_trace2_event_log_base import BaseEventLog
 import platform_utils
-from repo_trace import REPO_TRACE, IsTrace, Trace
+from repo_logging import RepoLogger
+from repo_trace import IsTrace
+from repo_trace import REPO_TRACE
+from repo_trace import Trace
 from wrapper import Wrapper
 
+
 GIT = "git"
-# NB: These do not need to be kept in sync with the repo launcher script.
-# These may be much newer as it allows the repo launcher to roll between
-# different repo releases while source versions might require a newer git.
-#
-# The soft version is when we start warning users that the version is old and
-# we'll be dropping support for it.  We'll refuse to work with versions older
-# than the hard version.
-#
-# git-1.7 is in (EOL) Ubuntu Precise.  git-1.9 is in Ubuntu Trusty.
-MIN_GIT_VERSION_SOFT = (1, 9, 1)
-MIN_GIT_VERSION_HARD = (1, 7, 2)
 GIT_DIR = "GIT_DIR"
 
 LAST_GITDIR = None
 LAST_CWD = None
+DEFAULT_GIT_FAIL_MESSAGE = "git command failure"
+ERROR_EVENT_LOGGING_PREFIX = "RepoGitCommandError"
+# Common line length limit
+GIT_ERROR_STDOUT_LINES = 1
+GIT_ERROR_STDERR_LINES = 10
+INVALID_GIT_EXIT_CODE = 126
+
+logger = RepoLogger(__file__)
 
 
-class _GitCall(object):
+class _GitCall:
     @functools.lru_cache(maxsize=None)
     def version_tuple(self):
         ret = Wrapper().ParseGitVersion()
         if ret is None:
-            print("fatal: unable to detect git version", file=sys.stderr)
-            sys.exit(1)
+            msg = "fatal: unable to detect git version"
+            logger.error(msg)
+            raise GitRequireError(msg)
         return ret
 
     def __getattr__(self, name):
@@ -57,7 +63,7 @@ class _GitCall(object):
         def fun(*cmdv):
             command = [name]
             command.extend(cmdv)
-            return GitCommand(None, command).Wait() == 0
+            return GitCommand(None, command, add_event_log=False).Wait() == 0
 
         return fun
 
@@ -95,7 +101,45 @@ def RepoSourceVersion():
     return ver
 
 
-class UserAgent(object):
+@functools.lru_cache(maxsize=None)
+def GetEventTargetPath():
+    """Get the 'trace2.eventtarget' path from git configuration.
+
+    Returns:
+        path: git config's 'trace2.eventtarget' path if it exists, or None
+    """
+    path = None
+    cmd = ["config", "--get", "trace2.eventtarget"]
+    # TODO(https://crbug.com/gerrit/13706): Use GitConfig when it supports
+    # system git config variables.
+    p = GitCommand(
+        None,
+        cmd,
+        capture_stdout=True,
+        capture_stderr=True,
+        bare=True,
+        add_event_log=False,
+    )
+    retval = p.Wait()
+    if retval == 0:
+        # Strip trailing carriage-return in path.
+        path = p.stdout.rstrip("\n")
+        if path == "":
+            return None
+    elif retval != 1:
+        # `git config --get` is documented to produce an exit status of `1`
+        # if the requested variable is not present in the configuration.
+        # Report any other return value as an error.
+        logger.error(
+            "repo: error: 'git config --get' call failed with return code: "
+            "%r, stderr: %r",
+            retval,
+            p.stderr,
+        )
+    return path
+
+
+class UserAgent:
     """Mange User-Agent settings when talking to external services
 
     We follow the style as documented here:
@@ -143,12 +187,10 @@ class UserAgent(object):
     def git(self):
         """The UA when running git."""
         if self._git_ua is None:
-            self._git_ua = "git/%s (%s) git-repo/%s" % (
-                git.version_tuple().full,
-                self.os,
-                RepoSourceVersion(),
+            self._git_ua = (
+                f"git/{git.version_tuple().full} ({self.os}) "
+                f"git-repo/{RepoSourceVersion()}"
             )
-
         return self._git_ua
 
 
@@ -163,10 +205,9 @@ def git_require(min_version, fail=False, msg=""):
         need = ".".join(map(str, min_version))
         if msg:
             msg = " for " + msg
-        print(
-            "fatal: git %s or later required%s" % (need, msg), file=sys.stderr
-        )
-        sys.exit(1)
+        error_msg = f"fatal: git {need} or later required{msg}"
+        logger.error(error_msg)
+        raise GitRequireError(error_msg)
     return False
 
 
@@ -191,15 +232,15 @@ def _build_env(
         env["GIT_SSH"] = ssh_proxy.proxy
         env["GIT_SSH_VARIANT"] = "ssh"
     if "http_proxy" in env and "darwin" == sys.platform:
-        s = "'http.proxy=%s'" % (env["http_proxy"],)
+        s = f"'http.proxy={env['http_proxy']}'"
         p = env.get("GIT_CONFIG_PARAMETERS")
         if p is not None:
             s = p + " " + s
         env["GIT_CONFIG_PARAMETERS"] = s
     if "GIT_ALLOW_PROTOCOL" not in env:
-        env[
-            "GIT_ALLOW_PROTOCOL"
-        ] = "file:git:http:https:ssh:persistent-http:persistent-https:sso:rpc"
+        env["GIT_ALLOW_PROTOCOL"] = (
+            "file:git:http:https:ssh:persistent-http:persistent-https:sso:rpc"
+        )
     env["GIT_HTTP_USER_AGENT"] = user_agent.git
 
     if objdir:
@@ -220,7 +261,7 @@ def _build_env(
     return env
 
 
-class GitCommand(object):
+class GitCommand:
     """Wrapper around a single git invocation."""
 
     def __init__(
@@ -237,6 +278,9 @@ class GitCommand(object):
         cwd=None,
         gitdir=None,
         objdir=None,
+        verify_command=False,
+        add_event_log=True,
+        log_as_error=True,
     ):
         if project:
             if not cwd:
@@ -244,6 +288,11 @@ class GitCommand(object):
             if not gitdir:
                 gitdir = project.gitdir
 
+        self.project = project
+        self.cmdv = cmdv
+        self.verify_command = verify_command
+        self.stdout, self.stderr = None, None
+
         # Git on Windows wants its paths only using / for reliability.
         if platform_utils.isWindows():
             if objdir:
@@ -262,15 +311,70 @@ class GitCommand(object):
         command = [GIT]
         if bare:
             cwd = None
-        command.append(cmdv[0])
-        # Need to use the --progress flag for fetch/clone so output will be
-        # displayed as by default git only does progress output if stderr is a
-        # TTY.
-        if sys.stderr.isatty() and cmdv[0] in ("fetch", "clone"):
-            if "--progress" not in cmdv and "--quiet" not in cmdv:
-                command.append("--progress")
+        command_name = cmdv[0]
+        command.append(command_name)
+
+        if command_name in ("fetch", "clone"):
+            env["GIT_TERMINAL_PROMPT"] = "0"
+            # Need to use the --progress flag for fetch/clone so output will be
+            # displayed as by default git only does progress output if stderr is
+            # a TTY.
+            if sys.stderr.isatty():
+                if "--progress" not in cmdv and "--quiet" not in cmdv:
+                    command.append("--progress")
         command.extend(cmdv[1:])
 
+        event_log = (
+            BaseEventLog(env=env, add_init_count=True)
+            if add_event_log
+            else None
+        )
+
+        try:
+            self._RunCommand(
+                command,
+                env,
+                capture_stdout=capture_stdout,
+                capture_stderr=capture_stderr,
+                merge_output=merge_output,
+                ssh_proxy=ssh_proxy,
+                cwd=cwd,
+                input=input,
+            )
+            self.VerifyCommand()
+        except GitCommandError as e:
+            if event_log is not None:
+                error_info = json.dumps(
+                    {
+                        "ErrorType": type(e).__name__,
+                        "Project": e.project,
+                        "CommandName": command_name,
+                        "Message": str(e),
+                        "ReturnCode": (
+                            str(e.git_rc) if e.git_rc is not None else None
+                        ),
+                        "IsError": log_as_error,
+                    }
+                )
+                event_log.ErrorEvent(
+                    f"{ERROR_EVENT_LOGGING_PREFIX}:{error_info}"
+                )
+                event_log.Write(GetEventTargetPath())
+            if isinstance(e, GitPopenCommandError):
+                raise
+
+    def _RunCommand(
+        self,
+        command,
+        env,
+        capture_stdout=False,
+        capture_stderr=False,
+        merge_output=False,
+        ssh_proxy=None,
+        cwd=None,
+        input=None,
+    ):
+        # Set subprocess.PIPE for streams that need to be captured.
         stdin = subprocess.PIPE if input else None
         stdout = subprocess.PIPE if capture_stdout else None
         stderr = (
@@ -279,6 +383,30 @@ class GitCommand(object):
             else (subprocess.PIPE if capture_stderr else None)
         )
 
+        # tee_stderr acts like a tee command for stderr, in that, it captures
+        # stderr from the subprocess and streams it back to sys.stderr, while
+        # keeping a copy in-memory.
+        # This allows us to store stderr logs from the subprocess into
+        # GitCommandError.
+        # Certain git operations, such as `git push`, writes diagnostic logs,
+        # such as, progress bar for pushing, into stderr. To ensure we don't
+        # break git's UX, we need to write to sys.stderr as we read from the
+        # subprocess. Setting encoding or errors makes subprocess return
+        # io.TextIOWrapper, which is line buffered. To avoid line-buffering
+        # while tee-ing stderr, we unset these kwargs. See GitCommand._Tee
+        # for tee-ing between the streams.
+        # We tee stderr iff the caller doesn't want to capture any stream to
+        # not disrupt the existing flow.
+        # See go/tee-repo-stderr for more context.
+        tee_stderr = False
+        kwargs = {"encoding": "utf-8", "errors": "backslashreplace"}
+        if not (stdin or stdout or stderr):
+            tee_stderr = True
+            # stderr will be written back to sys.stderr even though it is
+            # piped here.
+            stderr = subprocess.PIPE
+            kwargs = {}
+
         dbg = ""
         if IsTrace():
             global LAST_CWD
@@ -325,14 +453,17 @@ class GitCommand(object):
                     command,
                     cwd=cwd,
                     env=env,
-                    encoding="utf-8",
-                    errors="backslashreplace",
                     stdin=stdin,
                     stdout=stdout,
                     stderr=stderr,
+                    **kwargs,
                 )
             except Exception as e:
-                raise GitError("%s: %s" % (command[1], e))
+                raise GitPopenCommandError(
+                    message=f"{command[1]}: {e}",
+                    project=self.project.name if self.project else None,
+                    command_args=self.cmdv,
+                )
 
             if ssh_proxy:
                 ssh_proxy.add_client(p)
@@ -340,12 +471,45 @@ class GitCommand(object):
             self.process = p
 
             try:
-                self.stdout, self.stderr = p.communicate(input=input)
+                if tee_stderr:
+                    # tee_stderr streams stderr to sys.stderr while capturing
+                    # a copy within self.stderr. tee_stderr is only enabled
+                    # when the caller wants to pipe no stream.
+                    self.stderr = self._Tee(p.stderr, sys.stderr)
+                else:
+                    self.stdout, self.stderr = p.communicate(input=input)
             finally:
                 if ssh_proxy:
                     ssh_proxy.remove_client(p)
             self.rc = p.wait()
 
+    @staticmethod
+    def _Tee(in_stream, out_stream):
+        """Writes text from in_stream to out_stream while recording in buffer.
+
+        Args:
+            in_stream: I/O stream to be read from.
+            out_stream: I/O stream to write to.
+
+        Returns:
+            A str containing everything read from the in_stream.
+        """
+        buffer = ""
+        read_size = 1024 if sys.version_info < (3, 7) else -1
+        chunk = in_stream.read1(read_size)
+        while chunk:
+            # Convert to str.
+            if not hasattr(chunk, "encode"):
+                chunk = chunk.decode("utf-8", "backslashreplace")
+
+            buffer += chunk
+            out_stream.write(chunk)
+            out_stream.flush()
+
+            chunk = in_stream.read1(read_size)
+
+        return buffer
+
     @staticmethod
     def _GetBasicEnv():
         """Return a basic env for running git under.
@@ -365,5 +529,122 @@ class GitCommand(object):
             env.pop(key, None)
         return env
 
+    def VerifyCommand(self):
+        if self.rc == 0:
+            return None
+        stdout = (
+            "\n".join(self.stdout.split("\n")[:GIT_ERROR_STDOUT_LINES])
+            if self.stdout
+            else None
+        )
+        stderr = (
+            "\n".join(self.stderr.split("\n")[:GIT_ERROR_STDERR_LINES])
+            if self.stderr
+            else None
+        )
+        project = self.project.name if self.project else None
+        raise GitCommandError(
+            project=project,
+            command_args=self.cmdv,
+            git_rc=self.rc,
+            git_stdout=stdout,
+            git_stderr=stderr,
+        )
+
     def Wait(self):
+        if self.verify_command:
+            self.VerifyCommand()
         return self.rc
+
+
+class GitRequireError(RepoExitError):
+    """Error raised when git version is unavailable or invalid."""
+
+    def __init__(self, message, exit_code: int = INVALID_GIT_EXIT_CODE):
+        super().__init__(message, exit_code=exit_code)
+
+
+class GitCommandError(GitError):
+    """
+    Error raised from a failed git command.
+    Note that GitError can refer to any Git related error (e.g. branch not
+    specified for project.py 'UploadForReview'), while GitCommandError is
+    raised exclusively from non-zero exit codes returned from git commands.
+    """
+
+    # Tuples with error formats and suggestions for those errors.
+    _ERROR_TO_SUGGESTION = [
+        (
+            re.compile("couldn't find remote ref .*"),
+            "Check if the provided ref exists in the remote.",
+        ),
+        (
+            re.compile("unable to access '.*': .*"),
+            (
+                "Please make sure you have the correct access rights and the "
+                "repository exists."
+            ),
+        ),
+        (
+            re.compile("'.*' does not appear to be a git repository"),
+            "Are you running this repo command outside of a repo workspace?",
+        ),
+        (
+            re.compile("not a git repository"),
+            "Are you running this repo command outside of a repo workspace?",
+        ),
+    ]
+
+    def __init__(
+        self,
+        message: str = DEFAULT_GIT_FAIL_MESSAGE,
+        git_rc: int = None,
+        git_stdout: str = None,
+        git_stderr: str = None,
+        **kwargs,
+    ):
+        super().__init__(
+            message,
+            **kwargs,
+        )
+        self.git_rc = git_rc
+        self.git_stdout = git_stdout
+        self.git_stderr = git_stderr
+
+    @property
+    @functools.lru_cache(maxsize=None)
+    def suggestion(self):
+        """Returns helpful next steps for the given stderr."""
+        if not self.git_stderr:
+            return self.git_stderr
+
+        for err, suggestion in self._ERROR_TO_SUGGESTION:
+            if err.search(self.git_stderr):
+                return suggestion
+
+        return None
+
+    def __str__(self):
+        args = "[]" if not self.command_args else " ".join(self.command_args)
+        error_type = type(self).__name__
+        string = f"{error_type}: '{args}' on {self.project} failed"
+
+        if self.message != DEFAULT_GIT_FAIL_MESSAGE:
+            string += f": {self.message}"
+
+        if self.git_stdout:
+            string += f"\nstdout: {self.git_stdout}"
+
+        if self.git_stderr:
+            string += f"\nstderr: {self.git_stderr}"
+
+        if self.suggestion:
+            string += f"\nsuggestion: {self.suggestion}"
+
+        return string
+
+
+class GitPopenCommandError(GitError):
+    """
+    Error raised when subprocess.Popen fails for a GitCommand
+    """

git_config.py

@@ -15,7 +15,7 @@
 import contextlib
 import datetime
 import errno
-from http.client import HTTPException
+import http.client
 import json
 import os
 import re
@@ -26,11 +26,15 @@ from typing import Union
 import urllib.error
 import urllib.request
 
-from error import GitError, UploadError
+from error import GitError
+from error import UploadError
+from git_command import GitCommand
+from git_refs import R_CHANGES
+from git_refs import R_HEADS
+from git_refs import R_TAGS
 import platform_utils
 from repo_trace import Trace
-from git_command import GitCommand
-from git_refs import R_CHANGES, R_HEADS, R_TAGS
+
 
 # Prefix that is prepended to all the keys of SyncAnalysisState's data
 # that is saved in the config.
@@ -66,7 +70,7 @@ def _key(name):
     return ".".join(parts)
 
 
-class GitConfig(object):
+class GitConfig:
     _ForUser = None
 
     _ForSystem = None
@@ -86,6 +90,20 @@ class GitConfig(object):
 
     @staticmethod
     def _getUserConfig():
+        """Get the user-specific config file.
+
+        Prefers the XDG config location if available, with fallback to
+        ~/.gitconfig
+
+        This matches git behavior:
+        https://git-scm.com/docs/git-config#FILES
+        """
+        xdg_config_home = os.getenv(
+            "XDG_CONFIG_HOME", os.path.expanduser("~/.config")
+        )
+        xdg_config_file = os.path.join(xdg_config_home, "git", "config")
+        if os.path.exists(xdg_config_file):
+            return xdg_config_file
         return os.path.expanduser("~/.gitconfig")
 
     @classmethod
@@ -176,7 +194,7 @@ class GitConfig(object):
             config_dict[key] = self.GetString(key)
         return config_dict
 
-    def GetBoolean(self, name: str) -> Union[str, None]:
+    def GetBoolean(self, name: str) -> Union[bool, None]:
         """Returns a boolean from the configuration file.
 
         Returns:
@@ -366,7 +384,7 @@ class GitConfig(object):
             with Trace(": parsing %s", self.file):
                 with open(self._json) as fd:
                     return json.load(fd)
-        except (IOError, ValueError):
+        except (OSError, ValueError):
             platform_utils.remove(self._json, missing_ok=True)
             return None
 
@@ -374,7 +392,7 @@ class GitConfig(object):
         try:
             with open(self._json, "w") as fd:
                 json.dump(cache, fd, indent=2)
-        except (IOError, TypeError):
+        except (OSError, TypeError):
             platform_utils.remove(self._json, missing_ok=True)
 
     def _ReadGit(self):
@@ -414,7 +432,7 @@ class GitConfig(object):
         if p.Wait() == 0:
             return p.stdout
         else:
-            raise GitError("git config %s: %s" % (str(args), p.stderr))
+            raise GitError(f"git config {str(args)}: {p.stderr}")
 
 
 class RepoConfig(GitConfig):
@@ -426,7 +444,7 @@ class RepoConfig(GitConfig):
         return os.path.join(repo_config_dir, ".repoconfig/config")
 
 
-class RefSpec(object):
+class RefSpec:
     """A Git refspec line, split into its components:
 
     forced:  True if the line starts with '+'
@@ -537,7 +555,7 @@ def GetUrlCookieFile(url, quiet):
     yield cookiefile, None
 
 
-class Remote(object):
+class Remote:
     """Configuration options related to a remote."""
 
     def __init__(self, config, name):
@@ -647,13 +665,11 @@ class Remote(object):
                             userEmail, host, port
                         )
                 except urllib.error.HTTPError as e:
-                    raise UploadError("%s: %s" % (self.review, str(e)))
+                    raise UploadError(f"{self.review}: {str(e)}")
                 except urllib.error.URLError as e:
-                    raise UploadError("%s: %s" % (self.review, str(e)))
-                except HTTPException as e:
-                    raise UploadError(
-                        "%s: %s" % (self.review, e.__class__.__name__)
-                    )
+                    raise UploadError(f"{self.review}: {str(e)}")
+                except http.client.HTTPException as e:
+                    raise UploadError(f"{self.review}: {e.__class__.__name__}")
 
                 REVIEW_CACHE[u] = self._review_url
         return self._review_url + self.projectname
@@ -662,7 +678,7 @@ class Remote(object):
         username = self._config.GetString("review.%s.username" % self.review)
         if username is None:
             username = userEmail.split("@")[0]
-        return "ssh://%s@%s:%s/" % (username, host, port)
+        return f"ssh://{username}@{host}:{port}/"
 
     def ToLocal(self, rev):
         """Convert a remote revision string to something we have locally."""
@@ -711,15 +727,15 @@ class Remote(object):
         self._Set("fetch", list(map(str, self.fetch)))
 
     def _Set(self, key, value):
-        key = "remote.%s.%s" % (self.name, key)
+        key = f"remote.{self.name}.{key}"
         return self._config.SetString(key, value)
 
     def _Get(self, key, all_keys=False):
-        key = "remote.%s.%s" % (self.name, key)
+        key = f"remote.{self.name}.{key}"
         return self._config.GetString(key, all_keys=all_keys)
 
 
-class Branch(object):
+class Branch:
     """Configuration options related to a single branch."""
 
     def __init__(self, config, name):
@@ -758,11 +774,11 @@ class Branch(object):
                     fd.write("\tmerge = %s\n" % self.merge)
 
     def _Set(self, key, value):
-        key = "branch.%s.%s" % (self.name, key)
+        key = f"branch.{self.name}.{key}"
         return self._config.SetString(key, value)
 
     def _Get(self, key, all_keys=False):
-        key = "branch.%s.%s" % (self.name, key)
+        key = f"branch.{self.name}.{key}"
         return self._config.GetString(key, all_keys=all_keys)
 
 
@@ -791,8 +807,8 @@ class SyncAnalysisState:
                 to be logged.
         """
         self._config = config
-        now = datetime.datetime.utcnow()
-        self._Set("main.synctime", now.isoformat(timespec="microseconds") + "Z")
+        now = datetime.datetime.now(datetime.timezone.utc)
+        self._Set("main.synctime", now.isoformat(timespec="microseconds"))
         self._Set("main.version", "1")
         self._Set("sys.argv", sys.argv)
         for key, value in superproject_logging_data.items():

git_refs.py

@@ -13,8 +13,10 @@
 # limitations under the License.
 
 import os
-from repo_trace import Trace
+
 import platform_utils
+from repo_trace import Trace
+
 
 HEAD = "HEAD"
 R_CHANGES = "refs/changes/"
@@ -26,7 +28,7 @@ R_WORKTREE_M = R_WORKTREE + "m/"
 R_M = "refs/remotes/m/"
 
 
-class GitRefs(object):
+class GitRefs:
     def __init__(self, gitdir):
         self._gitdir = gitdir
         self._phyref = None
@@ -103,10 +105,8 @@ class GitRefs(object):
     def _ReadPackedRefs(self):
         path = os.path.join(self._gitdir, "packed-refs")
         try:
-            fd = open(path, "r")
+            fd = open(path)
             mtime = os.path.getmtime(path)
-        except IOError:
-            return
         except OSError:
             return
         try:

git_ssh

@@ -1,5 +1,4 @@
 #!/bin/sh
-#
 # Copyright (C) 2009 The Android Open Source Project
 #
 # Licensed under the Apache License, Version 2.0 (the "License");

git_superproject.py

@@ -22,17 +22,20 @@ Examples:
   UpdateProjectsResult = superproject.UpdateProjectsRevisionId(projects)
 """
 
-import hashlib
 import functools
+import hashlib
 import os
 import sys
 import time
 from typing import NamedTuple
+import urllib.parse
 
-from git_command import git_require, GitCommand
+from git_command import git_require
+from git_command import GitCommand
 from git_config import RepoConfig
 from git_refs import GitRefs
 
+
 _SUPERPROJECT_GIT_NAME = "superproject.git"
 _SUPERPROJECT_MANIFEST_NAME = "superproject_override.xml"
 
@@ -64,12 +67,12 @@ class UpdateProjectsResult(NamedTuple):
     fatal: bool
 
 
-class Superproject(object):
+class Superproject:
     """Get commit ids from superproject.
 
-    Initializes a local copy of a superproject for the manifest. This allows
-    lookup of commit ids for all projects. It contains _project_commit_ids which
-    is a dictionary with project/commit id entries.
+    Initializes a bare local copy of a superproject for the manifest. This
+    allows lookup of commit ids for all projects. It contains
+    _project_commit_ids which is a dictionary with project/commit id entries.
     """
 
     def __init__(
@@ -126,6 +129,30 @@ class Superproject(object):
         """Set the _print_messages attribute."""
         self._print_messages = value
 
+    @property
+    def commit_id(self):
+        """Returns the commit ID of the superproject checkout."""
+        cmd = ["rev-parse", self.revision]
+        p = GitCommand(
+            None,  # project
+            cmd,
+            gitdir=self._work_git,
+            bare=True,
+            capture_stdout=True,
+            capture_stderr=True,
+        )
+        retval = p.Wait()
+        if retval != 0:
+            self._LogWarning(
+                "git rev-parse call failed, command: git {}, "
+                "return code: {}, stderr: {}",
+                cmd,
+                retval,
+                p.stderr,
+            )
+            return None
+        return p.stdout
+
     @property
     def project_commit_ids(self):
         """Returns a dictionary of projects and their commit ids."""
@@ -138,12 +165,33 @@ class Superproject(object):
             self._manifest_path if os.path.exists(self._manifest_path) else None
         )
 
+    @property
+    def repo_id(self):
+        """Returns the repo ID for the superproject.
+
+        For example, if the superproject points to:
+            https://android-review.googlesource.com/platform/superproject/
+        Then the repo_id would be:
+            android/platform/superproject
+        """
+        review_url = self.remote.review
+        if review_url:
+            parsed_url = urllib.parse.urlparse(review_url)
+            netloc = parsed_url.netloc
+            if netloc:
+                parts = netloc.split("-review", 1)
+                host = parts[0]
+                rev = GitRefs(self._work_git).get("HEAD")
+                return f"{host}/{self.name}@{rev}"
+        return None
+
     def _LogMessage(self, fmt, *inputs):
         """Logs message to stderr and _git_event_log."""
         message = f"{self._LogMessagePrefix()} {fmt.format(*inputs)}"
         if self._print_messages:
             print(message, file=sys.stderr)
-        self._git_event_log.ErrorEvent(message, fmt)
+        if self._git_event_log:
+            self._git_event_log.ErrorEvent(message, fmt)
 
     def _LogMessagePrefix(self):
         """Returns the prefix string to be logged in each log message"""
@@ -233,7 +281,8 @@ class Superproject(object):
         p = GitCommand(
             None,
             cmd,
-            cwd=self._work_git,
+            gitdir=self._work_git,
+            bare=True,
             capture_stdout=True,
             capture_stderr=True,
         )
@@ -255,7 +304,7 @@ class Superproject(object):
         Works only in git repositories.
 
         Returns:
-            data: data returned from 'git ls-tree ...' instead of None.
+            data: data returned from 'git ls-tree ...'. None on error.
         """
         if not os.path.exists(self._work_git):
             self._LogWarning(
@@ -269,7 +318,8 @@ class Superproject(object):
         p = GitCommand(
             None,
             cmd,
-            cwd=self._work_git,
+            gitdir=self._work_git,
+            bare=True,
             capture_stdout=True,
             capture_stderr=True,
         )
@@ -284,6 +334,7 @@ class Superproject(object):
                 retval,
                 p.stderr,
             )
+            return None
         return data
 
     def Sync(self, git_event_log):
@@ -303,8 +354,6 @@ class Superproject(object):
             )
             return SyncResult(False, False)
 
-        _PrintBetaNotice()
-
         should_exit = True
         if not self._remote_url:
             self._LogWarning(
@@ -373,13 +422,14 @@ class Superproject(object):
             )
             return None
         manifest_str = self._manifest.ToXml(
-            groups=self._manifest.GetGroupsStr(), omit_local=True
+            filter_groups=self._manifest.GetManifestGroupsStr(),
+            omit_local=True,
         ).toxml()
         manifest_path = self._manifest_path
         try:
             with open(manifest_path, "w", encoding="utf-8") as fp:
                 fp.write(manifest_str)
-        except IOError as e:
+        except OSError as e:
             self._LogError("cannot write manifest to : {} {}", manifest_path, e)
             return None
         return manifest_path
@@ -448,16 +498,6 @@ class Superproject(object):
         return UpdateProjectsResult(manifest_path, False)
 
 
-@functools.lru_cache(maxsize=10)
-def _PrintBetaNotice():
-    """Print the notice of beta status."""
-    print(
-        "NOTICE: --use-superproject is in beta; report any issues to the "
-        "address described in `repo version`",
-        file=sys.stderr,
-    )
-
-
 @functools.lru_cache(maxsize=None)
 def _UseSuperprojectFromConfiguration():
     """Returns the user choice of whether to use superproject."""

git_trace2_event_log.py

@@ -12,35 +12,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Provide event logging in the git trace2 EVENT format.
+"""Event logging in the git trace2 EVENT format."""
 
-The git trace2 EVENT format is defined at:
-https://www.kernel.org/pub/software/scm/git/docs/technical/api-trace2.html#_event_format
-https://git-scm.com/docs/api-trace2#_the_event_format_target
-
-  Usage:
-
-  git_trace_log = EventLog()
-  git_trace_log.StartEvent()
-  ...
-  git_trace_log.ExitEvent()
-  git_trace_log.Write()
-"""
+from git_command import GetEventTargetPath
+from git_command import RepoSourceVersion
+from git_trace2_event_log_base import BaseEventLog
 
 
-import datetime
-import errno
-import json
-import os
-import socket
-import sys
-import tempfile
-import threading
-
-from git_command import GitCommand, RepoSourceVersion
-
-
-class EventLog(object):
+class EventLog(BaseEventLog):
     """Event log that records events that occurred during a repo invocation.
 
     Events are written to the log as a consecutive JSON entries, one per line.
@@ -57,314 +36,13 @@ class EventLog(object):
     https://git-scm.com/docs/api-trace2#_event_format
     """
 
-    def __init__(self, env=None):
-        """Initializes the event log."""
-        self._log = []
-        # Try to get session-id (sid) from environment (setup in repo launcher).
-        KEY = "GIT_TRACE2_PARENT_SID"
-        if env is None:
-            env = os.environ
+    def __init__(self, **kwargs):
+        super().__init__(repo_source_version=RepoSourceVersion(), **kwargs)
 
-        now = datetime.datetime.utcnow()
-
-        # Save both our sid component and the complete sid.
-        # We use our sid component (self._sid) as the unique filename prefix and
-        # the full sid (self._full_sid) in the log itself.
-        self._sid = "repo-%s-P%08x" % (
-            now.strftime("%Y%m%dT%H%M%SZ"),
-            os.getpid(),
-        )
-        parent_sid = env.get(KEY)
-        # Append our sid component to the parent sid (if it exists).
-        if parent_sid is not None:
-            self._full_sid = parent_sid + "/" + self._sid
-        else:
-            self._full_sid = self._sid
-
-        # Set/update the environment variable.
-        # Environment handling across systems is messy.
-        try:
-            env[KEY] = self._full_sid
-        except UnicodeEncodeError:
-            env[KEY] = self._full_sid.encode()
-
-        # Add a version event to front of the log.
-        self._AddVersionEvent()
-
-    @property
-    def full_sid(self):
-        return self._full_sid
-
-    def _AddVersionEvent(self):
-        """Adds a 'version' event at the beginning of current log."""
-        version_event = self._CreateEventDict("version")
-        version_event["evt"] = "2"
-        version_event["exe"] = RepoSourceVersion()
-        self._log.insert(0, version_event)
-
-    def _CreateEventDict(self, event_name):
-        """Returns a dictionary with common keys/values for git trace2 events.
-
-        Args:
-            event_name: The event name.
-
-        Returns:
-            Dictionary with the common event fields populated.
-        """
-        return {
-            "event": event_name,
-            "sid": self._full_sid,
-            "thread": threading.current_thread().name,
-            "time": datetime.datetime.utcnow().isoformat() + "Z",
-        }
-
-    def StartEvent(self):
-        """Append a 'start' event to the current log."""
-        start_event = self._CreateEventDict("start")
-        start_event["argv"] = sys.argv
-        self._log.append(start_event)
-
-    def ExitEvent(self, result):
-        """Append an 'exit' event to the current log.
-
-        Args:
-            result: Exit code of the event
-        """
-        exit_event = self._CreateEventDict("exit")
-
-        # Consider 'None' success (consistent with event_log result handling).
-        if result is None:
-            result = 0
-        exit_event["code"] = result
-        self._log.append(exit_event)
-
-    def CommandEvent(self, name, subcommands):
-        """Append a 'command' event to the current log.
-
-        Args:
-            name: Name of the primary command (ex: repo, git)
-            subcommands: List of the sub-commands (ex: version, init, sync)
-        """
-        command_event = self._CreateEventDict("command")
-        command_event["name"] = name
-        command_event["subcommands"] = subcommands
-        self._log.append(command_event)
-
-    def LogConfigEvents(self, config, event_dict_name):
-        """Append a |event_dict_name| event for each config key in |config|.
-
-        Args:
-            config: Configuration dictionary.
-            event_dict_name: Name of the event dictionary for items to be logged
-                under.
-        """
-        for param, value in config.items():
-            event = self._CreateEventDict(event_dict_name)
-            event["param"] = param
-            event["value"] = value
-            self._log.append(event)
-
-    def DefParamRepoEvents(self, config):
-        """Append 'def_param' events for repo config keys to the current log.
-
-        This appends one event for each repo.* config key.
-
-        Args:
-            config: Repo configuration dictionary
-        """
-        # Only output the repo.* config parameters.
-        repo_config = {k: v for k, v in config.items() if k.startswith("repo.")}
-        self.LogConfigEvents(repo_config, "def_param")
-
-    def GetDataEventName(self, value):
-        """Returns 'data-json' if the value is an array else returns 'data'."""
-        return "data-json" if value[0] == "[" and value[-1] == "]" else "data"
-
-    def LogDataConfigEvents(self, config, prefix):
-        """Append a 'data' event for each entry in |config| to the current log.
-
-        For each keyX and valueX of the config, "key" field of the event is
-        '|prefix|/keyX' and the "value" of the "key" field is valueX.
-
-        Args:
-            config: Configuration dictionary.
-            prefix: Prefix for each key that is logged.
-        """
-        for key, value in config.items():
-            event = self._CreateEventDict(self.GetDataEventName(value))
-            event["key"] = f"{prefix}/{key}"
-            event["value"] = value
-            self._log.append(event)
-
-    def ErrorEvent(self, msg, fmt):
-        """Append a 'error' event to the current log."""
-        error_event = self._CreateEventDict("error")
-        error_event["msg"] = msg
-        error_event["fmt"] = fmt
-        self._log.append(error_event)
-
-    def _GetEventTargetPath(self):
-        """Get the 'trace2.eventtarget' path from git configuration.
-
-        Returns:
-            path: git config's 'trace2.eventtarget' path if it exists, or None
-        """
-        path = None
-        cmd = ["config", "--get", "trace2.eventtarget"]
-        # TODO(https://crbug.com/gerrit/13706): Use GitConfig when it supports
-        # system git config variables.
-        p = GitCommand(
-            None, cmd, capture_stdout=True, capture_stderr=True, bare=True
-        )
-        retval = p.Wait()
-        if retval == 0:
-            # Strip trailing carriage-return in path.
-            path = p.stdout.rstrip("\n")
-        elif retval != 1:
-            # `git config --get` is documented to produce an exit status of `1`
-            # if the requested variable is not present in the configuration.
-            # Report any other return value as an error.
-            print(
-                "repo: error: 'git config --get' call failed with return code: "
-                "%r, stderr: %r" % (retval, p.stderr),
-                file=sys.stderr,
-            )
-        return path
-
-    def _WriteLog(self, write_fn):
-        """Writes the log out using a provided writer function.
-
-        Generate compact JSON output for each item in the log, and write it
-        using write_fn.
-
-        Args:
-            write_fn: A function that accepts byts and writes them to a
-                destination.
-        """
-
-        for e in self._log:
-            # Dump in compact encoding mode.
-            # See 'Compact encoding' in Python docs:
-            # https://docs.python.org/3/library/json.html#module-json
-            write_fn(
-                json.dumps(e, indent=None, separators=(",", ":")).encode(
-                    "utf-8"
-                )
-                + b"\n"
-            )
-
-    def Write(self, path=None):
-        """Writes the log out to a file or socket.
-
-        Log is only written if 'path' or 'git config --get trace2.eventtarget'
-        provide a valid path (or socket) to write logs to.
-
-        Logging filename format follows the git trace2 style of being a unique
-        (exclusive writable) file.
-
-        Args:
-            path: Path to where logs should be written. The path may have a
-                prefix of the form "af_unix:[{stream|dgram}:]", in which case
-                the path is treated as a Unix domain socket. See
-                https://git-scm.com/docs/api-trace2#_enabling_a_target for
-                details.
-
-        Returns:
-            log_path: Path to the log file or socket if log is written,
-                otherwise None
-        """
-        log_path = None
-        # If no logging path is specified, get the path from
-        # 'trace2.eventtarget'.
+    def Write(self, path=None, **kwargs):
         if path is None:
             path = self._GetEventTargetPath()
+        return super().Write(path=path, **kwargs)
 
-        # If no logging path is specified, exit.
-        if path is None:
-            return None
-
-        path_is_socket = False
-        socket_type = None
-        if isinstance(path, str):
-            parts = path.split(":", 1)
-            if parts[0] == "af_unix" and len(parts) == 2:
-                path_is_socket = True
-                path = parts[1]
-                parts = path.split(":", 1)
-                if parts[0] == "stream" and len(parts) == 2:
-                    socket_type = socket.SOCK_STREAM
-                    path = parts[1]
-                elif parts[0] == "dgram" and len(parts) == 2:
-                    socket_type = socket.SOCK_DGRAM
-                    path = parts[1]
-            else:
-                # Get absolute path.
-                path = os.path.abspath(os.path.expanduser(path))
-        else:
-            raise TypeError("path: str required but got %s." % type(path))
-
-        # Git trace2 requires a directory to write log to.
-
-        # TODO(https://crbug.com/gerrit/13706): Support file (append) mode also.
-        if not (path_is_socket or os.path.isdir(path)):
-            return None
-
-        if path_is_socket:
-            if socket_type == socket.SOCK_STREAM or socket_type is None:
-                try:
-                    with socket.socket(
-                        socket.AF_UNIX, socket.SOCK_STREAM
-                    ) as sock:
-                        sock.connect(path)
-                        self._WriteLog(sock.sendall)
-                    return f"af_unix:stream:{path}"
-                except OSError as err:
-                    # If we tried to connect to a DGRAM socket using STREAM,
-                    # ignore the attempt and continue to DGRAM below. Otherwise,
-                    # issue a warning.
-                    if err.errno != errno.EPROTOTYPE:
-                        print(
-                            f"repo: warning: git trace2 logging failed: {err}",
-                            file=sys.stderr,
-                        )
-                        return None
-            if socket_type == socket.SOCK_DGRAM or socket_type is None:
-                try:
-                    with socket.socket(
-                        socket.AF_UNIX, socket.SOCK_DGRAM
-                    ) as sock:
-                        self._WriteLog(lambda bs: sock.sendto(bs, path))
-                        return f"af_unix:dgram:{path}"
-                except OSError as err:
-                    print(
-                        f"repo: warning: git trace2 logging failed: {err}",
-                        file=sys.stderr,
-                    )
-                    return None
-            # Tried to open a socket but couldn't connect (SOCK_STREAM) or write
-            # (SOCK_DGRAM).
-            print(
-                "repo: warning: git trace2 logging failed: could not write to "
-                "socket",
-                file=sys.stderr,
-            )
-            return None
-
-        # Path is an absolute path
-        # Use NamedTemporaryFile to generate a unique filename as required by
-        # git trace2.
-        try:
-            with tempfile.NamedTemporaryFile(
-                mode="xb", prefix=self._sid, dir=path, delete=False
-            ) as f:
-                # TODO(https://crbug.com/gerrit/13706): Support writing events
-                # as they occur.
-                self._WriteLog(f.write)
-                log_path = f.name
-        except FileExistsError as err:
-            print(
-                "repo: warning: git trace2 logging failed: %r" % err,
-                file=sys.stderr,
-            )
-            return None
-        return log_path
+    def _GetEventTargetPath(self):
+        return GetEventTargetPath()

git_trace2_event_log_base.py

@@ -0,0 +1,356 @@
+# Copyright (C) 2020 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Provide event logging in the git trace2 EVENT format.
+
+The git trace2 EVENT format is defined at:
+https://www.kernel.org/pub/software/scm/git/docs/technical/api-trace2.html#_event_format
+https://git-scm.com/docs/api-trace2#_the_event_format_target
+
+  Usage:
+
+  git_trace_log = EventLog()
+  git_trace_log.StartEvent()
+  ...
+  git_trace_log.ExitEvent()
+  git_trace_log.Write()
+"""
+
+
+import datetime
+import errno
+import json
+import os
+import socket
+import sys
+import tempfile
+import threading
+
+
+# Timeout when sending events via socket (applies to connect, send)
+SOCK_TIMEOUT = 0.5  # in seconds
+# BaseEventLog __init__ Counter that is consistent within the same process
+p_init_count = 0
+
+
+class BaseEventLog:
+    """Event log that records events that occurred during a repo invocation.
+
+    Events are written to the log as a consecutive JSON entries, one per line.
+    Entries follow the git trace2 EVENT format.
+
+    Each entry contains the following common keys:
+    - event: The event name
+    - sid: session-id - Unique string to allow process instance to be
+          identified.
+    - thread: The thread name.
+    - time: is the UTC time of the event.
+
+    Valid 'event' names and event specific fields are documented here:
+    https://git-scm.com/docs/api-trace2#_event_format
+    """
+
+    def __init__(
+        self, env=None, repo_source_version=None, add_init_count=False
+    ):
+        """Initializes the event log."""
+        global p_init_count
+        p_init_count += 1
+        self._log = []
+        # Try to get session-id (sid) from environment (setup in repo launcher).
+        KEY = "GIT_TRACE2_PARENT_SID"
+        if env is None:
+            env = os.environ
+
+        self.start = datetime.datetime.now(datetime.timezone.utc)
+
+        # Save both our sid component and the complete sid.
+        # We use our sid component (self._sid) as the unique filename prefix and
+        # the full sid (self._full_sid) in the log itself.
+        self._sid = (
+            f"repo-{self.start.strftime('%Y%m%dT%H%M%SZ')}-P{os.getpid():08x}"
+        )
+
+        if add_init_count:
+            self._sid = f"{self._sid}-{p_init_count}"
+
+        parent_sid = env.get(KEY)
+        # Append our sid component to the parent sid (if it exists).
+        if parent_sid is not None:
+            self._full_sid = parent_sid + "/" + self._sid
+        else:
+            self._full_sid = self._sid
+
+        # Set/update the environment variable.
+        # Environment handling across systems is messy.
+        try:
+            env[KEY] = self._full_sid
+        except UnicodeEncodeError:
+            env[KEY] = self._full_sid.encode()
+
+        if repo_source_version is not None:
+            # Add a version event to front of the log.
+            self._AddVersionEvent(repo_source_version)
+
+    @property
+    def full_sid(self):
+        return self._full_sid
+
+    def _AddVersionEvent(self, repo_source_version):
+        """Adds a 'version' event at the beginning of current log."""
+        version_event = self._CreateEventDict("version")
+        version_event["evt"] = "2"
+        version_event["exe"] = repo_source_version
+        self._log.insert(0, version_event)
+
+    def _CreateEventDict(self, event_name):
+        """Returns a dictionary with common keys/values for git trace2 events.
+
+        Args:
+            event_name: The event name.
+
+        Returns:
+            Dictionary with the common event fields populated.
+        """
+        return {
+            "event": event_name,
+            "sid": self._full_sid,
+            "thread": threading.current_thread().name,
+            "time": datetime.datetime.now(datetime.timezone.utc).isoformat(),
+        }
+
+    def StartEvent(self, argv):
+        """Append a 'start' event to the current log."""
+        start_event = self._CreateEventDict("start")
+        start_event["argv"] = argv
+        self._log.append(start_event)
+
+    def ExitEvent(self, result):
+        """Append an 'exit' event to the current log.
+
+        Args:
+            result: Exit code of the event
+        """
+        exit_event = self._CreateEventDict("exit")
+
+        # Consider 'None' success (consistent with event_log result handling).
+        if result is None:
+            result = 0
+        exit_event["code"] = result
+        time_delta = datetime.datetime.now(datetime.timezone.utc) - self.start
+        exit_event["t_abs"] = time_delta.total_seconds()
+        self._log.append(exit_event)
+
+    def CommandEvent(self, name, subcommands):
+        """Append a 'command' event to the current log.
+
+        Args:
+            name: Name of the primary command (ex: repo, git)
+            subcommands: List of the sub-commands (ex: version, init, sync)
+        """
+        command_event = self._CreateEventDict("cmd_name")
+        name = f"{name}-"
+        name += "-".join(subcommands)
+        command_event["name"] = name
+        command_event["hierarchy"] = name
+        self._log.append(command_event)
+
+    def LogConfigEvents(self, config, event_dict_name):
+        """Append a |event_dict_name| event for each config key in |config|.
+
+        Args:
+            config: Configuration dictionary.
+            event_dict_name: Name of the event dictionary for items to be logged
+                under.
+        """
+        for param, value in config.items():
+            event = self._CreateEventDict(event_dict_name)
+            event["param"] = param
+            event["value"] = value
+            self._log.append(event)
+
+    def DefParamRepoEvents(self, config):
+        """Append 'def_param' events for repo config keys to the current log.
+
+        This appends one event for each repo.* config key.
+
+        Args:
+            config: Repo configuration dictionary
+        """
+        # Only output the repo.* config parameters.
+        repo_config = {k: v for k, v in config.items() if k.startswith("repo.")}
+        self.LogConfigEvents(repo_config, "def_param")
+
+    def GetDataEventName(self, value):
+        """Returns 'data-json' if the value is an array else returns 'data'."""
+        return "data-json" if value[0] == "[" and value[-1] == "]" else "data"
+
+    def LogDataConfigEvents(self, config, prefix):
+        """Append a 'data' event for each entry in |config| to the current log.
+
+        For each keyX and valueX of the config, "key" field of the event is
+        '|prefix|/keyX' and the "value" of the "key" field is valueX.
+
+        Args:
+            config: Configuration dictionary.
+            prefix: Prefix for each key that is logged.
+        """
+        for key, value in config.items():
+            event = self._CreateEventDict(self.GetDataEventName(value))
+            event["key"] = f"{prefix}/{key}"
+            event["value"] = value
+            self._log.append(event)
+
+    def ErrorEvent(self, msg, fmt=None):
+        """Append a 'error' event to the current log."""
+        error_event = self._CreateEventDict("error")
+        if fmt is None:
+            fmt = msg
+        error_event["msg"] = f"RepoErrorEvent:{msg}"
+        error_event["fmt"] = f"RepoErrorEvent:{fmt}"
+        self._log.append(error_event)
+
+    def _WriteLog(self, write_fn):
+        """Writes the log out using a provided writer function.
+
+        Generate compact JSON output for each item in the log, and write it
+        using write_fn.
+
+        Args:
+            write_fn: A function that accepts byts and writes them to a
+                destination.
+        """
+
+        for e in self._log:
+            # Dump in compact encoding mode.
+            # See 'Compact encoding' in Python docs:
+            # https://docs.python.org/3/library/json.html#module-json
+            write_fn(
+                json.dumps(e, indent=None, separators=(",", ":")).encode(
+                    "utf-8"
+                )
+                + b"\n"
+            )
+
+    def Write(self, path=None):
+        """Writes the log out to a file or socket.
+
+        Log is only written if 'path' or 'git config --get trace2.eventtarget'
+        provide a valid path (or socket) to write logs to.
+
+        Logging filename format follows the git trace2 style of being a unique
+        (exclusive writable) file.
+
+        Args:
+            path: Path to where logs should be written. The path may have a
+                prefix of the form "af_unix:[{stream|dgram}:]", in which case
+                the path is treated as a Unix domain socket. See
+                https://git-scm.com/docs/api-trace2#_enabling_a_target for
+                details.
+
+        Returns:
+            log_path: Path to the log file or socket if log is written,
+                otherwise None
+        """
+        log_path = None
+        # If no logging path is specified, exit.
+        if path is None:
+            return None
+
+        path_is_socket = False
+        socket_type = None
+        if isinstance(path, str):
+            parts = path.split(":", 1)
+            if parts[0] == "af_unix" and len(parts) == 2:
+                path_is_socket = True
+                path = parts[1]
+                parts = path.split(":", 1)
+                if parts[0] == "stream" and len(parts) == 2:
+                    socket_type = socket.SOCK_STREAM
+                    path = parts[1]
+                elif parts[0] == "dgram" and len(parts) == 2:
+                    socket_type = socket.SOCK_DGRAM
+                    path = parts[1]
+            else:
+                # Get absolute path.
+                path = os.path.abspath(os.path.expanduser(path))
+        else:
+            raise TypeError("path: str required but got %s." % type(path))
+
+        # Git trace2 requires a directory to write log to.
+
+        # TODO(https://crbug.com/gerrit/13706): Support file (append) mode also.
+        if not (path_is_socket or os.path.isdir(path)):
+            return None
+
+        if path_is_socket:
+            if socket_type == socket.SOCK_STREAM or socket_type is None:
+                try:
+                    with socket.socket(
+                        socket.AF_UNIX, socket.SOCK_STREAM
+                    ) as sock:
+                        sock.settimeout(SOCK_TIMEOUT)
+                        sock.connect(path)
+                        self._WriteLog(sock.sendall)
+                    return f"af_unix:stream:{path}"
+                except OSError as err:
+                    # If we tried to connect to a DGRAM socket using STREAM,
+                    # ignore the attempt and continue to DGRAM below. Otherwise,
+                    # issue a warning.
+                    if err.errno != errno.EPROTOTYPE:
+                        print(
+                            f"repo: warning: git trace2 logging failed: {err}",
+                            file=sys.stderr,
+                        )
+                        return None
+            if socket_type == socket.SOCK_DGRAM or socket_type is None:
+                try:
+                    with socket.socket(
+                        socket.AF_UNIX, socket.SOCK_DGRAM
+                    ) as sock:
+                        self._WriteLog(lambda bs: sock.sendto(bs, path))
+                        return f"af_unix:dgram:{path}"
+                except OSError as err:
+                    print(
+                        f"repo: warning: git trace2 logging failed: {err}",
+                        file=sys.stderr,
+                    )
+                    return None
+            # Tried to open a socket but couldn't connect (SOCK_STREAM) or write
+            # (SOCK_DGRAM).
+            print(
+                "repo: warning: git trace2 logging failed: could not write to "
+                "socket",
+                file=sys.stderr,
+            )
+            return None
+
+        # Path is an absolute path
+        # Use NamedTemporaryFile to generate a unique filename as required by
+        # git trace2.
+        try:
+            with tempfile.NamedTemporaryFile(
+                mode="xb", prefix=self._sid, dir=path, delete=False
+            ) as f:
+                # TODO(https://crbug.com/gerrit/13706): Support writing events
+                # as they occur.
+                self._WriteLog(f.write)
+                log_path = f.name
+        except FileExistsError as err:
+            print(
+                "repo: warning: git trace2 logging failed: %r" % err,
+                file=sys.stderr,
+            )
+            return None
+        return log_path

gitc_utils.py

@@ -1,166 +0,0 @@
-# Copyright (C) 2015 The Android Open Source Project
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import os
-import multiprocessing
-import re
-import sys
-import time
-
-import git_command
-import git_config
-import wrapper
-
-from error import ManifestParseError
-
-NUM_BATCH_RETRIEVE_REVISIONID = 32
-
-
-def get_gitc_manifest_dir():
-    return wrapper.Wrapper().get_gitc_manifest_dir()
-
-
-def parse_clientdir(gitc_fs_path):
-    return wrapper.Wrapper().gitc_parse_clientdir(gitc_fs_path)
-
-
-def _get_project_revision(args):
-    """Worker for _set_project_revisions to lookup one project remote."""
-    (i, url, expr) = args
-    gitcmd = git_command.GitCommand(
-        None, ["ls-remote", url, expr], capture_stdout=True, cwd="/tmp"
-    )
-    rc = gitcmd.Wait()
-    return (i, rc, gitcmd.stdout.split("\t", 1)[0])
-
-
-def _set_project_revisions(projects):
-    """Sets the revisionExpr for a list of projects.
-
-    Because of the limit of open file descriptors allowed, length of projects
-    should not be overly large. Recommend calling this function multiple times
-    with each call not exceeding NUM_BATCH_RETRIEVE_REVISIONID projects.
-
-    Args:
-        projects: List of project objects to set the revionExpr for.
-    """
-    # Retrieve the commit id for each project based off of its current
-    # revisionExpr and it is not already a commit id.
-    with multiprocessing.Pool(NUM_BATCH_RETRIEVE_REVISIONID) as pool:
-        results_iter = pool.imap_unordered(
-            _get_project_revision,
-            (
-                (i, project.remote.url, project.revisionExpr)
-                for i, project in enumerate(projects)
-                if not git_config.IsId(project.revisionExpr)
-            ),
-            chunksize=8,
-        )
-        for i, rc, revisionExpr in results_iter:
-            project = projects[i]
-            if rc:
-                print(
-                    "FATAL: Failed to retrieve revisionExpr for %s"
-                    % project.name
-                )
-                pool.terminate()
-                sys.exit(1)
-            if not revisionExpr:
-                pool.terminate()
-                raise ManifestParseError(
-                    "Invalid SHA-1 revision project %s (%s)"
-                    % (project.remote.url, project.revisionExpr)
-                )
-            project.revisionExpr = revisionExpr
-
-
-def generate_gitc_manifest(gitc_manifest, manifest, paths=None):
-    """Generate a manifest for shafsd to use for this GITC client.
-
-    Args:
-        gitc_manifest: Current gitc manifest, or None if there isn't one yet.
-        manifest: A GitcManifest object loaded with the current repo manifest.
-        paths: List of project paths we want to update.
-    """
-
-    print(
-        "Generating GITC Manifest by fetching revision SHAs for each "
-        "project."
-    )
-    if paths is None:
-        paths = list(manifest.paths.keys())
-
-    groups = [x for x in re.split(r"[,\s]+", manifest.GetGroupsStr()) if x]
-
-    # Convert the paths to projects, and filter them to the matched groups.
-    projects = [manifest.paths[p] for p in paths]
-    projects = [p for p in projects if p.MatchesGroups(groups)]
-
-    if gitc_manifest is not None:
-        for path, proj in manifest.paths.items():
-            if not proj.MatchesGroups(groups):
-                continue
-
-            if not proj.upstream and not git_config.IsId(proj.revisionExpr):
-                proj.upstream = proj.revisionExpr
-
-            if path not in gitc_manifest.paths:
-                # Any new projects need their first revision, even if we weren't
-                # asked for them.
-                projects.append(proj)
-            elif path not in paths:
-                # And copy revisions from the previous manifest if we're not
-                # updating them now.
-                gitc_proj = gitc_manifest.paths[path]
-                if gitc_proj.old_revision:
-                    proj.revisionExpr = None
-                    proj.old_revision = gitc_proj.old_revision
-                else:
-                    proj.revisionExpr = gitc_proj.revisionExpr
-
-    _set_project_revisions(projects)
-
-    if gitc_manifest is not None:
-        for path, proj in gitc_manifest.paths.items():
-            if proj.old_revision and path in paths:
-                # If we updated a project that has been started, keep the
-                # old-revision updated.
-                repo_proj = manifest.paths[path]
-                repo_proj.old_revision = repo_proj.revisionExpr
-                repo_proj.revisionExpr = None
-
-    # Convert URLs from relative to absolute.
-    for _name, remote in manifest.remotes.items():
-        remote.fetchUrl = remote.resolvedFetchUrl
-
-    # Save the manifest.
-    save_manifest(manifest)
-
-
-def save_manifest(manifest, client_dir=None):
-    """Save the manifest file in the client_dir.
-
-    Args:
-        manifest: Manifest object to save.
-        client_dir: Client directory to save the manifest in.
-    """
-    if not client_dir:
-        manifest_file = manifest.manifestFile
-    else:
-        manifest_file = os.path.join(client_dir, ".manifest")
-    with open(manifest_file, "w") as f:
-        manifest.Save(f, groups=manifest.GetGroupsStr())
-    # TODO(sbasi/jorg): Come up with a solution to remove the sleep below.
-    # Give the GITC filesystem time to register the manifest changes.
-    time.sleep(3)

hooks.py

@@ -12,11 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import errno
-import json
 import os
 import re
-import subprocess
 import sys
 import traceback
 import urllib.parse
@@ -25,7 +22,14 @@ from error import HookError
 from git_refs import HEAD
 
 
-class RepoHook(object):
+# The API we've documented to hook authors.  Keep in sync with repo-hooks.md.
+_API_ARGS = {
+    "pre-upload": {"project_list", "worktree_list"},
+    "post-sync": {"repo_topdir"},
+}
+
+
+class RepoHook:
     """A RepoHook contains information about a script to run as a hook.
 
     Hooks are used to run a python script before running an upload (for
@@ -59,6 +63,7 @@ class RepoHook(object):
         hooks_project,
         repo_topdir,
         manifest_url,
+        bug_url=None,
         bypass_hooks=False,
         allow_all_hooks=False,
         ignore_hooks=False,
@@ -78,6 +83,7 @@ class RepoHook(object):
                 run with CWD as this directory.
                 If you have a manifest, this is manifest.topdir.
             manifest_url: The URL to the manifest git repo.
+            bug_url: The URL to report issues.
             bypass_hooks: If True, then 'Do not run the hook'.
             allow_all_hooks: If True, then 'Run the hook without prompting'.
             ignore_hooks: If True, then 'Do not abort action if hooks fail'.
@@ -88,18 +94,18 @@ class RepoHook(object):
         self._hooks_project = hooks_project
         self._repo_topdir = repo_topdir
         self._manifest_url = manifest_url
+        self._bug_url = bug_url
         self._bypass_hooks = bypass_hooks
         self._allow_all_hooks = allow_all_hooks
         self._ignore_hooks = ignore_hooks
         self._abort_if_user_denies = abort_if_user_denies
 
         # Store the full path to the script for convenience.
-        if self._hooks_project:
+        self._script_fullpath = None
+        if self._hooks_project and self._hooks_project.worktree:
             self._script_fullpath = os.path.join(
                 self._hooks_project.worktree, self._hook_type + ".py"
             )
-        else:
-            self._script_fullpath = None
 
     def _GetHash(self):
         """Return a hash of the contents of the hooks directory.
@@ -183,7 +189,7 @@ class RepoHook(object):
                 abort_if_user_denies was passed to the consturctor.
         """
         hooks_config = self._hooks_project.config
-        git_approval_key = "repo.hooks.%s.%s" % (self._hook_type, subkey)
+        git_approval_key = f"repo.hooks.{self._hook_type}.{subkey}"
 
         # Get the last value that the user approved for this hook; may be None.
         old_val = hooks_config.GetString(git_approval_key)
@@ -196,7 +202,7 @@ class RepoHook(object):
             else:
                 # Give the user a reason why we're prompting, since they last
                 # told us to "never ask again".
-                prompt = "WARNING: %s\n\n" % (changed_prompt,)
+                prompt = f"WARNING: {changed_prompt}\n\n"
         else:
             prompt = ""
 
@@ -244,9 +250,8 @@ class RepoHook(object):
         return self._CheckForHookApprovalHelper(
             "approvedmanifest",
             self._manifest_url,
-            "Run hook scripts from %s" % (self._manifest_url,),
-            "Manifest URL has changed since %s was allowed."
-            % (self._hook_type,),
+            f"Run hook scripts from {self._manifest_url}",
+            f"Manifest URL has changed since {self._hook_type} was allowed.",
         )
 
     def _CheckForHookApprovalHash(self):
@@ -265,7 +270,7 @@ class RepoHook(object):
             "approvedhash",
             self._GetHash(),
             prompt % (self._GetMustVerb(), self._script_fullpath),
-            "Scripts have changed since %s was allowed." % (self._hook_type,),
+            f"Scripts have changed since {self._hook_type} was allowed.",
         )
 
     @staticmethod
@@ -298,43 +303,6 @@ class RepoHook(object):
 
         return interp
 
-    def _ExecuteHookViaReexec(self, interp, context, **kwargs):
-        """Execute the hook script through |interp|.
-
-        Note: Support for this feature should be dropped ~Jun 2021.
-
-        Args:
-            interp: The Python program to run.
-            context: Basic Python context to execute the hook inside.
-            kwargs: Arbitrary arguments to pass to the hook script.
-
-        Raises:
-            HookError: When the hooks failed for any reason.
-        """
-        # This logic needs to be kept in sync with _ExecuteHookViaImport below.
-        script = """
-import json, os, sys
-path = '''%(path)s'''
-kwargs = json.loads('''%(kwargs)s''')
-context = json.loads('''%(context)s''')
-sys.path.insert(0, os.path.dirname(path))
-data = open(path).read()
-exec(compile(data, path, 'exec'), context)
-context['main'](**kwargs)
-""" % {
-            "path": self._script_fullpath,
-            "kwargs": json.dumps(kwargs),
-            "context": json.dumps(context),
-        }
-
-        # We pass the script via stdin to avoid OS argv limits.  It also makes
-        # unhandled exception tracebacks less verbose/confusing for users.
-        cmd = [interp, "-c", "import sys; exec(sys.stdin.read())"]
-        proc = subprocess.Popen(cmd, stdin=subprocess.PIPE)
-        proc.communicate(input=script.encode("utf-8"))
-        if proc.returncode:
-            raise HookError("Failed to run %s hook." % (self._hook_type,))
-
     def _ExecuteHookViaImport(self, data, context, **kwargs):
         """Execute the hook code in |data| directly.
 
@@ -412,30 +380,13 @@ context['main'](**kwargs)
             # See what version of python the hook has been written against.
             data = open(self._script_fullpath).read()
             interp = self._ExtractInterpFromShebang(data)
-            reexec = False
             if interp:
                 prog = os.path.basename(interp)
-                if prog.startswith("python2") and sys.version_info.major != 2:
-                    reexec = True
-                elif prog.startswith("python3") and sys.version_info.major == 2:
-                    reexec = True
-
-            # Attempt to execute the hooks through the requested version of
-            # Python.
-            if reexec:
-                try:
-                    self._ExecuteHookViaReexec(interp, context, **kwargs)
-                except OSError as e:
-                    if e.errno == errno.ENOENT:
-                        # We couldn't find the interpreter, so fallback to
-                        # importing.
-                        reexec = False
-                    else:
-                        raise
+                if prog.startswith("python2"):
+                    raise HookError("Python 2 is not supported")
 
             # Run the hook by importing directly.
-            if not reexec:
-                self._ExecuteHookViaImport(data, context, **kwargs)
+            self._ExecuteHookViaImport(data, context, **kwargs)
         finally:
             # Restore sys.path and CWD.
             sys.path = orig_syspath
@@ -472,11 +423,26 @@ context['main'](**kwargs)
                 ignore the result through the option combinations as listed in
                 AddHookOptionGroup().
         """
+        # Make sure our own callers use the documented API.
+        exp_kwargs = _API_ARGS.get(self._hook_type, set())
+        got_kwargs = set(kwargs.keys())
+        if exp_kwargs != got_kwargs:
+            print(
+                "repo internal error: "
+                f"hook '{self._hook_type}' called incorrectly\n"
+                f"  got:      {sorted(got_kwargs)}\n"
+                f"  expected: {sorted(exp_kwargs)}\n"
+                f"Please file a bug: {self._bug_url}",
+                file=sys.stderr,
+            )
+            return False
+
         # Do not do anything in case bypass_hooks is set, or
         # no-op if there is no hooks project or if hook is disabled.
         if (
             self._bypass_hooks
             or not self._hooks_project
+            or not self._script_fullpath
             or self._hook_type not in self._hooks_project.enabled_repo_hooks
         ):
             return True
@@ -530,6 +496,7 @@ context['main'](**kwargs)
                 "manifest_url": manifest.manifestProject.GetRemote(
                     "origin"
                 ).url,
+                "bug_url": manifest.contactinfo.bugurl,
             }
         )
         return cls(*args, **kwargs)

hooks/commit-msg

@@ -1,5 +1,8 @@
 #!/bin/sh
-# From Gerrit Code Review 3.6.1 c67916dbdc07555c44e32a68f92ffc484b9b34f0
+# DO NOT EDIT THIS FILE
+# All updates should be sent upstream: https://gerrit.googlesource.com/gerrit/
+# This is synced from commit: 62f5bbea67f6dafa6e22a601a0c298214c510caf
+# DO NOT EDIT THIS FILE
 #
 # Part of Gerrit Code Review (https://www.gerritcodereview.com/)
 #
@@ -31,14 +34,20 @@ if test ! -f "$1" ; then
 fi
 
 # Do not create a change id if requested
-if test "false" = "$(git config --bool --get gerrit.createChangeId)" ; then
-  exit 0
-fi
+case "$(git config --get gerrit.createChangeId)" in
+  false)
+    exit 0
+    ;;
+  always)
+    ;;
+  *)
+    # Do not create a change id for squash/fixup commits.
+    if head -n1 "$1" | LC_ALL=C grep -q '^[a-z][a-z]*! '; then
+      exit 0
+    fi
+    ;;
+esac
 
-# Do not create a change id for squash commits.
-if head -n1 "$1" | grep -q '^squash! '; then
-  exit 0
-fi
 
 if git rev-parse --verify HEAD >/dev/null 2>&1; then
   refhash="$(git rev-parse HEAD)"
@@ -51,7 +60,7 @@ dest="$1.tmp.${random}"
 
 trap 'rm -f "$dest" "$dest-2"' EXIT
 
-if ! git stripspace --strip-comments < "$1" > "${dest}" ; then
+if ! cat "$1" | sed -e '/>8/q' | git stripspace --strip-comments > "${dest}" ; then
    echo "cannot strip comments from $1"
    exit 1
 fi
@@ -65,7 +74,7 @@ reviewurl="$(git config --get gerrit.reviewUrl)"
 if test -n "${reviewurl}" ; then
   token="Link"
   value="${reviewurl%/}/id/I$random"
-  pattern=".*/id/I[0-9a-f]\{40\}$"
+  pattern=".*/id/I[0-9a-f]\{40\}"
 else
   token="Change-Id"
   value="I$random"
@@ -92,7 +101,7 @@ fi
 # Avoid the --where option which only appeared in Git 2.15
 if ! git -c trailer.where=before interpret-trailers \
          --trailer "Signed-off-by: $token: $value" < "$dest-2" |
-     sed -re "s/^Signed-off-by: ($token: )/\1/" \
+     sed -e "s/^Signed-off-by: \($token: \)/\1/" \
          -e "/^Signed-off-by: SENTINEL/d" > "$dest" ; then
   echo "cannot insert $token line in $1"
   exit 1

hooks/pre-auto-gc

@@ -1,33 +1,25 @@
 #!/bin/sh
+# DO NOT EDIT THIS FILE
+# All updates should be sent upstream: https://github.com/git/git
+# This is synced from commit: 00e10ef10e161a913893b8cb33aa080d4ca5baa6
+# DO NOT EDIT THIS FILE
 #
 # An example hook script to verify if you are on battery, in case you
-# are running Windows, Linux or OS X. Called by git-gc --auto with no
-# arguments. The hook should exit with non-zero status after issuing an
-# appropriate message if it wants to stop the auto repacking.
-
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
+# are running Linux or OS X. Called by git-gc --auto with no arguments.
+# The hook should exit with non-zero status after issuing an appropriate
+# message if it wants to stop the auto repacking.
 #
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# This hook is stored in the contrib/hooks directory. Your distribution
+# may have put this somewhere else. If you want to use this hook, you
+# should make this script executable then link to it in the repository
+# you would like to use it in.
 #
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
-
-if uname -s | grep -q "_NT-"
-then
-	if test -x $SYSTEMROOT/System32/Wbem/wmic
-	then
-		STATUS=$(wmic path win32_battery get batterystatus /format:list | tr -d '\r\n')
-		[ "$STATUS" = "BatteryStatus=2" ] && exit 0 || exit 1
-	fi
-	exit 0
-fi
+# For example, if the hook is stored in
+# /usr/share/git-core/contrib/hooks/pre-auto-gc-battery:
+#
+# cd /path/to/your/repository.git
+# ln -sf /usr/share/git-core/contrib/hooks/pre-auto-gc-battery \
+#	hooks/pre-auto-gc
 
 if test -x /sbin/on_ac_power && (/sbin/on_ac_power;test $? -ne 1)
 then
@@ -48,11 +40,6 @@ elif test -x /usr/bin/pmset && /usr/bin/pmset -g batt |
 	grep -q "drawing from 'AC Power'"
 then
 	exit 0
-elif test -d /sys/bus/acpi/drivers/battery && test 0 = \
-  "$(find /sys/bus/acpi/drivers/battery/ -type l | wc -l)";
-then
-	# No battery exists.
-	exit 0
 fi
 
 echo "Auto packing deferred; not on AC"

main.py

@@ -1,5 +1,4 @@
 #!/usr/bin/env python3
-#
 # Copyright (C) 2008 The Android Open Source Project
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -21,30 +20,28 @@ which takes care of execing this entry point.
 """
 
 import getpass
+import json
 import netrc
 import optparse
 import os
 import shlex
+import signal
 import sys
 import textwrap
 import time
 import urllib.request
 
+from repo_logging import RepoLogger
+
+
 try:
     import kerberos
 except ImportError:
     kerberos = None
 
 from color import SetDefaultColoring
-import event_log
-from repo_trace import SetTrace, Trace, SetTraceToStderr
-from git_command import user_agent
-from git_config import RepoConfig
-from git_trace2_event_log import EventLog
 from command import InteractiveCommand
 from command import MirrorSafeCommand
-from command import GitcAvailableCommand, GitcClientCommand
-from subcmds.version import Version
 from editor import Editor
 from error import DownloadError
 from error import InvalidProjectGroupsError
@@ -53,12 +50,27 @@ from error import ManifestParseError
 from error import NoManifestException
 from error import NoSuchProjectError
 from error import RepoChangedException
-import gitc_utils
-from manifest_xml import GitcClient, RepoClient
-from pager import RunPager, TerminatePager
-from wrapper import WrapperPath, Wrapper
-
+from error import RepoError
+from error import RepoExitError
+from error import RepoUnhandledExceptionError
+from error import SilentRepoExitError
+import event_log
+from git_command import user_agent
+from git_config import RepoConfig
+from git_trace2_event_log import EventLog
+from manifest_xml import RepoClient
+from pager import RunPager
+from pager import TerminatePager
+from repo_trace import SetTrace
+from repo_trace import SetTraceToStderr
+from repo_trace import Trace
 from subcmds import all_commands
+from subcmds.version import Version
+from wrapper import Wrapper
+from wrapper import WrapperPath
+
+
+logger = RepoLogger(__file__)
 
 
 # NB: These do not need to be kept in sync with the repo launcher script.
@@ -73,28 +85,22 @@ from subcmds import all_commands
 MIN_PYTHON_VERSION_SOFT = (3, 6)
 MIN_PYTHON_VERSION_HARD = (3, 6)
 
-if sys.version_info.major < 3:
-    print(
-        "repo: error: Python 2 is no longer supported; "
-        "Please upgrade to Python {}.{}+.".format(*MIN_PYTHON_VERSION_SOFT),
-        file=sys.stderr,
+if sys.version_info < MIN_PYTHON_VERSION_HARD:
+    logger.error(
+        "repo: error: Python version is too old; "
+        "Please upgrade to Python %d.%d+.",
+        *MIN_PYTHON_VERSION_SOFT,
     )
     sys.exit(1)
-else:
-    if sys.version_info < MIN_PYTHON_VERSION_HARD:
-        print(
-            "repo: error: Python 3 version is too old; "
-            "Please upgrade to Python {}.{}+.".format(*MIN_PYTHON_VERSION_SOFT),
-            file=sys.stderr,
-        )
-        sys.exit(1)
-    elif sys.version_info < MIN_PYTHON_VERSION_SOFT:
-        print(
-            "repo: warning: your Python 3 version is no longer supported; "
-            "Please upgrade to Python {}.{}+.".format(*MIN_PYTHON_VERSION_SOFT),
-            file=sys.stderr,
-        )
+elif sys.version_info < MIN_PYTHON_VERSION_SOFT:
+    logger.error(
+        "repo: warning: your Python version is no longer supported; "
+        "Please upgrade to Python %d.%d+.",
+        *MIN_PYTHON_VERSION_SOFT,
+    )
 
+KEYBOARD_INTERRUPT_EXIT = 128 + signal.SIGINT
+MAX_PRINT_ERRORS = 5
 
 global_options = optparse.OptionParser(
     usage="repo [-p|--paginate|--no-pager] COMMAND [ARGS]",
@@ -179,7 +185,7 @@ global_options.add_option(
 )
 
 
-class _Repo(object):
+class _Repo:
     def __init__(self, repodir):
         self.repodir = repodir
         self.commands = all_commands
@@ -191,9 +197,8 @@ class _Repo(object):
         if short:
             commands = " ".join(sorted(self.commands))
             wrapped_commands = textwrap.wrap(commands, width=77)
-            print(
-                "Available commands:\n  %s" % ("\n  ".join(wrapped_commands),)
-            )
+            help_commands = "".join(f"\n  {x}" for x in wrapped_commands)
+            print(f"Available commands:{help_commands}")
             print("\nRun `repo help <command>` for command-specific details.")
             print("Bug reports:", Wrapper().BUG_URL)
         else:
@@ -229,7 +234,7 @@ class _Repo(object):
         if name in self.commands:
             return name, []
 
-        key = "alias.%s" % (name,)
+        key = f"alias.{name}"
         alias = RepoConfig.ForRepository(self.repodir).GetString(key)
         if alias is None:
             alias = RepoConfig.ForUser().GetString(key)
@@ -263,10 +268,14 @@ class _Repo(object):
             self._PrintHelp(short=True)
             return 1
 
-        run = lambda: self._RunLong(name, gopts, argv) or 0
+        git_trace2_event_log = EventLog()
+        run = (
+            lambda: self._RunLong(name, gopts, argv, git_trace2_event_log) or 0
+        )
         with Trace(
-            "starting new command: %s",
+            "starting new command: %s [sid=%s]",
             ", ".join([name] + argv),
+            git_trace2_event_log.full_sid,
             first_trace=True,
         ):
             if gopts.trace_python:
@@ -283,12 +292,11 @@ class _Repo(object):
                 result = run()
         return result
 
-    def _RunLong(self, name, gopts, argv):
+    def _RunLong(self, name, gopts, argv, git_trace2_event_log):
         """Execute the (longer running) requested subcommand."""
         result = 0
         SetDefaultColoring(gopts.color)
 
-        git_trace2_event_log = EventLog()
         outer_client = RepoClient(self.repodir)
         repo_client = outer_client
         if gopts.submanifest_path:
@@ -297,11 +305,6 @@ class _Repo(object):
                 submanifest_path=gopts.submanifest_path,
                 outer_client=outer_client,
             )
-        gitc_manifest = None
-        gitc_client_name = gitc_utils.parse_clientdir(os.getcwd())
-        if gitc_client_name:
-            gitc_manifest = GitcClient(self.repodir, gitc_client_name)
-            repo_client.isGitcClient = True
 
         try:
             cmd = self.commands[name](
@@ -310,50 +313,27 @@ class _Repo(object):
                 manifest=repo_client.manifest,
                 outer_client=outer_client,
                 outer_manifest=outer_client.manifest,
-                gitc_manifest=gitc_manifest,
                 git_event_log=git_trace2_event_log,
             )
         except KeyError:
-            print(
-                "repo: '%s' is not a repo command.  See 'repo help'." % name,
-                file=sys.stderr,
+            logger.error(
+                "repo: '%s' is not a repo command.  See 'repo help'.", name
             )
             return 1
 
         Editor.globalConfig = cmd.client.globalConfig
 
         if not isinstance(cmd, MirrorSafeCommand) and cmd.manifest.IsMirror:
-            print(
-                "fatal: '%s' requires a working directory" % name,
-                file=sys.stderr,
-            )
-            return 1
-
-        if (
-            isinstance(cmd, GitcAvailableCommand)
-            and not gitc_utils.get_gitc_manifest_dir()
-        ):
-            print(
-                "fatal: '%s' requires GITC to be available" % name,
-                file=sys.stderr,
-            )
-            return 1
-
-        if isinstance(cmd, GitcClientCommand) and not gitc_client_name:
-            print("fatal: '%s' requires a GITC client" % name, file=sys.stderr)
+            logger.error("fatal: '%s' requires a working directory", name)
             return 1
 
         try:
             copts, cargs = cmd.OptionParser.parse_args(argv)
             copts = cmd.ReadEnvironmentOptions(copts)
         except NoManifestException as e:
-            print(
-                "error: in `%s`: %s" % (" ".join([name] + argv), str(e)),
-                file=sys.stderr,
-            )
-            print(
-                "error: manifest missing or unreadable -- please run init",
-                file=sys.stderr,
+            logger.error("error: in `%s`: %s", " ".join([name] + argv), e)
+            logger.error(
+                "error: manifest missing or unreadable -- please run init"
             )
             return 1
 
@@ -371,10 +351,14 @@ class _Repo(object):
         start = time.time()
         cmd_event = cmd.event_log.Add(name, event_log.TASK_COMMAND, start)
         cmd.event_log.SetParent(cmd_event)
-        git_trace2_event_log.StartEvent()
+        git_trace2_event_log.StartEvent(["repo", name] + argv)
         git_trace2_event_log.CommandEvent(name="repo", subcommands=[name])
 
-        try:
+        def execute_command_helper():
+            """
+            Execute the subcommand.
+            """
+            nonlocal result
             cmd.CommonValidateOptions(copts, cargs)
             cmd.ValidateOptions(copts, cargs)
 
@@ -409,45 +393,90 @@ class _Repo(object):
                     if hasattr(copts, "manifest_branch"):
                         child_argv.extend(["--manifest-branch", spec.revision])
                     result = self._Run(name, gopts, child_argv) or result
+
+        def execute_command():
+            """
+            Execute the command and log uncaught exceptions.
+            """
+            try:
+                execute_command_helper()
+            except (
+                KeyboardInterrupt,
+                SystemExit,
+                Exception,
+                RepoExitError,
+            ) as e:
+                ok = isinstance(e, SystemExit) and not e.code
+                exception_name = type(e).__name__
+                if isinstance(e, RepoUnhandledExceptionError):
+                    exception_name = type(e.error).__name__
+                if isinstance(e, RepoExitError):
+                    aggregated_errors = e.aggregate_errors or []
+                    for error in aggregated_errors:
+                        project = None
+                        if isinstance(error, RepoError):
+                            project = error.project
+                        error_info = json.dumps(
+                            {
+                                "ErrorType": type(error).__name__,
+                                "Project": str(project),
+                                "Message": str(error),
+                            }
+                        )
+                        git_trace2_event_log.ErrorEvent(
+                            f"AggregateExitError:{error_info}"
+                        )
+                if not ok:
+                    git_trace2_event_log.ErrorEvent(
+                        f"RepoExitError:{exception_name}"
+                    )
+                raise
+
+        try:
+            execute_command()
         except (
             DownloadError,
             ManifestInvalidRevisionError,
+            ManifestParseError,
             NoManifestException,
         ) as e:
-            print(
-                "error: in `%s`: %s" % (" ".join([name] + argv), str(e)),
-                file=sys.stderr,
-            )
+            logger.error("error: in `%s`: %s", " ".join([name] + argv), e)
             if isinstance(e, NoManifestException):
-                print(
-                    "error: manifest missing or unreadable -- please run init",
-                    file=sys.stderr,
+                logger.error(
+                    "error: manifest missing or unreadable -- please run init"
                 )
-            result = 1
+            result = e.exit_code
         except NoSuchProjectError as e:
             if e.name:
-                print("error: project %s not found" % e.name, file=sys.stderr)
+                logger.error("error: project %s not found", e.name)
             else:
-                print("error: no project in current directory", file=sys.stderr)
-            result = 1
+                logger.error("error: no project in current directory")
+            result = e.exit_code
         except InvalidProjectGroupsError as e:
             if e.name:
-                print(
-                    "error: project group must be enabled for project %s"
-                    % e.name,
-                    file=sys.stderr,
+                logger.error(
+                    "error: project group must be enabled for project %s",
+                    e.name,
                 )
             else:
-                print(
+                logger.error(
                     "error: project group must be enabled for the project in "
-                    "the current directory",
-                    file=sys.stderr,
+                    "the current directory"
                 )
-            result = 1
+            result = e.exit_code
         except SystemExit as e:
             if e.code:
                 result = e.code
             raise
+        except KeyboardInterrupt:
+            result = KEYBOARD_INTERRUPT_EXIT
+            raise
+        except RepoExitError as e:
+            result = e.exit_code
+            raise
+        except Exception:
+            result = 1
+            raise
         finally:
             finish = time.time()
             elapsed = finish - start
@@ -499,7 +528,7 @@ def _CheckWrapperVersion(ver_str, repo_path):
         repo_path = "~/bin/repo"
 
     if not ver_str:
-        print("no --wrapper-version argument", file=sys.stderr)
+        logger.error("no --wrapper-version argument")
         sys.exit(1)
 
     # Pull out the version of the repo launcher we know about to compare.
@@ -508,7 +537,7 @@ def _CheckWrapperVersion(ver_str, repo_path):
 
     exp_str = ".".join(map(str, exp))
     if ver < MIN_REPO_VERSION:
-        print(
+        logger.error(
             """
 repo: error:
 !!! Your version of repo %s is too old.
@@ -517,42 +546,44 @@ repo: error:
 !!! You must upgrade before you can continue:
 
     cp %s %s
-"""
-            % (ver_str, min_str, exp_str, WrapperPath(), repo_path),
-            file=sys.stderr,
+""",
+            ver_str,
+            min_str,
+            exp_str,
+            WrapperPath(),
+            repo_path,
         )
         sys.exit(1)
 
     if exp > ver:
-        print(
-            "\n... A new version of repo (%s) is available." % (exp_str,),
-            file=sys.stderr,
+        logger.warning(
+            "\n... A new version of repo (%s) is available.", exp_str
         )
         if os.access(repo_path, os.W_OK):
-            print(
+            logger.warning(
                 """\
 ... You should upgrade soon:
     cp %s %s
-"""
-                % (WrapperPath(), repo_path),
-                file=sys.stderr,
+""",
+                WrapperPath(),
+                repo_path,
             )
         else:
-            print(
+            logger.warning(
                 """\
 ... New version is available at: %s
 ... The launcher is run from: %s
 !!! The launcher is not writable.  Please talk to your sysadmin or distro
 !!! to get an update installed.
-"""
-                % (WrapperPath(), repo_path),
-                file=sys.stderr,
+""",
+                WrapperPath(),
+                repo_path,
             )
 
 
 def _CheckRepoDir(repo_dir):
     if not repo_dir:
-        print("no --repo-dir argument", file=sys.stderr)
+        logger.error("no --repo-dir argument")
         sys.exit(1)
 
 
@@ -756,7 +787,7 @@ def init_http():
             mgr.add_password(p[1], "https://%s/" % host, p[0], p[2])
     except netrc.NetrcParseError:
         pass
-    except IOError:
+    except OSError:
         pass
     handlers.append(_BasicAuthHandler(mgr))
     handlers.append(_DigestAuthHandler(mgr))
@@ -811,19 +842,19 @@ def _Main(argv):
             SetTraceToStderr()
 
         result = repo._Run(name, gopts, argv) or 0
+    except RepoExitError as e:
+        if not isinstance(e, SilentRepoExitError):
+            logger.log_aggregated_errors(e)
+        result = e.exit_code
     except KeyboardInterrupt:
         print("aborted by user", file=sys.stderr)
-        result = 1
-    except ManifestParseError as mpe:
-        print("fatal: %s" % mpe, file=sys.stderr)
-        result = 1
+        result = KEYBOARD_INTERRUPT_EXIT
     except RepoChangedException as rce:
         # If repo changed, re-exec ourselves.
-        #
         argv = list(sys.argv)
         argv.extend(rce.extra_args)
         try:
-            os.execv(sys.executable, [__file__] + argv)
+            os.execv(sys.executable, [sys.executable, __file__] + argv)
         except OSError as e:
             print("fatal: cannot restart repo after upgrade", file=sys.stderr)
             print("fatal: %s" % e, file=sys.stderr)

man/repo-gc.1

@@ -1,21 +1,28 @@
 .\" DO NOT MODIFY THIS FILE!  It was generated by help2man.
-.TH REPO "1" "July 2022" "repo gitc-delete" "Repo Manual"
+.TH REPO "1" "April 2025" "repo gc" "Repo Manual"
 .SH NAME
-repo \- repo gitc-delete - manual page for repo gitc-delete
+repo \- repo gc - manual page for repo gc
 .SH SYNOPSIS
 .B repo
-\fI\,gitc-delete\/\fR
+\fI\,gc\/\fR
 .SH DESCRIPTION
 Summary
 .PP
-Delete a GITC Client.
+Cleaning up internal repo and Git state.
 .SH OPTIONS
 .TP
 \fB\-h\fR, \fB\-\-help\fR
 show this help message and exit
 .TP
-\fB\-f\fR, \fB\-\-force\fR
-force the deletion (no prompt)
+\fB\-n\fR, \fB\-\-dry\-run\fR
+do everything except actually delete
+.TP
+\fB\-y\fR, \fB\-\-yes\fR
+answer yes to all safe prompts
+.TP
+\fB\-\-repack\fR
+repack all projects that use partial clone with
+filter=blob:none
 .SS Logging options:
 .TP
 \fB\-v\fR, \fB\-\-verbose\fR
@@ -37,8 +44,4 @@ only operate on this (sub)manifest
 \fB\-\-no\-this\-manifest\-only\fR, \fB\-\-all\-manifests\fR
 operate on this manifest and its submanifests
 .PP
-Run `repo help gitc\-delete` to view the detailed manual.
-.SH DETAILS
-.PP
-This subcommand deletes the current GITC client, deleting the GITC manifest and
-all locally downloaded sources.
+Run `repo help gc` to view the detailed manual.

man/repo-gitc-init.1

@@ -1,175 +0,0 @@
-.\" DO NOT MODIFY THIS FILE!  It was generated by help2man.
-.TH REPO "1" "October 2022" "repo gitc-init" "Repo Manual"
-.SH NAME
-repo \- repo gitc-init - manual page for repo gitc-init
-.SH SYNOPSIS
-.B repo
-\fI\,gitc-init \/\fR[\fI\,options\/\fR] [\fI\,client name\/\fR]
-.SH DESCRIPTION
-Summary
-.PP
-Initialize a GITC Client.
-.SH OPTIONS
-.TP
-\fB\-h\fR, \fB\-\-help\fR
-show this help message and exit
-.SS Logging options:
-.TP
-\fB\-v\fR, \fB\-\-verbose\fR
-show all output
-.TP
-\fB\-q\fR, \fB\-\-quiet\fR
-only show errors
-.SS Manifest options:
-.TP
-\fB\-u\fR URL, \fB\-\-manifest\-url\fR=\fI\,URL\/\fR
-manifest repository location
-.TP
-\fB\-b\fR REVISION, \fB\-\-manifest\-branch\fR=\fI\,REVISION\/\fR
-manifest branch or revision (use HEAD for default)
-.TP
-\fB\-m\fR NAME.xml, \fB\-\-manifest\-name\fR=\fI\,NAME\/\fR.xml
-initial manifest file
-.TP
-\fB\-g\fR GROUP, \fB\-\-groups\fR=\fI\,GROUP\/\fR
-restrict manifest projects to ones with specified
-group(s) [default|all|G1,G2,G3|G4,\-G5,\-G6]
-.TP
-\fB\-p\fR PLATFORM, \fB\-\-platform\fR=\fI\,PLATFORM\/\fR
-restrict manifest projects to ones with a specified
-platform group [auto|all|none|linux|darwin|...]
-.TP
-\fB\-\-submodules\fR
-sync any submodules associated with the manifest repo
-.TP
-\fB\-\-standalone\-manifest\fR
-download the manifest as a static file rather then
-create a git checkout of the manifest repo
-.TP
-\fB\-\-manifest\-depth\fR=\fI\,DEPTH\/\fR
-create a shallow clone of the manifest repo with given
-depth (0 for full clone); see git clone (default: 0)
-.SS Manifest (only) checkout options:
-.TP
-\fB\-\-current\-branch\fR
-fetch only current manifest branch from server
-(default)
-.TP
-\fB\-\-no\-current\-branch\fR
-fetch all manifest branches from server
-.TP
-\fB\-\-tags\fR
-fetch tags in the manifest
-.TP
-\fB\-\-no\-tags\fR
-don't fetch tags in the manifest
-.SS Checkout modes:
-.TP
-\fB\-\-mirror\fR
-create a replica of the remote repositories rather
-than a client working directory
-.TP
-\fB\-\-archive\fR
-checkout an archive instead of a git repository for
-each project. See git archive.
-.TP
-\fB\-\-worktree\fR
-use git\-worktree to manage projects
-.SS Project checkout optimizations:
-.TP
-\fB\-\-reference\fR=\fI\,DIR\/\fR
-location of mirror directory
-.TP
-\fB\-\-dissociate\fR
-dissociate from reference mirrors after clone
-.TP
-\fB\-\-depth\fR=\fI\,DEPTH\/\fR
-create a shallow clone with given depth; see git clone
-.TP
-\fB\-\-partial\-clone\fR
-perform partial clone (https://gitscm.com/docs/gitrepositorylayout#_code_partialclone_code)
-.TP
-\fB\-\-no\-partial\-clone\fR
-disable use of partial clone (https://gitscm.com/docs/gitrepositorylayout#_code_partialclone_code)
-.TP
-\fB\-\-partial\-clone\-exclude\fR=\fI\,PARTIAL_CLONE_EXCLUDE\/\fR
-exclude the specified projects (a comma\-delimited
-project names) from partial clone (https://gitscm.com/docs/gitrepositorylayout#_code_partialclone_code)
-.TP
-\fB\-\-clone\-filter\fR=\fI\,CLONE_FILTER\/\fR
-filter for use with \fB\-\-partial\-clone\fR [default:
-blob:none]
-.TP
-\fB\-\-use\-superproject\fR
-use the manifest superproject to sync projects;
-implies \fB\-c\fR
-.TP
-\fB\-\-no\-use\-superproject\fR
-disable use of manifest superprojects
-.TP
-\fB\-\-clone\-bundle\fR
-enable use of \fI\,/clone.bundle\/\fP on HTTP/HTTPS (default if
-not \fB\-\-partial\-clone\fR)
-.TP
-\fB\-\-no\-clone\-bundle\fR
-disable use of \fI\,/clone.bundle\/\fP on HTTP/HTTPS (default if
-\fB\-\-partial\-clone\fR)
-.TP
-\fB\-\-git\-lfs\fR
-enable Git LFS support
-.TP
-\fB\-\-no\-git\-lfs\fR
-disable Git LFS support
-.SS repo Version options:
-.TP
-\fB\-\-repo\-url\fR=\fI\,URL\/\fR
-repo repository location ($REPO_URL)
-.TP
-\fB\-\-repo\-rev\fR=\fI\,REV\/\fR
-repo branch or revision ($REPO_REV)
-.TP
-\fB\-\-no\-repo\-verify\fR
-do not verify repo source code
-.SS Other options:
-.TP
-\fB\-\-config\-name\fR
-Always prompt for name/e\-mail
-.SS GITC options:
-.TP
-\fB\-f\fR MANIFEST_FILE, \fB\-\-manifest\-file\fR=\fI\,MANIFEST_FILE\/\fR
-Optional manifest file to use for this GITC client.
-.TP
-\fB\-c\fR GITC_CLIENT, \fB\-\-gitc\-client\fR=\fI\,GITC_CLIENT\/\fR
-Name of the gitc_client instance to create or modify.
-.SS Multi\-manifest:
-.TP
-\fB\-\-outer\-manifest\fR
-operate starting at the outermost manifest
-.TP
-\fB\-\-no\-outer\-manifest\fR
-do not operate on outer manifests
-.TP
-\fB\-\-this\-manifest\-only\fR
-only operate on this (sub)manifest
-.TP
-\fB\-\-no\-this\-manifest\-only\fR, \fB\-\-all\-manifests\fR
-operate on this manifest and its submanifests
-.PP
-Run `repo help gitc\-init` to view the detailed manual.
-.SH DETAILS
-.PP
-The 'repo gitc\-init' command is ran to initialize a new GITC client for use with
-the GITC file system.
-.PP
-This command will setup the client directory, initialize repo, just like repo
-init does, and then downloads the manifest collection and installs it in the
-\&.repo/directory of the GITC client.
-.PP
-Once this is done, a GITC manifest is generated by pulling the HEAD SHA for each
-project and generates the properly formatted XML file and installs it as
-\&.manifest in the GITC client directory.
-.PP
-The \fB\-c\fR argument is required to specify the GITC client name.
-.PP
-The optional \fB\-f\fR argument can be used to specify the manifest file to use for
-this GITC client.

man/repo-init.1

@@ -1,5 +1,5 @@
 .\" DO NOT MODIFY THIS FILE!  It was generated by help2man.
-.TH REPO "1" "October 2022" "repo init" "Repo Manual"
+.TH REPO "1" "September 2024" "repo init" "Repo Manual"
 .SH NAME
 repo \- repo init - manual page for repo init
 .SH SYNOPSIS
@@ -28,6 +28,11 @@ manifest repository location
 \fB\-b\fR REVISION, \fB\-\-manifest\-branch\fR=\fI\,REVISION\/\fR
 manifest branch or revision (use HEAD for default)
 .TP
+\fB\-\-manifest\-upstream\-branch\fR=\fI\,BRANCH\/\fR
+when a commit is provided to \fB\-\-manifest\-branch\fR, this
+is the name of the git ref in which the commit can be
+found
+.TP
 \fB\-m\fR NAME.xml, \fB\-\-manifest\-name\fR=\fI\,NAME\/\fR.xml
 initial manifest file
 .TP
@@ -163,6 +168,10 @@ The optional \fB\-b\fR argument can be used to select the manifest branch to che
 and use. If no branch is specified, the remote's default branch is used. This is
 equivalent to using \fB\-b\fR HEAD.
 .PP
+The optional \fB\-\-manifest\-upstream\-branch\fR argument can be used when a commit is
+provided to \fB\-\-manifest\-branch\fR (or \fB\-b\fR), to specify the name of the git ref in
+which the commit can be found.
+.PP
 The optional \fB\-m\fR argument can be used to specify an alternate manifest to be
 used. If no manifest is specified, the manifest default.xml will be used.
 .PP

man/repo-manifest.1

@@ -1,5 +1,5 @@
 .\" DO NOT MODIFY THIS FILE!  It was generated by help2man.
-.TH REPO "1" "October 2022" "repo manifest" "Repo Manual"
+.TH REPO "1" "December 2025" "repo manifest" "Repo Manual"
 .SH NAME
 repo \- repo manifest - manual page for repo manifest
 .SH SYNOPSIS
@@ -30,8 +30,8 @@ if in \fB\-r\fR mode, do not write the dest\-branch field
 (only of use if the branch names for a sha1 manifest
 are sensitive)
 .TP
-\fB\-\-json\fR
-output manifest in JSON format (experimental)
+\fB\-\-format\fR=\fI\,FORMAT\/\fR
+output format: xml, json (default: xml)
 .TP
 \fB\-\-pretty\fR
 format output for humans to read
@@ -78,6 +78,10 @@ set to the ref we were on when the manifest was generated. The 'dest\-branch'
 attribute is set to indicate the remote ref to push changes to via 'repo
 upload'.
 .PP
+Multiple output formats are supported via \fB\-\-format\fR. The default output is XML,
+and formats are generally "condensed". Use \fB\-\-pretty\fR for more human\-readable
+variations.
+.PP
 repo Manifest Format
 .PP
 A repo manifest describes the structure of a repo client; that is the
@@ -135,7 +139,7 @@ include*)>
 <!ATTLIST manifest\-server url CDATA #REQUIRED>
 .IP
 <!ELEMENT submanifest EMPTY>
-<!ATTLIST submanifest name           ID #REQUIRED>
+<!ATTLIST submanifest name           ID    #REQUIRED>
 <!ATTLIST submanifest remote         IDREF #IMPLIED>
 <!ATTLIST submanifest project        CDATA #IMPLIED>
 <!ATTLIST submanifest manifest\-name  CDATA #IMPLIED>
@@ -166,9 +170,9 @@ CDATA #IMPLIED>
 <!ATTLIST project sync\-c      CDATA #IMPLIED>
 <!ATTLIST project sync\-s      CDATA #IMPLIED>
 <!ATTLIST project sync\-tags   CDATA #IMPLIED>
-<!ATTLIST project upstream CDATA #IMPLIED>
+<!ATTLIST project upstream    CDATA #IMPLIED>
 <!ATTLIST project clone\-depth CDATA #IMPLIED>
-<!ATTLIST project force\-path CDATA #IMPLIED>
+<!ATTLIST project force\-path  CDATA #IMPLIED>
 .IP
 <!ELEMENT annotation EMPTY>
 <!ATTLIST annotation name  CDATA #REQUIRED>
@@ -180,25 +184,43 @@ CDATA #IMPLIED>
 <!ATTLIST copyfile dest CDATA #REQUIRED>
 .IP
 <!ELEMENT linkfile EMPTY>
-<!ATTLIST linkfile src CDATA #REQUIRED>
+<!ATTLIST linkfile src  CDATA #REQUIRED>
 <!ATTLIST linkfile dest CDATA #REQUIRED>
+.TP
+<!ELEMENT extend\-project (annotation*,
+copyfile*,
+linkfile*)>
+.TP
+<!ATTLIST extend\-project name
+CDATA #REQUIRED>
+.TP
+<!ATTLIST extend\-project path
+CDATA #IMPLIED>
+.TP
+<!ATTLIST extend\-project dest\-path
+CDATA #IMPLIED>
+.TP
+<!ATTLIST extend\-project groups
+CDATA #IMPLIED>
+.TP
+<!ATTLIST extend\-project revision
+CDATA #IMPLIED>
+.TP
+<!ATTLIST extend\-project remote
+CDATA #IMPLIED>
 .IP
-<!ELEMENT extend\-project EMPTY>
-<!ATTLIST extend\-project name CDATA #REQUIRED>
-<!ATTLIST extend\-project path CDATA #IMPLIED>
-<!ATTLIST extend\-project dest\-path CDATA #IMPLIED>
-<!ATTLIST extend\-project groups CDATA #IMPLIED>
-<!ATTLIST extend\-project revision CDATA #IMPLIED>
-<!ATTLIST extend\-project remote CDATA #IMPLIED>
 <!ATTLIST extend\-project dest\-branch CDATA #IMPLIED>
-<!ATTLIST extend\-project upstream CDATA #IMPLIED>
+<!ATTLIST extend\-project upstream    CDATA #IMPLIED>
+<!ATTLIST extend\-project base\-rev    CDATA #IMPLIED>
 .IP
 <!ELEMENT remove\-project EMPTY>
-<!ATTLIST remove\-project name  CDATA #REQUIRED>
-<!ATTLIST remove\-project optional  CDATA #IMPLIED>
+<!ATTLIST remove\-project name     CDATA #IMPLIED>
+<!ATTLIST remove\-project path     CDATA #IMPLIED>
+<!ATTLIST remove\-project optional CDATA #IMPLIED>
+<!ATTLIST remove\-project base\-rev CDATA #IMPLIED>
 .IP
 <!ELEMENT repo\-hooks EMPTY>
-<!ATTLIST repo\-hooks in\-project CDATA #REQUIRED>
+<!ATTLIST repo\-hooks in\-project   CDATA #REQUIRED>
 <!ATTLIST repo\-hooks enabled\-list CDATA #REQUIRED>
 .IP
 <!ELEMENT superproject EMPTY>
@@ -207,11 +229,12 @@ CDATA #IMPLIED>
 <!ATTLIST superproject revision CDATA #IMPLIED>
 .IP
 <!ELEMENT contactinfo EMPTY>
-<!ATTLIST contactinfo bugurl  CDATA #REQUIRED>
+<!ATTLIST contactinfo bugurl CDATA #REQUIRED>
 .IP
 <!ELEMENT include EMPTY>
-<!ATTLIST include name   CDATA #REQUIRED>
-<!ATTLIST include groups CDATA #IMPLIED>
+<!ATTLIST include name     CDATA #REQUIRED>
+<!ATTLIST include groups   CDATA #IMPLIED>
+<!ATTLIST include revision CDATA #IMPLIED>
 .PP
 ]>
 ```
@@ -302,25 +325,7 @@ Element manifest\-server
 At most one manifest\-server may be specified. The url attribute is used to
 specify the URL of a manifest server, which is an XML RPC service.
 .PP
-The manifest server should implement the following RPC methods:
-.IP
-GetApprovedManifest(branch, target)
-.PP
-Return a manifest in which each project is pegged to a known good revision for
-the current branch and target. This is used by repo sync when the \fB\-\-smart\-sync\fR
-option is given.
-.PP
-The target to use is defined by environment variables TARGET_PRODUCT and
-TARGET_BUILD_VARIANT. These variables are used to create a string of the form
-$TARGET_PRODUCT\-$TARGET_BUILD_VARIANT, e.g. passion\-userdebug. If one of those
-variables or both are not present, the program will call GetApprovedManifest
-without the target parameter and the manifest server should choose a reasonable
-default target.
-.IP
-GetManifest(tag)
-.PP
-Return a manifest in which each project is pegged to the revision at the
-specified tag. This is used by repo sync when the \fB\-\-smart\-tag\fR option is given.
+See the [smart sync documentation](./smart\-sync.md) for more details.
 .PP
 Element submanifest
 .PP
@@ -372,7 +377,7 @@ supplied, `revision` is used.
 .PP
 `path` may not be an absolute path or use "." or ".." path components.
 .PP
-Attribute `groups`: List of additional groups to which all projects in the
+Attribute `groups`: Set of additional groups to which all projects in the
 included submanifest belong. This appends and recurses, meaning all projects in
 submanifests carry all parent submanifest groups. Same syntax as the
 corresponding element of `project`.
@@ -434,7 +439,7 @@ Attribute `dest\-branch`: Name of a Git branch (e.g. `main`). When using `repo
 upload`, changes will be submitted for code review on this branch. If
 unspecified both here and in the default element, `revision` is used instead.
 .PP
-Attribute `groups`: List of groups to which this project belongs, whitespace or
+Attribute `groups`: Set of groups to which this project belongs, whitespace or
 comma separated. All projects belong to the group "all", and each project
 automatically belongs to a group of its name:`name` and path:`path`. E.g. for
 `<project name="monkeys" path="barrel\-of"/>`, that project definition is
@@ -478,8 +483,8 @@ repo client where the Git working directory for this project should be placed.
 This is used to move a project in the checkout by overriding the existing `path`
 setting.
 .PP
-Attribute `groups`: List of additional groups to which this project belongs.
-Same syntax as the corresponding element of `project`.
+Attribute `groups`: Set of additional groups to which this project belongs. Same
+syntax as the corresponding element of `project`.
 .PP
 Attribute `revision`: If specified, overrides the revision of the original
 project. Same syntax as the corresponding element of `project`.
@@ -493,21 +498,31 @@ project. Same syntax as the corresponding element of `project`.
 Attribute `upstream`: If specified, overrides the upstream of the original
 project. Same syntax as the corresponding element of `project`.
 .PP
+Attribute `base\-rev`: If specified, adds a check against the revision to be
+extended. Manifest parse will fail and give a list of mismatch extends if the
+revisions being extended have changed since base\-rev was set. Intended for use
+with layered manifests using hash revisions to prevent patch branches hiding
+newer upstream revisions. Also compares named refs like branches or tags but is
+misleading if branches are used as base\-rev. Same syntax as the corresponding
+element of `project`.
+.PP
 Element annotation
 .PP
-Zero or more annotation elements may be specified as children of a project or
-remote element. Each element describes a name\-value pair. For projects, this
-name\-value pair will be exported into each project's environment during a
-\&'forall' command, prefixed with `REPO__`. In addition, there is an optional
-attribute "keep" which accepts the case insensitive values "true" (default) or
-"false". This attribute determines whether or not the annotation will be kept
-when exported with the manifest subcommand.
+Zero or more annotation elements may be specified as children of a project
+element, an extend\-project element, or a remote element. Each element describes
+a name\-value pair. For projects, this name\-value pair will be exported into each
+project's environment during a 'forall' command, prefixed with `REPO__`. In
+addition, there is an optional attribute "keep" which accepts the case
+insensitive values "true" (default) or "false". This attribute determines
+whether or not the annotation will be kept when exported with the manifest
+subcommand.
 .PP
 Element copyfile
 .PP
 Zero or more copyfile elements may be specified as children of a project
-element. Each element describes a src\-dest pair of files; the "src" file will be
-copied to the "dest" place during `repo sync` command.
+element, or an extend\-project element. Each element describes a src\-dest pair of
+files; the "src" file will be copied to the "dest" place during `repo sync`
+command.
 .PP
 "src" is project relative, "dest" is relative to the top of the tree. Copying
 from paths outside of the project or to paths outside of the repo client is not
@@ -518,10 +533,14 @@ Intermediate paths must not be symlinks either.
 .PP
 Parent directories of "dest" will be automatically created if missing.
 .PP
+The files are copied in the order they are specified in the manifests. If
+multiple elements specify the same source and destination, they will only be
+applied as one, based on the first occurence. Files are copied before any links
+specified via linkfile elements are created.
+.PP
 Element linkfile
 .PP
-It's just like copyfile and runs at the same time as copyfile but instead of
-copying it creates a symlink.
+It's just like copyfile, but instead of copying it creates a symlink.
 .PP
 The symlink is created at "dest" (relative to the top of the tree) and points to
 the path specified by "src" which is a path in the project.
@@ -531,18 +550,42 @@ Parent directories of "dest" will be automatically created if missing.
 The symlink target may be a file or directory, but it may not point outside of
 the repo client.
 .PP
+The links are created in the order they are specified in the manifests. If
+multiple elements specify the same source and destination, they will only be
+applied as one, based on the first occurence. Links are created after any files
+specified via copyfile elements are copied.
+.PP
 Element remove\-project
 .PP
-Deletes the named project from the internal manifest table, possibly allowing a
+Deletes a project from the internal manifest table, possibly allowing a
 subsequent project element in the same manifest file to replace the project with
 a different source.
 .PP
 This element is mostly useful in a local manifest file, where the user can
 remove a project, and possibly replace it with their own definition.
 .PP
+The project `name` or project `path` can be used to specify the remove target
+meaning one of them is required. If only name is specified, all projects with
+that name are removed.
+.PP
+If both name and path are specified, only projects with the same name and path
+are removed, meaning projects with the same name but in other locations are
+kept.
+.PP
+If only path is specified, a matching project is removed regardless of its name.
+Logic otherwise behaves like both are specified.
+.PP
 Attribute `optional`: Set to true to ignore remove\-project elements with no
 matching `project` element.
 .PP
+Attribute `base\-rev`: If specified, adds a check against the revision to be
+removed. Manifest parse will fail and give a list of mismatch removes if the
+revisions being removed have changed since base\-rev was set. Intended for use
+with layered manifests using hash revisions to prevent patch branches hiding
+newer upstream revisions. Also compares named refs like branches or tags but is
+misleading if branches are used as base\-rev. Same syntax as the corresponding
+element of `project`.
+.PP
 Element repo\-hooks
 .PP
 NB: See the [practical documentation](./repo\-hooks.md) for using repo hooks.
@@ -603,12 +646,18 @@ repository's root.
 "name" may not be an absolute path or use "." or ".." path components. These
 restrictions are not enforced for [Local Manifests].
 .PP
-Attribute `groups`: List of additional groups to which all projects in the
+Attribute `groups`: Set of additional groups to which all projects in the
 included manifest belong. This appends and recurses, meaning all projects in
-included manifests carry all parent include groups. Same syntax as the
+included manifests carry all parent include groups. This also applies to all
+extend\-project elements in the included manifests. Same syntax as the
 corresponding element of `project`.
 .PP
-Local Manifests 
+Attribute `revision`: Name of a Git branch (e.g. `main` or `refs/heads/main`)
+default to which all projects in the included manifest belong. This recurses,
+meaning it will apply to all projects in all manifests included as a result of
+this element.
+.PP
+Local Manifests
 .PP
 Additional remotes and projects may be added through local manifest files stored
 in `$TOP_DIR/.repo/local_manifests/*.xml`.

man/repo-smartsync.1

@@ -1,5 +1,5 @@
 .\" DO NOT MODIFY THIS FILE!  It was generated by help2man.
-.TH REPO "1" "November 2022" "repo smartsync" "Repo Manual"
+.TH REPO "1" "August 2025" "repo smartsync" "Repo Manual"
 .SH NAME
 repo \- repo smartsync - manual page for repo smartsync
 .SH SYNOPSIS
@@ -20,11 +20,11 @@ number of CPU cores)
 .TP
 \fB\-\-jobs\-network\fR=\fI\,JOBS\/\fR
 number of network jobs to run in parallel (defaults to
-\fB\-\-jobs\fR or 1)
+\fB\-\-jobs\fR or 1). Ignored unless \fB\-\-no\-interleaved\fR is set
 .TP
 \fB\-\-jobs\-checkout\fR=\fI\,JOBS\/\fR
 number of local checkout jobs to run in parallel
-(defaults to \fB\-\-jobs\fR or 8)
+(defaults to \fB\-\-jobs\fR or 8). Ignored unless \fB\-\-nointerleaved\fR is set
 .TP
 \fB\-f\fR, \fB\-\-force\-broken\fR
 obsolete option (to be deleted in the future)
@@ -37,11 +37,20 @@ overwrite an existing git directory if it needs to
 point to a different object directory. WARNING: this
 may cause loss of data
 .TP
+\fB\-\-force\-checkout\fR
+force checkout even if it results in throwing away
+uncommitted modifications. WARNING: this may cause
+loss of data
+.TP
 \fB\-\-force\-remove\-dirty\fR
 force remove projects with uncommitted modifications
 if projects no longer exist in the manifest. WARNING:
 this may cause loss of data
 .TP
+\fB\-\-rebase\fR
+rebase local commits regardless of whether they are
+published
+.TP
 \fB\-l\fR, \fB\-\-local\-only\fR
 only update working tree, don't fetch
 .TP
@@ -49,6 +58,12 @@ only update working tree, don't fetch
 use the existing manifest checkout as\-is. (do not
 update to the latest revision)
 .TP
+\fB\-\-interleaved\fR
+fetch and checkout projects in parallel (default)
+.TP
+\fB\-\-no\-interleaved\fR
+fetch and checkout projects in phases
+.TP
 \fB\-n\fR, \fB\-\-network\-only\fR
 fetch only, don't update working tree
 .TP
@@ -136,6 +151,16 @@ operate on this manifest and its submanifests
 .TP
 \fB\-\-no\-repo\-verify\fR
 do not verify repo source code
+.SS post\-sync hooks:
+.TP
+\fB\-\-no\-verify\fR
+Do not run the post\-sync hook.
+.TP
+\fB\-\-verify\fR
+Run the post\-sync hook without prompting.
+.TP
+\fB\-\-ignore\-hooks\fR
+Do not abort if post\-sync hooks fail.
 .PP
 Run `repo help smartsync` to view the detailed manual.
 .SH DETAILS

man/repo-sync.1

@@ -1,5 +1,5 @@
 .\" DO NOT MODIFY THIS FILE!  It was generated by help2man.
-.TH REPO "1" "November 2022" "repo sync" "Repo Manual"
+.TH REPO "1" "August 2025" "repo sync" "Repo Manual"
 .SH NAME
 repo \- repo sync - manual page for repo sync
 .SH SYNOPSIS
@@ -20,11 +20,11 @@ number of CPU cores)
 .TP
 \fB\-\-jobs\-network\fR=\fI\,JOBS\/\fR
 number of network jobs to run in parallel (defaults to
-\fB\-\-jobs\fR or 1)
+\fB\-\-jobs\fR or 1). Ignored unless \fB\-\-no\-interleaved\fR is set
 .TP
 \fB\-\-jobs\-checkout\fR=\fI\,JOBS\/\fR
 number of local checkout jobs to run in parallel
-(defaults to \fB\-\-jobs\fR or 8)
+(defaults to \fB\-\-jobs\fR or 8). Ignored unless \fB\-\-nointerleaved\fR is set
 .TP
 \fB\-f\fR, \fB\-\-force\-broken\fR
 obsolete option (to be deleted in the future)
@@ -37,11 +37,20 @@ overwrite an existing git directory if it needs to
 point to a different object directory. WARNING: this
 may cause loss of data
 .TP
+\fB\-\-force\-checkout\fR
+force checkout even if it results in throwing away
+uncommitted modifications. WARNING: this may cause
+loss of data
+.TP
 \fB\-\-force\-remove\-dirty\fR
 force remove projects with uncommitted modifications
 if projects no longer exist in the manifest. WARNING:
 this may cause loss of data
 .TP
+\fB\-\-rebase\fR
+rebase local commits regardless of whether they are
+published
+.TP
 \fB\-l\fR, \fB\-\-local\-only\fR
 only update working tree, don't fetch
 .TP
@@ -49,6 +58,12 @@ only update working tree, don't fetch
 use the existing manifest checkout as\-is. (do not
 update to the latest revision)
 .TP
+\fB\-\-interleaved\fR
+fetch and checkout projects in parallel (default)
+.TP
+\fB\-\-no\-interleaved\fR
+fetch and checkout projects in phases
+.TP
 \fB\-n\fR, \fB\-\-network\-only\fR
 fetch only, don't update working tree
 .TP
@@ -143,6 +158,16 @@ operate on this manifest and its submanifests
 .TP
 \fB\-\-no\-repo\-verify\fR
 do not verify repo source code
+.SS post\-sync hooks:
+.TP
+\fB\-\-no\-verify\fR
+Do not run the post\-sync hook.
+.TP
+\fB\-\-verify\fR
+Run the post\-sync hook without prompting.
+.TP
+\fB\-\-ignore\-hooks\fR
+Do not abort if post\-sync hooks fail.
 .PP
 Run `repo help sync` to view the detailed manual.
 .SH DETAILS
@@ -185,6 +210,11 @@ The \fB\-\-force\-sync\fR option can be used to overwrite existing git directori
 they have previously been linked to a different object directory. WARNING: This
 may cause data to be lost since refs may be removed when overwriting.
 .PP
+The \fB\-\-force\-checkout\fR option can be used to force git to switch revs even if the
+index or the working tree differs from HEAD, and if there are untracked files.
+WARNING: This may cause data to be lost since uncommitted changes may be
+removed.
+.PP
 The \fB\-\-force\-remove\-dirty\fR option can be used to remove previously used projects
 with uncommitted changes. WARNING: This may cause data to be lost since
 uncommitted changes may be removed with projects that no longer exist in the

man/repo-upload.1

@@ -1,5 +1,5 @@
 .\" DO NOT MODIFY THIS FILE!  It was generated by help2man.
-.TH REPO "1" "August 2022" "repo upload" "Repo Manual"
+.TH REPO "1" "June 2024" "repo upload" "Repo Manual"
 .SH NAME
 repo \- repo upload - manual page for repo upload
 .SH SYNOPSIS
@@ -18,8 +18,11 @@ show this help message and exit
 number of jobs to run in parallel (default: based on
 number of CPU cores)
 .TP
-\fB\-t\fR
-send local branch name to Gerrit Code Review
+\fB\-t\fR, \fB\-\-topic\-branch\fR
+set the topic to the local branch name
+.TP
+\fB\-\-topic\fR=\fI\,TOPIC\/\fR
+set topic for the change
 .TP
 \fB\-\-hashtag\fR=\fI\,HASHTAGS\/\fR, \fB\-\-ht\fR=\fI\,HASHTAGS\/\fR
 add hashtags (comma delimited) to the review
@@ -30,6 +33,9 @@ add local branch name as a hashtag
 \fB\-l\fR LABELS, \fB\-\-label\fR=\fI\,LABELS\/\fR
 add a label when uploading
 .TP
+\fB\-\-pd\fR=\fI\,PATCHSET_DESCRIPTION\/\fR, \fB\-\-patchset\-description\fR=\fI\,PATCHSET_DESCRIPTION\/\fR
+description for patchset
+.TP
 \fB\-\-re\fR=\fI\,REVIEWERS\/\fR, \fB\-\-reviewers\fR=\fI\,REVIEWERS\/\fR
 request reviews from these people
 .TP
@@ -198,6 +204,12 @@ review.URL.uploadnotify:
 Control e\-mail notifications when uploading.
 https://gerrit\-review.googlesource.com/Documentation/user\-upload.html#notify
 .PP
+review.URL.uploadwarningthreshold:
+.PP
+Repo will warn you if you are attempting to upload a large number of commits in
+one or more branches. By default, the threshold is five commits. This option
+allows you to override the warning threshold to a different value.
+.PP
 References
 .PP
 Gerrit Code Review: https://www.gerritcodereview.com/

man/repo-wipe.1

@@ -0,0 +1,61 @@
+.\" DO NOT MODIFY THIS FILE!  It was generated by help2man.
+.TH REPO "1" "November 2025" "repo wipe" "Repo Manual"
+.SH NAME
+repo \- repo wipe - manual page for repo wipe
+.SH SYNOPSIS
+.B repo
+\fI\,wipe <project>\/\fR...
+.SH DESCRIPTION
+Summary
+.PP
+Wipe projects from the worktree
+.SH OPTIONS
+.TP
+\fB\-h\fR, \fB\-\-help\fR
+show this help message and exit
+.TP
+\fB\-f\fR, \fB\-\-force\fR
+force wipe shared projects and uncommitted changes
+.TP
+\fB\-\-force\-uncommitted\fR
+force wipe even if there are uncommitted changes
+.TP
+\fB\-\-force\-shared\fR
+force wipe even if the project shares an object
+directory
+.SS Logging options:
+.TP
+\fB\-v\fR, \fB\-\-verbose\fR
+show all output
+.TP
+\fB\-q\fR, \fB\-\-quiet\fR
+only show errors
+.SS Multi\-manifest options:
+.TP
+\fB\-\-outer\-manifest\fR
+operate starting at the outermost manifest
+.TP
+\fB\-\-no\-outer\-manifest\fR
+do not operate on outer manifests
+.TP
+\fB\-\-this\-manifest\-only\fR
+only operate on this (sub)manifest
+.TP
+\fB\-\-no\-this\-manifest\-only\fR, \fB\-\-all\-manifests\fR
+operate on this manifest and its submanifests
+.PP
+Run `repo help wipe` to view the detailed manual.
+.SH DETAILS
+.PP
+The 'repo wipe' command removes the specified projects from the worktree (the
+checked out source code) and deletes the project's git data from `.repo`.
+.PP
+This is a destructive operation and cannot be undone.
+.PP
+Projects can be specified either by name, or by a relative or absolute path to
+the project's local directory.
+.SH EXAMPLES
+.SS # Wipe the project "platform/build" by name:
+$ repo wipe platform/build
+.SS # Wipe the project at the path "build/make":
+$ repo wipe build/make

man/repo.1

@@ -1,5 +1,5 @@
 .\" DO NOT MODIFY THIS FILE!  It was generated by help2man.
-.TH REPO "1" "November 2022" "repo" "Repo Manual"
+.TH REPO "1" "November 2025" "repo" "Repo Manual"
 .SH NAME
 repo \- repository management tool built on top of git
 .SH SYNOPSIS
@@ -79,11 +79,8 @@ Download and checkout a change
 forall
 Run a shell command in each project
 .TP
-gitc\-delete
-Delete a GITC Client.
-.TP
-gitc\-init
-Initialize a GITC Client.
+gc
+Cleaning up internal repo and Git state.
 .TP
 grep
 Print lines matching a pattern
@@ -135,6 +132,9 @@ Upload changes for code review
 .TP
 version
 Display the version of repo
+.TP
+wipe
+Wipe projects from the worktree
 .PP
 See 'repo help <command>' for more information on a specific command.
-Bug reports: https://bugs.chromium.org/p/gerrit/issues/entry?template=Repo+tool+issue
+Bug reports: https://issues.gerritcodereview.com/issues/new?component=1370071

manifest_xml.py

@@ -18,28 +18,25 @@ import os
 import platform
 import re
 import sys
-import xml.dom.minidom
 import urllib.parse
+import xml.dom.minidom
 
-import gitc_utils
+from error import ManifestInvalidPathError
+from error import ManifestInvalidRevisionError
+from error import ManifestParseError
 from git_config import GitConfig
-from git_refs import R_HEADS, HEAD
+from git_refs import HEAD
+from git_refs import R_HEADS
 from git_superproject import Superproject
 import platform_utils
-from project import (
-    Annotation,
-    RemoteSpec,
-    Project,
-    RepoProject,
-    ManifestProject,
-)
-from error import (
-    ManifestParseError,
-    ManifestInvalidPathError,
-    ManifestInvalidRevisionError,
-)
+from project import Annotation
+from project import ManifestProject
+from project import Project
+from project import RemoteSpec
+from project import RepoProject
 from wrapper import Wrapper
 
+
 MANIFEST_FILE_NAME = "manifest.xml"
 LOCAL_MANIFEST_NAME = "local_manifest.xml"
 LOCAL_MANIFESTS_DIR_NAME = "local_manifests"
@@ -117,12 +114,40 @@ def XmlInt(node, attr, default=None):
     try:
         return int(value)
     except ValueError:
-        raise ManifestParseError(
-            'manifest: invalid %s="%s" integer' % (attr, value)
-        )
+        raise ManifestParseError(f'manifest: invalid {attr}="{value}" integer')
 
 
-class _Default(object):
+def normalize_url(url: str) -> str:
+    """Mutate input 'url' into normalized form:
+
+    * remove trailing slashes
+    * convert SCP-like syntax to SSH URL
+
+    Args:
+        url: URL to modify
+
+    Returns:
+        The normalized URL.
+    """
+
+    url = url.rstrip("/")
+    parsed_url = urllib.parse.urlparse(url)
+
+    # This matches patterns like "git@github.com:foo".
+    scp_like_url_re = r"^[^/:]+@[^/:]+:[^/]+"
+
+    # If our URL is missing a schema and matches git's
+    # SCP-like syntax we should convert it to a proper
+    # SSH URL instead to make urljoin() happier.
+    #
+    # See: https://git-scm.com/docs/git-clone#URLS
+    if not parsed_url.scheme and re.match(scp_like_url_re, url):
+        return "ssh://" + url.replace(":", "/", 1)
+
+    return url
+
+
+class _Default:
     """Project defaults within the manifest."""
 
     revisionExpr = None
@@ -145,7 +170,7 @@ class _Default(object):
         return self.__dict__ != other.__dict__
 
 
-class _XmlRemote(object):
+class _XmlRemote:
     def __init__(
         self,
         name,
@@ -185,20 +210,22 @@ class _XmlRemote(object):
     def _resolveFetchUrl(self):
         if self.fetchUrl is None:
             return ""
-        url = self.fetchUrl.rstrip("/")
-        manifestUrl = self.manifestUrl.rstrip("/")
-        # urljoin will gets confused over quite a few things.  The ones we care
-        # about here are:
-        # * no scheme in the base url, like <hostname:port>
-        # We handle no scheme by replacing it with an obscure protocol, gopher
-        # and then replacing it with the original when we are done.
 
-        if manifestUrl.find(":") != manifestUrl.find("/") - 1:
-            url = urllib.parse.urljoin("gopher://" + manifestUrl, url)
-            url = re.sub(r"^gopher://", "", url)
+        fetch_url = normalize_url(self.fetchUrl)
+        manifest_url = normalize_url(self.manifestUrl)
+
+        # urljoin doesn't like URLs with no scheme in the base URL
+        # such as file paths.  We handle this by prefixing it with
+        # an obscure protocol, gopher, and replacing it with the
+        # original after urljoin
+        if manifest_url.find(":") != manifest_url.find("/") - 1:
+            fetch_url = urllib.parse.urljoin(
+                "gopher://" + manifest_url, fetch_url
+            )
+            fetch_url = re.sub(r"^gopher://", "", fetch_url)
         else:
-            url = urllib.parse.urljoin(manifestUrl, url)
-        return url
+            fetch_url = urllib.parse.urljoin(manifest_url, fetch_url)
+        return fetch_url
 
     def ToRemoteSpec(self, projectName):
         fetchUrl = self.resolvedFetchUrl.rstrip("/")
@@ -228,7 +255,7 @@ class _XmlSubmanifest:
       project: a string, the name of the manifest project.
       revision: a string, the commitish.
       manifestName: a string, the submanifest file name.
-      groups: a list of strings, the groups to add to all projects in the
+      groups: a set of strings, the groups to add to all projects in the
           submanifest.
       default_groups: a list of strings, the default groups to sync.
       path: a string, the relative path for the submanifest checkout.
@@ -254,7 +281,7 @@ class _XmlSubmanifest:
         self.project = project
         self.revision = revision
         self.manifestName = manifestName
-        self.groups = groups
+        self.groups = groups or set()
         self.default_groups = default_groups
         self.path = path
         self.parent = parent
@@ -277,8 +304,8 @@ class _XmlSubmanifest:
         self.repo_client = RepoClient(
             parent.repodir,
             linkFile,
-            parent_groups=",".join(groups) or "",
-            submanifest_path=self.relpath,
+            parent_groups=groups,
+            submanifest_path=os.path.join(parent.path_prefix, self.relpath),
             outer_client=outer_client,
             default_groups=default_groups,
         )
@@ -318,7 +345,7 @@ class _XmlSubmanifest:
         manifestName = self.manifestName or "default.xml"
         revision = self.revision or self.name
         path = self.path or revision.split("/")[-1]
-        groups = self.groups or []
+        groups = self.groups
 
         return SubmanifestSpec(
             self.name, manifestUrl, manifestName, revision, path, groups
@@ -332,9 +359,7 @@ class _XmlSubmanifest:
 
     def GetGroupsStr(self):
         """Returns the `groups` given for this submanifest."""
-        if self.groups:
-            return ",".join(self.groups)
-        return ""
+        return ",".join(sorted(self.groups))
 
     def GetDefaultGroupsStr(self):
         """Returns the `default-groups` given for this submanifest."""
@@ -354,10 +379,10 @@ class SubmanifestSpec:
         self.manifestName = manifestName
         self.revision = revision
         self.path = path
-        self.groups = groups or []
+        self.groups = groups
 
 
-class XmlManifest(object):
+class XmlManifest:
     """manages the repo configuration file"""
 
     def __init__(
@@ -366,7 +391,7 @@ class XmlManifest(object):
         manifest_file,
         local_manifests=None,
         outer_client=None,
-        parent_groups="",
+        parent_groups=None,
         submanifest_path="",
         default_groups=None,
     ):
@@ -382,7 +407,8 @@ class XmlManifest(object):
                 manifests. This will usually be
                 |repodir|/|LOCAL_MANIFESTS_DIR_NAME|.
             outer_client: RepoClient of the outer manifest.
-            parent_groups: a string, the groups to apply to this projects.
+            parent_groups: a set of strings, the groups to apply to this
+                manifest.
             submanifest_path: The submanifest root relative to the repo root.
             default_groups: a string, the default manifest groups to use.
         """
@@ -405,14 +431,9 @@ class XmlManifest(object):
             self.manifestFileOverrides = {}
         self.local_manifests = local_manifests
         self._load_local_manifests = True
-        self.parent_groups = parent_groups
+        self.parent_groups = parent_groups or set()
         self.default_groups = default_groups
 
-        if outer_client and self.isGitcClient:
-            raise ManifestParseError(
-                "Multi-manifest is incompatible with `gitc-init`"
-            )
-
         if submanifest_path and not outer_client:
             # If passing a submanifest_path, there must be an outer_client.
             raise ManifestParseError(f"Bad call to {self.__class__.__name__}")
@@ -545,21 +566,29 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         """
         return [x for x in re.split(r"[,\s]+", field) if x]
 
+    def _ParseSet(self, field):
+        """Parse fields that contain flattened sets.
+
+        These are whitespace & comma separated.  Empty elements will be
+        discarded.
+        """
+        return set(self._ParseList(field))
+
     def ToXml(
         self,
         peg_rev=False,
         peg_rev_upstream=True,
         peg_rev_dest_branch=True,
-        groups=None,
+        filter_groups=None,
         omit_local=False,
     ):
         """Return the current manifest XML."""
         mp = self.manifestProject
 
-        if groups is None:
-            groups = mp.manifest_groups
-        if groups:
-            groups = self._ParseList(groups)
+        if filter_groups is None:
+            filter_groups = mp.manifest_groups
+        if filter_groups:
+            filter_groups = self._ParseList(filter_groups)
 
         doc = xml.dom.minidom.Document()
         root = doc.createElement("manifest")
@@ -632,7 +661,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                     output_project(parent, parent_node, project)
 
         def output_project(parent, parent_node, p):
-            if not p.MatchesGroups(groups):
+            if not p.MatchesGroups(filter_groups):
                 return
 
             if omit_local and self.IsFromLocalManifest(p):
@@ -703,10 +732,9 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                 le.setAttribute("dest", lf.dest)
                 e.appendChild(le)
 
-            default_groups = ["all", "name:%s" % p.name, "path:%s" % p.relpath]
-            egroups = [g for g in p.groups if g not in default_groups]
-            if egroups:
-                e.setAttribute("groups", ",".join(egroups))
+            groups = p.groups - {"all", f"name:{p.name}", f"path:{p.relpath}"}
+            if groups:
+                e.setAttribute("groups", ",".join(sorted(groups)))
 
             for a in p.annotations:
                 if a.keep == "true":
@@ -730,10 +758,10 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
             self._output_manifest_project_extras(p, e)
 
             if p.subprojects:
-                subprojects = set(subp.name for subp in p.subprojects)
+                subprojects = {subp.name for subp in p.subprojects}
                 output_projects(p, e, list(sorted(subprojects)))
 
-        projects = set(p.name for p in self._paths.values() if not p.parent)
+        projects = {p.name for p in self._paths.values() if not p.parent}
         output_projects(None, root, list(sorted(projects)))
 
         if self._repo_hooks_project:
@@ -803,17 +831,17 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
             for child in node.childNodes:
                 if child.nodeType == xml.dom.Node.ELEMENT_NODE:
                     attrs = child.attributes
-                    element = dict(
-                        (attrs.item(i).localName, attrs.item(i).value)
+                    element = {
+                        attrs.item(i).localName: attrs.item(i).value
                         for i in range(attrs.length)
-                    )
+                    }
                     if child.nodeName in SINGLE_ELEMENTS:
                         ret[child.nodeName] = element
                     elif child.nodeName in MULTI_ELEMENTS:
                         ret.setdefault(child.nodeName, []).append(element)
                     else:
                         raise ManifestParseError(
-                            'Unhandled element "%s"' % (child.nodeName,)
+                            f'Unhandled element "{child.nodeName}"'
                         )
 
                     append_children(element, child)
@@ -860,8 +888,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         self._Load()
         outer = self._outer_client
         yield outer
-        for tree in outer.all_children:
-            yield tree
+        yield from outer.all_children
 
     @property
     def all_children(self):
@@ -870,8 +897,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         for child in self._submanifests.values():
             if child.repo_client:
                 yield child.repo_client
-                for tree in child.repo_client.all_children:
-                    yield tree
+                yield from child.repo_client.all_children
 
     @property
     def path_prefix(self):
@@ -981,16 +1007,22 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
             return self.manifestProject.clone_filter
         return None
 
+    @property
+    def CloneFilterForDepth(self):
+        if self.manifestProject.clone_filter_for_depth:
+            return self.manifestProject.clone_filter_for_depth
+        return None
+
     @property
     def PartialCloneExclude(self):
         exclude = self.manifest.manifestProject.partial_clone_exclude or ""
-        return set(x.strip() for x in exclude.split(","))
+        return {x.strip() for x in exclude.split(",")}
 
     def SetManifestOverride(self, path):
         """Override manifestFile.  The caller must call Unload()"""
-        self._outer_client.manifest.manifestFileOverrides[
-            self.path_prefix
-        ] = path
+        self._outer_client.manifest.manifestFileOverrides[self.path_prefix] = (
+            path
+        )
 
     @property
     def UseLocalManifests(self):
@@ -1090,7 +1122,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
             groups += f",platform-{platform.system().lower()}"
         return groups
 
-    def GetGroupsStr(self):
+    def GetManifestGroupsStr(self):
         """Returns the manifest group string that should be synced."""
         return (
             self.manifestProject.manifest_groups or self.GetDefaultGroupsStr()
@@ -1145,12 +1177,12 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                     b = b[len(R_HEADS) :]
                 self.branch = b
 
-                parent_groups = self.parent_groups
+                parent_groups = self.parent_groups.copy()
                 if self.path_prefix:
-                    parent_groups = (
+                    parent_groups |= {
                         f"{SUBMANIFEST_GROUP_PREFIX}:path:"
-                        f"{self.path_prefix},{parent_groups}"
-                    )
+                        f"{self.path_prefix}"
+                    }
 
                 # The manifestFile was specified by the user which is why we
                 # allow include paths to point anywhere.
@@ -1176,16 +1208,16 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                                 # Since local manifests are entirely managed by
                                 # the user, allow them to point anywhere the
                                 # user wants.
-                                local_group = (
+                                local_group = {
                                     f"{LOCAL_MANIFEST_GROUP_PREFIX}:"
                                     f"{local_file[:-4]}"
-                                )
+                                }
                                 nodes.append(
                                     self._ParseManifestXml(
                                         local,
                                         self.subdir,
                                         parent_groups=(
-                                            f"{local_group},{parent_groups}"
+                                            local_group | parent_groups
                                         ),
                                         restrict_includes=False,
                                     )
@@ -1236,7 +1268,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         self,
         path,
         include_root,
-        parent_groups="",
+        parent_groups=None,
         restrict_includes=True,
         parent_node=None,
     ):
@@ -1245,11 +1277,11 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         Args:
             path: The XML file to read & parse.
             include_root: The path to interpret include "name"s relative to.
-            parent_groups: The groups to apply to this projects.
+            parent_groups: The set of groups to apply to this manifest.
             restrict_includes: Whether to constrain the "name" attribute of
                 includes.
-            parent_node: The parent include node, to apply attribute to this
-                projects.
+            parent_node: The parent include node, to apply attributes to this
+                manifest.
 
         Returns:
             List of XML nodes.
@@ -1257,35 +1289,42 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         try:
             root = xml.dom.minidom.parse(path)
         except (OSError, xml.parsers.expat.ExpatError) as e:
-            raise ManifestParseError(
-                "error parsing manifest %s: %s" % (path, e)
-            )
+            raise ManifestParseError(f"error parsing manifest {path}: {e}")
 
         if not root or not root.childNodes:
-            raise ManifestParseError("no root node in %s" % (path,))
+            raise ManifestParseError(f"no root node in {path}")
 
         for manifest in root.childNodes:
-            if manifest.nodeName == "manifest":
+            if (
+                manifest.nodeType == manifest.ELEMENT_NODE
+                and manifest.nodeName == "manifest"
+            ):
                 break
         else:
-            raise ManifestParseError("no <manifest> in %s" % (path,))
+            raise ManifestParseError(f"no <manifest> in {path}")
 
         nodes = []
         for node in manifest.childNodes:
+            if (
+                parent_node
+                and node.nodeName in ("include", "project")
+                and not node.hasAttribute("revision")
+            ):
+                node.setAttribute(
+                    "revision", parent_node.getAttribute("revision")
+                )
             if node.nodeName == "include":
                 name = self._reqatt(node, "name")
                 if restrict_includes:
                     msg = self._CheckLocalPath(name)
                     if msg:
                         raise ManifestInvalidPathError(
-                            '<include> invalid "name": %s: %s' % (name, msg)
+                            f'<include> invalid "name": {name}: {msg}'
                         )
-                include_groups = ""
-                if parent_groups:
-                    include_groups = parent_groups
+                include_groups = (parent_groups or set()).copy()
                 if node.hasAttribute("groups"):
-                    include_groups = (
-                        node.getAttribute("groups") + "," + include_groups
+                    include_groups |= self._ParseSet(
+                        node.getAttribute("groups")
                     )
                 fp = os.path.join(include_root, name)
                 if not os.path.isfile(fp):
@@ -1301,33 +1340,23 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                     )
                 # should isolate this to the exact exception, but that's
                 # tricky.  actual parsing implementation may vary.
-                except (
-                    KeyboardInterrupt,
-                    RuntimeError,
-                    SystemExit,
-                    ManifestParseError,
-                ):
+                except (RuntimeError, ManifestParseError):
                     raise
                 except Exception as e:
                     raise ManifestParseError(
-                        "failed parsing included manifest %s: %s" % (name, e)
+                        f"failed parsing included manifest {name}: {e}"
                     )
             else:
-                if parent_groups and node.nodeName == "project":
-                    nodeGroups = parent_groups
-                    if node.hasAttribute("groups"):
-                        nodeGroups = (
-                            node.getAttribute("groups") + "," + nodeGroups
-                        )
-                    node.setAttribute("groups", nodeGroups)
-                if (
-                    parent_node
-                    and node.nodeName == "project"
-                    and not node.hasAttribute("revision")
+                if parent_groups and node.nodeName in (
+                    "project",
+                    "extend-project",
                 ):
-                    node.setAttribute(
-                        "revision", parent_node.getAttribute("revision")
-                    )
+                    nodeGroups = parent_groups.copy()
+                    if node.hasAttribute("groups"):
+                        nodeGroups |= self._ParseSet(
+                            node.getAttribute("groups")
+                        )
+                    node.setAttribute("groups", ",".join(sorted(nodeGroups)))
                 nodes.append(node)
         return nodes
 
@@ -1418,6 +1447,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
 
         repo_hooks_project = None
         enabled_repo_hooks = None
+        failed_revision_changes = []
         for node in itertools.chain(*node_list):
             if node.nodeName == "project":
                 project = self._ParseProject(node)
@@ -1435,7 +1465,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                 dest_path = node.getAttribute("dest-path")
                 groups = node.getAttribute("groups")
                 if groups:
-                    groups = self._ParseList(groups)
+                    groups = self._ParseSet(groups or "")
                 revision = node.getAttribute("revision")
                 remote_name = node.getAttribute("remote")
                 if not remote_name:
@@ -1444,6 +1474,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                     remote = self._get_remote(node)
                 dest_branch = node.getAttribute("dest-branch")
                 upstream = node.getAttribute("upstream")
+                base_revision = node.getAttribute("base-rev")
 
                 named_projects = self._projects[name]
                 if dest_path and not path and len(named_projects) > 1:
@@ -1455,8 +1486,15 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                     if path and p.relpath != path:
                         continue
                     if groups:
-                        p.groups.extend(groups)
+                        p.groups |= groups
                     if revision:
+                        if base_revision:
+                            if p.revisionExpr != base_revision:
+                                failed_revision_changes.append(
+                                    "extend-project name %s mismatch base "
+                                    "%s vs revision %s"
+                                    % (name, base_revision, p.revisionExpr)
+                                )
                         p.SetRevision(revision)
 
                     if remote_name:
@@ -1478,6 +1516,14 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                         p.UpdatePaths(relpath, worktree, gitdir, objdir)
                         self._paths[p.relpath] = p
 
+                    for n in node.childNodes:
+                        if n.nodeName == "copyfile":
+                            self._ParseCopyFile(p, n)
+                        elif n.nodeName == "linkfile":
+                            self._ParseLinkFile(p, n)
+                        elif n.nodeName == "annotation":
+                            self._ParseAnnotation(p, n)
+
             if node.nodeName == "repo-hooks":
                 # Only one project can be the hooks project
                 if repo_hooks_project is not None:
@@ -1529,24 +1575,73 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                 self._contactinfo = ContactInfo(bugurl)
 
             if node.nodeName == "remove-project":
-                name = self._reqatt(node, "name")
+                name = node.getAttribute("name")
+                path = node.getAttribute("path")
+                base_revision = node.getAttribute("base-rev")
 
-                if name in self._projects:
-                    for p in self._projects[name]:
-                        del self._paths[p.relpath]
-                    del self._projects[name]
+                # Name or path needed.
+                if not name and not path:
+                    raise ManifestParseError(
+                        "remove-project must have name and/or path"
+                    )
 
-                    # If the manifest removes the hooks project, treat it as if
-                    # it deleted
-                    # the repo-hooks element too.
-                    if repo_hooks_project == name:
-                        repo_hooks_project = None
-                elif not XmlBool(node, "optional", False):
+                removed_project = ""
+
+                # Find and remove projects based on name and/or path.
+                for projname, projects in list(self._projects.items()):
+                    for p in projects:
+                        if name == projname and not path:
+                            if base_revision:
+                                if p.revisionExpr != base_revision:
+                                    failed_revision_changes.append(
+                                        "remove-project name %s mismatch base "
+                                        "%s vs revision %s"
+                                        % (name, base_revision, p.revisionExpr)
+                                    )
+                            del self._paths[p.relpath]
+                            if not removed_project:
+                                del self._projects[name]
+                                removed_project = name
+                        elif path == p.relpath and (
+                            name == projname or not name
+                        ):
+                            if base_revision:
+                                if p.revisionExpr != base_revision:
+                                    failed_revision_changes.append(
+                                        "remove-project path %s mismatch base "
+                                        "%s vs revision %s"
+                                        % (
+                                            p.relpath,
+                                            base_revision,
+                                            p.revisionExpr,
+                                        )
+                                    )
+                            self._projects[projname].remove(p)
+                            del self._paths[p.relpath]
+                            removed_project = p.name
+
+                # If the manifest removes the hooks project, treat it as if
+                # it deleted the repo-hooks element too.
+                if (
+                    removed_project
+                    and removed_project not in self._projects
+                    and repo_hooks_project == removed_project
+                ):
+                    repo_hooks_project = None
+
+                if not removed_project and not XmlBool(node, "optional", False):
                     raise ManifestParseError(
                         "remove-project element specifies non-existent "
-                        "project: %s" % name
+                        "project: %s" % node.toxml()
                     )
 
+        if failed_revision_changes:
+            raise ManifestParseError(
+                "revision base check failed, rebase patches and update "
+                "base revs for: ",
+                failed_revision_changes,
+            )
+
         # Store repo hooks project information.
         if repo_hooks_project:
             # Store a reference to the Project.
@@ -1722,7 +1817,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         groups = ""
         if node.hasAttribute("groups"):
             groups = node.getAttribute("groups")
-        groups = self._ParseList(groups)
+        groups = self._ParseSet(groups)
         default_groups = self._ParseList(node.getAttribute("default-groups"))
         path = node.getAttribute("path")
         if path == "":
@@ -1738,13 +1833,13 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                 msg = self._CheckLocalPath(name)
                 if msg:
                     raise ManifestInvalidPathError(
-                        '<submanifest> invalid "name": %s: %s' % (name, msg)
+                        f'<submanifest> invalid "name": {name}: {msg}'
                     )
         else:
             msg = self._CheckLocalPath(path)
             if msg:
                 raise ManifestInvalidPathError(
-                    '<submanifest> invalid "path": %s: %s' % (path, msg)
+                    f'<submanifest> invalid "path": {path}: {msg}'
                 )
 
         submanifest = _XmlSubmanifest(
@@ -1779,7 +1874,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         msg = self._CheckLocalPath(name, dir_ok=True)
         if msg:
             raise ManifestInvalidPathError(
-                '<project> invalid "name": %s: %s' % (name, msg)
+                f'<project> invalid "name": {name}: {msg}'
             )
         if parent:
             name = self._JoinName(parent.name, name)
@@ -1789,7 +1884,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
             remote = self._default.remote
         if remote is None:
             raise ManifestParseError(
-                "no remote for project %s within %s" % (name, self.manifestFile)
+                f"no remote for project {name} within {self.manifestFile}"
             )
 
         revisionExpr = node.getAttribute("revision") or remote.revision
@@ -1810,7 +1905,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
             msg = self._CheckLocalPath(path, dir_ok=True, cwd_dot_ok=True)
             if msg:
                 raise ManifestInvalidPathError(
-                    '<project> invalid "path": %s: %s' % (path, msg)
+                    f'<project> invalid "path": {path}: {msg}'
                 )
 
         rebase = XmlBool(node, "rebase", True)
@@ -1831,11 +1926,6 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
 
         upstream = node.getAttribute("upstream") or self._default.upstreamExpr
 
-        groups = ""
-        if node.hasAttribute("groups"):
-            groups = node.getAttribute("groups")
-        groups = self._ParseList(groups)
-
         if parent is None:
             (
                 relpath,
@@ -1850,8 +1940,11 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                 parent, name, path
             )
 
-        default_groups = ["all", "name:%s" % name, "path:%s" % relpath]
-        groups.extend(set(default_groups).difference(groups))
+        groups = ""
+        if node.hasAttribute("groups"):
+            groups = node.getAttribute("groups")
+        groups = self._ParseSet(groups)
+        groups |= {"all", f"name:{name}", f"path:{relpath}"}
 
         if self.IsMirror and node.hasAttribute("force-path"):
             if XmlBool(node, "force-path", False):
@@ -1883,11 +1976,11 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         for n in node.childNodes:
             if n.nodeName == "copyfile":
                 self._ParseCopyFile(project, n)
-            if n.nodeName == "linkfile":
+            elif n.nodeName == "linkfile":
                 self._ParseLinkFile(project, n)
-            if n.nodeName == "annotation":
+            elif n.nodeName == "annotation":
                 self._ParseAnnotation(project, n)
-            if n.nodeName == "project":
+            elif n.nodeName == "project":
                 project.subprojects.append(
                     self._ParseProject(n, parent=project)
                 )
@@ -1971,7 +2064,12 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         path = path.rstrip("/")
         name = name.rstrip("/")
         relpath = self._JoinRelpath(parent.relpath, path)
-        gitdir = os.path.join(parent.gitdir, "subprojects", "%s.git" % path)
+        subprojects = os.path.join(parent.gitdir, "subprojects", f"{path}.git")
+        modules = os.path.join(parent.gitdir, "modules", path)
+        if platform_utils.isdir(subprojects):
+            gitdir = subprojects
+        else:
+            gitdir = modules
         objdir = os.path.join(
             parent.gitdir, "subproject-objects", "%s.git" % name
         )
@@ -2022,22 +2120,22 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         # implementation:
         # https://eclipse.googlesource.com/jgit/jgit/+/9110037e3e9461ff4dac22fee84ef3694ed57648/org.eclipse.jgit/src/org/eclipse/jgit/lib/ObjectChecker.java#884
         BAD_CODEPOINTS = {
-            "\u200C",  # ZERO WIDTH NON-JOINER
-            "\u200D",  # ZERO WIDTH JOINER
-            "\u200E",  # LEFT-TO-RIGHT MARK
-            "\u200F",  # RIGHT-TO-LEFT MARK
-            "\u202A",  # LEFT-TO-RIGHT EMBEDDING
-            "\u202B",  # RIGHT-TO-LEFT EMBEDDING
-            "\u202C",  # POP DIRECTIONAL FORMATTING
-            "\u202D",  # LEFT-TO-RIGHT OVERRIDE
-            "\u202E",  # RIGHT-TO-LEFT OVERRIDE
-            "\u206A",  # INHIBIT SYMMETRIC SWAPPING
-            "\u206B",  # ACTIVATE SYMMETRIC SWAPPING
-            "\u206C",  # INHIBIT ARABIC FORM SHAPING
-            "\u206D",  # ACTIVATE ARABIC FORM SHAPING
-            "\u206E",  # NATIONAL DIGIT SHAPES
-            "\u206F",  # NOMINAL DIGIT SHAPES
-            "\uFEFF",  # ZERO WIDTH NO-BREAK SPACE
+            "\u200c",  # ZERO WIDTH NON-JOINER
+            "\u200d",  # ZERO WIDTH JOINER
+            "\u200e",  # LEFT-TO-RIGHT MARK
+            "\u200f",  # RIGHT-TO-LEFT MARK
+            "\u202a",  # LEFT-TO-RIGHT EMBEDDING
+            "\u202b",  # RIGHT-TO-LEFT EMBEDDING
+            "\u202c",  # POP DIRECTIONAL FORMATTING
+            "\u202d",  # LEFT-TO-RIGHT OVERRIDE
+            "\u202e",  # RIGHT-TO-LEFT OVERRIDE
+            "\u206a",  # INHIBIT SYMMETRIC SWAPPING
+            "\u206b",  # ACTIVATE SYMMETRIC SWAPPING
+            "\u206c",  # INHIBIT ARABIC FORM SHAPING
+            "\u206d",  # ACTIVATE ARABIC FORM SHAPING
+            "\u206e",  # NATIONAL DIGIT SHAPES
+            "\u206f",  # NOMINAL DIGIT SHAPES
+            "\ufeff",  # ZERO WIDTH NO-BREAK SPACE
         }
         if BAD_CODEPOINTS & path_codepoints:
             # This message is more expansive than reality, but should be fine.
@@ -2067,7 +2165,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         if not cwd_dot_ok or parts != ["."]:
             for part in set(parts):
                 if part in {".", "..", ".git"} or part.startswith(".repo"):
-                    return "bad component: %s" % (part,)
+                    return f"bad component: {part}"
 
         if not dir_ok and resep.match(path[-1]):
             return "dirs not allowed"
@@ -2103,7 +2201,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         msg = cls._CheckLocalPath(dest)
         if msg:
             raise ManifestInvalidPathError(
-                '<%s> invalid "dest": %s: %s' % (element, dest, msg)
+                f'<{element}> invalid "dest": {dest}: {msg}'
             )
 
         # |src| is the file we read from or path we point to for symlinks.
@@ -2114,7 +2212,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         )
         if msg:
             raise ManifestInvalidPathError(
-                '<%s> invalid "src": %s: %s' % (element, src, msg)
+                f'<{element}> invalid "src": {src}: {msg}'
             )
 
     def _ParseCopyFile(self, project, node):
@@ -2158,7 +2256,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         v = self._remotes.get(name)
         if not v:
             raise ManifestParseError(
-                "remote %s not defined in %s" % (name, self.manifestFile)
+                f"remote {name} not defined in {self.manifestFile}"
             )
         return v
 
@@ -2184,7 +2282,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         toProjects = manifest.paths
 
         fromKeys = sorted(fromProjects.keys())
-        toKeys = sorted(toProjects.keys())
+        toKeys = set(toProjects.keys())
 
         diff = {
             "added": [],
@@ -2195,13 +2293,13 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
         }
 
         for proj in fromKeys:
+            fromProj = fromProjects[proj]
             if proj not in toKeys:
-                diff["removed"].append(fromProjects[proj])
-            elif not fromProjects[proj].Exists:
+                diff["removed"].append(fromProj)
+            elif not fromProj.Exists:
                 diff["missing"].append(toProjects[proj])
                 toKeys.remove(proj)
             else:
-                fromProj = fromProjects[proj]
                 toProj = toProjects[proj]
                 try:
                     fromRevId = fromProj.GetCommitRevisionId()
@@ -2213,27 +2311,11 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
                         diff["changed"].append((fromProj, toProj))
                 toKeys.remove(proj)
 
-        for proj in toKeys:
-            diff["added"].append(toProjects[proj])
+        diff["added"].extend(toProjects[proj] for proj in sorted(toKeys))
 
         return diff
 
 
-class GitcManifest(XmlManifest):
-    """Parser for GitC (git-in-the-cloud) manifests."""
-
-    def _ParseProject(self, node, parent=None):
-        """Override _ParseProject and add support for GITC specific attributes."""  # noqa: E501
-        return super()._ParseProject(
-            node, parent=parent, old_revision=node.getAttribute("old-revision")
-        )
-
-    def _output_manifest_project_extras(self, p, e):
-        """Output GITC Specific Project attributes"""
-        if p.old_revision:
-            e.setAttribute("old-revision", str(p.old_revision))
-
-
 class RepoClient(XmlManifest):
     """Manages a repo client checkout."""
 
@@ -2251,7 +2333,6 @@ class RepoClient(XmlManifest):
             submanifest_path: The submanifest root relative to the repo root.
             **kwargs: Additional keyword arguments, passed to XmlManifest.
         """
-        self.isGitcClient = False
         submanifest_path = submanifest_path or ""
         if submanifest_path:
             self._CheckLocalPath(submanifest_path)
@@ -2286,19 +2367,3 @@ class RepoClient(XmlManifest):
 
         # TODO: Completely separate manifest logic out of the client.
         self.manifest = self
-
-
-class GitcClient(RepoClient, GitcManifest):
-    """Manages a GitC client checkout."""
-
-    def __init__(self, repodir, gitc_client_name):
-        """Initialize the GitcManifest object."""
-        self.gitc_client_name = gitc_client_name
-        self.gitc_client_dir = os.path.join(
-            gitc_utils.get_gitc_manifest_dir(), gitc_client_name
-        )
-
-        super().__init__(
-            repodir, os.path.join(self.gitc_client_dir, ".manifest")
-        )
-        self.isGitcClient = True

pager.py

@@ -19,6 +19,7 @@ import sys
 
 import platform_utils
 
+
 active = False
 pager_process = None
 old_stdout = None
@@ -39,7 +40,7 @@ def RunPager(globalConfig):
 
 
 def TerminatePager():
-    global pager_process, old_stdout, old_stderr
+    global pager_process
     if pager_process:
         sys.stdout.flush()
         sys.stderr.flush()
@@ -118,7 +119,10 @@ def _BecomePager(pager):
     # available versions of 'less', a better 'more'.
     _a, _b, _c = select.select([0], [], [0])
 
-    os.environ["LESS"] = "FRSX"
+    # This matches the behavior of git, which sets $LESS to `FRX` if it is not
+    # set. See:
+    # https://git-scm.com/docs/git-config#Documentation/git-config.txt-corepager
+    os.environ.setdefault("LESS", "FRX")
 
     try:
         os.execvp(pager, [pager])

platform_utils.py

@@ -57,8 +57,8 @@ def _validate_winpath(path):
     if _winpath_is_valid(path):
         return path
     raise ValueError(
-        'Path "{}" must be a relative path or an absolute '
-        "path starting with a drive letter".format(path)
+        f'Path "{path}" must be a relative path or an absolute '
+        "path starting with a drive letter"
     )
 
 
@@ -156,6 +156,12 @@ def remove(path, missing_ok=False):
                 os.rmdir(longpath)
             else:
                 os.remove(longpath)
+        elif (
+            e.errno == errno.EROFS
+            and missing_ok
+            and not os.path.exists(longpath)
+        ):
+            pass
         elif missing_ok and e.errno == errno.ENOENT:
             pass
         else:
@@ -193,10 +199,9 @@ def _walk_windows_impl(top, topdown, onerror, followlinks):
     for name in dirs:
         new_path = os.path.join(top, name)
         if followlinks or not islink(new_path):
-            for x in _walk_windows_impl(
+            yield from _walk_windows_impl(
                 new_path, topdown, onerror, followlinks
-            ):
-                yield x
+            )
     if not topdown:
         yield top, dirs, nondirs
 
@@ -252,32 +257,3 @@ def readlink(path):
         return platform_utils_win32.readlink(_makelongpath(path))
     else:
         return os.readlink(path)
-
-
-def realpath(path):
-    """Return the canonical path of the specified filename, eliminating
-    any symbolic links encountered in the path.
-
-    Availability: Windows, Unix.
-    """
-    if isWindows():
-        current_path = os.path.abspath(path)
-        path_tail = []
-        for c in range(0, 100):  # Avoid cycles
-            if islink(current_path):
-                target = readlink(current_path)
-                current_path = os.path.join(
-                    os.path.dirname(current_path), target
-                )
-            else:
-                basename = os.path.basename(current_path)
-                if basename == "":
-                    path_tail.append(current_path)
-                    break
-                path_tail.append(basename)
-                current_path = os.path.dirname(current_path)
-        path_tail.reverse()
-        result = os.path.normpath(os.path.join(*path_tail))
-        return result
-    else:
-        return os.path.realpath(path)

platform_utils_win32.py

@@ -12,12 +12,28 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from ctypes import addressof
+from ctypes import byref
+from ctypes import c_buffer
+from ctypes import c_ubyte
+from ctypes import FormatError
+from ctypes import get_last_error
+from ctypes import Structure
+from ctypes import Union
+from ctypes import WinDLL
+from ctypes import WinError
+from ctypes.wintypes import BOOL
+from ctypes.wintypes import BOOLEAN
+from ctypes.wintypes import DWORD
+from ctypes.wintypes import HANDLE
+from ctypes.wintypes import LPCWSTR
+from ctypes.wintypes import LPDWORD
+from ctypes.wintypes import LPVOID
+from ctypes.wintypes import ULONG
+from ctypes.wintypes import USHORT
+from ctypes.wintypes import WCHAR
 import errno
 
-from ctypes import WinDLL, get_last_error, FormatError, WinError, addressof
-from ctypes import c_buffer, c_ubyte, Structure, Union, byref
-from ctypes.wintypes import BOOL, BOOLEAN, LPCWSTR, DWORD, HANDLE
-from ctypes.wintypes import WCHAR, USHORT, LPVOID, ULONG, LPDWORD
 
 kernel32 = WinDLL("kernel32", use_last_error=True)
 
@@ -170,9 +186,7 @@ def _create_symlink(source, link_name, dwFlags):
             error_desc = FormatError(code).strip()
             if code == ERROR_PRIVILEGE_NOT_HELD:
                 raise OSError(errno.EPERM, error_desc, link_name)
-            _raise_winerror(
-                code, 'Error creating symbolic link "{}"'.format(link_name)
-            )
+            _raise_winerror(code, f'Error creating symbolic link "{link_name}"')
 
 
 def islink(path):
@@ -194,7 +208,7 @@ def readlink(path):
     )
     if reparse_point_handle == INVALID_HANDLE_VALUE:
         _raise_winerror(
-            get_last_error(), 'Error opening symbolic link "{}"'.format(path)
+            get_last_error(), f'Error opening symbolic link "{path}"'
         )
     target_buffer = c_buffer(MAXIMUM_REPARSE_DATA_BUFFER_SIZE)
     n_bytes_returned = DWORD()
@@ -211,7 +225,7 @@ def readlink(path):
     CloseHandle(reparse_point_handle)
     if not io_result:
         _raise_winerror(
-            get_last_error(), 'Error reading symbolic link "{}"'.format(path)
+            get_last_error(), f'Error reading symbolic link "{path}"'
         )
     rdb = REPARSE_DATA_BUFFER.from_buffer(target_buffer)
     if rdb.ReparseTag == IO_REPARSE_TAG_SYMLINK:
@@ -220,11 +234,11 @@ def readlink(path):
         return rdb.MountPointReparseBuffer.PrintName
     # Unsupported reparse point type.
     _raise_winerror(
-        ERROR_NOT_SUPPORTED, 'Error reading symbolic link "{}"'.format(path)
+        ERROR_NOT_SUPPORTED, f'Error reading symbolic link "{path}"'
     )
 
 
 def _raise_winerror(code, error_desc):
     win_error_desc = FormatError(code).strip()
-    error_desc = "{0}: {1}".format(error_desc, win_error_desc)
+    error_desc = f"{error_desc}: {win_error_desc}"
     raise WinError(code, error_desc)

progress.py

@@ -16,6 +16,7 @@ import os
 import sys
 import time
 
+
 try:
     import threading as _threading
 except ImportError:
@@ -23,7 +24,11 @@ except ImportError:
 
 from repo_trace import IsTraceToStderr
 
-_NOT_TTY = not os.isatty(2)
+
+# Capture the original stderr stream. We use this exclusively for progress
+# updates to ensure we talk to the terminal even if stderr is redirected.
+_STDERR = sys.stderr
+_TTY = _STDERR.isatty()
 
 # This will erase all content in the current line (wherever the cursor is).
 # It does not move the cursor, so this is usually followed by \r to move to
@@ -50,11 +55,11 @@ def duration_str(total):
     uses microsecond resolution.  This makes for noisy output.
     """
     hours, mins, secs = convert_to_hms(total)
-    ret = "%.3fs" % (secs,)
+    ret = f"{secs:.3f}s"
     if mins:
-        ret = "%im%s" % (mins, ret)
+        ret = f"{mins}m{ret}"
     if hours:
-        ret = "%ih%s" % (hours, ret)
+        ret = f"{hours}h{ret}"
     return ret
 
 
@@ -76,16 +81,20 @@ def elapsed_str(total):
     return ret
 
 
-class Progress(object):
+def jobs_str(total):
+    return f"{total} job{'s' if total > 1 else ''}"
+
+
+class Progress:
     def __init__(
         self,
         title,
         total=0,
         units="",
-        print_newline=False,
         delay=True,
         quiet=False,
         show_elapsed=False,
+        elide=False,
     ):
         self._title = title
         self._total = total
@@ -93,7 +102,10 @@ class Progress(object):
         self._start = time.time()
         self._show = not delay
         self._units = units
-        self._print_newline = print_newline
+        self._elide = elide and _TTY
+        self._quiet = quiet
+        self._ended = False
+
         # Only show the active jobs section if we run more than one in parallel.
         self._show_jobs = False
         self._active = 0
@@ -107,21 +119,28 @@ class Progress(object):
         )
         self._update_thread.daemon = True
 
-        # When quiet, never show any output.  It's a bit hacky, but reusing the
-        # existing logic that delays initial output keeps the rest of the class
-        # clean.  Basically we set the start time to years in the future.
-        if quiet:
-            self._show = False
-            self._start += 2**32
-        elif show_elapsed:
+        if not quiet and show_elapsed:
             self._update_thread.start()
 
+    def update_total(self, new_total):
+        """Updates the total if the new total is larger."""
+        if new_total > self._total:
+            self._total = new_total
+
     def _update_loop(self):
         while True:
-            if self._update_event.is_set():
+            self.update(inc=0)
+            if self._update_event.wait(timeout=1):
                 return
-            self.update(inc=0, msg=self._last_msg)
-            time.sleep(1)
+
+    def _write(self, s):
+        s = "\r" + s
+        if self._elide:
+            col = os.get_terminal_size(_STDERR.fileno()).columns
+            if len(s) > col:
+                s = s[: col - 1] + ".."
+        _STDERR.write(s)
+        _STDERR.flush()
 
     def start(self, name):
         self._active += 1
@@ -133,11 +152,19 @@ class Progress(object):
         self.update(msg="finished " + name)
         self._active -= 1
 
-    def update(self, inc=1, msg=""):
+    def update(self, inc=1, msg=None):
+        """Updates the progress indicator.
+
+        Args:
+            inc: The number of items completed.
+            msg: The message to display. If None, use the last message.
+        """
         self._done += inc
+        if msg is None:
+            msg = self._last_msg
         self._last_msg = msg
 
-        if _NOT_TTY or IsTraceToStderr():
+        if not _TTY or IsTraceToStderr() or self._quiet:
             return
 
         elapsed_sec = time.time() - self._start
@@ -148,25 +175,21 @@ class Progress(object):
                 return
 
         if self._total <= 0:
-            sys.stderr.write(
-                "\r%s: %d,%s" % (self._title, self._done, CSI_ERASE_LINE_AFTER)
+            self._write(
+                "%s: %d,%s" % (self._title, self._done, CSI_ERASE_LINE_AFTER)
             )
-            sys.stderr.flush()
         else:
             p = (100 * self._done) / self._total
             if self._show_jobs:
-                jobs = "[%d job%s] " % (
-                    self._active,
-                    "s" if self._active > 1 else "",
-                )
+                jobs = f"[{jobs_str(self._active)}] "
             else:
                 jobs = ""
             if self._show_elapsed:
                 elapsed = f" {elapsed_str(elapsed_sec)} |"
             else:
                 elapsed = ""
-            sys.stderr.write(
-                "\r%s: %2d%% %s(%d%s/%d%s)%s %s%s%s"
+            self._write(
+                "%s: %2d%% %s(%d%s/%d%s)%s %s%s"
                 % (
                     self._title,
                     p,
@@ -178,27 +201,43 @@ class Progress(object):
                     elapsed,
                     msg,
                     CSI_ERASE_LINE_AFTER,
-                    "\n" if self._print_newline else "",
                 )
             )
-            sys.stderr.flush()
+
+    def display_message(self, msg):
+        """Clears the current progress line and prints a message above it.
+
+        The progress bar is then redrawn on the next line.
+        """
+        if not _TTY or IsTraceToStderr() or self._quiet:
+            return
+
+        # Erase the current line, print the message with a newline,
+        # and then immediately redraw the progress bar on the new line.
+        _STDERR.write("\r" + CSI_ERASE_LINE)
+        _STDERR.write(msg + "\n")
+        _STDERR.flush()
+        self.update(inc=0)
 
     def end(self):
+        if self._ended:
+            return
+        self._ended = True
+
         self._update_event.set()
-        if _NOT_TTY or IsTraceToStderr() or not self._show:
+        if not _TTY or IsTraceToStderr() or self._quiet:
             return
 
         duration = duration_str(time.time() - self._start)
         if self._total <= 0:
-            sys.stderr.write(
-                "\r%s: %d, done in %s%s\n"
+            self._write(
+                "%s: %d, done in %s%s\n"
                 % (self._title, self._done, duration, CSI_ERASE_LINE_AFTER)
             )
-            sys.stderr.flush()
         else:
             p = (100 * self._done) / self._total
-            sys.stderr.write(
-                "\r%s: %3d%% (%d%s/%d%s), done in %s%s\n"
+            self._write(
+                "%s: %3d%% (%d%s/%d%s), done in %s%s\n"
                 % (
                     self._title,
                     p,
@@ -210,4 +249,3 @@ class Progress(object):
                     CSI_ERASE_LINE_AFTER,
                 )
             )
-            sys.stderr.flush()

pyproject.toml

@@ -1,4 +1,4 @@
-# Copyright 2023 The Android Open Source Project
+# Copyright (C) 2023 The Android Open Source Project
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,4 +15,9 @@
 [tool.black]
 line-length = 80
 # NB: Keep in sync with tox.ini.
-target-version = ['py36', 'py37', 'py38', 'py39', 'py310']
+target-version = ['py36', 'py37', 'py38', 'py39', 'py310', 'py311'] #, 'py312'
+
+[tool.pytest.ini_options]
+markers = """
+    skip_cq: Skip tests in the CQ. Should be rarely used!
+"""

release/check-metadata.py

@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+# Copyright (C) 2025 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helper tool to check various metadata (e.g. licensing) in source files."""
+
+import argparse
+from pathlib import Path
+import re
+import sys
+
+import util
+
+
+_FILE_HEADER_RE = re.compile(
+    r"""# Copyright \(C\) 20[0-9]{2} The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2\.0 \(the "License"\);
+# you may not use this file except in compliance with the License\.
+# You may obtain a copy of the License at
+#
+#      http://www\.apache\.org/licenses/LICENSE-2\.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied\.
+# See the License for the specific language governing permissions and
+# limitations under the License\.
+"""
+)
+
+
+def check_license(path: Path, lines: list[str]) -> bool:
+    """Check license header."""
+    # Enforce licensing on configs & scripts.
+    if not (
+        path.suffix in (".bash", ".cfg", ".ini", ".py", ".toml")
+        or lines[0] in ("#!/bin/bash", "#!/bin/sh", "#!/usr/bin/env python3")
+    ):
+        return True
+
+    # Extract the file header.
+    header_lines = []
+    for line in lines:
+        if line.startswith("#"):
+            header_lines.append(line)
+        else:
+            break
+    if not header_lines:
+        print(
+            f"error: {path.relative_to(util.TOPDIR)}: "
+            "missing file header (copyright+licensing)",
+            file=sys.stderr,
+        )
+        return False
+
+    # Skip the shebang.
+    if header_lines[0].startswith("#!"):
+        header_lines.pop(0)
+
+    # If this file is imported into the tree, then leave it be.
+    if header_lines[0] == "# DO NOT EDIT THIS FILE":
+        return True
+
+    header = "".join(f"{x}\n" for x in header_lines)
+    if not _FILE_HEADER_RE.match(header):
+        print(
+            f"error: {path.relative_to(util.TOPDIR)}: "
+            "file header incorrectly formatted",
+            file=sys.stderr,
+        )
+        print(
+            "".join(f"> {x}\n" for x in header_lines), end="", file=sys.stderr
+        )
+        return False
+
+    return True
+
+
+def check_path(opts: argparse.Namespace, path: Path) -> bool:
+    """Check a single path."""
+    data = path.read_text(encoding="utf-8")
+    lines = data.splitlines()
+    # NB: Use list comprehension and not a generator so we run all the checks.
+    return all(
+        [
+            check_license(path, lines),
+        ]
+    )
+
+
+def check_paths(opts: argparse.Namespace, paths: list[Path]) -> bool:
+    """Check all the paths."""
+    # NB: Use list comprehension and not a generator so we check all paths.
+    return all([check_path(opts, x) for x in paths])
+
+
+def find_files(opts: argparse.Namespace) -> list[Path]:
+    """Find all the files in the source tree."""
+    result = util.run(
+        opts,
+        ["git", "ls-tree", "-r", "-z", "--name-only", "HEAD"],
+        cwd=util.TOPDIR,
+        capture_output=True,
+        encoding="utf-8",
+    )
+    return [util.TOPDIR / x for x in result.stdout.split("\0")[:-1]]
+
+
+def get_parser() -> argparse.ArgumentParser:
+    """Get a CLI parser."""
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "-n",
+        "--dry-run",
+        dest="dryrun",
+        action="store_true",
+        help="show everything that would be done",
+    )
+    parser.add_argument(
+        "paths",
+        nargs="*",
+        help="the paths to scan",
+    )
+    return parser
+
+
+def main(argv: list[str]) -> int:
+    """The main func!"""
+    parser = get_parser()
+    opts = parser.parse_args(argv)
+
+    paths = opts.paths
+    if not opts.paths:
+        paths = find_files(opts)
+
+    return 0 if check_paths(opts, paths) else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))

release/update-hooks

@@ -0,0 +1,143 @@
+#!/usr/bin/env python3
+# Copyright (C) 2024 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helper tool for updating hooks from their various upstreams."""
+
+import argparse
+import base64
+import json
+from pathlib import Path
+import sys
+from typing import List, Optional
+import urllib.request
+
+
+assert sys.version_info >= (3, 8), "Python 3.8+ required"
+
+
+TOPDIR = Path(__file__).resolve().parent.parent
+HOOKS_DIR = TOPDIR / "hooks"
+
+
+def update_hook_commit_msg() -> None:
+    """Update commit-msg hook from Gerrit."""
+    hook = HOOKS_DIR / "commit-msg"
+    print(
+        f"{hook.name}: Updating from https://gerrit.googlesource.com/gerrit/"
+        "+/HEAD/resources/com/google/gerrit/server/tools/root/hooks/commit-msg"
+    )
+
+    # Get the current commit.
+    url = "https://gerrit.googlesource.com/gerrit/+/HEAD?format=JSON"
+    with urllib.request.urlopen(url) as fp:
+        data = fp.read()
+    # Discard the xss protection.
+    data = data.split(b"\n", 1)[1]
+    data = json.loads(data)
+    commit = data["commit"]
+
+    # Fetch the data for that commit.
+    url = (
+        f"https://gerrit.googlesource.com/gerrit/+/{commit}/"
+        "resources/com/google/gerrit/server/tools/root/hooks/commit-msg"
+    )
+    with urllib.request.urlopen(f"{url}?format=TEXT") as fp:
+        data = fp.read()
+
+    # gitiles base64 encodes text data.
+    data = base64.b64decode(data)
+
+    # Inject header into the hook.
+    lines = data.split(b"\n")
+    lines = (
+        lines[:1]
+        + [
+            b"# DO NOT EDIT THIS FILE",
+            (
+                b"# All updates should be sent upstream: "
+                b"https://gerrit.googlesource.com/gerrit/"
+            ),
+            f"# This is synced from commit: {commit}".encode("utf-8"),
+            b"# DO NOT EDIT THIS FILE",
+        ]
+        + lines[1:]
+    )
+    data = b"\n".join(lines)
+
+    # Update the hook.
+    hook.write_bytes(data)
+    hook.chmod(0o755)
+
+
+def update_hook_pre_auto_gc() -> None:
+    """Update pre-auto-gc hook from git."""
+    hook = HOOKS_DIR / "pre-auto-gc"
+    print(
+        f"{hook.name}: Updating from https://github.com/git/git/"
+        "HEAD/contrib/hooks/pre-auto-gc-battery"
+    )
+
+    # Get the current commit.
+    headers = {
+        "Accept": "application/vnd.github+json",
+        "X-GitHub-Api-Version": "2022-11-28",
+    }
+    url = "https://api.github.com/repos/git/git/git/refs/heads/master"
+    req = urllib.request.Request(url, headers=headers)
+    with urllib.request.urlopen(req) as fp:
+        data = fp.read()
+    data = json.loads(data)
+
+    # Fetch the data for that commit.
+    commit = data["object"]["sha"]
+    url = (
+        f"https://raw.githubusercontent.com/git/git/{commit}/"
+        "contrib/hooks/pre-auto-gc-battery"
+    )
+    with urllib.request.urlopen(url) as fp:
+        data = fp.read()
+
+    # Inject header into the hook.
+    lines = data.split(b"\n")
+    lines = (
+        lines[:1]
+        + [
+            b"# DO NOT EDIT THIS FILE",
+            (
+                b"# All updates should be sent upstream: "
+                b"https://github.com/git/git/"
+            ),
+            f"# This is synced from commit: {commit}".encode("utf-8"),
+            b"# DO NOT EDIT THIS FILE",
+        ]
+        + lines[1:]
+    )
+    data = b"\n".join(lines)
+
+    # Update the hook.
+    hook.write_bytes(data)
+    hook.chmod(0o755)
+
+
+def main(argv: Optional[List[str]] = None) -> Optional[int]:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.parse_args(argv)
+
+    update_hook_commit_msg()
+    update_hook_pre_auto_gc()
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))

release/update-manpages

@@ -22,4 +22,5 @@ import sys
 
 import update_manpages
 
+
 sys.exit(update_manpages.main(sys.argv[1:]))

release/update_manpages.py

@@ -17,18 +17,21 @@
 Most code lives in this module so it can be unittested.
 """
 
-from pathlib import Path
-from functools import partial
 import argparse
+import functools
 import multiprocessing
 import os
+from pathlib import Path
 import re
 import shutil
 import subprocess
 import sys
 import tempfile
+from typing import List
 
-TOPDIR = Path(__file__).resolve().parent.parent
+
+THIS_FILE = Path(__file__).resolve()
+TOPDIR = THIS_FILE.parent.parent
 MANDIR = TOPDIR.joinpath("man")
 
 # Load repo local modules.
@@ -41,9 +44,23 @@ def worker(cmd, **kwargs):
     subprocess.run(cmd, **kwargs)
 
 
-def main(argv):
+def get_parser() -> argparse.ArgumentParser:
+    """Get argument parser."""
     parser = argparse.ArgumentParser(description=__doc__)
-    parser.parse_args(argv)
+    parser.add_argument(
+        "-n",
+        "--check",
+        "--dry-run",
+        action="store_const",
+        const=True,
+        help="Check if changes are necessary; don't actually change files",
+    )
+    return parser
+
+
+def main(argv: List[str]) -> int:
+    parser = get_parser()
+    opts = parser.parse_args(argv)
 
     if not shutil.which("help2man"):
         sys.exit("Please install help2man to continue.")
@@ -112,8 +129,11 @@ def main(argv):
 
         # Run all cmd in parallel, and wait for them to finish.
         with multiprocessing.Pool() as pool:
-            pool.map(partial(worker, cwd=tempdir, check=True), cmdlist)
+            pool.map(
+                functools.partial(worker, cwd=tempdir, check=True), cmdlist
+            )
 
+    ret = 0
     for tmp_path in MANDIR.glob("*.1.tmp"):
         path = tmp_path.parent / tmp_path.stem
         old_data = path.read_text() if path.exists() else ""
@@ -130,7 +150,17 @@ def main(argv):
         )
         new_data = re.sub(r'^(\.TH REPO "1" ")([^"]+)', r"\1", data, flags=re.M)
         if old_data != new_data:
-            path.write_text(data)
+            if opts.check:
+                ret = 1
+                print(
+                    f"{THIS_FILE.name}: {path.name}: "
+                    "man page needs regenerating",
+                    file=sys.stderr,
+                )
+            else:
+                path.write_text(data)
+
+    return ret
 
 
 def replace_regex(data):

release/util.py

@@ -14,8 +14,9 @@
 
 """Random utility code for release tools."""
 
-import os
+from pathlib import Path
 import re
+import shlex
 import subprocess
 import sys
 
@@ -23,8 +24,9 @@ import sys
 assert sys.version_info >= (3, 6), "This module requires Python 3.6+"
 
 
-TOPDIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-HOMEDIR = os.path.expanduser("~")
+THIS_FILE = Path(__file__).resolve()
+TOPDIR = THIS_FILE.parent.parent
+HOMEDIR = Path("~").expanduser()
 
 
 # These are the release keys we sign with.
@@ -35,12 +37,7 @@ KEYID_ECC = "E1F9040D7A3F6DAFAC897CD3D3B95DA243E48A39"
 
 def cmdstr(cmd):
     """Get a nicely quoted shell command."""
-    ret = []
-    for arg in cmd:
-        if not re.match(r"^[a-zA-Z0-9/_.=-]+$", arg):
-            arg = f'"{arg}"'
-        ret.append(arg)
-    return " ".join(ret)
+    return " ".join(shlex.quote(x) for x in cmd)
 
 
 def run(opts, cmd, check=True, **kwargs):
@@ -58,7 +55,7 @@ def run(opts, cmd, check=True, **kwargs):
 def import_release_key(opts):
     """Import the public key of the official release repo signing key."""
     # Extract the key from our repo launcher.
-    launcher = getattr(opts, "launcher", os.path.join(TOPDIR, "repo"))
+    launcher = getattr(opts, "launcher", TOPDIR / "repo")
     print(f'Importing keys from "{launcher}" launcher script')
     with open(launcher, encoding="utf-8") as fp:
         data = fp.read()

repo_logging.py

@@ -0,0 +1,93 @@
+# Copyright (C) 2023 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Logic for printing user-friendly logs in repo."""
+
+import logging
+
+from color import Coloring
+from error import RepoExitError
+
+
+SEPARATOR = "=" * 80
+MAX_PRINT_ERRORS = 5
+
+
+class _ConfigMock:
+    """Default coloring config to use when Logging.config is not set."""
+
+    def __init__(self):
+        self.default_values = {"color.ui": "auto"}
+
+    def GetString(self, x):
+        return self.default_values.get(x, None)
+
+
+class _LogColoring(Coloring):
+    """Coloring outstream for logging."""
+
+    def __init__(self, config):
+        super().__init__(config, "logs")
+        self.error = self.nofmt_colorer("error", fg="red")
+        self.warning = self.nofmt_colorer("warn", fg="yellow")
+        self.levelMap = {
+            "WARNING": self.warning,
+            "ERROR": self.error,
+        }
+
+
+class _LogColoringFormatter(logging.Formatter):
+    """Coloring formatter for logging."""
+
+    def __init__(self, config=None, *args, **kwargs):
+        self.config = config if config else _ConfigMock()
+        self.colorer = _LogColoring(self.config)
+        super().__init__(*args, **kwargs)
+
+    def format(self, record):
+        """Formats |record| with color."""
+        msg = super().format(record)
+        colorer = self.colorer.levelMap.get(record.levelname)
+        return msg if not colorer else colorer(msg)
+
+
+class RepoLogger(logging.Logger):
+    """Repo Logging Module."""
+
+    def __init__(self, name: str, config=None, **kwargs):
+        super().__init__(name, **kwargs)
+        handler = logging.StreamHandler()
+        handler.setFormatter(_LogColoringFormatter(config))
+        self.addHandler(handler)
+
+    def log_aggregated_errors(self, err: RepoExitError):
+        """Print all aggregated logs."""
+        self.error(SEPARATOR)
+
+        if not err.aggregate_errors:
+            self.error("Repo command failed: %s", type(err).__name__)
+            self.error("\t%s", str(err))
+            return
+
+        self.error(
+            "Repo command failed due to the following `%s` errors:",
+            type(err).__name__,
+        )
+        self.error(
+            "\n".join(str(e) for e in err.aggregate_errors[:MAX_PRINT_ERRORS])
+        )
+
+        diff = len(err.aggregate_errors) - MAX_PRINT_ERRORS
+        if diff > 0:
+            self.error("+%d additional errors...", diff)

repo_trace.py

@@ -20,14 +20,15 @@ Temporary: Tracing is always on. Set `REPO_TRACE=0` to turn off.
 To also include trace outputs in stderr do `repo --trace_to_stderr ...`
 """
 
-import sys
+import contextlib
 import os
-import time
+import sys
 import tempfile
-from contextlib import ContextDecorator
+import time
 
 import platform_utils
 
+
 # Env var to implicitly turn on tracing.
 REPO_TRACE = "REPO_TRACE"
 
@@ -68,7 +69,7 @@ def _SetTraceFile(quiet):
     _TRACE_FILE = _GetTraceFile(quiet)
 
 
-class Trace(ContextDecorator):
+class Trace(contextlib.ContextDecorator):
     """Used to capture and save git traces."""
 
     def _time(self):
@@ -141,7 +142,7 @@ def _GetTraceFile(quiet):
 def _ClearOldTraces():
     """Clear the oldest commands if trace file is too big."""
     try:
-        with open(_TRACE_FILE, "r", errors="ignore") as f:
+        with open(_TRACE_FILE, errors="ignore") as f:
             if os.path.getsize(f.name) / (1024 * 1024) <= _MAX_SIZE:
                 return
             trace_lines = f.readlines()

requirements.json

@@ -46,12 +46,14 @@
 
   # Supported git versions.
   #
-  # git-1.7.2 is in Debian Squeeze.
-  # git-1.7.9 is in Ubuntu Precise.
   # git-1.9.1 is in Ubuntu Trusty.
-  # git-1.7.10 is in Debian Wheezy.
+  # git-2.1.4 is in Debian Jessie.
+  # git-2.7.4 is in Ubuntu Xenial.
+  # git-2.11.0 is in Debian Stretch.
+  # git-2.17.0 is in Ubuntu Bionic.
+  # git-2.20.1 is in Debian Buster.
   "git": {
-    "hard": [1, 7, 2],
-    "soft": [1, 9, 1]
+    "hard": [1, 9, 1],
+    "soft": [2, 7, 4]
   }
 }

run_tests

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-# Copyright 2019 The Android Open Source Project
+# Copyright (C) 2019 The Android Open Source Project
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,37 +15,148 @@
 
 """Wrapper to run linters and pytest with the right settings."""
 
+import functools
 import os
+import shlex
+import shutil
 import subprocess
 import sys
-import pytest
+from typing import List
 
 
 ROOT_DIR = os.path.dirname(os.path.realpath(__file__))
 
 
+def log_cmd(cmd: str, argv: list[str]) -> None:
+    """Log a debug message to make history easier to track."""
+    print("+", cmd, shlex.join(argv), file=sys.stderr)
+
+
+@functools.lru_cache()
+def is_ci() -> bool:
+    """Whether we're running in our CI system."""
+    return os.getenv("LUCI_CQ") == "yes"
+
+
+def run_pytest(argv: List[str]) -> int:
+    """Returns the exit code from pytest."""
+    if is_ci():
+        argv = ["-m", "not skip_cq"] + argv
+
+    log_cmd("pytest", argv)
+    return subprocess.run(
+        [sys.executable, "-m", "pytest"] + argv,
+        check=False,
+        cwd=ROOT_DIR,
+    ).returncode
+
+
+def run_pytest_py38(argv: List[str]) -> int:
+    """Returns the exit code from pytest under Python 3.8."""
+    if is_ci():
+        argv = ["-m", "not skip_cq"] + argv
+
+    log_cmd("[vpython 3.8] pytest", argv)
+    try:
+        return subprocess.run(
+            [
+                "vpython3",
+                "-vpython-spec",
+                "run_tests.vpython3.8",
+                "-m",
+                "pytest",
+            ]
+            + argv,
+            check=False,
+            cwd=ROOT_DIR,
+        ).returncode
+    except FileNotFoundError:
+        # Skip if the user doesn't have vpython from depot_tools.
+        return 0
+
+
 def run_black():
     """Returns the exit code from black."""
+    # Black by default only matches .py files.  We have to list standalone
+    # scripts manually.
+    extra_programs = [
+        "repo",
+        "run_tests",
+        "release/update-hooks",
+        "release/update-manpages",
+    ]
+    argv = ["--diff", "--check", ROOT_DIR] + extra_programs
+    log_cmd("black", argv)
     return subprocess.run(
-        [sys.executable, "-m", "black", "--check", ROOT_DIR], check=False
+        [sys.executable, "-m", "black"] + argv,
+        check=False,
+        cwd=ROOT_DIR,
     ).returncode
 
 
 def run_flake8():
     """Returns the exit code from flake8."""
+    argv = [ROOT_DIR]
+    log_cmd("flake8", argv)
     return subprocess.run(
-        [sys.executable, "-m", "flake8", ROOT_DIR], check=False
+        [sys.executable, "-m", "flake8"] + argv,
+        check=False,
+        cwd=ROOT_DIR,
+    ).returncode
+
+
+def run_isort():
+    """Returns the exit code from isort."""
+    argv = ["--check", ROOT_DIR]
+    log_cmd("isort", argv)
+    return subprocess.run(
+        [sys.executable, "-m", "isort"] + argv,
+        check=False,
+        cwd=ROOT_DIR,
+    ).returncode
+
+
+def run_check_metadata():
+    """Returns the exit code from check-metadata."""
+    argv = []
+    log_cmd("release/check-metadata.py", argv)
+    return subprocess.run(
+        [sys.executable, "release/check-metadata.py"] + argv,
+        check=False,
+        cwd=ROOT_DIR,
+    ).returncode
+
+
+def run_update_manpages() -> int:
+    """Returns the exit code from release/update-manpages."""
+    # Allow this to fail on CI, but not local devs.
+    if is_ci() and not shutil.which("help2man"):
+        print("update-manpages: help2man not found; skipping test")
+        return 0
+
+    argv = ["--check"]
+    log_cmd("release/update-manpages", argv)
+    return subprocess.run(
+        [sys.executable, "release/update-manpages"] + argv,
+        check=False,
+        cwd=ROOT_DIR,
     ).returncode
 
 
 def main(argv):
     """The main entry."""
     checks = (
-        lambda: pytest.main(argv),
+        functools.partial(run_pytest, argv),
+        functools.partial(run_pytest_py38, argv),
         run_black,
         run_flake8,
+        run_isort,
+        run_check_metadata,
+        run_update_manpages,
     )
-    return 0 if all(not c() for c in checks) else 1
+    # Run all the tests all the time to get full feedback.  Don't exit on the
+    # first error as that makes it more difficult to iterate in the CQ.
+    return 1 if sum(c() for c in checks) else 0
 
 
 if __name__ == "__main__":

run_tests.vpython3

@@ -5,97 +5,92 @@
 # List of available wheels:
 #   https://chromium.googlesource.com/infra/infra/+/main/infra/tools/dockerbuild/wheels.md
 
-python_version: "3.8"
+python_version: "3.11"
 
 wheel: <
   name: "infra/python/wheels/pytest-py3"
-  version: "version:6.2.2"
+  version: "version:8.3.4"
 >
 
-# Required by pytest==6.2.2
+# Required by pytest==8.3.4
 wheel: <
   name: "infra/python/wheels/py-py2_py3"
-  version: "version:1.10.0"
+  version: "version:1.11.0"
 >
 
-# Required by pytest==6.2.2
+# Required by pytest==8.3.4
 wheel: <
   name: "infra/python/wheels/iniconfig-py3"
   version: "version:1.1.1"
 >
 
-# Required by pytest==6.2.2
+# Required by pytest==8.3.4
 wheel: <
   name: "infra/python/wheels/packaging-py3"
   version: "version:23.0"
 >
 
-# Required by pytest==6.2.2
+# Required by pytest==8.3.4
 wheel: <
   name: "infra/python/wheels/pluggy-py3"
-  version: "version:0.13.1"
+  version: "version:1.5.0"
 >
 
-# Required by pytest==6.2.2
+# Required by pytest==8.3.4
 wheel: <
   name: "infra/python/wheels/toml-py3"
   version: "version:0.10.1"
 >
 
-# Required by pytest==6.2.2
+# Required by pytest==8.3.4
 wheel: <
   name: "infra/python/wheels/pyparsing-py3"
   version: "version:3.0.7"
 >
 
-# Required by pytest==6.2.2
+# Required by pytest==8.3.4
 wheel: <
   name: "infra/python/wheels/attrs-py2_py3"
   version: "version:21.4.0"
 >
 
-# Required by packaging==16.8
-wheel: <
-  name: "infra/python/wheels/six-py2_py3"
-  version: "version:1.16.0"
->
-
+# NB: Keep in sync with constraints.txt.
 wheel: <
   name: "infra/python/wheels/black-py3"
-  version: "version:23.1.0"
+  version: "version:25.1.0"
 >
 
-# Required by black==23.1.0
+# Required by black==25.1.0
 wheel: <
   name: "infra/python/wheels/mypy-extensions-py3"
   version: "version:0.4.3"
 >
 
-# Required by black==23.1.0
+# Required by black==25.1.0
 wheel: <
   name: "infra/python/wheels/tomli-py3"
   version: "version:2.0.1"
 >
 
-# Required by black==23.1.0
+# Required by black==25.1.0
 wheel: <
   name: "infra/python/wheels/platformdirs-py3"
   version: "version:2.5.2"
 >
 
-# Required by black==23.1.0
+# Required by black==25.1.0
 wheel: <
   name: "infra/python/wheels/pathspec-py3"
   version: "version:0.9.0"
 >
 
-# Required by black==23.1.0
+# Required by black==25.1.0
 wheel: <
   name: "infra/python/wheels/typing-extensions-py3"
   version: "version:4.3.0"
 >
 
-# Required by black==23.1.0
+# Required by black==25.1.0
 wheel: <
   name: "infra/python/wheels/click-py3"
   version: "version:8.0.3"
@@ -123,3 +118,8 @@ wheel: <
   name: "infra/python/wheels/pycodestyle-py2_py3"
   version: "version:2.10.0"
 >
+
+wheel: <
+  name: "infra/python/wheels/isort-py3"
+  version: "version:5.10.1"
+>

run_tests.vpython3.8

@@ -0,0 +1,67 @@
+# This is a vpython "spec" file.
+#
+# Read more about `vpython` and how to modify this file here:
+#   https://chromium.googlesource.com/infra/infra/+/main/doc/users/vpython.md
+# List of available wheels:
+#   https://chromium.googlesource.com/infra/infra/+/main/infra/tools/dockerbuild/wheels.md
+
+python_version: "3.8"
+
+wheel: <
+  name: "infra/python/wheels/pytest-py3"
+  version: "version:8.3.4"
+>
+
+# Required by pytest==8.3.4
+wheel: <
+  name: "infra/python/wheels/py-py2_py3"
+  version: "version:1.11.0"
+>
+
+# Required by pytest==8.3.4
+wheel: <
+  name: "infra/python/wheels/iniconfig-py3"
+  version: "version:1.1.1"
+>
+
+# Required by pytest==8.3.4
+wheel: <
+  name: "infra/python/wheels/packaging-py3"
+  version: "version:23.0"
+>
+
+# Required by pytest==8.3.4
+wheel: <
+  name: "infra/python/wheels/pluggy-py3"
+  version: "version:1.5.0"
+>
+
+# Required by pytest==8.3.4
+wheel: <
+  name: "infra/python/wheels/toml-py3"
+  version: "version:0.10.1"
+>
+
+# Required by pytest==8.3.4
+wheel: <
+  name: "infra/python/wheels/tomli-py3"
+  version: "version:2.1.0"
+>
+
+# Required by pytest==8.3.4
+wheel: <
+  name: "infra/python/wheels/pyparsing-py3"
+  version: "version:3.0.7"
+>
+
+# Required by pytest==8.3.4
+wheel: <
+  name: "infra/python/wheels/attrs-py2_py3"
+  version: "version:21.4.0"
+>
+
+# Required by pytest==8.3.4
+wheel: <
+  name: "infra/python/wheels/exceptiongroup-py3"
+  version: "version:1.1.2"
+>

setup.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
-# Copyright 2019 The Android Open Source Project
+# Copyright (C) 2019 The Android Open Source Project
 #
-# Licensed under the Apache License, Version 2.0 (the 'License");
+# Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
@@ -16,6 +16,7 @@
 """Python packaging for repo."""
 
 import os
+
 import setuptools
 
 
@@ -40,7 +41,7 @@ setuptools.setup(
     long_description_content_type="text/plain",
     url="https://gerrit.googlesource.com/git-repo/",
     project_urls={
-        "Bug Tracker": "https://bugs.chromium.org/p/gerrit/issues/list?q=component:Applications%3Erepo",  # noqa: E501
+        "Bug Tracker": "https://issues.gerritcodereview.com/issues?q=is:open%20componentid:1370071",  # noqa: E501
     },
     # https://pypi.org/classifiers/
     classifiers=[

ssh.py

@@ -24,6 +24,7 @@ import sys
 import tempfile
 import time
 
+from git_command import git
 import platform_utils
 from repo_trace import Trace
 
@@ -42,7 +43,7 @@ def _parse_ssh_version(ver_str=None):
     """parse a ssh version string into a tuple"""
     if ver_str is None:
         ver_str = _run_ssh_version()
-    m = re.match(r"^OpenSSH_([0-9.]+)(p[0-9]+)?\s", ver_str)
+    m = re.match(r"^OpenSSH_([0-9.]+)(p[0-9]+)?[\s,]", ver_str)
     if m:
         return tuple(int(x) for x in m.group(1).split("."))
     else:
@@ -57,8 +58,12 @@ def version():
     except FileNotFoundError:
         print("fatal: ssh not installed", file=sys.stderr)
         sys.exit(1)
-    except subprocess.CalledProcessError:
-        print("fatal: unable to detect ssh version", file=sys.stderr)
+    except subprocess.CalledProcessError as e:
+        print(
+            "fatal: unable to detect ssh version"
+            f" (code={e.returncode}, output={e.stdout})",
+            file=sys.stderr,
+        )
         sys.exit(1)
 
 
@@ -165,7 +170,7 @@ class ProxyManager:
         # Check to see whether we already think that the master is running; if
         # we think it's already running, return right away.
         if port is not None:
-            key = "%s:%s" % (host, port)
+            key = f"{host}:{port}"
         else:
             key = host
 
@@ -207,7 +212,33 @@ class ProxyManager:
                 # and print to the log there.
                 pass
 
-        command = command_base[:1] + ["-M", "-N"] + command_base[1:]
+        # Git protocol V2 is a new feature in git 2.18.0, made default in
+        # git 2.26.0
+        # It is faster and more efficient than V1.
+        # To enable it when using SSH, the environment variable GIT_PROTOCOL
+        # must be set in the SSH side channel when establishing the connection
+        # to the git server.
+        # See https://git-scm.com/docs/protocol-v2#_ssh_and_file_transport
+        # Normally git does this by itself. But here, where the SSH connection
+        # is established manually over ControlMaster via the repo-tool, it must
+        # be passed in explicitly instead.
+        # Based on https://git-scm.com/docs/gitprotocol-pack#_extra_parameters,
+        # GIT_PROTOCOL is considered an "Extra Parameter" and must be ignored
+        # by servers that do not understand it. This means that it is safe to
+        # set it even when connecting to older servers.
+        # It should also be safe to set the environment variable for older
+        # local git versions, since it is only part of the ssh side channel.
+        git_protocol_version = _get_git_protocol_version()
+        ssh_git_protocol_args = [
+            "-o",
+            f"SetEnv GIT_PROTOCOL=version={git_protocol_version}",
+        ]
+
+        command = (
+            command_base[:1]
+            + ["-M", "-N", *ssh_git_protocol_args]
+            + command_base[1:]
+        )
         p = None
         try:
             with Trace("Call to ssh: %s", " ".join(command)):
@@ -289,3 +320,32 @@ class ProxyManager:
                 tempfile.mkdtemp("", "ssh-", tmp_dir), "master-" + tokens
             )
         return self._sock_path
+
+
+@functools.lru_cache(maxsize=1)
+def _get_git_protocol_version() -> str:
+    """Return the git protocol version.
+
+    The version is found by first reading the global git config.
+    If no git config for protocol version exists, try to deduce the default
+    protocol version based on the git version.
+
+    See https://git-scm.com/docs/gitprotocol-v2 for details.
+    """
+    try:
+        return subprocess.check_output(
+            ["git", "config", "--get", "--global", "protocol.version"],
+            encoding="utf-8",
+            stderr=subprocess.PIPE,
+        ).strip()
+    except subprocess.CalledProcessError as e:
+        if e.returncode == 1:
+            # Exit code 1 means that the git config key was not found.
+            # Try to imitate the defaults that git would have used.
+            git_version = git.version_tuple()
+            if git_version >= (2, 26, 0):
+                # Since git version 2.26, protocol v2 is the default.
+                return "2"
+            return "1"
+        # Other exit codes indicate error with reading the config.
+        raise

subcmds/__init__.py

@@ -14,8 +14,10 @@
 
 import os
 
+
 # A mapping of the subcommand name to the class that implements it.
 all_commands = {}
+all_modules = []
 
 my_dir = os.path.dirname(__file__)
 for py in os.listdir(my_dir):
@@ -35,13 +37,12 @@ for py in os.listdir(my_dir):
         try:
             cmd = getattr(mod, clsn)
         except AttributeError:
-            raise SyntaxError(
-                "%s/%s does not define class %s" % (__name__, py, clsn)
-            )
+            raise SyntaxError(f"{__name__}/{py} does not define class {clsn}")
 
         name = name.replace("_", "-")
         cmd.NAME = name
         all_commands[name] = cmd
+        all_modules.append(mod)
 
 # Add 'branch' as an alias for 'branches'.
 all_commands["branch"] = all_commands["branches"]

subcmds/abandon.py

@@ -12,14 +12,24 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from collections import defaultdict
+import collections
 import functools
 import itertools
-import sys
 
-from command import Command, DEFAULT_LOCAL_JOBS
+from command import Command
+from command import DEFAULT_LOCAL_JOBS
+from error import RepoError
+from error import RepoExitError
 from git_command import git
 from progress import Progress
+from repo_logging import RepoLogger
+
+
+logger = RepoLogger(__file__)
+
+
+class AbandonError(RepoExitError):
+    """Exit error when abandon command fails."""
 
 
 class Abandon(Command):
@@ -38,7 +48,6 @@ It is equivalent to "git branch -D <branchname>".
     def _Options(self, p):
         p.add_option(
             "--all",
-            dest="all",
             action="store_true",
             help="delete all branches in all projects",
         )
@@ -60,47 +69,62 @@ It is equivalent to "git branch -D <branchname>".
         else:
             args.insert(0, "'All local branches'")
 
-    def _ExecuteOne(self, all_branches, nb, project):
+    @classmethod
+    def _ExecuteOne(cls, all_branches, nb, project_idx):
         """Abandon one project."""
+        project = cls.get_parallel_context()["projects"][project_idx]
         if all_branches:
             branches = project.GetBranches()
         else:
             branches = nb
 
         ret = {}
+        errors = []
         for name in branches:
-            status = project.AbandonBranch(name)
+            status = None
+            try:
+                status = project.AbandonBranch(name)
+            except RepoError as e:
+                status = False
+                errors.append(e)
             if status is not None:
                 ret[name] = status
-        return (ret, project)
+
+        return (ret, project_idx, errors)
 
     def Execute(self, opt, args):
         nb = args[0].split()
-        err = defaultdict(list)
-        success = defaultdict(list)
+        err = collections.defaultdict(list)
+        success = collections.defaultdict(list)
+        aggregate_errors = []
         all_projects = self.GetProjects(
             args[1:], all_manifests=not opt.this_manifest_only
         )
         _RelPath = lambda p: p.RelPath(local=opt.this_manifest_only)
 
         def _ProcessResults(_pool, pm, states):
-            for results, project in states:
+            for results, project_idx, errors in states:
+                project = all_projects[project_idx]
                 for branch, status in results.items():
                     if status:
                         success[branch].append(project)
                     else:
                         err[branch].append(project)
-                pm.update()
+                aggregate_errors.extend(errors)
+                pm.update(msg="")
 
-        self.ExecuteInParallel(
-            opt.jobs,
-            functools.partial(self._ExecuteOne, opt.all, nb),
-            all_projects,
-            callback=_ProcessResults,
-            output=Progress(
-                "Abandon %s" % (nb,), len(all_projects), quiet=opt.quiet
-            ),
-        )
+        with self.ParallelContext():
+            self.get_parallel_context()["projects"] = all_projects
+            self.ExecuteInParallel(
+                opt.jobs,
+                functools.partial(self._ExecuteOne, opt.all, nb),
+                range(len(all_projects)),
+                callback=_ProcessResults,
+                output=Progress(
+                    f"Abandon {nb}", len(all_projects), quiet=opt.quiet
+                ),
+                chunksize=1,
+            )
 
         width = max(
             itertools.chain(
@@ -110,19 +134,13 @@ It is equivalent to "git branch -D <branchname>".
         if err:
             for br in err.keys():
                 err_msg = "error: cannot abandon %s" % br
-                print(err_msg, file=sys.stderr)
+                logger.error(err_msg)
                 for proj in err[br]:
-                    print(
-                        " " * len(err_msg) + " | %s" % _RelPath(proj),
-                        file=sys.stderr,
-                    )
-            sys.exit(1)
+                    logger.error(" " * len(err_msg) + " | %s", _RelPath(proj))
+            raise AbandonError(aggregate_errors=aggregate_errors)
         elif not success:
-            print(
-                "error: no project has local branch(es) : %s" % nb,
-                file=sys.stderr,
-            )
-            sys.exit(1)
+            logger.error("error: no project has local branch(es) : %s", nb)
+            raise AbandonError(aggregate_errors=aggregate_errors)
         else:
             # Everything below here is displaying status.
             if opt.quiet:
@@ -139,4 +157,4 @@ It is equivalent to "git branch -D <branchname>".
                             _RelPath(p) for p in success[br]
                         )
                     )
-                print("%s%s| %s\n" % (br, " " * (width - len(br)), result))
+                print(f"{br}{' ' * (width - len(br))}| {result}\n")

subcmds/branches.py

@@ -16,7 +16,8 @@ import itertools
 import sys
 
 from color import Coloring
-from command import Command, DEFAULT_LOCAL_JOBS
+from command import Command
+from command import DEFAULT_LOCAL_JOBS
 
 
 class BranchColoring(Coloring):
@@ -27,7 +28,7 @@ class BranchColoring(Coloring):
         self.notinproject = self.printer("notinproject", fg="red")
 
 
-class BranchInfo(object):
+class BranchInfo:
     def __init__(self, name):
         self.name = name
         self.current = 0
@@ -97,6 +98,22 @@ is shown, then the branch appears in all projects.
 """
     PARALLEL_JOBS = DEFAULT_LOCAL_JOBS
 
+    @classmethod
+    def _ExpandProjectToBranches(cls, project_idx):
+        """Expands a project into a list of branch names & associated info.
+
+        Args:
+            project_idx: project.Project index
+
+        Returns:
+            List[Tuple[str, git_config.Branch, int]]
+        """
+        branches = []
+        project = cls.get_parallel_context()["projects"][project_idx]
+        for name, b in project.GetBranches().items():
+            branches.append((name, b, project_idx))
+        return branches
+
     def Execute(self, opt, args):
         projects = self.GetProjects(
             args, all_manifests=not opt.this_manifest_only
@@ -106,17 +123,20 @@ is shown, then the branch appears in all projects.
         project_cnt = len(projects)
 
         def _ProcessResults(_pool, _output, results):
-            for name, b in itertools.chain.from_iterable(results):
+            for name, b, project_idx in itertools.chain.from_iterable(results):
+                b.project = projects[project_idx]
                 if name not in all_branches:
                     all_branches[name] = BranchInfo(name)
                 all_branches[name].add(b)
 
-        self.ExecuteInParallel(
-            opt.jobs,
-            expand_project_to_branches,
-            projects,
-            callback=_ProcessResults,
-        )
+        with self.ParallelContext():
+            self.get_parallel_context()["projects"] = projects
+            self.ExecuteInParallel(
+                opt.jobs,
+                self._ExpandProjectToBranches,
+                range(len(projects)),
+                callback=_ProcessResults,
+            )
 
         names = sorted(all_branches)
 
@@ -147,7 +167,10 @@ is shown, then the branch appears in all projects.
             else:
                 published = " "
 
-            hdr("%c%c %-*s" % (current, published, width, name))
+            # A branch name can contain a percent sign, so we need to escape it.
+            # Escape after f-string formatting to properly account for leading
+            # spaces.
+            hdr(f"{current}{published} {name:{width}}".replace("%", "%%"))
             out.write(" |")
 
             _RelPath = lambda p: p.RelPath(local=opt.this_manifest_only)
@@ -173,7 +196,7 @@ is shown, then the branch appears in all projects.
                         if _RelPath(p) not in have:
                             paths.append(_RelPath(p))
 
-                s = " %s %s" % (in_type, ", ".join(paths))
+                s = f" {in_type} {', '.join(paths)}"
                 if not i.IsSplitCurrent and (width + 7 + len(s) < 80):
                     fmt = out.current if i.IsCurrent else fmt
                     fmt(s)
@@ -190,19 +213,3 @@ is shown, then the branch appears in all projects.
             else:
                 out.write(" in all projects")
             out.nl()
-
-
-def expand_project_to_branches(project):
-    """Expands a project into a list of branch names & associated information.
-
-    Args:
-        project: project.Project
-
-    Returns:
-        List[Tuple[str, git_config.Branch]]
-    """
-    branches = []
-    for name, b in project.GetBranches().items():
-        b.project = project
-        branches.append((name, b))
-    return branches

subcmds/checkout.py

@@ -13,10 +13,32 @@
 # limitations under the License.
 
 import functools
-import sys
+from typing import NamedTuple
 
-from command import Command, DEFAULT_LOCAL_JOBS
+from command import Command
+from command import DEFAULT_LOCAL_JOBS
+from error import GitError
+from error import RepoExitError
 from progress import Progress
+from repo_logging import RepoLogger
+
+
+logger = RepoLogger(__file__)
+
+
+class CheckoutBranchResult(NamedTuple):
+    # Whether the Project is on the branch (i.e. branch exists and no errors)
+    result: bool
+    project_idx: int
+    error: Exception
+
+
+class CheckoutCommandError(RepoExitError):
+    """Exception thrown when checkout command fails."""
+
+
+class MissingBranchError(RepoExitError):
+    """Exception thrown when no project has specified branch."""
 
 
 class Checkout(Command):
@@ -39,44 +61,54 @@ The command is equivalent to:
         if not args:
             self.Usage()
 
-    def _ExecuteOne(self, nb, project):
+    @classmethod
+    def _ExecuteOne(cls, nb, project_idx):
         """Checkout one project."""
-        return (project.CheckoutBranch(nb), project)
+        error = None
+        result = None
+        project = cls.get_parallel_context()["projects"][project_idx]
+        try:
+            result = project.CheckoutBranch(nb)
+        except GitError as e:
+            error = e
+        return CheckoutBranchResult(result, project_idx, error)
 
     def Execute(self, opt, args):
         nb = args[0]
         err = []
+        err_projects = []
         success = []
         all_projects = self.GetProjects(
             args[1:], all_manifests=not opt.this_manifest_only
         )
 
         def _ProcessResults(_pool, pm, results):
-            for status, project in results:
-                if status is not None:
-                    if status:
-                        success.append(project)
-                    else:
-                        err.append(project)
-                pm.update()
+            for result in results:
+                project = all_projects[result.project_idx]
+                if result.error is not None:
+                    err.append(result.error)
+                    err_projects.append(project)
+                elif result.result:
+                    success.append(project)
+                pm.update(msg="")
 
-        self.ExecuteInParallel(
-            opt.jobs,
-            functools.partial(self._ExecuteOne, nb),
-            all_projects,
-            callback=_ProcessResults,
-            output=Progress(
-                "Checkout %s" % (nb,), len(all_projects), quiet=opt.quiet
-            ),
-        )
+        with self.ParallelContext():
+            self.get_parallel_context()["projects"] = all_projects
+            self.ExecuteInParallel(
+                opt.jobs,
+                functools.partial(self._ExecuteOne, nb),
+                range(len(all_projects)),
+                callback=_ProcessResults,
+                output=Progress(
+                    f"Checkout {nb}", len(all_projects), quiet=opt.quiet
+                ),
+            )
 
-        if err:
-            for p in err:
-                print(
-                    "error: %s/: cannot checkout %s" % (p.relpath, nb),
-                    file=sys.stderr,
-                )
-            sys.exit(1)
+        if err_projects:
+            for p in err_projects:
+                logger.error("error: %s/: cannot checkout %s", p.relpath, nb)
+            raise CheckoutCommandError(aggregate_errors=err)
         elif not success:
-            print("error: no project has branch %s" % nb, file=sys.stderr)
-            sys.exit(1)
+            msg = f"error: no project has branch {nb}"
+            logger.error(msg)
+            raise MissingBranchError(msg)

subcmds/cherry_pick.py

@@ -14,10 +14,15 @@
 
 import re
 import sys
+
 from command import Command
+from error import GitError
 from git_command import GitCommand
+from repo_logging import RepoLogger
+
 
 CHANGE_ID_RE = re.compile(r"^\s*Change-Id: I([0-9a-f]{40})\s*$")
+logger = RepoLogger(__file__)
 
 
 class CherryPick(Command):
@@ -44,18 +49,29 @@ change id will be added.
             ["rev-parse", "--verify", reference],
             capture_stdout=True,
             capture_stderr=True,
+            verify_command=True,
         )
-        if p.Wait() != 0:
-            print(p.stderr, file=sys.stderr)
-            sys.exit(1)
+        try:
+            p.Wait()
+        except GitError:
+            logger.error(p.stderr)
+            raise
+
         sha1 = p.stdout.strip()
 
-        p = GitCommand(None, ["cat-file", "commit", sha1], capture_stdout=True)
-        if p.Wait() != 0:
-            print(
-                "error: Failed to retrieve old commit message", file=sys.stderr
-            )
-            sys.exit(1)
+        p = GitCommand(
+            None,
+            ["cat-file", "commit", sha1],
+            capture_stdout=True,
+            verify_command=True,
+        )
+
+        try:
+            p.Wait()
+        except GitError:
+            logger.error("error: Failed to retrieve old commit message")
+            raise
+
         old_msg = self._StripHeader(p.stdout)
 
         p = GitCommand(
@@ -63,37 +79,43 @@ change id will be added.
             ["cherry-pick", sha1],
             capture_stdout=True,
             capture_stderr=True,
+            verify_command=True,
         )
-        status = p.Wait()
+
+        try:
+            p.Wait()
+        except GitError as e:
+            logger.error(e)
+            logger.warning(
+                "NOTE: When committing (please see above) and editing the "
+                "commit message, please remove the old Change-Id-line and "
+                "add:\n%s",
+                self._GetReference(sha1),
+            )
+            raise
 
         if p.stdout:
             print(p.stdout.strip(), file=sys.stdout)
         if p.stderr:
             print(p.stderr.strip(), file=sys.stderr)
 
-        if status == 0:
-            # The cherry-pick was applied correctly. We just need to edit the
-            # commit message.
-            new_msg = self._Reformat(old_msg, sha1)
+        # The cherry-pick was applied correctly. We just need to edit
+        # the commit message.
+        new_msg = self._Reformat(old_msg, sha1)
 
-            p = GitCommand(
-                None,
-                ["commit", "--amend", "-F", "-"],
-                input=new_msg,
-                capture_stdout=True,
-                capture_stderr=True,
-            )
-            if p.Wait() != 0:
-                print("error: Failed to update commit message", file=sys.stderr)
-                sys.exit(1)
-
-        else:
-            print(
-                "NOTE: When committing (please see above) and editing the "
-                "commit message, please remove the old Change-Id-line and add:"
-            )
-            print(self._GetReference(sha1), file=sys.stderr)
-            print(file=sys.stderr)
+        p = GitCommand(
+            None,
+            ["commit", "--amend", "-F", "-"],
+            input=new_msg,
+            capture_stdout=True,
+            capture_stderr=True,
+            verify_command=True,
+        )
+        try:
+            p.Wait()
+        except GitError:
+            logger.error("error: Failed to update commit message")
+            raise
 
     def _IsChangeId(self, line):
         return CHANGE_ID_RE.match(line)

subcmds/diff.py

@@ -15,7 +15,8 @@
 import functools
 import io
 
-from command import DEFAULT_LOCAL_JOBS, PagedCommand
+from command import DEFAULT_LOCAL_JOBS
+from command import PagedCommand
 
 
 class Diff(PagedCommand):
@@ -34,12 +35,12 @@ to the Unix 'patch' command.
         p.add_option(
             "-u",
             "--absolute",
-            dest="absolute",
             action="store_true",
             help="paths are relative to the repository root",
         )
 
-    def _ExecuteOne(self, absolute, local, project):
+    @classmethod
+    def _ExecuteOne(cls, absolute, local, project_idx):
         """Obtains the diff for a specific project.
 
         Args:
@@ -47,12 +48,13 @@ to the Unix 'patch' command.
             local: a boolean, if True, the path is relative to the local
                 (sub)manifest.  If false, the path is relative to the outermost
                 manifest.
-            project: Project to get status of.
+            project_idx: Project index to get status of.
 
         Returns:
             The status of the project.
         """
         buf = io.StringIO()
+        project = cls.get_parallel_context()["projects"][project_idx]
         ret = project.PrintWorkTreeDiff(absolute, output_redir=buf, local=local)
         return (ret, buf.getvalue())
 
@@ -70,12 +72,15 @@ to the Unix 'patch' command.
                     ret = 1
             return ret
 
-        return self.ExecuteInParallel(
-            opt.jobs,
-            functools.partial(
-                self._ExecuteOne, opt.absolute, opt.this_manifest_only
-            ),
-            all_projects,
-            callback=_ProcessResults,
-            ordered=True,
-        )
+        with self.ParallelContext():
+            self.get_parallel_context()["projects"] = all_projects
+            return self.ExecuteInParallel(
+                opt.jobs,
+                functools.partial(
+                    self._ExecuteOne, opt.absolute, opt.this_manifest_only
+                ),
+                range(len(all_projects)),
+                callback=_ProcessResults,
+                ordered=True,
+                chunksize=1,
+            )

subcmds/diffmanifests.py

@@ -67,7 +67,9 @@ synced and their revisions won't be found.
 
     def _Options(self, p):
         p.add_option(
-            "--raw", dest="raw", action="store_true", help="display raw diff"
+            "--raw",
+            action="store_true",
+            help="display raw diff",
         )
         p.add_option(
             "--no-color",
@@ -78,7 +80,6 @@ synced and their revisions won't be found.
         )
         p.add_option(
             "--pretty-format",
-            dest="pretty_format",
             action="store",
             metavar="<FORMAT>",
             help="print the log using a custom git pretty format string",
@@ -87,25 +88,17 @@ synced and their revisions won't be found.
     def _printRawDiff(self, diff, pretty_format=None, local=False):
         _RelPath = lambda p: p.RelPath(local=local)
         for project in diff["added"]:
-            self.printText(
-                "A %s %s" % (_RelPath(project), project.revisionExpr)
-            )
+            self.printText(f"A {_RelPath(project)} {project.revisionExpr}")
             self.out.nl()
 
         for project in diff["removed"]:
-            self.printText(
-                "R %s %s" % (_RelPath(project), project.revisionExpr)
-            )
+            self.printText(f"R {_RelPath(project)} {project.revisionExpr}")
             self.out.nl()
 
         for project, otherProject in diff["changed"]:
             self.printText(
-                "C %s %s %s"
-                % (
-                    _RelPath(project),
-                    project.revisionExpr,
-                    otherProject.revisionExpr,
-                )
+                f"C {_RelPath(project)} {project.revisionExpr} "
+                f"{otherProject.revisionExpr}"
             )
             self.out.nl()
             self._printLogs(
@@ -118,12 +111,8 @@ synced and their revisions won't be found.
 
         for project, otherProject in diff["unreachable"]:
             self.printText(
-                "U %s %s %s"
-                % (
-                    _RelPath(project),
-                    project.revisionExpr,
-                    otherProject.revisionExpr,
-                )
+                f"U {_RelPath(project)} {project.revisionExpr} "
+                f"{otherProject.revisionExpr}"
             )
             self.out.nl()
 
@@ -245,9 +234,9 @@ synced and their revisions won't be found.
             )
             self.printRevision = self.out.nofmt_printer("revision", fg="yellow")
         else:
-            self.printProject = (
-                self.printAdded
-            ) = self.printRemoved = self.printRevision = self.printText
+            self.printProject = self.printAdded = self.printRemoved = (
+                self.printRevision
+            ) = self.printText
 
         manifest1 = RepoClient(self.repodir)
         manifest1.Override(args[0], load_local_manifests=False)

subcmds/download.py

@@ -16,9 +16,18 @@ import re
 import sys
 
 from command import Command
-from error import GitError, NoSuchProjectError
+from error import GitError
+from error import NoSuchProjectError
+from error import RepoExitError
+from repo_logging import RepoLogger
+
 
 CHANGE_RE = re.compile(r"^([1-9][0-9]*)(?:[/\.-]([1-9][0-9]*))?$")
+logger = RepoLogger(__file__)
+
+
+class DownloadCommandError(RepoExitError):
+    """Error raised when download command fails."""
 
 
 class Download(Command):
@@ -51,7 +60,6 @@ If no project is specified try to use current directory as a project.
         p.add_option(
             "-r",
             "--revert",
-            dest="revert",
             action="store_true",
             help="revert instead of checkout",
         )
@@ -102,23 +110,18 @@ If no project is specified try to use current directory as a project.
                     except NoSuchProjectError:
                         project = None
                     if project not in projects:
-                        print(
+                        logger.error(
                             "error: %s matches too many projects; please "
-                            "re-run inside the project checkout." % (a,),
-                            file=sys.stderr,
+                            "re-run inside the project checkout.",
+                            a,
                         )
                         for project in projects:
-                            print(
-                                "  %s/ @ %s"
-                                % (
-                                    project.RelPath(
-                                        local=opt.this_manifest_only
-                                    ),
-                                    project.revisionExpr,
-                                ),
-                                file=sys.stderr,
+                            logger.error(
+                                "  %s/ @ %s",
+                                project.RelPath(local=opt.this_manifest_only),
+                                project.revisionExpr,
                             )
-                        sys.exit(1)
+                        raise NoSuchProjectError()
                 else:
                     project = projects[0]
                     print("Defaulting to cwd project", project.name)
@@ -137,29 +140,33 @@ If no project is specified try to use current directory as a project.
                 )
 
     def Execute(self, opt, args):
+        try:
+            self._ExecuteHelper(opt, args)
+        except Exception as e:
+            if isinstance(e, RepoExitError):
+                raise e
+            raise DownloadCommandError(aggregate_errors=[e])
+
+    def _ExecuteHelper(self, opt, args):
         for project, change_id, ps_id in self._ParseChangeIds(opt, args):
             dl = project.DownloadPatchSet(change_id, ps_id)
-            if not dl:
-                print(
-                    "[%s] change %d/%d not found"
-                    % (project.name, change_id, ps_id),
-                    file=sys.stderr,
-                )
-                sys.exit(1)
 
             if not opt.revert and not dl.commits:
-                print(
-                    "[%s] change %d/%d has already been merged"
-                    % (project.name, change_id, ps_id),
-                    file=sys.stderr,
+                logger.error(
+                    "[%s] change %d/%d has already been merged",
+                    project.name,
+                    change_id,
+                    ps_id,
                 )
                 continue
 
             if len(dl.commits) > 1:
-                print(
-                    "[%s] %d/%d depends on %d unmerged changes:"
-                    % (project.name, change_id, ps_id, len(dl.commits)),
-                    file=sys.stderr,
+                logger.error(
+                    "[%s] %d/%d depends on %d unmerged changes:",
+                    project.name,
+                    change_id,
+                    ps_id,
+                    len(dl.commits),
                 )
                 for c in dl.commits:
                     print("  %s" % (c), file=sys.stderr)
@@ -196,9 +203,10 @@ If no project is specified try to use current directory as a project.
                         project._Checkout(dl.commit)
 
             except GitError:
-                print(
-                    "[%s] Could not complete the %s of %s"
-                    % (project.name, mode, dl.commit),
-                    file=sys.stderr,
+                logger.error(
+                    "[%s] Could not complete the %s of %s",
+                    project.name,
+                    mode,
+                    dl.commit,
                 )
-                sys.exit(1)
+                raise

subcmds/forall.py

@@ -15,22 +15,21 @@
 import errno
 import functools
 import io
-import multiprocessing
-import re
 import os
+import re
 import signal
-import sys
 import subprocess
+import sys
 
 from color import Coloring
-from command import (
-    DEFAULT_LOCAL_JOBS,
-    Command,
-    MirrorSafeCommand,
-    WORKER_BATCH_SIZE,
-)
+from command import Command
+from command import DEFAULT_LOCAL_JOBS
+from command import MirrorSafeCommand
 from error import ManifestInvalidRevisionError
+from repo_logging import RepoLogger
 
+
+logger = RepoLogger(__file__)
 _CAN_COLOR = [
     "branch",
     "diff",
@@ -142,7 +141,6 @@ without iterating through the remaining projects.
         p.add_option(
             "-r",
             "--regex",
-            dest="regex",
             action="store_true",
             help="execute the command only on projects matching regex or "
             "wildcard expression",
@@ -150,7 +148,6 @@ without iterating through the remaining projects.
         p.add_option(
             "-i",
             "--inverse-regex",
-            dest="inverse_regex",
             action="store_true",
             help="execute the command only on projects not matching regex or "
             "wildcard expression",
@@ -158,22 +155,20 @@ without iterating through the remaining projects.
         p.add_option(
             "-g",
             "--groups",
-            dest="groups",
             help="execute the command only on projects matching the specified "
             "groups",
         )
         p.add_option(
             "-c",
             "--command",
-            help="command (and arguments) to execute",
             dest="command",
+            help="command (and arguments) to execute",
             action="callback",
             callback=self._cmd_option,
         )
         p.add_option(
             "-e",
             "--abort-on-errors",
-            dest="abort_on_errors",
             action="store_true",
             help="abort if a command exits unsuccessfully",
         )
@@ -240,7 +235,6 @@ without iterating through the remaining projects.
                     cmd.insert(cmd.index(cn) + 1, "--color")
 
         mirror = self.manifest.IsMirror
-        rc = 0
 
         smart_sync_manifest_name = "smart_sync_override.xml"
         smart_sync_manifest_path = os.path.join(
@@ -263,71 +257,80 @@ without iterating through the remaining projects.
 
         os.environ["REPO_COUNT"] = str(len(projects))
 
+        def _ProcessResults(_pool, _output, results):
+            rc = 0
+            first = True
+            for r, output in results:
+                if output:
+                    if first:
+                        first = False
+                    elif opt.project_header:
+                        print()
+                    # To simplify the DoWorkWrapper, take care of automatic
+                    # newlines.
+                    end = "\n"
+                    if output[-1] == "\n":
+                        end = ""
+                    print(output, end=end)
+                rc = rc or r
+                if r != 0 and opt.abort_on_errors:
+                    raise Exception("Aborting due to previous error")
+            return rc
+
         try:
             config = self.manifest.manifestProject.config
-            with multiprocessing.Pool(opt.jobs, InitWorker) as pool:
-                results_it = pool.imap(
+            with self.ParallelContext():
+                self.get_parallel_context()["projects"] = projects
+                rc = self.ExecuteInParallel(
+                    opt.jobs,
                     functools.partial(
-                        DoWorkWrapper, mirror, opt, cmd, shell, config
+                        self.DoWorkWrapper, mirror, opt, cmd, shell, config
                     ),
-                    enumerate(projects),
-                    chunksize=WORKER_BATCH_SIZE,
+                    range(len(projects)),
+                    callback=_ProcessResults,
+                    ordered=True,
+                    initializer=self.InitWorker,
+                    chunksize=1,
                 )
-                first = True
-                for r, output in results_it:
-                    if output:
-                        if first:
-                            first = False
-                        elif opt.project_header:
-                            print()
-                        # To simplify the DoWorkWrapper, take care of automatic
-                        # newlines.
-                        end = "\n"
-                        if output[-1] == "\n":
-                            end = ""
-                        print(output, end=end)
-                    rc = rc or r
-                    if r != 0 and opt.abort_on_errors:
-                        raise Exception("Aborting due to previous error")
         except (KeyboardInterrupt, WorkerKeyboardInterrupt):
             # Catch KeyboardInterrupt raised inside and outside of workers
-            rc = rc or errno.EINTR
+            rc = errno.EINTR
         except Exception as e:
             # Catch any other exceptions raised
-            print(
-                "forall: unhandled error, terminating the pool: %s: %s"
-                % (type(e).__name__, e),
-                file=sys.stderr,
+            logger.error(
+                "forall: unhandled error, terminating the pool: %s: %s",
+                type(e).__name__,
+                e,
             )
-            rc = rc or getattr(e, "errno", 1)
+            rc = getattr(e, "errno", 1)
         if rc != 0:
             sys.exit(rc)
 
+    @classmethod
+    def InitWorker(cls):
+        signal.signal(signal.SIGINT, signal.SIG_IGN)
+
+    @classmethod
+    def DoWorkWrapper(cls, mirror, opt, cmd, shell, config, project_idx):
+        """A wrapper around the DoWork() method.
+
+        Catch the KeyboardInterrupt exceptions here and re-raise them as a
+        different, ``Exception``-based exception to stop it flooding the console
+        with stacktraces and making the parent hang indefinitely.
+
+        """
+        project = cls.get_parallel_context()["projects"][project_idx]
+        try:
+            return DoWork(project, mirror, opt, cmd, shell, project_idx, config)
+        except KeyboardInterrupt:
+            print("%s: Worker interrupted" % project.name)
+            raise WorkerKeyboardInterrupt()
+
 
 class WorkerKeyboardInterrupt(Exception):
     """Keyboard interrupt exception for worker processes."""
 
 
-def InitWorker():
-    signal.signal(signal.SIGINT, signal.SIG_IGN)
-
-
-def DoWorkWrapper(mirror, opt, cmd, shell, config, args):
-    """A wrapper around the DoWork() method.
-
-    Catch the KeyboardInterrupt exceptions here and re-raise them as a
-    different, ``Exception``-based exception to stop it flooding the console
-    with stacktraces and making the parent hang indefinitely.
-
-    """
-    cnt, project = args
-    try:
-        return DoWork(project, mirror, opt, cmd, shell, cnt, config)
-    except KeyboardInterrupt:
-        print("%s: Worker interrupted" % project.name)
-        raise WorkerKeyboardInterrupt()
-
-
 def DoWork(project, mirror, opt, cmd, shell, cnt, config):
     env = os.environ.copy()
 

subcmds/gc.py

@@ -0,0 +1,294 @@
+# Copyright (C) 2024 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+from typing import List, Set
+
+from command import Command
+from git_command import GitCommand
+import platform_utils
+from progress import Progress
+from project import Project
+
+
+class Gc(Command):
+    COMMON = True
+    helpSummary = "Cleaning up internal repo and Git state."
+    helpUsage = """
+%prog
+"""
+
+    def _Options(self, p):
+        p.add_option(
+            "-n",
+            "--dry-run",
+            dest="dryrun",
+            default=False,
+            action="store_true",
+            help="do everything except actually delete",
+        )
+        p.add_option(
+            "-y",
+            "--yes",
+            default=False,
+            action="store_true",
+            help="answer yes to all safe prompts",
+        )
+        p.add_option(
+            "--repack",
+            default=False,
+            action="store_true",
+            help="repack all projects that use partial clone with "
+            "filter=blob:none",
+        )
+
+    def _find_git_to_delete(
+        self, to_keep: Set[str], start_dir: str
+    ) -> Set[str]:
+        """Searches no longer needed ".git" directories.
+
+        Scans the file system starting from `start_dir` and removes all
+        directories that end with ".git" that are not in the `to_keep` set.
+        """
+        to_delete = set()
+        for root, dirs, _ in platform_utils.walk(start_dir):
+            for directory in dirs:
+                if not directory.endswith(".git"):
+                    continue
+
+                path = os.path.join(root, directory)
+                if path not in to_keep:
+                    to_delete.add(path)
+
+        return to_delete
+
+    def delete_unused_projects(self, projects: List[Project], opt):
+        print(f"Scanning filesystem under {self.repodir}...")
+
+        project_paths = set()
+        project_object_paths = set()
+
+        for project in projects:
+            project_paths.add(project.gitdir)
+            project_object_paths.add(project.objdir)
+
+        to_delete = self._find_git_to_delete(
+            project_paths, os.path.join(self.repodir, "projects")
+        )
+
+        to_delete.update(
+            self._find_git_to_delete(
+                project_object_paths,
+                os.path.join(self.repodir, "project-objects"),
+            )
+        )
+
+        if not to_delete:
+            print("Nothing to clean up.")
+            return 0
+
+        print("Identified the following projects are no longer used:")
+        print("\n".join(to_delete))
+        print("")
+        if not opt.yes:
+            print(
+                "If you proceed, any local commits in those projects will be "
+                "destroyed!"
+            )
+            ask = input("Proceed? [y/N] ")
+            if ask.lower() != "y":
+                return 1
+
+        pm = Progress(
+            "Deleting",
+            len(to_delete),
+            delay=False,
+            quiet=opt.quiet,
+            show_elapsed=True,
+            elide=True,
+        )
+
+        for path in to_delete:
+            if opt.dryrun:
+                print(f"\nWould have deleted ${path}")
+            else:
+                tmp_path = os.path.join(
+                    os.path.dirname(path),
+                    f"to_be_deleted_{os.path.basename(path)}",
+                )
+                platform_utils.rename(path, tmp_path)
+                platform_utils.rmtree(tmp_path)
+            pm.update(msg=path)
+        pm.end()
+
+        return 0
+
+    def _generate_promisor_files(self, pack_dir: str):
+        """Generates promisor files for all pack files in the given directory.
+
+        Promisor files are empty files with the same name as the corresponding
+        pack file but with the ".promisor" extension. They are used by Git.
+        """
+        for root, _, files in platform_utils.walk(pack_dir):
+            for file in files:
+                if not file.endswith(".pack"):
+                    continue
+                with open(os.path.join(root, f"{file[:-4]}promisor"), "w"):
+                    pass
+
+    def repack_projects(self, projects: List[Project], opt):
+        repack_projects = []
+        # Find all projects eligible for repacking:
+        # - can't be shared
+        # - have a specific fetch filter
+        for project in projects:
+            if project.config.GetBoolean("extensions.preciousObjects"):
+                continue
+            if not project.clone_depth:
+                continue
+            if project.manifest.CloneFilterForDepth != "blob:none":
+                continue
+
+            repack_projects.append(project)
+
+        if opt.dryrun:
+            print(f"Would have repacked {len(repack_projects)} projects.")
+            return 0
+
+        pm = Progress(
+            "Repacking (this will take a while)",
+            len(repack_projects),
+            delay=False,
+            quiet=opt.quiet,
+            show_elapsed=True,
+            elide=True,
+        )
+
+        for project in repack_projects:
+            pm.update(msg=f"{project.name}")
+
+            pack_dir = os.path.join(project.gitdir, "tmp_repo_repack")
+            if os.path.isdir(pack_dir):
+                platform_utils.rmtree(pack_dir)
+            os.mkdir(pack_dir)
+
+            # Prepare workspace for repacking - remove all unreachable refs and
+            # their objects.
+            GitCommand(
+                project,
+                ["reflog", "expire", "--expire-unreachable=all"],
+                verify_command=True,
+            ).Wait()
+            pm.update(msg=f"{project.name} | gc", inc=0)
+            GitCommand(
+                project,
+                ["gc"],
+                verify_command=True,
+            ).Wait()
+
+            # Get all objects that are reachable from the remote, and pack them.
+            pm.update(msg=f"{project.name} | generating list of objects", inc=0)
+            remote_objects_cmd = GitCommand(
+                project,
+                [
+                    "rev-list",
+                    "--objects",
+                    f"--remotes={project.remote.name}",
+                    "--filter=blob:none",
+                    "--tags",
+                ],
+                capture_stdout=True,
+                verify_command=True,
+            )
+
+            # Get all local objects and pack them.
+            local_head_objects_cmd = GitCommand(
+                project,
+                ["rev-list", "--objects", "HEAD^{tree}"],
+                capture_stdout=True,
+                verify_command=True,
+            )
+            local_objects_cmd = GitCommand(
+                project,
+                [
+                    "rev-list",
+                    "--objects",
+                    "--all",
+                    "--reflog",
+                    "--indexed-objects",
+                    "--not",
+                    f"--remotes={project.remote.name}",
+                    "--tags",
+                ],
+                capture_stdout=True,
+                verify_command=True,
+            )
+
+            remote_objects_cmd.Wait()
+
+            pm.update(msg=f"{project.name} | remote repack", inc=0)
+            GitCommand(
+                project,
+                ["pack-objects", os.path.join(pack_dir, "pack")],
+                input=remote_objects_cmd.stdout,
+                capture_stderr=True,
+                capture_stdout=True,
+                verify_command=True,
+            ).Wait()
+
+            # create promisor file for each pack file
+            self._generate_promisor_files(pack_dir)
+
+            local_head_objects_cmd.Wait()
+            local_objects_cmd.Wait()
+
+            pm.update(msg=f"{project.name} | local repack", inc=0)
+            GitCommand(
+                project,
+                ["pack-objects", os.path.join(pack_dir, "pack")],
+                input=local_head_objects_cmd.stdout + local_objects_cmd.stdout,
+                capture_stderr=True,
+                capture_stdout=True,
+                verify_command=True,
+            ).Wait()
+
+            # Swap the old pack directory with the new one.
+            platform_utils.rename(
+                os.path.join(project.objdir, "objects", "pack"),
+                os.path.join(project.objdir, "objects", "pack_old"),
+            )
+            platform_utils.rename(
+                pack_dir,
+                os.path.join(project.objdir, "objects", "pack"),
+            )
+            platform_utils.rmtree(
+                os.path.join(project.objdir, "objects", "pack_old")
+            )
+
+        pm.end()
+        return 0
+
+    def Execute(self, opt, args):
+        projects: List[Project] = self.GetProjects(
+            args, all_manifests=not opt.this_manifest_only
+        )
+
+        ret = self.delete_unused_projects(projects, opt)
+        if ret != 0:
+            return ret
+
+        if not opt.repack:
+            return
+
+        return self.repack_projects(projects, opt)

subcmds/gitc_delete.py

@@ -1,52 +0,0 @@
-# Copyright (C) 2015 The Android Open Source Project
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import sys
-
-from command import Command, GitcClientCommand
-import platform_utils
-
-
-class GitcDelete(Command, GitcClientCommand):
-    COMMON = True
-    visible_everywhere = False
-    helpSummary = "Delete a GITC Client."
-    helpUsage = """
-%prog
-"""
-    helpDescription = """
-This subcommand deletes the current GITC client, deleting the GITC manifest
-and all locally downloaded sources.
-"""
-
-    def _Options(self, p):
-        p.add_option(
-            "-f",
-            "--force",
-            dest="force",
-            action="store_true",
-            help="force the deletion (no prompt)",
-        )
-
-    def Execute(self, opt, args):
-        if not opt.force:
-            prompt = (
-                "This will delete GITC client: %s\nAre you sure? (yes/no) "
-                % self.gitc_manifest.gitc_client_name
-            )
-            response = input(prompt).lower()
-            if not response == "yes":
-                print('Response was not "yes"\n Exiting...')
-                sys.exit(1)
-        platform_utils.rmtree(self.gitc_manifest.gitc_client_dir)

subcmds/gitc_init.py

@@ -1,87 +0,0 @@
-# Copyright (C) 2015 The Android Open Source Project
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import os
-import sys
-
-import gitc_utils
-from command import GitcAvailableCommand
-from manifest_xml import GitcManifest
-from subcmds import init
-import wrapper
-
-
-class GitcInit(init.Init, GitcAvailableCommand):
-    COMMON = True
-    MULTI_MANIFEST_SUPPORT = False
-    helpSummary = "Initialize a GITC Client."
-    helpUsage = """
-%prog [options] [client name]
-"""
-    helpDescription = """
-The '%prog' command is ran to initialize a new GITC client for use
-with the GITC file system.
-
-This command will setup the client directory, initialize repo, just
-like repo init does, and then downloads the manifest collection
-and installs it in the .repo/directory of the GITC client.
-
-Once this is done, a GITC manifest is generated by pulling the HEAD
-SHA for each project and generates the properly formatted XML file
-and installs it as .manifest in the GITC client directory.
-
-The -c argument is required to specify the GITC client name.
-
-The optional -f argument can be used to specify the manifest file to
-use for this GITC client.
-"""
-
-    def _Options(self, p):
-        super()._Options(p, gitc_init=True)
-
-    def Execute(self, opt, args):
-        gitc_client = gitc_utils.parse_clientdir(os.getcwd())
-        if not gitc_client or (
-            opt.gitc_client and gitc_client != opt.gitc_client
-        ):
-            print(
-                "fatal: Please update your repo command. See go/gitc for "
-                "instructions.",
-                file=sys.stderr,
-            )
-            sys.exit(1)
-        self.client_dir = os.path.join(
-            gitc_utils.get_gitc_manifest_dir(), gitc_client
-        )
-        super().Execute(opt, args)
-
-        manifest_file = self.manifest.manifestFile
-        if opt.manifest_file:
-            if not os.path.exists(opt.manifest_file):
-                print(
-                    "fatal: Specified manifest file %s does not exist."
-                    % opt.manifest_file
-                )
-                sys.exit(1)
-            manifest_file = opt.manifest_file
-
-        manifest = GitcManifest(
-            self.repodir, os.path.join(self.client_dir, ".manifest")
-        )
-        manifest.Override(manifest_file)
-        gitc_utils.generate_gitc_manifest(None, manifest)
-        print(
-            "Please run `cd %s` to view your GITC client."
-            % os.path.join(wrapper.Wrapper().GITC_FS_ROOT_DIR, gitc_client)
-        )

subcmds/grep.py

@@ -14,11 +14,19 @@
 
 import functools
 import sys
+from typing import NamedTuple
 
 from color import Coloring
-from command import DEFAULT_LOCAL_JOBS, PagedCommand
+from command import DEFAULT_LOCAL_JOBS
+from command import PagedCommand
 from error import GitError
+from error import InvalidArgumentsError
+from error import SilentRepoExitError
 from git_command import GitCommand
+from repo_logging import RepoLogger
+
+
+logger = RepoLogger(__file__)
 
 
 class GrepColoring(Coloring):
@@ -28,6 +36,22 @@ class GrepColoring(Coloring):
         self.fail = self.printer("fail", fg="red")
 
 
+class ExecuteOneResult(NamedTuple):
+    """Result from an execute instance."""
+
+    project_idx: int
+    rc: int
+    stdout: str
+    stderr: str
+    error: GitError
+
+
+class GrepCommandError(SilentRepoExitError):
+    """Grep command failure. Since Grep command
+    output already outputs errors ensure that
+    aggregate errors exit silently."""
+
+
 class Grep(PagedCommand):
     COMMON = True
     helpSummary = "Print lines matching a pattern"
@@ -96,7 +120,6 @@ contain a line that matches both expressions:
         g.add_option(
             "-r",
             "--revision",
-            dest="revision",
             action="append",
             metavar="TREEish",
             help="Search TREEish, instead of the work tree",
@@ -237,8 +260,10 @@ contain a line that matches both expressions:
             help="Show only file names not containing matching lines",
         )
 
-    def _ExecuteOne(self, cmd_argv, project):
+    @classmethod
+    def _ExecuteOne(cls, cmd_argv, project_idx):
         """Process one project."""
+        project = cls.get_parallel_context()["projects"][project_idx]
         try:
             p = GitCommand(
                 project,
@@ -246,43 +271,60 @@ contain a line that matches both expressions:
                 bare=False,
                 capture_stdout=True,
                 capture_stderr=True,
+                verify_command=True,
             )
         except GitError as e:
-            return (project, -1, None, str(e))
+            return ExecuteOneResult(project_idx, -1, None, str(e), e)
 
-        return (project, p.Wait(), p.stdout, p.stderr)
+        try:
+            error = None
+            rc = p.Wait()
+        except GitError as e:
+            rc = 1
+            error = e
+        return ExecuteOneResult(project_idx, rc, p.stdout, p.stderr, error)
 
     @staticmethod
-    def _ProcessResults(full_name, have_rev, opt, _pool, out, results):
+    def _ProcessResults(
+        full_name, have_rev, opt, projects, _pool, out, results
+    ):
         git_failed = False
         bad_rev = False
         have_match = False
         _RelPath = lambda p: p.RelPath(local=opt.this_manifest_only)
+        errors = []
 
-        for project, rc, stdout, stderr in results:
-            if rc < 0:
+        for result in results:
+            project = projects[result.project_idx]
+            if result.rc < 0:
                 git_failed = True
                 out.project("--- project %s ---" % _RelPath(project))
                 out.nl()
-                out.fail("%s", stderr)
+                out.fail("%s", result.stderr)
                 out.nl()
+                errors.append(result.error)
                 continue
 
-            if rc:
+            if result.rc:
                 # no results
-                if stderr:
-                    if have_rev and "fatal: ambiguous argument" in stderr:
+                if result.stderr:
+                    if (
+                        have_rev
+                        and "fatal: ambiguous argument" in result.stderr
+                    ):
                         bad_rev = True
                     else:
                         out.project("--- project %s ---" % _RelPath(project))
                         out.nl()
-                        out.fail("%s", stderr.strip())
+                        out.fail("%s", result.stderr.strip())
                         out.nl()
+                    if result.error is not None:
+                        errors.append(result.error)
                 continue
             have_match = True
 
             # We cut the last element, to avoid a blank line.
-            r = stdout.split("\n")
+            r = result.stdout.split("\n")
             r = r[0:-1]
 
             if have_rev and full_name:
@@ -304,7 +346,7 @@ contain a line that matches both expressions:
                 for line in r:
                     print(line)
 
-        return (git_failed, bad_rev, have_match)
+        return (git_failed, bad_rev, have_match, errors)
 
     def Execute(self, opt, args):
         out = GrepColoring(self.manifest.manifestProject.config)
@@ -333,33 +375,34 @@ contain a line that matches both expressions:
         have_rev = False
         if opt.revision:
             if "--cached" in cmd_argv:
-                print(
-                    "fatal: cannot combine --cached and --revision",
-                    file=sys.stderr,
-                )
-                sys.exit(1)
+                msg = "fatal: cannot combine --cached and --revision"
+                logger.error(msg)
+                raise InvalidArgumentsError(msg)
             have_rev = True
             cmd_argv.extend(opt.revision)
         cmd_argv.append("--")
 
-        git_failed, bad_rev, have_match = self.ExecuteInParallel(
-            opt.jobs,
-            functools.partial(self._ExecuteOne, cmd_argv),
-            projects,
-            callback=functools.partial(
-                self._ProcessResults, full_name, have_rev, opt
-            ),
-            output=out,
-            ordered=True,
-        )
+        with self.ParallelContext():
+            self.get_parallel_context()["projects"] = projects
+            git_failed, bad_rev, have_match, errors = self.ExecuteInParallel(
+                opt.jobs,
+                functools.partial(self._ExecuteOne, cmd_argv),
+                range(len(projects)),
+                callback=functools.partial(
+                    self._ProcessResults, full_name, have_rev, opt, projects
+                ),
+                output=out,
+                ordered=True,
+                chunksize=1,
+            )
 
         if git_failed:
-            sys.exit(1)
+            raise GrepCommandError(
+                "error: git failures", aggregate_errors=errors
+            )
         elif have_match:
             sys.exit(0)
         elif have_rev and bad_rev:
             for r in opt.revision:
-                print("error: can't search revision %s" % r, file=sys.stderr)
-            sys.exit(1)
-        else:
-            sys.exit(1)
+                logger.error("error: can't search revision %s", r)
+        raise GrepCommandError(aggregate_errors=errors)

subcmds/help.py

@@ -16,18 +16,18 @@ import re
 import sys
 import textwrap
 
-from subcmds import all_commands
 from color import Coloring
-from command import (
-    PagedCommand,
-    MirrorSafeCommand,
-    GitcAvailableCommand,
-    GitcClientCommand,
-)
-import gitc_utils
+from command import MirrorSafeCommand
+from command import PagedCommand
+from error import RepoExitError
+from subcmds import all_commands
 from wrapper import Wrapper
 
 
+class InvalidHelpCommand(RepoExitError):
+    """Invalid command passed into help."""
+
+
 class Help(PagedCommand, MirrorSafeCommand):
     COMMON = False
     helpSummary = "Display detailed help on a command"
@@ -74,26 +74,9 @@ Displays detailed usage information about a command.
     def PrintCommonCommandsBody(self):
         print("The most commonly used repo commands are:")
 
-        def gitc_supported(cmd):
-            if not isinstance(cmd, GitcAvailableCommand) and not isinstance(
-                cmd, GitcClientCommand
-            ):
-                return True
-            if self.client.isGitcClient:
-                return True
-            if isinstance(cmd, GitcClientCommand):
-                return False
-            if gitc_utils.get_gitc_manifest_dir():
-                return True
-            return False
-
         commandNames = list(
             sorted(
-                [
-                    name
-                    for name, command in all_commands.items()
-                    if command.COMMON and gitc_supported(command)
-                ]
+                name for name, command in all_commands.items() if command.COMMON
             )
         )
         self._PrintCommands(commandNames)
@@ -167,7 +150,7 @@ Displays detailed usage information about a command.
     def _PrintAllCommandHelp(self):
         for name in sorted(all_commands):
             cmd = all_commands[name](manifest=self.manifest)
-            self._PrintCommandHelp(cmd, header_prefix="[%s] " % (name,))
+            self._PrintCommandHelp(cmd, header_prefix=f"[{name}] ")
 
     def _Options(self, p):
         p.add_option(
@@ -202,7 +185,7 @@ Displays detailed usage information about a command.
                 print(
                     "repo: '%s' is not a repo command." % name, file=sys.stderr
                 )
-                sys.exit(1)
+                raise InvalidHelpCommand(name)
 
             self._PrintCommandHelp(cmd)
 

subcmds/info.py

@@ -14,9 +14,10 @@
 
 import optparse
 
-from command import PagedCommand
 from color import Coloring
-from git_refs import R_M, R_HEADS
+from command import PagedCommand
+from git_refs import R_HEADS
+from git_refs import R_M
 
 
 class _Coloring(Coloring):
@@ -42,14 +43,12 @@ class Info(PagedCommand):
         p.add_option(
             "-o",
             "--overview",
-            dest="overview",
             action="store_true",
             help="show overview of all local commits",
         )
         p.add_option(
             "-c",
             "--current-branch",
-            dest="current_branch",
             action="store_true",
             help="consider only checked out branches",
         )
@@ -89,18 +88,25 @@ class Info(PagedCommand):
             self.manifest = self.manifest.outer_client
         manifestConfig = self.manifest.manifestProject.config
         mergeBranch = manifestConfig.GetBranch("default").merge
-        manifestGroups = self.manifest.GetGroupsStr()
+        manifestGroups = self.manifest.GetManifestGroupsStr()
 
         self.heading("Manifest branch: ")
         if self.manifest.default.revisionExpr:
             self.headtext(self.manifest.default.revisionExpr)
         self.out.nl()
         self.heading("Manifest merge branch: ")
-        self.headtext(mergeBranch)
+        # The manifest might not have a merge branch if it isn't in a git repo,
+        # e.g. if `repo init --standalone-manifest` is used.
+        self.headtext(mergeBranch or "")
         self.out.nl()
         self.heading("Manifest groups: ")
         self.headtext(manifestGroups)
         self.out.nl()
+        sp = self.manifest.superproject
+        srev = sp.commit_id if sp and sp.commit_id else "None"
+        self.heading("Superproject revision: ")
+        self.headtext(srev)
+        self.out.nl()
 
         self.printSeparator()
 
@@ -247,7 +253,7 @@ class Info(PagedCommand):
 
             for commit in commits:
                 split = commit.split()
-                self.text("{0:38}{1} ".format("", "-"))
+                self.text(f"{'':38}{'-'} ")
                 self.sha(split[0] + " ")
                 self.text(" ".join(split[1:]))
                 self.out.nl()

subcmds/init.py

@@ -16,9 +16,19 @@ import os
 import sys
 
 from color import Coloring
-from command import InteractiveCommand, MirrorSafeCommand
-from git_command import git_require, MIN_GIT_VERSION_SOFT, MIN_GIT_VERSION_HARD
+from command import InteractiveCommand
+from command import MirrorSafeCommand
+from error import RepoUnhandledExceptionError
+from error import UpdateManifestError
+from git_command import git_require
+from repo_logging import RepoLogger
 from wrapper import Wrapper
+from wrapper import WrapperDir
+
+
+logger = RepoLogger(__file__)
+
+_REPO_ALLOW_SHALLOW = os.environ.get("REPO_ALLOW_SHALLOW")
 
 
 class Init(InteractiveCommand, MirrorSafeCommand):
@@ -42,6 +52,10 @@ The optional -b argument can be used to select the manifest branch
 to checkout and use.  If no branch is specified, the remote's default
 branch is used.  This is equivalent to using -b HEAD.
 
+The optional --manifest-upstream-branch argument can be used when a commit is
+provided to --manifest-branch (or -b), to specify the name of the git ref in
+which the commit can be found.
+
 The optional -m argument can be used to specify an alternate manifest
 to be used. If no manifest is specified, the manifest default.xml
 will be used.
@@ -80,8 +94,8 @@ to update the working directory files.
     def _CommonOptions(self, p):
         """Disable due to re-use of Wrapper()."""
 
-    def _Options(self, p, gitc_init=False):
-        Wrapper().InitParser(p, gitc_init=gitc_init)
+    def _Options(self, p):
+        Wrapper().InitParser(p)
         m = p.add_option_group("Multi-manifest")
         m.add_option(
             "--outer-manifest",
@@ -113,6 +127,7 @@ to update the working directory files.
         return {
             "REPO_MANIFEST_URL": "manifest_url",
             "REPO_MIRROR_LOCATION": "reference",
+            "REPO_GIT_LFS": "git_lfs",
         }
 
     def _SyncManifest(self, opt):
@@ -125,6 +140,10 @@ to update the working directory files.
         # manifest project is special and is created when instantiating the
         # manifest which happens before we parse options.
         self.manifest.manifestProject.clone_depth = opt.manifest_depth
+        self.manifest.manifestProject.upstream = opt.manifest_upstream_branch
+        clone_filter_for_depth = (
+            "blob:none" if (_REPO_ALLOW_SHALLOW == "0") else None
+        )
         if not self.manifest.manifestProject.Sync(
             manifest_url=opt.manifest_url,
             manifest_branch=opt.manifest_branch,
@@ -140,6 +159,7 @@ to update the working directory files.
             partial_clone=opt.partial_clone,
             clone_filter=opt.clone_filter,
             partial_clone_exclude=opt.partial_clone_exclude,
+            clone_filter_for_depth=clone_filter_for_depth,
             clone_bundle=opt.clone_bundle,
             git_lfs=opt.git_lfs,
             use_superproject=opt.use_superproject,
@@ -150,7 +170,10 @@ to update the working directory files.
             git_event_log=self.git_event_log,
             manifest_name=opt.manifest_name,
         ):
-            sys.exit(1)
+            manifest_name = opt.manifest_name
+            raise UpdateManifestError(
+                f"Unable to sync manifest {manifest_name}"
+            )
 
     def _Prompt(self, prompt, value):
         print("%-10s [%s]: " % (prompt, value), end="", flush=True)
@@ -197,7 +220,7 @@ to update the working directory files.
 
             if not opt.quiet:
                 print()
-            print("Your identity is: %s <%s>" % (name, email))
+            print(f"Your identity is: {name} <{email}>")
             print("is this correct [y/N]? ", end="", flush=True)
             a = sys.stdin.readline().strip().lower()
             if a in ("yes", "y", "t", "true"):
@@ -300,6 +323,12 @@ to update the working directory files.
                 " be used with --standalone-manifest."
             )
 
+        if opt.manifest_upstream_branch and opt.manifest_branch is None:
+            self.OptionParser.error(
+                "--manifest-upstream-branch cannot be used without "
+                "--manifest-branch."
+            )
+
         if args:
             if opt.manifest_url:
                 self.OptionParser.error(
@@ -313,13 +342,17 @@ to update the working directory files.
                 self.OptionParser.error("too many arguments to init")
 
     def Execute(self, opt, args):
-        git_require(MIN_GIT_VERSION_HARD, fail=True)
-        if not git_require(MIN_GIT_VERSION_SOFT):
-            print(
-                "repo: warning: git-%s+ will soon be required; please upgrade "
-                "your version of git to maintain support."
-                % (".".join(str(x) for x in MIN_GIT_VERSION_SOFT),),
-                file=sys.stderr,
+        wrapper = Wrapper()
+
+        reqs = wrapper.Requirements.from_dir(WrapperDir())
+        git_require(reqs.get_hard_ver("git"), fail=True)
+        min_git_version_soft = reqs.get_soft_ver("git")
+        if not git_require(min_git_version_soft):
+            logger.warning(
+                "repo: warning: git-%s+ will soon be required; "
+                "please upgrade your version of git to maintain "
+                "support.",
+                ".".join(str(x) for x in min_git_version_soft),
             )
 
         rp = self.manifest.repoProject
@@ -332,20 +365,19 @@ to update the working directory files.
 
         # Handle new --repo-rev requests.
         if opt.repo_rev:
-            wrapper = Wrapper()
             try:
                 remote_ref, rev = wrapper.check_repo_rev(
-                    rp.gitdir,
+                    rp.worktree,
                     opt.repo_rev,
                     repo_verify=opt.repo_verify,
                     quiet=opt.quiet,
                 )
-            except wrapper.CloneFailure:
-                print(
-                    "fatal: double check your --repo-rev setting.",
-                    file=sys.stderr,
-                )
-                sys.exit(1)
+            except wrapper.CloneFailure as e:
+                err_msg = "fatal: double check your --repo-rev setting."
+                logger.error(err_msg)
+                self.git_event_log.ErrorEvent(err_msg)
+                raise RepoUnhandledExceptionError(e)
+
             branch = rp.GetBranch("default")
             branch.merge = remote_ref
             rp.work_git.reset("--hard", rev)

subcmds/list.py

@@ -14,7 +14,8 @@
 
 import os
 
-from command import Command, MirrorSafeCommand
+from command import Command
+from command import MirrorSafeCommand
 
 
 class List(Command, MirrorSafeCommand):
@@ -39,7 +40,6 @@ This is similar to running: repo forall -c 'echo "$REPO_PATH : $REPO_PROJECT"'.
         p.add_option(
             "-r",
             "--regex",
-            dest="regex",
             action="store_true",
             help="filter the project list based on regex or wildcard matching "
             "of strings",
@@ -47,7 +47,6 @@ This is similar to running: repo forall -c 'echo "$REPO_PATH : $REPO_PROJECT"'.
         p.add_option(
             "-g",
             "--groups",
-            dest="groups",
             help="filter the project list based on the groups the project is "
             "in",
         )
@@ -60,21 +59,18 @@ This is similar to running: repo forall -c 'echo "$REPO_PATH : $REPO_PROJECT"'.
         p.add_option(
             "-n",
             "--name-only",
-            dest="name_only",
             action="store_true",
             help="display only the name of the repository",
         )
         p.add_option(
             "-p",
             "--path-only",
-            dest="path_only",
             action="store_true",
             help="display only the path of the repository",
         )
         p.add_option(
             "-f",
             "--fullpath",
-            dest="fullpath",
             action="store_true",
             help="display the full work tree path instead of the relative path",
         )
@@ -130,7 +126,7 @@ This is similar to running: repo forall -c 'echo "$REPO_PATH : $REPO_PROJECT"'.
             elif opt.path_only and not opt.name_only:
                 lines.append("%s" % (_getpath(project)))
             else:
-                lines.append("%s : %s" % (_getpath(project), project.name))
+                lines.append(f"{_getpath(project)} : {project.name}")
 
         if lines:
             lines.sort()

subcmds/manifest.py

@@ -12,11 +12,27 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import enum
 import json
+import optparse
 import os
 import sys
 
 from command import PagedCommand
+from repo_logging import RepoLogger
+
+
+logger = RepoLogger(__file__)
+
+
+class OutputFormat(enum.Enum):
+    """Type for the requested output format."""
+
+    # Canonicalized manifest in XML format.
+    XML = enum.auto()
+
+    # Canonicalized manifest in JSON format.
+    JSON = enum.auto()
 
 
 class Manifest(PagedCommand):
@@ -38,6 +54,10 @@ revisions set to the current commit hash.  These are known as
 In this case, the 'upstream' attribute is set to the ref we were on
 when the manifest was generated.  The 'dest-branch' attribute is set
 to indicate the remote ref to push changes to via 'repo upload'.
+
+Multiple output formats are supported via --format.  The default output
+is XML, and formats are generally "condensed".  Use --pretty for more
+human-readable variations.
 """
 
     @property
@@ -82,11 +102,21 @@ to indicate the remote ref to push changes to via 'repo upload'.
             "(only of use if the branch names for a sha1 manifest are "
             "sensitive)",
         )
+        # Replaced with --format=json.  Kept for backwards compatibility.
+        # Can delete in Jun 2026 or later.
         p.add_option(
             "--json",
-            default=False,
-            action="store_true",
-            help="output manifest in JSON format (experimental)",
+            action="store_const",
+            dest="format",
+            const=OutputFormat.JSON.name.lower(),
+            help=optparse.SUPPRESS_HELP,
+        )
+        formats = tuple(x.lower() for x in OutputFormat.__members__.keys())
+        p.add_option(
+            "--format",
+            default=OutputFormat.XML.name.lower(),
+            choices=formats,
+            help=f"output format: {', '.join(formats)} (default: %default)",
         )
         p.add_option(
             "--pretty",
@@ -104,7 +134,6 @@ to indicate the remote ref to push changes to via 'repo upload'.
         p.add_option(
             "-o",
             "--output-file",
-            dest="output_file",
             default="-",
             help="file to save the manifest to. (Filename prefix for "
             "multi-tree.)",
@@ -117,6 +146,8 @@ to indicate the remote ref to push changes to via 'repo upload'.
         if opt.manifest_name:
             self.manifest.Override(opt.manifest_name, False)
 
+        output_format = OutputFormat[opt.format.upper()]
+
         for manifest in self.ManifestList(opt):
             output_file = opt.output_file
             if output_file == "-":
@@ -131,8 +162,7 @@ to indicate the remote ref to push changes to via 'repo upload'.
 
             manifest.SetUseLocalManifests(not opt.ignore_local_manifests)
 
-            if opt.json:
-                print("warning: --json is experimental!", file=sys.stderr)
+            if output_format == OutputFormat.JSON:
                 doc = manifest.ToDict(
                     peg_rev=opt.peg_rev,
                     peg_rev_upstream=opt.peg_rev_upstream,
@@ -148,7 +178,7 @@ to indicate the remote ref to push changes to via 'repo upload'.
                     "separators": (",", ": ") if opt.pretty else (",", ":"),
                     "sort_keys": True,
                 }
-                fd.write(json.dumps(doc, **json_settings))
+                fd.write(json.dumps(doc, **json_settings) + "\n")
             else:
                 manifest.Save(
                     fd,
@@ -159,13 +189,13 @@ to indicate the remote ref to push changes to via 'repo upload'.
             if output_file != "-":
                 fd.close()
                 if manifest.path_prefix:
-                    print(
-                        f"Saved {manifest.path_prefix} submanifest to "
-                        f"{output_file}",
-                        file=sys.stderr,
+                    logger.warning(
+                        "Saved %s submanifest to %s",
+                        manifest.path_prefix,
+                        output_file,
                     )
                 else:
-                    print(f"Saved manifest to {output_file}", file=sys.stderr)
+                    logger.warning("Saved manifest to %s", output_file)
 
     def ValidateOptions(self, opt, args):
         if args:

subcmds/overview.py

@@ -37,7 +37,6 @@ are displayed.
         p.add_option(
             "-c",
             "--current-branch",
-            dest="current_branch",
             action="store_true",
             help="consider only checked out branches",
         )

subcmds/prune.py

@@ -15,7 +15,8 @@
 import itertools
 
 from color import Coloring
-from command import DEFAULT_LOCAL_JOBS, PagedCommand
+from command import DEFAULT_LOCAL_JOBS
+from command import PagedCommand
 
 
 class Prune(PagedCommand):
@@ -26,8 +27,10 @@ class Prune(PagedCommand):
 """
     PARALLEL_JOBS = DEFAULT_LOCAL_JOBS
 
-    def _ExecuteOne(self, project):
+    @classmethod
+    def _ExecuteOne(cls, project_idx):
         """Process one project."""
+        project = cls.get_parallel_context()["projects"][project_idx]
         return project.PruneHeads()
 
     def Execute(self, opt, args):
@@ -40,13 +43,15 @@ class Prune(PagedCommand):
         def _ProcessResults(_pool, _output, results):
             return list(itertools.chain.from_iterable(results))
 
-        all_branches = self.ExecuteInParallel(
-            opt.jobs,
-            self._ExecuteOne,
-            projects,
-            callback=_ProcessResults,
-            ordered=True,
-        )
+        with self.ParallelContext():
+            self.get_parallel_context()["projects"] = projects
+            all_branches = self.ExecuteInParallel(
+                opt.jobs,
+                self._ExecuteOne,
+                range(len(projects)),
+                callback=_ProcessResults,
+                ordered=True,
+            )
 
         if not all_branches:
             return
@@ -82,9 +87,7 @@ class Prune(PagedCommand):
             )
 
             if not branch.base_exists:
-                print(
-                    "(ignoring: tracking branch is gone: %s)" % (branch.base,)
-                )
+                print(f"(ignoring: tracking branch is gone: {branch.base})")
             else:
                 commits = branch.commits
                 date = branch.date

subcmds/rebase.py

@@ -17,6 +17,10 @@ import sys
 from color import Coloring
 from command import Command
 from git_command import GitCommand
+from repo_logging import RepoLogger
+
+
+logger = RepoLogger(__file__)
 
 
 class RebaseColoring(Coloring):
@@ -43,21 +47,18 @@ branch but need to incorporate new upstream changes "underneath" them.
         g.add_option(
             "-i",
             "--interactive",
-            dest="interactive",
             action="store_true",
             help="interactive rebase (single project only)",
         )
 
         p.add_option(
             "--fail-fast",
-            dest="fail_fast",
             action="store_true",
             help="stop rebasing after first error is hit",
         )
         p.add_option(
             "-f",
             "--force-rebase",
-            dest="force_rebase",
             action="store_true",
             help="pass --force-rebase to git rebase",
         )
@@ -70,27 +71,23 @@ branch but need to incorporate new upstream changes "underneath" them.
         )
         p.add_option(
             "--autosquash",
-            dest="autosquash",
             action="store_true",
             help="pass --autosquash to git rebase",
         )
         p.add_option(
             "--whitespace",
-            dest="whitespace",
             action="store",
             metavar="WS",
             help="pass --whitespace to git rebase",
         )
         p.add_option(
             "--auto-stash",
-            dest="auto_stash",
             action="store_true",
             help="stash local modifications before starting",
         )
         p.add_option(
             "-m",
             "--onto-manifest",
-            dest="onto_manifest",
             action="store_true",
             help="rebase onto the manifest version instead of upstream "
             "HEAD (this helps to make sure the local tree stays "
@@ -104,17 +101,15 @@ branch but need to incorporate new upstream changes "underneath" them.
         one_project = len(all_projects) == 1
 
         if opt.interactive and not one_project:
-            print(
-                "error: interactive rebase not supported with multiple "
-                "projects",
-                file=sys.stderr,
+            logger.error(
+                "error: interactive rebase not supported with multiple projects"
             )
+
             if len(args) == 1:
-                print(
-                    "note: project %s is mapped to more than one path"
-                    % (args[0],),
-                    file=sys.stderr,
+                logger.warning(
+                    "note: project %s is mapped to more than one path", args[0]
                 )
+
             return 1
 
         # Setup the common git rebase args that we use for all projects.
@@ -145,10 +140,9 @@ branch but need to incorporate new upstream changes "underneath" them.
             cb = project.CurrentBranch
             if not cb:
                 if one_project:
-                    print(
-                        "error: project %s has a detached HEAD"
-                        % _RelPath(project),
-                        file=sys.stderr,
+                    logger.error(
+                        "error: project %s has a detached HEAD",
+                        _RelPath(project),
                     )
                     return 1
                 # Ignore branches with detached HEADs.
@@ -157,10 +151,9 @@ branch but need to incorporate new upstream changes "underneath" them.
             upbranch = project.GetBranch(cb)
             if not upbranch.LocalMerge:
                 if one_project:
-                    print(
-                        "error: project %s does not track any remote branches"
-                        % _RelPath(project),
-                        file=sys.stderr,
+                    logger.error(
+                        "error: project %s does not track any remote branches",
+                        _RelPath(project),
                     )
                     return 1
                 # Ignore branches without remotes.
@@ -206,7 +199,9 @@ branch but need to incorporate new upstream changes "underneath" them.
                     ret += 1
 
         if ret:
-            out.fail("%i projects had errors", ret)
+            msg_fmt = "%d projects had errors"
+            self.git_event_log.ErrorEvent(msg_fmt % (ret), msg_fmt)
+            out.fail(msg_fmt, ret)
             out.nl()
 
         return ret

subcmds/selfupdate.py

@@ -12,12 +12,21 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from optparse import SUPPRESS_HELP
-import sys
+import optparse
 
-from command import Command, MirrorSafeCommand
-from subcmds.sync import _PostRepoUpgrade
+from command import Command
+from command import MirrorSafeCommand
+from error import RepoExitError
+from repo_logging import RepoLogger
 from subcmds.sync import _PostRepoFetch
+from subcmds.sync import _PostRepoUpgrade
+
+
+logger = RepoLogger(__file__)
+
+
+class SelfupdateError(RepoExitError):
+    """Exit error for failed selfupdate command."""
 
 
 class Selfupdate(Command, MirrorSafeCommand):
@@ -45,9 +54,8 @@ need to be performed by an end-user.
         )
         g.add_option(
             "--repo-upgraded",
-            dest="repo_upgraded",
             action="store_true",
-            help=SUPPRESS_HELP,
+            help=optparse.SUPPRESS_HELP,
         )
 
     def Execute(self, opt, args):
@@ -58,9 +66,10 @@ need to be performed by an end-user.
             _PostRepoUpgrade(self.manifest)
 
         else:
-            if not rp.Sync_NetworkHalf().success:
-                print("error: can't update repo", file=sys.stderr)
-                sys.exit(1)
+            result = rp.Sync_NetworkHalf()
+            if result.error:
+                logger.error("error: can't update repo")
+                raise SelfupdateError(aggregate_errors=[result.error])
 
             rp.bare_git.gc("--auto")
             _PostRepoFetch(rp, repo_verify=opt.repo_verify, verbose=True)

subcmds/stage.py

@@ -17,6 +17,10 @@ import sys
 from color import Coloring
 from command import InteractiveCommand
 from git_command import GitCommand
+from repo_logging import RepoLogger
+
+
+logger = RepoLogger(__file__)
 
 
 class _ProjectList(Coloring):
@@ -42,7 +46,6 @@ The '%prog' command stages files to prepare the next commit.
         g.add_option(
             "-i",
             "--interactive",
-            dest="interactive",
             action="store_true",
             help="use interactive staging",
         )
@@ -62,7 +65,7 @@ The '%prog' command stages files to prepare the next commit.
             if p.IsDirty()
         ]
         if not all_projects:
-            print("no projects have uncommitted modifications", file=sys.stderr)
+            logger.error("no projects have uncommitted modifications")
             return
 
         out = _ProjectList(self.manifest.manifestProject.config)

subcmds/start.py

@@ -13,15 +13,27 @@
 # limitations under the License.
 
 import functools
-import os
-import sys
+from typing import NamedTuple
 
-from command import Command, DEFAULT_LOCAL_JOBS
-from git_config import IsImmutable
+from command import Command
+from command import DEFAULT_LOCAL_JOBS
+from error import RepoExitError
 from git_command import git
-import gitc_utils
+from git_config import IsImmutable
 from progress import Progress
-from project import SyncBuffer
+from repo_logging import RepoLogger
+
+
+logger = RepoLogger(__file__)
+
+
+class ExecuteOneResult(NamedTuple):
+    project_idx: int
+    error: Exception
+
+
+class StartError(RepoExitError):
+    """Exit error for failed start command."""
 
 
 class Start(Command):
@@ -39,7 +51,6 @@ revision specified in the manifest.
     def _Options(self, p):
         p.add_option(
             "--all",
-            dest="all",
             action="store_true",
             help="begin branch in all projects",
         )
@@ -67,32 +78,33 @@ revision specified in the manifest.
         if not git.check_ref_format("heads/%s" % nb):
             self.OptionParser.error("'%s' is not a valid name" % nb)
 
-    def _ExecuteOne(self, revision, nb, project):
+    @classmethod
+    def _ExecuteOne(cls, revision, nb, default_revisionExpr, project_idx):
         """Start one project."""
         # If the current revision is immutable, such as a SHA1, a tag or
         # a change, then we can't push back to it. Substitute with
         # dest_branch, if defined; or with manifest default revision instead.
         branch_merge = ""
+        error = None
+        project = cls.get_parallel_context()["projects"][project_idx]
         if IsImmutable(project.revisionExpr):
             if project.dest_branch:
                 branch_merge = project.dest_branch
             else:
-                branch_merge = self.manifest.default.revisionExpr
+                branch_merge = default_revisionExpr
 
         try:
-            ret = project.StartBranch(
+            project.StartBranch(
                 nb, branch_merge=branch_merge, revision=revision
             )
         except Exception as e:
-            print(
-                "error: unable to checkout %s: %s" % (project.name, e),
-                file=sys.stderr,
-            )
-            ret = False
-        return (ret, project)
+            logger.error("error: unable to checkout %s: %s", project.name, e)
+            error = e
+        return ExecuteOneResult(project_idx, error)
 
     def Execute(self, opt, args):
         nb = args[0]
+        err_projects = []
         err = []
         projects = []
         if not opt.all:
@@ -102,70 +114,44 @@ revision specified in the manifest.
 
         all_projects = self.GetProjects(
             projects,
-            missing_ok=bool(self.gitc_manifest),
             all_manifests=not opt.this_manifest_only,
         )
 
-        # This must happen after we find all_projects, since GetProjects may
-        # need the local directory, which will disappear once we save the GITC
-        # manifest.
-        if self.gitc_manifest:
-            gitc_projects = self.GetProjects(
-                projects, manifest=self.gitc_manifest, missing_ok=True
-            )
-            for project in gitc_projects:
-                if project.old_revision:
-                    project.already_synced = True
-                else:
-                    project.already_synced = False
-                    project.old_revision = project.revisionExpr
-                project.revisionExpr = None
-            # Save the GITC manifest.
-            gitc_utils.save_manifest(self.gitc_manifest)
-
-            # Make sure we have a valid CWD.
-            if not os.path.exists(os.getcwd()):
-                os.chdir(self.manifest.topdir)
-
-            pm = Progress("Syncing %s" % nb, len(all_projects), quiet=opt.quiet)
-            for project in all_projects:
-                gitc_project = self.gitc_manifest.paths[project.relpath]
-                # Sync projects that have not been opened.
-                if not gitc_project.already_synced:
-                    proj_localdir = os.path.join(
-                        self.gitc_manifest.gitc_client_dir, project.relpath
-                    )
-                    project.worktree = proj_localdir
-                    if not os.path.exists(proj_localdir):
-                        os.makedirs(proj_localdir)
-                    project.Sync_NetworkHalf()
-                    sync_buf = SyncBuffer(self.manifest.manifestProject.config)
-                    project.Sync_LocalHalf(sync_buf)
-                    project.revisionId = gitc_project.old_revision
-                pm.update()
-            pm.end()
-
         def _ProcessResults(_pool, pm, results):
-            for result, project in results:
-                if not result:
-                    err.append(project)
-                pm.update()
+            for result in results:
+                if result.error:
+                    project = all_projects[result.project_idx]
+                    err_projects.append(project)
+                    err.append(result.error)
+                pm.update(msg="")
 
-        self.ExecuteInParallel(
-            opt.jobs,
-            functools.partial(self._ExecuteOne, opt.revision, nb),
-            all_projects,
-            callback=_ProcessResults,
-            output=Progress(
-                "Starting %s" % (nb,), len(all_projects), quiet=opt.quiet
-            ),
-        )
+        with self.ParallelContext():
+            self.get_parallel_context()["projects"] = all_projects
+            self.ExecuteInParallel(
+                opt.jobs,
+                functools.partial(
+                    self._ExecuteOne,
+                    opt.revision,
+                    nb,
+                    self.manifest.default.revisionExpr,
+                ),
+                range(len(all_projects)),
+                callback=_ProcessResults,
+                output=Progress(
+                    f"Starting {nb}", len(all_projects), quiet=opt.quiet
+                ),
+                chunksize=1,
+            )
 
-        if err:
-            for p in err:
-                print(
-                    "error: %s/: cannot start %s"
-                    % (p.RelPath(local=opt.this_manifest_only), nb),
-                    file=sys.stderr,
+        if err_projects:
+            for p in err_projects:
+                logger.error(
+                    "error: %s/: cannot start %s",
+                    p.RelPath(local=opt.this_manifest_only),
+                    nb,
                 )
-            sys.exit(1)
+            msg_fmt = "cannot start %d project(s)"
+            self.git_event_log.ErrorEvent(
+                msg_fmt % (len(err_projects)), msg_fmt
+            )
+            raise StartError(aggregate_errors=err)

subcmds/status.py

@@ -17,9 +17,9 @@ import glob
 import io
 import os
 
-from command import DEFAULT_LOCAL_JOBS, PagedCommand
-
 from color import Coloring
+from command import DEFAULT_LOCAL_JOBS
+from command import PagedCommand
 import platform_utils
 
 
@@ -82,13 +82,13 @@ the following meanings:
         p.add_option(
             "-o",
             "--orphans",
-            dest="orphans",
             action="store_true",
             help="include objects in working directory outside of repo "
             "projects",
         )
 
-    def _StatusHelper(self, quiet, local, project):
+    @classmethod
+    def _StatusHelper(cls, quiet, local, project_idx):
         """Obtains the status for a specific project.
 
         Obtains the status for a project, redirecting the output to
@@ -99,12 +99,13 @@ the following meanings:
             local: a boolean, if True, the path is relative to the local
                 (sub)manifest.  If false, the path is relative to the outermost
                 manifest.
-            project: Project to get status of.
+            project_idx: Project index to get status of.
 
         Returns:
             The status of the project.
         """
         buf = io.StringIO()
+        project = cls.get_parallel_context()["projects"][project_idx]
         ret = project.PrintWorkTreeStatus(
             quiet=quiet, output_redir=buf, local=local
         )
@@ -143,15 +144,18 @@ the following meanings:
                     ret += 1
             return ret
 
-        counter = self.ExecuteInParallel(
-            opt.jobs,
-            functools.partial(
-                self._StatusHelper, opt.quiet, opt.this_manifest_only
-            ),
-            all_projects,
-            callback=_ProcessResults,
-            ordered=True,
-        )
+        with self.ParallelContext():
+            self.get_parallel_context()["projects"] = all_projects
+            counter = self.ExecuteInParallel(
+                opt.jobs,
+                functools.partial(
+                    self._StatusHelper, opt.quiet, opt.this_manifest_only
+                ),
+                range(len(all_projects)),
+                callback=_ProcessResults,
+                ordered=True,
+                chunksize=1,
+            )
 
         if not opt.quiet and len(all_projects) == counter:
             print("nothing to commit (working directory clean)")

subcmds/upload.py

@@ -19,16 +19,26 @@ import re
 import sys
 from typing import List
 
-from command import DEFAULT_LOCAL_JOBS, InteractiveCommand
+from command import DEFAULT_LOCAL_JOBS
+from command import InteractiveCommand
 from editor import Editor
+from error import GitError
+from error import SilentRepoExitError
 from error import UploadError
 from git_command import GitCommand
 from git_refs import R_HEADS
 from hooks import RepoHook
 from project import ReviewableBranch
+from repo_logging import RepoLogger
+from subcmds.sync import LocalSyncState
 
 
 _DEFAULT_UNUSUAL_COMMIT_THRESHOLD = 5
+logger = RepoLogger(__file__)
+
+
+class UploadExitError(SilentRepoExitError):
+    """Indicates that there is an upload command error requiring a sys exit."""
 
 
 def _VerifyPendingCommits(branches: List[ReviewableBranch]) -> bool:
@@ -62,16 +72,16 @@ def _VerifyPendingCommits(branches: List[ReviewableBranch]) -> bool:
     # If any branch has many commits, prompt the user.
     if many_commits:
         if len(branches) > 1:
-            print(
+            logger.warning(
                 "ATTENTION: One or more branches has an unusually high number "
                 "of commits."
             )
         else:
-            print(
+            logger.warning(
                 "ATTENTION: You are uploading an unusually high number of "
                 "commits."
             )
-        print(
+        logger.warning(
             "YOU PROBABLY DO NOT MEAN TO DO THIS. (Did you rebase across "
             "branches?)"
         )
@@ -85,8 +95,8 @@ def _VerifyPendingCommits(branches: List[ReviewableBranch]) -> bool:
 
 def _die(fmt, *args):
     msg = fmt % args
-    print("error: %s" % msg, file=sys.stderr)
-    sys.exit(1)
+    logger.error("error: %s", msg)
+    raise UploadExitError(msg)
 
 
 def _SplitEmails(values):
@@ -208,9 +218,14 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
     def _Options(self, p):
         p.add_option(
             "-t",
+            "--topic-branch",
             dest="auto_topic",
             action="store_true",
-            help="send local branch name to Gerrit Code Review",
+            help="set the topic to the local branch name",
+        )
+        p.add_option(
+            "--topic",
+            help="set topic for the change",
         )
         p.add_option(
             "--hashtag",
@@ -234,6 +249,12 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
             default=[],
             help="add a label when uploading",
         )
+        p.add_option(
+            "--pd",
+            "--patchset-description",
+            dest="patchset_description",
+            help="description for patchset",
+        )
         p.add_option(
             "--re",
             "--reviewers",
@@ -246,7 +267,6 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
             "--cc",
             type="string",
             action="append",
-            dest="cc",
             help="also send email to these email addresses",
         )
         p.add_option(
@@ -260,7 +280,6 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
         p.add_option(
             "-c",
             "--current-branch",
-            dest="current_branch",
             action="store_true",
             help="upload current git branch",
         )
@@ -289,7 +308,6 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
             "-p",
             "--private",
             action="store_true",
-            dest="private",
             default=False,
             help="upload as a private change (deprecated; use --wip)",
         )
@@ -297,7 +315,6 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
             "-w",
             "--wip",
             action="store_true",
-            dest="wip",
             default=False,
             help="upload as a work-in-progress change",
         )
@@ -528,136 +545,123 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
         except (AttributeError, IndexError):
             return ""
 
-    def _UploadAndReport(self, opt, todo, original_people):
+    def _UploadBranch(self, opt, branch, original_people):
+        """Upload Branch."""
+        people = copy.deepcopy(original_people)
+        self._AppendAutoList(branch, people)
+
+        # Check if topic branches should be sent to the server during
+        # upload.
+        if opt.topic is None:
+            if opt.auto_topic is not True:
+                key = "review.%s.uploadtopic" % branch.project.remote.review
+                opt.auto_topic = branch.project.config.GetBoolean(key)
+            if opt.auto_topic:
+                opt.topic = branch.name
+
+        def _ExpandCommaList(value):
+            """Split |value| up into comma delimited entries."""
+            if not value:
+                return
+            for ret in value.split(","):
+                ret = ret.strip()
+                if ret:
+                    yield ret
+
+        # Check if hashtags should be included.
+        key = "review.%s.uploadhashtags" % branch.project.remote.review
+        hashtags = set(_ExpandCommaList(branch.project.config.GetString(key)))
+        for tag in opt.hashtags:
+            hashtags.update(_ExpandCommaList(tag))
+        if opt.hashtag_branch:
+            hashtags.add(branch.name)
+
+        # Check if labels should be included.
+        key = "review.%s.uploadlabels" % branch.project.remote.review
+        labels = set(_ExpandCommaList(branch.project.config.GetString(key)))
+        for label in opt.labels:
+            labels.update(_ExpandCommaList(label))
+
+        # Handle e-mail notifications.
+        if opt.notify is False:
+            notify = "NONE"
+        else:
+            key = "review.%s.uploadnotify" % branch.project.remote.review
+            notify = branch.project.config.GetString(key)
+
+        destination = opt.dest_branch or branch.project.dest_branch
+
+        if branch.project.dest_branch and not opt.dest_branch:
+            merge_branch = self._GetMergeBranch(
+                branch.project, local_branch=branch.name
+            )
+
+            full_dest = destination
+            if not full_dest.startswith(R_HEADS):
+                full_dest = R_HEADS + full_dest
+            full_revision = branch.project.revisionExpr
+            if not full_revision.startswith(R_HEADS):
+                full_revision = R_HEADS + full_revision
+
+            # If the merge branch of the local branch is different from
+            # the project's revision AND destination, this might not be
+            # intentional.
+            if (
+                merge_branch
+                and merge_branch != full_revision
+                and merge_branch != full_dest
+            ):
+                print(
+                    f"For local branch {branch.name}: merge branch "
+                    f"{merge_branch} does not match destination branch "
+                    f"{destination} and revision {branch.project.revisionExpr}"
+                )
+                print("skipping upload.")
+                print(
+                    f"Please use `--destination {destination}` if this "
+                    "is intentional"
+                )
+                branch.uploaded = False
+                return
+
+        # If using superproject, add the root repo as a push option.
+        manifest = branch.project.manifest
+        push_options = list(opt.push_options)
+        if manifest.manifestProject.use_superproject:
+            sp = manifest.superproject
+            if sp:
+                r_id = sp.repo_id
+                if r_id:
+                    push_options.append(f"custom-keyed-value=rootRepo:{r_id}")
+
+        branch.UploadForReview(
+            people,
+            dryrun=opt.dryrun,
+            topic=opt.topic,
+            hashtags=hashtags,
+            labels=labels,
+            private=opt.private,
+            notify=notify,
+            wip=opt.wip,
+            ready=opt.ready,
+            dest_branch=destination,
+            validate_certs=opt.validate_certs,
+            push_options=push_options,
+            patchset_description=opt.patchset_description,
+        )
+
+        branch.uploaded = True
+
+    def _UploadAndReport(self, opt, todo, people):
         have_errors = False
+        aggregate_errors = []
         for branch in todo:
             try:
-                people = copy.deepcopy(original_people)
-                self._AppendAutoList(branch, people)
-
-                # Check if there are local changes that may have been forgotten.
-                changes = branch.project.UncommitedFiles()
-                if opt.ignore_untracked_files:
-                    untracked = set(branch.project.UntrackedFiles())
-                    changes = [x for x in changes if x not in untracked]
-
-                if changes:
-                    key = "review.%s.autoupload" % branch.project.remote.review
-                    answer = branch.project.config.GetBoolean(key)
-
-                    # If they want to auto upload, let's not ask because it
-                    # could be automated.
-                    if answer is None:
-                        print()
-                        print(
-                            "Uncommitted changes in %s (did you forget to "
-                            "amend?):" % branch.project.name
-                        )
-                        print("\n".join(changes))
-                        print("Continue uploading? (y/N) ", end="", flush=True)
-                        if opt.yes:
-                            print("<--yes>")
-                            a = "yes"
-                        else:
-                            a = sys.stdin.readline().strip().lower()
-                        if a not in ("y", "yes", "t", "true", "on"):
-                            print("skipping upload", file=sys.stderr)
-                            branch.uploaded = False
-                            branch.error = "User aborted"
-                            continue
-
-                # Check if topic branches should be sent to the server during
-                # upload.
-                if opt.auto_topic is not True:
-                    key = "review.%s.uploadtopic" % branch.project.remote.review
-                    opt.auto_topic = branch.project.config.GetBoolean(key)
-
-                def _ExpandCommaList(value):
-                    """Split |value| up into comma delimited entries."""
-                    if not value:
-                        return
-                    for ret in value.split(","):
-                        ret = ret.strip()
-                        if ret:
-                            yield ret
-
-                # Check if hashtags should be included.
-                key = "review.%s.uploadhashtags" % branch.project.remote.review
-                hashtags = set(
-                    _ExpandCommaList(branch.project.config.GetString(key))
-                )
-                for tag in opt.hashtags:
-                    hashtags.update(_ExpandCommaList(tag))
-                if opt.hashtag_branch:
-                    hashtags.add(branch.name)
-
-                # Check if labels should be included.
-                key = "review.%s.uploadlabels" % branch.project.remote.review
-                labels = set(
-                    _ExpandCommaList(branch.project.config.GetString(key))
-                )
-                for label in opt.labels:
-                    labels.update(_ExpandCommaList(label))
-
-                # Handle e-mail notifications.
-                if opt.notify is False:
-                    notify = "NONE"
-                else:
-                    key = (
-                        "review.%s.uploadnotify" % branch.project.remote.review
-                    )
-                    notify = branch.project.config.GetString(key)
-
-                destination = opt.dest_branch or branch.project.dest_branch
-
-                if branch.project.dest_branch and not opt.dest_branch:
-                    merge_branch = self._GetMergeBranch(
-                        branch.project, local_branch=branch.name
-                    )
-
-                    full_dest = destination
-                    if not full_dest.startswith(R_HEADS):
-                        full_dest = R_HEADS + full_dest
-
-                    # If the merge branch of the local branch is different from
-                    # the project's revision AND destination, this might not be
-                    # intentional.
-                    if (
-                        merge_branch
-                        and merge_branch != branch.project.revisionExpr
-                        and merge_branch != full_dest
-                    ):
-                        print(
-                            f"For local branch {branch.name}: merge branch "
-                            f"{merge_branch} does not match destination branch "
-                            f"{destination}"
-                        )
-                        print("skipping upload.")
-                        print(
-                            f"Please use `--destination {destination}` if this "
-                            "is intentional"
-                        )
-                        branch.uploaded = False
-                        continue
-
-                branch.UploadForReview(
-                    people,
-                    dryrun=opt.dryrun,
-                    auto_topic=opt.auto_topic,
-                    hashtags=hashtags,
-                    labels=labels,
-                    private=opt.private,
-                    notify=notify,
-                    wip=opt.wip,
-                    ready=opt.ready,
-                    dest_branch=destination,
-                    validate_certs=opt.validate_certs,
-                    push_options=opt.push_options,
-                )
-
-                branch.uploaded = True
-            except UploadError as e:
+                self._UploadBranch(opt, branch, people)
+            except (UploadError, GitError) as e:
+                self.git_event_log.ErrorEvent(f"upload error: {e}")
                 branch.error = e
+                aggregate_errors.append(e)
                 branch.uploaded = False
                 have_errors = True
 
@@ -696,7 +700,7 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
                 )
 
         if have_errors:
-            sys.exit(1)
+            raise UploadExitError(aggregate_errors=aggregate_errors)
 
     def _GetMergeBranch(self, project, local_branch=None):
         if local_branch is None:
@@ -718,16 +722,17 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
         merge_branch = p.stdout.strip()
         return merge_branch
 
-    @staticmethod
-    def _GatherOne(opt, project):
+    @classmethod
+    def _GatherOne(cls, opt, project_idx):
         """Figure out the upload status for |project|."""
+        project = cls.get_parallel_context()["projects"][project_idx]
         if opt.current_branch:
             cbr = project.CurrentBranch
             up_branch = project.GetUploadableBranch(cbr)
             avail = [up_branch] if up_branch else None
         else:
             avail = project.GetUploadableBranches(opt.branch)
-        return (project, avail)
+        return (project_idx, avail)
 
     def Execute(self, opt, args):
         projects = self.GetProjects(
@@ -737,40 +742,37 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
         def _ProcessResults(_pool, _out, results):
             pending = []
             for result in results:
-                project, avail = result
+                project_idx, avail = result
+                project = projects[project_idx]
                 if avail is None:
-                    print(
+                    logger.error(
                         'repo: error: %s: Unable to upload branch "%s". '
                         "You might be able to fix the branch by running:\n"
-                        "  git branch --set-upstream-to m/%s"
-                        % (
-                            project.RelPath(local=opt.this_manifest_only),
-                            project.CurrentBranch,
-                            project.manifest.branch,
-                        ),
-                        file=sys.stderr,
+                        "  git branch --set-upstream-to m/%s",
+                        project.RelPath(local=opt.this_manifest_only),
+                        project.CurrentBranch,
+                        project.manifest.branch,
                     )
                 elif avail:
-                    pending.append(result)
+                    pending.append((project, avail))
             return pending
 
-        pending = self.ExecuteInParallel(
-            opt.jobs,
-            functools.partial(self._GatherOne, opt),
-            projects,
-            callback=_ProcessResults,
-        )
+        with self.ParallelContext():
+            self.get_parallel_context()["projects"] = projects
+            pending = self.ExecuteInParallel(
+                opt.jobs,
+                functools.partial(self._GatherOne, opt),
+                range(len(projects)),
+                callback=_ProcessResults,
+            )
 
         if not pending:
             if opt.branch is None:
-                print(
-                    "repo: error: no branches ready for upload", file=sys.stderr
-                )
+                logger.error("repo: error: no branches ready for upload")
             else:
-                print(
-                    'repo: error: no branches named "%s" ready for upload'
-                    % (opt.branch,),
-                    file=sys.stderr,
+                logger.error(
+                    'repo: error: no branches named "%s" ready for upload',
+                    opt.branch,
                 )
             return 1
 
@@ -799,6 +801,11 @@ Gerrit Code Review:  https://www.gerritcodereview.com/
             if not hook.Run(
                 project_list=pending_proj_names, worktree_list=pending_worktrees
             ):
+                if LocalSyncState(manifest).IsPartiallySynced():
+                    logger.error(
+                        "Partially synced tree detected. Syncing all projects "
+                        "may resolve issues you're seeing."
+                    )
                 ret = 1
         if ret:
             return ret

subcmds/version.py

@@ -15,8 +15,11 @@
 import platform
 import sys
 
-from command import Command, MirrorSafeCommand
-from git_command import git, RepoSourceVersion, user_agent
+from command import Command
+from command import MirrorSafeCommand
+from git_command import git
+from git_command import RepoSourceVersion
+from git_command import user_agent
 from git_refs import HEAD
 from wrapper import Wrapper
 
@@ -39,35 +42,28 @@ class Version(Command, MirrorSafeCommand):
         # These might not be the same.  Report them both.
         src_ver = RepoSourceVersion()
         rp_ver = rp.bare_git.describe(HEAD)
-        print("repo version %s" % rp_ver)
-        print("       (from %s)" % rem.url)
-        print("       (tracking %s)" % branch.merge)
-        print("       (%s)" % rp.bare_git.log("-1", "--format=%cD", HEAD))
+        print(f"repo version {rp_ver}")
+        print(f"       (from {rem.url})")
+        print(f"       (tracking {branch.merge})")
+        print(f"       ({rp.bare_git.log('-1', '--format=%cD', HEAD)})")
 
         if self.wrapper_path is not None:
-            print("repo launcher version %s" % self.wrapper_version)
-            print("       (from %s)" % self.wrapper_path)
+            print(f"repo launcher version {self.wrapper_version}")
+            print(f"       (from {self.wrapper_path})")
 
             if src_ver != rp_ver:
-                print("       (currently at %s)" % src_ver)
+                print(f"       (currently at {src_ver})")
 
-        print("repo User-Agent %s" % user_agent.repo)
-        print("git %s" % git.version_tuple().full)
-        print("git User-Agent %s" % user_agent.git)
-        print("Python %s" % sys.version)
+        print(f"repo User-Agent {user_agent.repo}")
+        print(f"git {git.version_tuple().full}")
+        print(f"git User-Agent {user_agent.git}")
+        print(f"Python {sys.version}")
         uname = platform.uname()
         if sys.version_info.major < 3:
             # Python 3 returns a named tuple, but Python 2 is simpler.
             print(uname)
         else:
-            print(
-                "OS %s %s (%s)" % (uname.system, uname.release, uname.version)
-            )
-            print(
-                "CPU %s (%s)"
-                % (
-                    uname.machine,
-                    uname.processor if uname.processor else "unknown",
-                )
-            )
+            print(f"OS {uname.system} {uname.release} ({uname.version})")
+            processor = uname.processor if uname.processor else "unknown"
+            print(f"CPU {uname.machine} ({processor})")
         print("Bug reports:", Wrapper().BUG_URL)

subcmds/wipe.py

@@ -0,0 +1,184 @@
+# Copyright (C) 2025 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+from typing import List
+
+from command import Command
+from error import GitError
+from error import RepoExitError
+import platform_utils
+from project import DeleteWorktreeError
+
+
+class Error(RepoExitError):
+    """Exit error when wipe command fails."""
+
+
+class Wipe(Command):
+    """Delete projects from the worktree and .repo"""
+
+    COMMON = True
+    helpSummary = "Wipe projects from the worktree"
+    helpUsage = """
+%prog <project>...
+"""
+    helpDescription = """
+The '%prog' command removes the specified projects from the worktree
+(the checked out source code) and deletes the project's git data from `.repo`.
+
+This is a destructive operation and cannot be undone.
+
+Projects can be specified either by name, or by a relative or absolute path
+to the project's local directory.
+
+Examples:
+
+  # Wipe the project "platform/build" by name:
+  $ repo wipe platform/build
+
+  # Wipe the project at the path "build/make":
+  $ repo wipe build/make
+"""
+
+    def _Options(self, p):
+        # TODO(crbug.com/gerrit/393383056): Add --broken option to scan and
+        # wipe broken projects.
+        p.add_option(
+            "-f",
+            "--force",
+            action="store_true",
+            help="force wipe shared projects and uncommitted changes",
+        )
+        p.add_option(
+            "--force-uncommitted",
+            action="store_true",
+            help="force wipe even if there are uncommitted changes",
+        )
+        p.add_option(
+            "--force-shared",
+            action="store_true",
+            help="force wipe even if the project shares an object directory",
+        )
+
+    def ValidateOptions(self, opt, args: List[str]):
+        if not args:
+            self.Usage()
+
+    def Execute(self, opt, args: List[str]):
+        # Get all projects to handle shared object directories.
+        all_projects = self.GetProjects(None, all_manifests=True, groups="all")
+        projects_to_wipe = self.GetProjects(args, all_manifests=True)
+        relpaths_to_wipe = {p.relpath for p in projects_to_wipe}
+
+        # Build a map from objdir to the relpaths of projects that use it.
+        objdir_map = {}
+        for p in all_projects:
+            objdir_map.setdefault(p.objdir, set()).add(p.relpath)
+
+        uncommitted_projects = []
+        shared_objdirs = {}
+        objdirs_to_delete = set()
+
+        for project in projects_to_wipe:
+            if project == self.manifest.manifestProject:
+                raise Error(
+                    f"error: cannot wipe the manifest project: {project.name}"
+                )
+
+            try:
+                if project.HasChanges():
+                    uncommitted_projects.append(project.name)
+            except GitError:
+                uncommitted_projects.append(f"{project.name} (corrupted)")
+
+            users = objdir_map.get(project.objdir, {project.relpath})
+            is_shared = not users.issubset(relpaths_to_wipe)
+            if is_shared:
+                shared_objdirs.setdefault(project.objdir, set()).update(users)
+            else:
+                objdirs_to_delete.add(project.objdir)
+
+        block_uncommitted = uncommitted_projects and not (
+            opt.force or opt.force_uncommitted
+        )
+        block_shared = shared_objdirs and not (opt.force or opt.force_shared)
+
+        if block_uncommitted or block_shared:
+            error_messages = []
+            if block_uncommitted:
+                error_messages.append(
+                    "The following projects have uncommitted changes or are "
+                    "corrupted:\n"
+                    + "\n".join(f" - {p}" for p in sorted(uncommitted_projects))
+                )
+            if block_shared:
+                shared_dir_messages = []
+                for objdir, users in sorted(shared_objdirs.items()):
+                    other_users = users - relpaths_to_wipe
+                    projects_to_wipe_in_dir = users & relpaths_to_wipe
+                    message = f"""Object directory {objdir} is shared by:
+  Projects to be wiped: {', '.join(sorted(projects_to_wipe_in_dir))}
+  Projects not to be wiped: {', '.join(sorted(other_users))}"""
+                    shared_dir_messages.append(message)
+                error_messages.append(
+                    "The following projects have shared object directories:\n"
+                    + "\n".join(sorted(shared_dir_messages))
+                )
+
+            if block_uncommitted and block_shared:
+                error_messages.append(
+                    "Use --force to wipe anyway, or --force-uncommitted and "
+                    "--force-shared to specify."
+                )
+            elif block_uncommitted:
+                error_messages.append("Use --force-uncommitted to wipe anyway.")
+            else:
+                error_messages.append("Use --force-shared to wipe anyway.")
+
+            raise Error("\n\n".join(error_messages))
+
+        # If we are here, either there were no issues, or --force was used.
+        # Proceed with wiping.
+        successful_wipes = set()
+
+        for project in projects_to_wipe:
+            try:
+                # Force the delete here since we've already performed our
+                # own safety checks above.
+                project.DeleteWorktree(force=True, verbose=opt.verbose)
+                successful_wipes.add(project.relpath)
+            except DeleteWorktreeError as e:
+                print(
+                    f"error: failed to wipe {project.name}: {e}",
+                    file=sys.stderr,
+                )
+
+        # Clean up object directories only if all projects using them were
+        # successfully wiped.
+        for objdir in objdirs_to_delete:
+            users = objdir_map.get(objdir, set())
+            # Check if every project that uses this objdir has been
+            # successfully processed. If a project failed to be wiped, don't
+            # delete the object directory, or we'll corrupt the remaining
+            # project.
+            if users.issubset(successful_wipes):
+                if os.path.exists(objdir):
+                    if opt.verbose:
+                        print(
+                            f"Deleting objects directory: {objdir}",
+                            file=sys.stderr,
+                        )
+                    platform_utils.rmtree(objdir)

tests/conftest.py

@@ -1,4 +1,4 @@
-# Copyright 2022 The Android Open Source Project
+# Copyright (C) 2022 The Android Open Source Project
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,8 +14,11 @@
 
 """Common fixtures for pytests."""
 
+import pathlib
+
 import pytest
 
+import platform_utils
 import repo_trace
 
 
@@ -23,3 +26,58 @@ import repo_trace
 def disable_repo_trace(tmp_path):
     """Set an environment marker to relax certain strict checks for test code."""  # noqa: E501
     repo_trace._TRACE_FILE = str(tmp_path / "TRACE_FILE_from_test")
+
+
+# adapted from pytest-home 0.5.1
+def _set_home(monkeypatch, path: pathlib.Path):
+    """
+    Set the home dir using a pytest monkeypatch context.
+    """
+    win = platform_utils.isWindows()
+    vars = ["HOME"] + win * ["USERPROFILE"]
+    for var in vars:
+        monkeypatch.setenv(var, str(path))
+    return path
+
+
+# copied from
+# https://github.com/pytest-dev/pytest/issues/363#issuecomment-1335631998
+@pytest.fixture(scope="session")
+def monkeysession():
+    with pytest.MonkeyPatch.context() as mp:
+        yield mp
+
+
+@pytest.fixture(autouse=True, scope="session")
+def session_tmp_home_dir(tmp_path_factory, monkeysession):
+    """Set HOME to a temporary directory, avoiding user's .gitconfig.
+
+    b/302797407
+
+    Set home at session scope to take effect prior to
+    ``test_wrapper.GitCheckoutTestCase.setUpClass``.
+    """
+    return _set_home(monkeysession, tmp_path_factory.mktemp("home"))
+
+
+# adapted from pytest-home 0.5.1
+@pytest.fixture(autouse=True)
+def tmp_home_dir(monkeypatch, tmp_path_factory):
+    """Set HOME to a temporary directory.
+
+    Ensures that state doesn't accumulate in $HOME across tests.
+
+    Note that in conjunction with session_tmp_homedir, the HOME
+    dir is patched twice, once at session scope, and then again at
+    the function scope.
+    """
+    return _set_home(monkeypatch, tmp_path_factory.mktemp("home"))
+
+
+@pytest.fixture(autouse=True)
+def setup_user_identity(monkeysession, scope="session"):
+    """Set env variables for author and committer name and email."""
+    monkeysession.setenv("GIT_AUTHOR_NAME", "Foo Bar")
+    monkeysession.setenv("GIT_COMMITTER_NAME", "Foo Bar")
+    monkeysession.setenv("GIT_AUTHOR_EMAIL", "foo@bar.baz")
+    monkeysession.setenv("GIT_COMMITTER_EMAIL", "foo@bar.baz")

tests/fixtures/gitc_config

@@ -1 +0,0 @@
-gitc_dir=/test/usr/local/google/gitc

tests/fixtures/test.gitconfig

@@ -11,3 +11,11 @@
 	intk = 10k
 	intm = 10m
 	intg = 10g
+
+[color "status"]
+	one = yellow
+	two = magenta cyan
+	three = black red ul
+	reset = reset
+	none
+	empty =

tests/test_color.py

@@ -0,0 +1,74 @@
+# Copyright (C) 2024 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unittests for the color.py module."""
+
+import os
+import unittest
+
+import color
+import git_config
+
+
+def fixture(*paths):
+    """Return a path relative to test/fixtures."""
+    return os.path.join(os.path.dirname(__file__), "fixtures", *paths)
+
+
+class ColoringTests(unittest.TestCase):
+    """tests of the Coloring class."""
+
+    def setUp(self):
+        """Create a GitConfig object using the test.gitconfig fixture."""
+        config_fixture = fixture("test.gitconfig")
+        self.config = git_config.GitConfig(config_fixture)
+        color.SetDefaultColoring("true")
+        self.color = color.Coloring(self.config, "status")
+
+    def test_Color_Parse_all_params_none(self):
+        """all params are None"""
+        val = self.color._parse(None, None, None, None)
+        self.assertEqual("", val)
+
+    def test_Color_Parse_first_parameter_none(self):
+        """check fg & bg & attr"""
+        val = self.color._parse(None, "black", "red", "ul")
+        self.assertEqual("\x1b[4;30;41m", val)
+
+    def test_Color_Parse_one_entry(self):
+        """check fg"""
+        val = self.color._parse("one", None, None, None)
+        self.assertEqual("\033[33m", val)
+
+    def test_Color_Parse_two_entry(self):
+        """check fg & bg"""
+        val = self.color._parse("two", None, None, None)
+        self.assertEqual("\033[35;46m", val)
+
+    def test_Color_Parse_three_entry(self):
+        """check fg & bg & attr"""
+        val = self.color._parse("three", None, None, None)
+        self.assertEqual("\033[4;30;41m", val)
+
+    def test_Color_Parse_reset_entry(self):
+        """check reset entry"""
+        val = self.color._parse("reset", None, None, None)
+        self.assertEqual("\033[m", val)
+
+    def test_Color_Parse_empty_entry(self):
+        """check empty entry"""
+        val = self.color._parse("none", "blue", "white", "dim")
+        self.assertEqual("\033[2;34;47m", val)
+        val = self.color._parse("empty", "green", "white", "bold")
+        self.assertEqual("\033[1;32;47m", val)

tests/test_error.py

@@ -1,4 +1,4 @@
-# Copyright 2021 The Android Open Source Project
+# Copyright (C) 2021 The Android Open Source Project
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,7 +18,21 @@ import inspect
 import pickle
 import unittest
 
+import command
 import error
+import fetch
+import git_command
+import project
+from subcmds import all_modules
+
+
+imports = all_modules + [
+    error,
+    project,
+    git_command,
+    fetch,
+    command,
+]
 
 
 class PickleTests(unittest.TestCase):
@@ -26,10 +40,11 @@ class PickleTests(unittest.TestCase):
 
     def getExceptions(self):
         """Return all our custom exceptions."""
-        for name in dir(error):
-            cls = getattr(error, name)
-            if isinstance(cls, type) and issubclass(cls, Exception):
-                yield cls
+        for entry in imports:
+            for name in dir(entry):
+                cls = getattr(entry, name)
+                if isinstance(cls, type) and issubclass(cls, Exception):
+                    yield cls
 
     def testExceptionLookup(self):
         """Make sure our introspection logic works."""

tests/test_git_command.py

@@ -1,4 +1,4 @@
-# Copyright 2019 The Android Open Source Project
+# Copyright (C) 2019 The Android Open Source Project
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,14 +14,14 @@
 
 """Unittests for the git_command.py module."""
 
-import re
+import io
 import os
+import re
+import subprocess
 import unittest
+from unittest import mock
 
-try:
-    from unittest import mock
-except ImportError:
-    import mock
+import pytest
 
 import git_command
 import wrapper
@@ -65,6 +65,169 @@ class GitCommandTest(unittest.TestCase):
         )
 
 
+class GitCommandWaitTest(unittest.TestCase):
+    """Tests the GitCommand class .Wait()"""
+
+    def setUp(self):
+        class MockPopen:
+            rc = 0
+
+            def __init__(self):
+                self.stdout = io.BufferedReader(io.BytesIO())
+                self.stderr = io.BufferedReader(io.BytesIO())
+
+            def communicate(
+                self, input: str = None, timeout: float = None
+            ) -> [str, str]:
+                """Mock communicate fn."""
+                return ["", ""]
+
+            def wait(self, timeout=None):
+                return self.rc
+
+        self.popen = popen = MockPopen()
+
+        def popen_mock(*args, **kwargs):
+            return popen
+
+        def realpath_mock(val):
+            return val
+
+        mock.patch.object(subprocess, "Popen", side_effect=popen_mock).start()
+
+        mock.patch.object(
+            os.path, "realpath", side_effect=realpath_mock
+        ).start()
+
+    def tearDown(self):
+        mock.patch.stopall()
+
+    def test_raises_when_verify_non_zero_result(self):
+        self.popen.rc = 1
+        r = git_command.GitCommand(None, ["status"], verify_command=True)
+        with self.assertRaises(git_command.GitCommandError):
+            r.Wait()
+
+    def test_returns_when_no_verify_non_zero_result(self):
+        self.popen.rc = 1
+        r = git_command.GitCommand(None, ["status"], verify_command=False)
+        self.assertEqual(1, r.Wait())
+
+    def test_default_returns_non_zero_result(self):
+        self.popen.rc = 1
+        r = git_command.GitCommand(None, ["status"])
+        self.assertEqual(1, r.Wait())
+
+
+class GitCommandStreamLogsTest(unittest.TestCase):
+    """Tests the GitCommand class stderr log streaming cases."""
+
+    def setUp(self):
+        self.mock_process = mock.MagicMock()
+        self.mock_process.communicate.return_value = (None, None)
+        self.mock_process.wait.return_value = 0
+
+        self.mock_popen = mock.MagicMock()
+        self.mock_popen.return_value = self.mock_process
+        mock.patch("subprocess.Popen", self.mock_popen).start()
+
+    def tearDown(self):
+        mock.patch.stopall()
+
+    def test_does_not_stream_logs_when_input_is_set(self):
+        git_command.GitCommand(None, ["status"], input="foo")
+
+        self.mock_popen.assert_called_once_with(
+            ["git", "status"],
+            cwd=None,
+            env=mock.ANY,
+            encoding="utf-8",
+            errors="backslashreplace",
+            stdin=subprocess.PIPE,
+            stdout=None,
+            stderr=None,
+        )
+        self.mock_process.communicate.assert_called_once_with(input="foo")
+        self.mock_process.stderr.read1.assert_not_called()
+
+    def test_does_not_stream_logs_when_stdout_is_set(self):
+        git_command.GitCommand(None, ["status"], capture_stdout=True)
+
+        self.mock_popen.assert_called_once_with(
+            ["git", "status"],
+            cwd=None,
+            env=mock.ANY,
+            encoding="utf-8",
+            errors="backslashreplace",
+            stdin=None,
+            stdout=subprocess.PIPE,
+            stderr=None,
+        )
+        self.mock_process.communicate.assert_called_once_with(input=None)
+        self.mock_process.stderr.read1.assert_not_called()
+
+    def test_does_not_stream_logs_when_stderr_is_set(self):
+        git_command.GitCommand(None, ["status"], capture_stderr=True)
+
+        self.mock_popen.assert_called_once_with(
+            ["git", "status"],
+            cwd=None,
+            env=mock.ANY,
+            encoding="utf-8",
+            errors="backslashreplace",
+            stdin=None,
+            stdout=None,
+            stderr=subprocess.PIPE,
+        )
+        self.mock_process.communicate.assert_called_once_with(input=None)
+        self.mock_process.stderr.read1.assert_not_called()
+
+    def test_does_not_stream_logs_when_merge_output_is_set(self):
+        git_command.GitCommand(None, ["status"], merge_output=True)
+
+        self.mock_popen.assert_called_once_with(
+            ["git", "status"],
+            cwd=None,
+            env=mock.ANY,
+            encoding="utf-8",
+            errors="backslashreplace",
+            stdin=None,
+            stdout=None,
+            stderr=subprocess.STDOUT,
+        )
+        self.mock_process.communicate.assert_called_once_with(input=None)
+        self.mock_process.stderr.read1.assert_not_called()
+
+    @mock.patch("sys.stderr")
+    def test_streams_stderr_when_no_stream_is_set(self, mock_stderr):
+        logs = "\n".join(
+            [
+                "Enumerating objects: 5, done.",
+                "Counting objects: 100% (5/5), done.",
+                "Writing objects: 100% (3/3), 330 bytes | 330 KiB/s, done.",
+                "remote: Processing changes: refs: 1, new: 1, done ",
+                "remote: SUCCESS",
+            ]
+        )
+        self.mock_process.stderr = io.BufferedReader(
+            io.BytesIO(bytes(logs, "utf-8"))
+        )
+
+        cmd = git_command.GitCommand(None, ["push"])
+
+        self.mock_popen.assert_called_once_with(
+            ["git", "push"],
+            cwd=None,
+            env=mock.ANY,
+            stdin=None,
+            stdout=None,
+            stderr=subprocess.PIPE,
+        )
+        self.mock_process.communicate.assert_not_called()
+        mock_stderr.write.assert_called_once_with(logs)
+        self.assertEqual(cmd.stderr, logs)
+
+
 class GitCallUnitTest(unittest.TestCase):
     """Tests the _GitCall class (via git_command.git)."""
 
@@ -102,6 +265,7 @@ class UserAgentUnitTest(unittest.TestCase):
         m = re.match(r"^[^ ]+$", os_name)
         self.assertIsNotNone(m)
 
+    @pytest.mark.skip_cq("TODO(b/266734831): Find out why this fails in CQ")
     def test_smoke_repo(self):
         """Make sure repo UA returns something useful."""
         ua = git_command.user_agent.repo
@@ -110,6 +274,7 @@ class UserAgentUnitTest(unittest.TestCase):
         m = re.match(r"^git-repo/[^ ]+ ([^ ]+) git/[^ ]+ Python/[0-9.]+", ua)
         self.assertIsNotNone(m)
 
+    @pytest.mark.skip_cq("TODO(b/266734831): Find out why this fails in CQ")
     def test_smoke_git(self):
         """Make sure git UA returns something useful."""
         ua = git_command.user_agent.git
@@ -153,12 +318,31 @@ class GitRequireTests(unittest.TestCase):
 
     def test_older_fatal(self):
         """Test fatal require calls with old versions."""
-        with self.assertRaises(SystemExit) as e:
+        with self.assertRaises(git_command.GitRequireError) as e:
             git_command.git_require((2,), fail=True)
             self.assertNotEqual(0, e.code)
 
     def test_older_fatal_msg(self):
         """Test fatal require calls with old versions and message."""
-        with self.assertRaises(SystemExit) as e:
+        with self.assertRaises(git_command.GitRequireError) as e:
             git_command.git_require((2,), fail=True, msg="so sad")
             self.assertNotEqual(0, e.code)
+
+
+class GitCommandErrorTest(unittest.TestCase):
+    """Test for the GitCommandError class."""
+
+    def test_augument_stderr(self):
+        self.assertEqual(
+            git_command.GitCommandError(
+                git_stderr="couldn't find remote ref refs/heads/foo"
+            ).suggestion,
+            "Check if the provided ref exists in the remote.",
+        )
+
+        self.assertEqual(
+            git_command.GitCommandError(
+                git_stderr="'foobar' does not appear to be a git repository"
+            ).suggestion,
+            "Are you running this repo command outside of a repo workspace?",
+        )

tests/test_git_config.py

@@ -100,7 +100,7 @@ class GitConfigReadOnlyTests(unittest.TestCase):
             ("intg", 10737418240),
         )
         for key, value in TESTS:
-            self.assertEqual(value, self.config.GetInt("section.%s" % (key,)))
+            self.assertEqual(value, self.config.GetInt(f"section.{key}"))
 
 
 class GitConfigReadWriteTests(unittest.TestCase):

tests/test_git_superproject.py

@@ -21,10 +21,12 @@ import tempfile
 import unittest
 from unittest import mock
 
+import pytest
+from test_manifest_xml import sort_attributes
+
 import git_superproject
 import git_trace2_event_log
 import manifest_xml
-from test_manifest_xml import sort_attributes
 
 
 class SuperprojectTestCase(unittest.TestCase):
@@ -33,7 +35,7 @@ class SuperprojectTestCase(unittest.TestCase):
     PARENT_SID_KEY = "GIT_TRACE2_PARENT_SID"
     PARENT_SID_VALUE = "parent_sid"
     SELF_SID_REGEX = r"repo-\d+T\d+Z-.*"
-    FULL_SID_REGEX = r"^%s/%s" % (PARENT_SID_VALUE, SELF_SID_REGEX)
+    FULL_SID_REGEX = rf"^{PARENT_SID_VALUE}/{SELF_SID_REGEX}"
 
     def setUp(self):
         """Set up superproject every time."""
@@ -107,7 +109,9 @@ class SuperprojectTestCase(unittest.TestCase):
             self.assertRegex(log_entry["sid"], self.FULL_SID_REGEX)
         else:
             self.assertRegex(log_entry["sid"], self.SELF_SID_REGEX)
-        self.assertRegex(log_entry["time"], r"^\d+-\d+-\d+T\d+:\d+:\d+\.\d+Z$")
+        self.assertRegex(
+            log_entry["time"], r"^\d+-\d+-\d+T\d+:\d+:\d+\.\d+\+00:00$"
+        )
 
     def readLog(self, log_path):
         """Helper function to read log data into a list."""
@@ -142,6 +146,7 @@ class SuperprojectTestCase(unittest.TestCase):
         )
         self.assertIsNone(manifest.superproject)
 
+    @pytest.mark.skip_cq("TODO(b/266734831): Find out why this takes 8m+ in CQ")
     def test_superproject_get_superproject_invalid_url(self):
         """Test with an invalid url."""
         manifest = self.getXmlManifest(
@@ -165,6 +170,7 @@ class SuperprojectTestCase(unittest.TestCase):
         self.assertFalse(sync_result.success)
         self.assertTrue(sync_result.fatal)
 
+    @pytest.mark.skip_cq("TODO(b/266734831): Find out why this takes 8m+ in CQ")
     def test_superproject_get_superproject_invalid_branch(self):
         """Test with an invalid branch."""
         manifest = self.getXmlManifest(
@@ -246,7 +252,7 @@ class SuperprojectTestCase(unittest.TestCase):
         os.mkdir(self._superproject._superproject_path)
         manifest_path = self._superproject._WriteManifestFile()
         self.assertIsNotNone(manifest_path)
-        with open(manifest_path, "r") as fp:
+        with open(manifest_path) as fp:
             manifest_xml_data = fp.read()
         self.assertEqual(
             sort_attributes(manifest_xml_data),
@@ -281,7 +287,7 @@ class SuperprojectTestCase(unittest.TestCase):
                     )
                     self.assertIsNotNone(update_result.manifest_path)
                     self.assertFalse(update_result.fatal)
-                    with open(update_result.manifest_path, "r") as fp:
+                    with open(update_result.manifest_path) as fp:
                         manifest_xml_data = fp.read()
                     self.assertEqual(
                         sort_attributes(manifest_xml_data),
@@ -368,7 +374,7 @@ class SuperprojectTestCase(unittest.TestCase):
                     )
                     self.assertIsNotNone(update_result.manifest_path)
                     self.assertFalse(update_result.fatal)
-                    with open(update_result.manifest_path, "r") as fp:
+                    with open(update_result.manifest_path) as fp:
                         manifest_xml_data = fp.read()
                     # Verify platform/vendor/x's project revision hasn't
                     # changed.
@@ -433,7 +439,7 @@ class SuperprojectTestCase(unittest.TestCase):
                     )
                     self.assertIsNotNone(update_result.manifest_path)
                     self.assertFalse(update_result.fatal)
-                    with open(update_result.manifest_path, "r") as fp:
+                    with open(update_result.manifest_path) as fp:
                         manifest_xml_data = fp.read()
                     # Verify platform/vendor/x's project revision hasn't
                     # changed.
@@ -489,7 +495,9 @@ class SuperprojectTestCase(unittest.TestCase):
 
                     self.assertTrue(self._superproject._Fetch())
                     self.assertEqual(
-                        mock_git_command.call_args.args,
+                        # TODO: Once we require Python 3.8+,
+                        #  use 'mock_git_command.call_args.args'.
+                        mock_git_command.call_args[0],
                         (
                             None,
                             [
@@ -509,7 +517,9 @@ class SuperprojectTestCase(unittest.TestCase):
                     # If branch for revision exists, set as --negotiation-tip.
                     self.assertTrue(self._superproject._Fetch())
                     self.assertEqual(
-                        mock_git_command.call_args.args,
+                        # TODO: Once we require Python 3.8+,
+                        #  use 'mock_git_command.call_args.args'.
+                        mock_git_command.call_args[0],
                         (
                             None,
                             [

tests/test_git_trace2_event_log.py

@@ -61,7 +61,7 @@ class EventLogTestCase(unittest.TestCase):
     PARENT_SID_KEY = "GIT_TRACE2_PARENT_SID"
     PARENT_SID_VALUE = "parent_sid"
     SELF_SID_REGEX = r"repo-\d+T\d+Z-.*"
-    FULL_SID_REGEX = r"^%s/%s" % (PARENT_SID_VALUE, SELF_SID_REGEX)
+    FULL_SID_REGEX = rf"^{PARENT_SID_VALUE}/{SELF_SID_REGEX}"
 
     def setUp(self):
         """Load the event_log module every time."""
@@ -90,7 +90,9 @@ class EventLogTestCase(unittest.TestCase):
             self.assertRegex(log_entry["sid"], self.FULL_SID_REGEX)
         else:
             self.assertRegex(log_entry["sid"], self.SELF_SID_REGEX)
-        self.assertRegex(log_entry["time"], r"^\d+-\d+-\d+T\d+:\d+:\d+\.\d+Z$")
+        self.assertRegex(
+            log_entry["time"], r"^\d+-\d+-\d+T\d+:\d+:\d+\.\d+\+00:00$"
+        )
 
     def readLog(self, log_path):
         """Helper function to read log data into a list."""
@@ -148,7 +150,7 @@ class EventLogTestCase(unittest.TestCase):
         <version event>
         <start event>
         """
-        self._event_log_module.StartEvent()
+        self._event_log_module.StartEvent([])
         with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
             log_path = self._event_log_module.Write(path=tempdir)
             self._log_data = self.readLog(log_path)
@@ -211,10 +213,8 @@ class EventLogTestCase(unittest.TestCase):
         <version event>
         <command event>
         """
-        name = "repo"
-        subcommands = ["init" "this"]
         self._event_log_module.CommandEvent(
-            name="repo", subcommands=subcommands
+            name="repo", subcommands=["init", "this"]
         )
         with tempfile.TemporaryDirectory(prefix="event_log_tests") as tempdir:
             log_path = self._event_log_module.Write(path=tempdir)
@@ -223,12 +223,10 @@ class EventLogTestCase(unittest.TestCase):
         self.assertEqual(len(self._log_data), 2)
         command_event = self._log_data[1]
         self.verifyCommonKeys(self._log_data[0], expected_event_name="version")
-        self.verifyCommonKeys(command_event, expected_event_name="command")
+        self.verifyCommonKeys(command_event, expected_event_name="cmd_name")
         # Check for 'command' event specific fields.
         self.assertIn("name", command_event)
-        self.assertIn("subcommands", command_event)
-        self.assertEqual(command_event["name"], name)
-        self.assertEqual(command_event["subcommands"], subcommands)
+        self.assertEqual(command_event["name"], "repo-init-this")
 
     def test_def_params_event_repo_config(self):
         """Test 'def_params' event data outputs only repo config keys.
@@ -338,8 +336,8 @@ class EventLogTestCase(unittest.TestCase):
         # Check for 'error' event specific fields.
         self.assertIn("msg", error_event)
         self.assertIn("fmt", error_event)
-        self.assertEqual(error_event["msg"], msg)
-        self.assertEqual(error_event["fmt"], fmt)
+        self.assertEqual(error_event["msg"], f"RepoErrorEvent:{msg}")
+        self.assertEqual(error_event["fmt"], f"RepoErrorEvent:{fmt}")
 
     def test_write_with_filename(self):
         """Test Write() with a path to a file exits with None."""
@@ -380,17 +378,17 @@ class EventLogTestCase(unittest.TestCase):
             socket_path = os.path.join(tempdir, "server.sock")
             server_ready = threading.Condition()
             # Start "server" listening on Unix domain socket at socket_path.
+            server_thread = threading.Thread(
+                target=serverLoggingThread,
+                args=(socket_path, server_ready, received_traces),
+            )
             try:
-                server_thread = threading.Thread(
-                    target=serverLoggingThread,
-                    args=(socket_path, server_ready, received_traces),
-                )
                 server_thread.start()
 
                 with server_ready:
                     server_ready.wait(timeout=120)
 
-                self._event_log_module.StartEvent()
+                self._event_log_module.StartEvent([])
                 path = self._event_log_module.Write(
                     path=f"af_unix:{socket_path}"
                 )