Adding sparse checkout functionality

Sparse checkout allows Git to check out only a portion of the
repository. This is particularly helpful for monorepository workflows
where CI jobs only need a portion of the repository to run tests.

This speeds up the CI jobs, and puts less load on the Git server as it
relieves it from having to pack up all the objects.

In the Git project, though it is still labeled experimental, the feature
has been stable for a number of years and is unlikely to change at this
point.

This change allows configuring Git to use sparse checkout and provide a
list of directories for the sync command to retrieve.

Projects with sparse paths default to a blob:none partial clone so the
excluded paths are never fetched, unless a filter is already configured
or the project is excluded from partial clones.  Both the plain and the
git-worktree checkout flows configure sparse-checkout before populating
the working tree.

Change-Id: I772c7db5209b345306be378771c8f2040832bca1
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/535061
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: John Cai <johncai86@gmail.com>
Commit-Queue: John Cai <johncai86@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>

docs/manifest-format.md

@@ -74,7 +74,8 @@ following DTD:
   <!ELEMENT project (annotation*,
                      project*,
                      copyfile*,
-                     linkfile*)>
+                     linkfile*,
+                     sparse-path*)>
   <!ATTLIST project name          CDATA #REQUIRED>
   <!ATTLIST project path          CDATA #IMPLIED>
   <!ATTLIST project remote        IDREF #IMPLIED>
@@ -102,9 +103,13 @@ following DTD:
   <!ATTLIST linkfile src  CDATA #REQUIRED>
   <!ATTLIST linkfile dest CDATA #REQUIRED>
 
+  <!ELEMENT sparse-path EMPTY>
+  <!ATTLIST sparse-path path CDATA #REQUIRED>
+
   <!ELEMENT extend-project (annotation*,
                             copyfile*,
-                            linkfile*)>
+                            linkfile*,
+                            sparse-path*)>
   <!ATTLIST extend-project name        CDATA #REQUIRED>
   <!ATTLIST extend-project path        CDATA #IMPLIED>
   <!ATTLIST extend-project dest-path   CDATA #IMPLIED>
@@ -424,10 +429,11 @@ attributes of an existing project without completely replacing the
 existing project definition.  This makes the local manifest more robust
 against changes to the original manifest.
 
-The `extend-project` element can also contain `annotation`, `copyfile`, and
-`linkfile` child elements. These are added to the project's definition. A
-`copyfile` or `linkfile` with a `dest` that already exists in the project
-will overwrite the original.
+The `extend-project` element can also contain `annotation`, `copyfile`,
+`linkfile`, and `sparse-path` child elements. These are added to the project's
+definition. A `copyfile` or `linkfile` with a `dest` that already exists in the
+project will overwrite the original. There is no way to remove a `sparse-path`
+that the original project definition declared.
 
 Attribute `path`: If specified, limit the change to projects checked out
 at the specified path, rather than all projects with the given name.
@@ -471,6 +477,63 @@ attribute "keep" which accepts the case insensitive values "true"
 (default) or "false". This attribute determines whether or not the
 annotation will be kept when exported with the manifest subcommand.
 
+### Element sparse-path
+
+Zero or more sparse-path elements may be specified as children of a
+project element or an extend-project element. Each element specifies a
+path to include in the working directory.
+
+Requires Git 2.25.0 or later, which is when `git sparse-checkout` and cone mode
+were introduced. Declaring a sparse-path with an older Git fails the sync for
+that project.
+
+Declaring at least one sparse-path is what enables Git sparse-checkout for
+the project: the listed paths are checked out, and other directories are left
+out of the working directory. Cone mode also checks out the files sitting
+directly in the project's top-level directory and in each leading directory of
+a sparse-path, so `src/backend` brings along the top-level files and the files
+directly under `src/` too. A project with no sparse-path elements is checked
+out in full.
+The checkout uses [cone mode](https://git-scm.com/docs/git-sparse-checkout#_internalscone_mode_handling)
+for better performance, which means only directory paths are supported;
+individual files and globs are not. Cone mode does no pattern matching, so `*`,
+`?`, `[`, `]`, and `\` are not glob syntax in a `path`. In fact, Git 2.36.0 and
+later reject them with `specify directories rather than patterns`, which
+surfaces as a sync failure for the project. Earlier versions starting with
+2.25.0 have no such check and are treated as literals.
+
+Sparse-checkout is configured only when the project's working directory is
+first created. Changes to `sparse-path` in the manifest (added, changed, or
+removed) are not applied to a project that is already checked out. To change
+the paths in an existing checkout, run `git sparse-checkout` in the project
+directly, or remove the project's working directory and sync again.
+
+Because sparse-checkout only prunes the working directory, a project with at
+least one sparse-path is fetched with `--filter=blob:none` so that blobs which
+will never be checked out are not downloaded. Without the filter Git still
+downloads every blob in the project and sparse-checkout merely declines to
+write most of them to disk, so the filter is what turns sparse-checkout into a
+saving in transfer size and server load. The filter from
+`repo init --partial-clone --clone-filter` takes precedence, and projects listed
+in `repo init --partial-clone-exclude` are fetched in full as usual. Like
+sparse-checkout itself, the implied filter only applies when the project is
+first fetched (or is already a partial clone), so adding a sparse-path to an
+existing full checkout does not convert it to a partial clone. Mirrors are never
+filtered.
+
+Because that filter is only useful if the objects it excludes are not fetched
+by some other route, a project that takes the implied `--filter=blob:none` also
+skips the `/clone.bundle` download: the bundle is an unfiltered pack of every
+object, so fetching it would undo the filter. Projects that set the filter
+explicitly, or that are listed in `repo init --partial-clone-exclude`, keep
+using the clone bundle.
+
+Attribute `path`: A path relative to the project root to include in the
+sparse-checkout. The path is validated like `<copyfile>` and `<linkfile>`
+paths: it must stay inside the project, so absolute paths, `.` or `..`
+components, `~`, and `.git` or `.repo*` components are all rejected. A trailing
+`/` is allowed.
+
 ### Element copyfile
 
 Zero or more copyfile elements may be specified as children of a