git-repo/docs/smart-sync.md

repo Smart Syncing

Repo normally fetches & syncs manifests from the same URL specified during repo init, and that often fetches the latest revisions of all projects in the manifest. This flow works well for tracking and developing with the latest code, but often it's desirable to sync to other points. For example, to get a local build matching a specific release or build to reproduce bugs reported by other people.

Repo's sync subcommand has support for fetching manifests from a server over an XML-RPC connection. The local configuration and network API are defined by repo, but individual projects have to host their own server for the client to communicate with.

This process is called "smart syncing" -- instead of blindly fetching the latest revision of all projects and getting an unknown state to develop against, the client passes a request to the server and is given a matching manifest that typically specifies specific commits for every project to fetch a known source state.

[TOC]

Manifest Configuration

The manifest specifies the server to communicate with via the the <manifest-server> element element. This is how the client knows what service to talk to.

  <manifest-server url="https://example.com/your/manifest/server/url"
                   helper="repo-remote-helper-name" />

If the URL starts with persistent-, then the git-remote-persistent-https helper is used to communicate with the server.

Pluggable Remote Helpers

For custom proxying or authentication, Repo supports pluggable remote helpers. You can declare a helper binary via the optional helper attribute on the <manifest-server> element.

If the helper attribute is present in <manifest-server>:

1. Repo searches your system PATH for the specified helper binary (e.g., repo-remote-sso). If the helper cannot be found in PATH, Repo will raise a SmartSyncError and abort the sync.
2. The helper is executed with the manifest server URL as its first argument:

   <helper-binary-name> <url>
   

3. The helper must output a single-line JSON object on stdout and exit:
   • On success, return status: "ok" and a loopback proxy URL (including scheme, e.g., http://127.0.0.1:999):

     {"status": "ok", "message": "http://127.0.0.1:999"}
     

   • On failure, return status: "error" and a detailed error message:

     {"status": "error", "message": "unauthorized"}
     

4. Repo routes the XML-RPC request through the returned proxy address.

Timeout Constraint: The remote helper must complete and exit within 10 seconds. If it hangs or exceeds this limit, Repo will force-terminate (kill) the process and abort the synchronization.

Credentials

Credentials may be specified directly in typical username:password URI syntax in the <manifest-server> element directly in the manifest.

If they are not specified, repo sync has --manifest-server-username=USERNAME and --manifest-server-password=PASSWORD options.

If those are not used, then repo will look up the host in your ~/.netrc database.

When making the connection, cookies matching the host are automatically loaded from the cookiejar specified in Git's http.cookiefile setting.

Manifest Server

Unfortunately, there are no public reference implementations. Google has an internal one for Android, but it is written using Google's internal systems, so wouldn't be that helpful as a reference.

That said, the XML-RPC API is pretty simple, so any standard XML-RPC server example would do. Google's internal server uses Python's xmlrpc.server.SimpleXMLRPCDispatcher.

Network API

The manifest server should implement the following RPC methods.

GetApprovedManifestV2

GetApprovedManifestV2(request: dict) -> dict

Like GetApprovedManifest, but takes a single struct of named options so new options can be added without changing the method signature. Unset options are omitted from the struct. Servers should ignore options they do not understand.

Request options:

• branch (str, required): Same as in GetApprovedManifest.
• target (str, optional): Same as in GetApprovedManifest.
• best_effort (bool, optional, default False): The client can proceed without a manifest, so the server may decline the request (e.g. during a staged rollout or to shed load). Repo sets this for Smart Syncs enabled by the manifest's sync-smartsync default, and falls back to a regular ToT sync if the request is declined. Servers must not decline requests without this option for these reasons.

Response fields:

• success (bool): Whether a manifest was returned.
• manifest_xml (str): The manifest. Required if success is true.
• error_message (str): Why the request failed or was declined. Repo shows it to the user like GetApprovedManifest errors.

If the server does not implement this method, repo calls GetApprovedManifest instead and drops the options it cannot express.

GetApprovedManifest

GetApprovedManifest(branch: str, target: Optional[str]) -> str

The meaning of branch and target is not strictly defined. The server may interpret them however it wants. The recommended interpretation is that the branch matches the manifest branch, and target is an identifier for your project that matches something users would build.

See the client section below for how repo typically generates these values.

The server will return a manifest or an error. If it's an error, repo will show the output directly to the user to provide a limited feedback channel.

If the user's request is ambiguous and could match multiple manifests, the server has to decide whether to pick one automatically (and silently such that the user won't know there were multiple matches), or return an error and force the user to be more specific.

GetManifest

GetManifest(tag: str) -> str

The meaning of tag is not strictly defined. Projects are encouraged to use a system where the tag matches a unique source state.

See the client section below for how repo typically generates these values.

The server will return a manifest or an error. If it's an error, repo will show the output directly to the user to provide a limited feedback channel.

If the user's request is ambiguous and could match multiple manifests, the server has to decide whether to pick one automatically (and silently such that the user won't know there were multiple matches), or return an error and force the user to be more specific.

Client Options

Once repo has successfully downloaded the manifest from the server, it saves a copy into .repo/manifests/smart_sync_override.xml so users can examine it. The next time repo sync is run, this file is automatically replaced or removed based on the current set of options.

With -l/--local-only and no explicit -s/--smart-sync or -t/--smart-tag, repo does not contact the manifest server. Instead, it keeps using the existing smart_sync_override.xml, if there is one, whether it came from an explicit -s/--smart-sync or from the manifest's sync-smartsync default. This way, repo sync -n followed by repo sync -l ends up on the same revisions as the equivalent single repo sync (e.g. repo sync -s -n + repo sync -l matches repo sync -s). Pass --no-smart-sync to discard it and use the default manifest instead.

--no-manifest-update only skips updating the manifest checkout. It does not affect Smart Sync: the default Smart Sync from sync-smartsync (see below) still runs, and an existing smart_sync_override.xml is replaced or removed as in a regular sync.

sync-smartsync (Manifest Attribute)

The manifest can enable Smart Sync by default for all repo sync invocations by setting sync-smartsync="true" on the <default> element:

  <default sync-smartsync="true" ... />

When enabled in the manifest, repo sync will automatically perform a Smart Sync unless explicitly overridden on the command line. Since the user did not ask for a Smart Sync, any failure to obtain a manifest from the manifest server (e.g. the server is unreachable, or it returns a manifest that cannot be parsed) is silently ignored and repo sync falls back to a regular ToT sync; pass --verbose to see the reason. Repo marks these requests as best_effort (see GetApprovedManifestV2), so the server may also decline them, e.g. during a staged rollout. This is only a fallback for the manifest-driven default: an explicit -s/--smart-sync or -t/--smart-tag still fails the sync, as before. Options that specify an explicit target or manifest source (such as -t/--smart-tag, -m/--manifest-name or --superproject-revision) will also disable the default Smart Sync behavior.

Users can override this manifest default on their workstation with the sync.smartsync user setting; see Per-user default below.

--smart-sync / -s

Explicitly enables Smart Sync. Repo will call GetApprovedManifestV2({branch[, target]}), or GetApprovedManifest(branch[, target]) if the server does not implement it.

The branch is determined by the current manifest branch as specified by --manifest-branch=BRANCH when running repo init. If ${SYNC_BRANCH} is defined and non-empty, its value is sent instead. This allows selecting a branch other than the one the checkout tracks, e.g. a server-side (CI) branch name that builds from the same manifest branch.

The target is defined by environment variables in the order below. If none of them match, then target is omitted. These variables were decided as they match the settings Android build environments automatically setup.

1. ${SYNC_TARGET}: If defined, the value is used directly.
2. ${TARGET_PRODUCT}-${TARGET_RELEASE}-${TARGET_BUILD_VARIANT}: If these variables are all defined, then they are merged with - and used.
3. ${TARGET_PRODUCT}-${TARGET_BUILD_VARIANT}: If these variables are all defined, then they are merged with - and used.

--no-smart-sync

Explicitly disables Smart Sync, overriding any sync-smartsync="true" setting declared in the manifest <default> element.

Per-user default (sync.smartsync)

A user can decide once, for all their checkouts, whether repo sync should Smart Sync by default by setting the boolean sync.smartsync key in their per-user repo config.

Command line flags (-s, --no-smart-sync, -t) override the user setting, which in turn overrides the manifest's sync-smartsync default. A Smart Sync enabled this way is treated as an explicit request, so it is subject to the same restrictions as -s and fails the sync if no manifest can be obtained from the server instead of silently falling back to ToT.

--smart-tag=TAG / -t

Repo will call GetManifest(TAG).