gitc: Lower concurrent ls-projects requests

Too many requests at the same time is causing 502 errors. Change-Id: Ic8fbb2fbb7fb6014733fa5be018d2dc02472f704
RepoHook: do not list options twice during hash based approval
2026-04-20 03:08:20 +00:00 · 2016-08-23 14:19:00 -07:00 · 2016-08-18 17:06:26 -07:00 · 2016-08-17 13:58:17 +09:00 · 2016-08-17 04:57:28 +00:00 · 2016-08-17 04:04:14 +00:00
26 changed files with 785 additions and 316 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -1,4 +1,4 @@
 # Prevent /bin/sh scripts from being clobbered by autocrlf=true
 git_ssh text eol=lf
-main.py text eol=lf
 repo text eol=lf
+hooks/* text eol=lf
--- a/.mailmap
+++ b/.mailmap
@@ -0,0 +1,8 @@
+Anthony Newnam <anthony.newnam@garmin.com>    Anthony <anthony@bnovc.com>
+Shawn Pearce <sop@google.com>                 Shawn O. Pearce <sop@google.com>
+Jia Bi <bijia@xiaomi.com>                     bijia <bijia@xiaomi.com>
+JoonCheol Park <jooncheol@gmail.com>          Jooncheol Park <jooncheol@gmail.com>
+Sergii Pylypenko <x.pelya.x@gmail.com>        pelya <x.pelya.x@gmail.com>
+Ulrik Sjölin <ulrik.sjolin@sonyericsson.com>  Ulrik Sjolin <ulrik.sjolin@gmail.com>
+Ulrik Sjölin <ulrik.sjolin@sonyericsson.com>  Ulrik Sjolin <ulrik.sjolin@sonyericsson.com>
+Ulrik Sjölin <ulrik.sjolin@sonyericsson.com>  Ulrik Sjölin <ulrik.sjolin@sonyericsson.com>
--- a/.pylintrc
+++ b/.pylintrc
@@ -53,7 +53,7 @@ load-plugins=
 enable=RP0004

 # Disable the message(s) with the given id(s).
-disable=R0903,R0912,R0913,R0914,R0915,W0141,C0111,C0103,W0603,W0703,R0911,C0301,C0302,R0902,R0904,W0142,W0212,E1101,E1103,R0201,W0201,W0122,W0232,RP0001,RP0003,RP0101,RP0002,RP0401,RP0701,RP0801,F0401,E0611,R0801,I0011
+disable=C0326,R0903,R0912,R0913,R0914,R0915,W0141,C0111,C0103,W0603,W0703,R0911,C0301,C0302,R0902,R0904,W0142,W0212,E1101,E1103,R0201,W0201,W0122,W0232,RP0001,RP0003,RP0101,RP0002,RP0401,RP0701,RP0801,F0401,E0611,R0801,I0011

 [REPORTS]

--- a/README.md
+++ b/README.md
@@ -0,0 +1,14 @@
+# repo
+
+Repo is a tool built on top of Git.  Repo helps manage many Git repositories,
+does the uploads to revision control systems, and automates parts of the
+development workflow.  Repo is not meant to replace Git, only to make it
+easier to work with Git.  The repo command is an executable Python script
+that you can put anywhere in your path.
+
+* Homepage: https://code.google.com/p/git-repo/
+* Bug reports: https://code.google.com/p/git-repo/issues/
+* Source: https://code.google.com/p/git-repo/
+* Overview: https://source.android.com/source/developing.html
+* Docs: https://source.android.com/source/using-repo.html
+* [Submitting patches](./SUBMITTING_PATCHES.md)
--- a/SUBMITTING_PATCHES.md
+++ b/SUBMITTING_PATCHES.md
@@ -1,15 +1,17 @@
-Short Version:
+# Short Version

 - Make small logical changes.
 - Provide a meaningful commit message.
 - Check for coding errors with pylint
 - Make sure all code is under the Apache License, 2.0.
- - Publish your changes for review:
+ - Publish your changes for review.
+ - Make corrections if requested.
+ - Verify your changes on gerrit so they can be submitted.

-   git push https://gerrit-review.googlesource.com/git-repo HEAD:refs/for/master
+   `git push https://gerrit-review.googlesource.com/git-repo HEAD:refs/for/master`


-Long Version:
+# Long Version

 I wanted a file describing how to submit patches for repo,
 so I started with the one found in the core Git distribution
@@ -17,10 +19,10 @@ so I started with the one found in the core Git distribution
 patch submission guidelines for the Linux kernel.

 However there are some differences, so please review and familiarize
-yourself with the following relevant bits:
+yourself with the following relevant bits.


-(1) Make separate commits for logically separate changes.
+## Make separate commits for logically separate changes.

 Unless your patch is really trivial, you should not be sending
 out a patch that was generated between your working tree and your
@@ -34,14 +36,14 @@ If your description starts to get too long, that's a sign that you
 probably need to split up your commit to finer grained pieces.


-(2) Check for coding errors with pylint
+## Check for coding errors with pylint

 Run pylint on changed modules using the provided configuration:

-  pylint --rcfile=.pylintrc file.py
+    pylint --rcfile=.pylintrc file.py


-(3) Check the license
+## Check the license

 repo is licensed under the Apache License, 2.0.

@@ -57,7 +59,7 @@ your patch.  It is virtually impossible to remove a patch once it
 has been applied and pushed out.


-(4) Sending your patches.
+## Sending your patches.

 Do not email your patches to anyone.

@@ -75,13 +77,39 @@ Ensure you have obtained an HTTP password to authenticate:

  https://gerrit-review.googlesource.com/new-password

+Ensure that you have the local commit hook installed to automatically
+add a ChangeId to your commits:
+
+    curl -Lo `git rev-parse --git-dir`/hooks/commit-msg https://gerrit-review.googlesource.com/tools/hooks/commit-msg
+    chmod +x `git rev-parse --git-dir`/hooks/commit-msg
+
+If you have already committed your changes you will need to amend the commit
+to get the ChangeId added.
+
+    git commit --amend
+
 Push your patches over HTTPS to the review server, possibly through
 a remembered remote to make this easier in the future:

-   git config remote.review.url https://gerrit-review.googlesource.com/git-repo
-   git config remote.review.push HEAD:refs/for/master
+    git config remote.review.url https://gerrit-review.googlesource.com/git-repo
+    git config remote.review.push HEAD:refs/for/master

-   git push review
+    git push review

 You will be automatically emailed a copy of your commits, and any
 comments made by the project maintainers.
+
+
+## Make changes if requested
+
+The project maintainer who reviews your changes might request changes to your
+commit. If you make the requested changes you will need to amend your commit
+and push it to the review server again.
+
+
+## Verify your changes on gerrit
+
+After you receive a Code-Review+2 from the maintainer, select the Verified
+button on the gerrit page for the change. This verifies that you have tested
+your changes and notifies the maintainer that they are ready to be submitted.
+The maintainer will then submit your changes to the repository.
--- a/command.py
+++ b/command.py
@@ -31,7 +31,7 @@ class Command(object):
  manifest = None
  _optparse = None

-  def WantPager(self, opt):
+  def WantPager(self, _opt):
    return False

  def ReadEnvironmentOptions(self, opts):
@@ -63,7 +63,7 @@ class Command(object):
        usage = self.helpUsage.strip().replace('%prog', me)
      except AttributeError:
        usage = 'repo %s' % self.NAME
-      self._optparse = optparse.OptionParser(usage = usage)
+      self._optparse = optparse.OptionParser(usage=usage)
      self._Options(self._optparse)
    return self._optparse

@@ -110,15 +110,20 @@ class Command(object):
    project = None
    if os.path.exists(path):
      oldpath = None
-      while path \
-        and path != oldpath \
-        and path != manifest.topdir:
+      while path and \
+            path != oldpath and \
+            path != manifest.topdir:
        try:
          project = self._by_path[path]
          break
        except KeyError:
          oldpath = path
          path = os.path.dirname(path)
+      if not project and path == manifest.topdir:
+        try:
+          project = self._by_path[path]
+        except KeyError:
+          pass
    else:
      try:
        project = self._by_path[path]
@@ -138,7 +143,7 @@ class Command(object):
    mp = manifest.manifestProject

    if not groups:
-        groups = mp.config.GetString('manifest.groups')
+      groups = mp.config.GetString('manifest.groups')
    if not groups:
      groups = 'default,platform-' + platform.system().lower()
    groups = [x for x in re.split(r'[,\s]+', groups) if x]
@@ -151,8 +156,7 @@ class Command(object):
                                  for p in project.GetDerivedSubprojects())
      all_projects_list.extend(derived_projects.values())
      for project in all_projects_list:
-        if ((missing_ok or project.Exists) and
-            project.MatchesGroups(groups)):
+        if (missing_ok or project.Exists) and project.MatchesGroups(groups):
          result.append(project)
    else:
      self._ResetPathToProjectMap(all_projects_list)
@@ -166,8 +170,8 @@ class Command(object):

          # If it's not a derived project, update path->project mapping and
          # search again, as arg might actually point to a derived subproject.
-          if (project and not project.Derived and
-              (submodules_ok or project.sync_s)):
+          if (project and not project.Derived and (submodules_ok or
+                                                   project.sync_s)):
            search_again = False
            for subproject in project.GetDerivedSubprojects():
              self._UpdatePathToProjectMap(subproject)
@@ -194,17 +198,24 @@ class Command(object):
    result.sort(key=_getpath)
    return result

-  def FindProjects(self, args):
+  def FindProjects(self, args, inverse=False):
    result = []
    patterns = [re.compile(r'%s' % a, re.IGNORECASE) for a in args]
    for project in self.GetProjects(''):
      for pattern in patterns:
-        if pattern.search(project.name) or pattern.search(project.relpath):
+        match = pattern.search(project.name) or pattern.search(project.relpath)
+        if not inverse and match:
          result.append(project)
          break
+        if inverse and match:
+          break
+      else:
+        if inverse:
+          result.append(project)
    result.sort(key=lambda project: project.relpath)
    return result

+
 # pylint: disable=W0223
 # Pylint warns that the `InteractiveCommand` and `PagedCommand` classes do not
 # override method `Execute` which is abstract in `Command`.  Since that method
@@ -214,24 +225,33 @@ class InteractiveCommand(Command):
  """Command which requires user interaction on the tty and
     must not run within a pager, even if the user asks to.
  """
-  def WantPager(self, opt):
+  def WantPager(self, _opt):
    return False

+
 class PagedCommand(Command):
  """Command which defaults to output in a pager, as its
     display tends to be larger than one screen full.
  """
-  def WantPager(self, opt):
+  def WantPager(self, _opt):
    return True

 # pylint: enable=W0223

+
 class MirrorSafeCommand(object):
  """Command permits itself to run within a mirror,
     and does not require a working directory.
  """

-class RequiresGitcCommand(object):
+
+class GitcAvailableCommand(object):
  """Command that requires GITC to be available, but does
     not require the local client to be a GITC client.
  """
+
+
+class GitcClientCommand(object):
+  """Command that requires the local client to be a GITC
+     client.
+  """
--- a/docs/manifest-format.txt
+++ b/docs/manifest-format.txt
@@ -47,10 +47,12 @@ following DTD:
    <!ATTLIST default sync-s      CDATA #IMPLIED>

    <!ELEMENT manifest-server (EMPTY)>
-    <!ATTLIST url              CDATA #REQUIRED>
+    <!ATTLIST manifest-server url CDATA #REQUIRED>

    <!ELEMENT project (annotation*,
-                       project*)>
+                       project*,
+                       copyfile*,
+                       linkfile*)>
    <!ATTLIST project name        CDATA #REQUIRED>
    <!ATTLIST project path        CDATA #IMPLIED>
    <!ATTLIST project remote      IDREF #IMPLIED>
@@ -68,7 +70,15 @@ following DTD:
    <!ATTLIST annotation value CDATA #REQUIRED>
    <!ATTLIST annotation keep  CDATA "true">

-    <!ELEMENT extend-project>
+    <!ELEMENT copyfile (EMPTY)>
+    <!ATTLIST copyfile src  CDATA #REQUIRED>
+    <!ATTLIST copyfile dest CDATA #REQUIRED>
+
+    <!ELEMENT linkfile (EMPTY)>
+    <!ATTLIST linkfile src CDATA #REQUIRED>
+    <!ATTLIST linkfile dest CDATA #REQUIRED>
+
+    <!ELEMENT extend-project (EMPTY)>
    <!ATTLIST extend-project name CDATA #REQUIRED>
    <!ATTLIST extend-project path CDATA #IMPLIED>
    <!ATTLIST extend-project groups CDATA #IMPLIED>
@@ -165,7 +175,8 @@ The manifest server should implement the following RPC methods:
  GetApprovedManifest(branch, target)

 Return a manifest in which each project is pegged to a known good revision
-for the current branch and target.
+for the current branch and target. This is used by repo sync when the
+--smart-sync option is given.

 The target to use is defined by environment variables TARGET_PRODUCT
 and TARGET_BUILD_VARIANT. These variables are used to create a string
@@ -177,7 +188,8 @@ should choose a reasonable default target.
  GetManifest(tag)

 Return a manifest in which each project is pegged to the revision at
-the specified tag.
+the specified tag. This is used by repo sync when the --smart-tag option
+is given.


 Element project
@@ -285,6 +297,21 @@ prefixed with REPO__.  In addition, there is an optional attribute
 "false".  This attribute determines whether or not the annotation will
 be kept when exported with the manifest subcommand.

+Element copyfile
+----------------
+
+Zero or more copyfile elements may be specified as children of a
+project element. Each element describes a src-dest pair of files;
+the "src" file will be copied to the "dest" place during 'repo sync'
+command.
+"src" is project relative, "dest" is relative to the top of the tree.
+
+Element linkfile
+----------------
+
+It's just like copyfile and runs at the same time as copyfile but
+instead of copying it creates a symlink.
+
 Element remove-project
 ----------------------

--- a/git_command.py
+++ b/git_command.py
@@ -168,6 +168,9 @@ class GitCommand(object):
      if p is not None:
        s = p + ' ' + s
      _setenv(env, 'GIT_CONFIG_PARAMETERS', s)
+    if 'GIT_ALLOW_PROTOCOL' not in env:
+      _setenv(env, 'GIT_ALLOW_PROTOCOL',
+              'file:git:http:https:ssh:persistent-http:persistent-https:sso:rpc')

    if project:
      if not cwd:
--- a/gitc_utils.py
+++ b/gitc_utils.py
@@ -24,25 +24,15 @@ import git_command
 import git_config
 import wrapper

-from manifest_xml import GitcManifest
+from error import ManifestParseError

-GITC_FS_ROOT_DIR = '/gitc/manifest-rw/'
-NUM_BATCH_RETRIEVE_REVISIONID = 300
+NUM_BATCH_RETRIEVE_REVISIONID = 32

 def get_gitc_manifest_dir():
  return wrapper.Wrapper().get_gitc_manifest_dir()

 def parse_clientdir(gitc_fs_path):
-  """Parse a path in the GITC FS and return its client name.
-
-  @param gitc_fs_path: A subdirectory path within the GITC_FS_ROOT_DIR.
-
-  @returns: The GITC client name
-  """
-  if (gitc_fs_path == GITC_FS_ROOT_DIR or
-      not gitc_fs_path.startswith(GITC_FS_ROOT_DIR)):
-    return None
-  return gitc_fs_path.split(GITC_FS_ROOT_DIR)[1].split('/')[0]
+  return wrapper.Wrapper().gitc_parse_clientdir(gitc_fs_path)

 def _set_project_revisions(projects):
  """Sets the revisionExpr for a list of projects.
@@ -66,7 +56,11 @@ def _set_project_revisions(projects):
    if gitcmd.Wait():
      print('FATAL: Failed to retrieve revisionExpr for %s' % proj)
      sys.exit(1)
-    proj.revisionExpr = gitcmd.stdout.split('\t')[0]
+    revisionExpr = gitcmd.stdout.split('\t')[0]
+    if not revisionExpr:
+      raise(ManifestParseError('Invalid SHA-1 revision project %s (%s)' %
+                               (proj.remote.url, proj.revisionExpr)))
+    proj.revisionExpr = revisionExpr

 def _manifest_groups(manifest):
  """Returns the manifest group string that should be synced
@@ -82,17 +76,13 @@ def _manifest_groups(manifest):
    groups = 'default,platform-' + platform.system().lower()
  return groups

-def generate_gitc_manifest(repodir, client_name, gitc_manifest, repo_manifest_file, paths=None):
+def generate_gitc_manifest(gitc_manifest, manifest, paths=None):
  """Generate a manifest for shafsd to use for this GITC client.

-  @param repodir: The repo directory
-  @param client_name: The gitc client name
-  @param gitc_manifest: Current gitc manifest, or None if there isn't one yet
-  @param repo_manifest_file: The file used by the main repo manifest
+  @param gitc_manifest: Current gitc manifest, or None if there isn't one yet.
+  @param manifest: A GitcManifest object loaded with the current repo manifest.
  @param paths: List of project paths we want to update.
  """
-  manifest = GitcManifest(repodir, client_name)
-  manifest.Override(repo_manifest_file)

  print('Generating GITC Manifest by fetching revision SHAs for each '
        'project.')
@@ -143,7 +133,7 @@ def generate_gitc_manifest(repodir, client_name, gitc_manifest, repo_manifest_fi
        repo_proj.revisionExpr = None

  # Convert URLs from relative to absolute.
-  for name, remote in manifest.remotes.iteritems():
+  for _name, remote in manifest.remotes.iteritems():
    remote.fetchUrl = remote.resolvedFetchUrl

  # Save the manifest.
--- a/hooks/commit-msg
+++ b/hooks/commit-msg
@@ -1,6 +1,7 @@
 #!/bin/sh
+# From Gerrit Code Review 2.12.1
 #
-# Part of Gerrit Code Review (http://code.google.com/p/gerrit/)
+# Part of Gerrit Code Review (https://www.gerritcodereview.com/)
 #
 # Copyright (C) 2009 The Android Open Source Project
 #
@@ -19,7 +20,7 @@

 unset GREP_OPTIONS

-CHANGE_ID_AFTER="Bug|Issue"
+CHANGE_ID_AFTER="Bug|Issue|Test"
 MSG="$1"

 # Check for, and add if missing, a unique Change-Id
@@ -38,6 +39,12 @@ add_ChangeId() {
 		return
 	fi

+	# Do not add Change-Id to temp commits
+	if echo "$clean_message" | head -1 | grep -q '^\(fixup\|squash\)!'
+	then
+		return
+	fi
+
 	if test "false" = "`git config --bool --get gerrit.createChangeId`"
 	then
 		return
@@ -57,6 +64,10 @@ add_ChangeId() {
 		AWK=/usr/xpg4/bin/awk
 	fi

+	# Get core.commentChar from git config or use default symbol
+	commentChar=`git config --get core.commentChar`
+	commentChar=${commentChar:-#}
+
 	# How this works:
 	# - parse the commit message as (textLine+ blankLine*)*
 	# - assume textLine+ to be a footer until proven otherwise
@@ -75,8 +86,8 @@ add_ChangeId() {
 		blankLines = 0
 	}

-	# Skip lines starting with "#" without any spaces before it.
-	/^#/ { next }
+	# Skip lines starting with commentChar without any spaces before it.
+	/^'"$commentChar"'/ { next }

 	# Skip the line starting with the diff command and everything after it,
 	# up to the end of the file, assuming it is only patch data.
--- a/main.py
+++ b/main.py
@@ -42,7 +42,7 @@ from git_command import git, GitCommand
 from git_config import init_ssh, close_ssh
 from command import InteractiveCommand
 from command import MirrorSafeCommand
-from command import RequiresGitcCommand
+from command import GitcAvailableCommand, GitcClientCommand
 from subcmds.version import Version
 from editor import Editor
 from error import DownloadError
@@ -144,11 +144,16 @@ class _Repo(object):
            file=sys.stderr)
      return 1

-    if isinstance(cmd, RequiresGitcCommand) and not gitc_utils.get_gitc_manifest_dir():
+    if isinstance(cmd, GitcAvailableCommand) and not gitc_utils.get_gitc_manifest_dir():
      print("fatal: '%s' requires GITC to be available" % name,
            file=sys.stderr)
      return 1

+    if isinstance(cmd, GitcClientCommand) and not gitc_client_name:
+      print("fatal: '%s' requires a GITC client" % name,
+            file=sys.stderr)
+      return 1
+
    try:
      copts, cargs = cmd.OptionParser.parse_args(argv)
      copts = cmd.ReadEnvironmentOptions(copts)
@@ -374,7 +379,7 @@ class _KerberosAuthHandler(urllib.request.BaseHandler):
    self.context = None
    self.handler_order = urllib.request.BaseHandler.handler_order - 50

-  def http_error_401(self, req, fp, code, msg, headers):
+  def http_error_401(self, req, fp, code, msg, headers): # pylint:disable=unused-argument
    host = req.get_host()
    retry = self.http_error_auth_reqed('www-authenticate', host, req, headers)
    return retry
--- a/manifest_xml.py
+++ b/manifest_xml.py
@@ -102,7 +102,10 @@ class _XmlRemote(object):
    remoteName = self.name
    if self.remoteAlias:
      remoteName = self.remoteAlias
-    return RemoteSpec(remoteName, url, self.reviewUrl)
+    return RemoteSpec(remoteName,
+                      url=url,
+                      review=self.reviewUrl,
+                      orig_name=self.name)

 class XmlManifest(object):
  """manages the repo configuration file"""
@@ -249,9 +252,9 @@ class XmlManifest(object):
        e.setAttribute('path', relpath)
      remoteName = None
      if d.remote:
-        remoteName = d.remote.remoteAlias or d.remote.name
-      if not d.remote or p.remote.name != remoteName:
-        remoteName = p.remote.name
+        remoteName = d.remote.name
+      if not d.remote or p.remote.orig_name != remoteName:
+        remoteName = p.remote.orig_name
        e.setAttribute('remote', remoteName)
      if peg_rev:
        if self.IsMirror:
@@ -267,7 +270,7 @@ class XmlManifest(object):
            # isn't our value
            e.setAttribute('upstream', p.revisionExpr)
      else:
-        revision = self.remotes[remoteName].revision or d.revisionExpr
+        revision = self.remotes[p.remote.orig_name].revision or d.revisionExpr
        if not revision or revision != p.revisionExpr:
          e.setAttribute('revision', p.revisionExpr)
        if p.upstream and p.upstream != p.revisionExpr:
@@ -969,5 +972,5 @@ class GitcManifest(XmlManifest):
  def _output_manifest_project_extras(self, p, e):
    """Output GITC Specific Project attributes"""
    if p.old_revision:
-        e.setAttribute('old-revision', str(p.old_revision))
+      e.setAttribute('old-revision', str(p.old_revision))

--- a/project.py
+++ b/project.py
@@ -30,7 +30,8 @@ import traceback

 from color import Coloring
 from git_command import GitCommand, git_require
-from git_config import GitConfig, IsId, GetSchemeFromUrl, GetUrlCookieFile, ID_RE
+from git_config import GitConfig, IsId, GetSchemeFromUrl, GetUrlCookieFile, \
+    ID_RE
 from error import GitError, HookError, UploadError, DownloadError
 from error import ManifestInvalidRevisionError
 from error import NoManifestException
@@ -39,11 +40,18 @@ from trace import IsTrace, Trace
 from git_refs import GitRefs, HEAD, R_HEADS, R_TAGS, R_PUB, R_M

 from pyversion import is_python3
-if not is_python3():
+if is_python3():
+  import urllib.parse
+else:
+  import imp
+  import urlparse
+  urllib = imp.new_module('urllib')
+  urllib.parse = urlparse
  # pylint:disable=W0622
  input = raw_input
  # pylint:enable=W0622

+
 def _lwrite(path, content):
  lock = '%s.lock' % path

@@ -59,21 +67,27 @@ def _lwrite(path, content):
    os.remove(lock)
    raise

+
 def _error(fmt, *args):
  msg = fmt % args
  print('error: %s' % msg, file=sys.stderr)

+
 def _warn(fmt, *args):
  msg = fmt % args
  print('warn: %s' % msg, file=sys.stderr)

+
 def not_rev(r):
  return '^' + r

+
 def sq(r):
  return "'" + r.replace("'", "'\''") + "'"

 _project_hook_list = None
+
+
 def _ProjectHooks():
  """List the hooks present in the 'hooks' directory.

@@ -107,15 +121,14 @@ class DownloadedChange(object):
  @property
  def commits(self):
    if self._commit_cache is None:
-      self._commit_cache = self.project.bare_git.rev_list(
-        '--abbrev=8',
-        '--abbrev-commit',
-        '--pretty=oneline',
-        '--reverse',
-        '--date-order',
-        not_rev(self.base),
-        self.commit,
-        '--')
+      self._commit_cache = self.project.bare_git.rev_list('--abbrev=8',
+                                                          '--abbrev-commit',
+                                                          '--pretty=oneline',
+                                                          '--reverse',
+                                                          '--date-order',
+                                                          not_rev(self.base),
+                                                          self.commit,
+                                                          '--')
    return self._commit_cache


@@ -134,36 +147,36 @@ class ReviewableBranch(object):
  @property
  def commits(self):
    if self._commit_cache is None:
-      self._commit_cache = self.project.bare_git.rev_list(
-        '--abbrev=8',
-        '--abbrev-commit',
-        '--pretty=oneline',
-        '--reverse',
-        '--date-order',
-        not_rev(self.base),
-        R_HEADS + self.name,
-        '--')
+      self._commit_cache = self.project.bare_git.rev_list('--abbrev=8',
+                                                          '--abbrev-commit',
+                                                          '--pretty=oneline',
+                                                          '--reverse',
+                                                          '--date-order',
+                                                          not_rev(self.base),
+                                                          R_HEADS + self.name,
+                                                          '--')
    return self._commit_cache

  @property
  def unabbrev_commits(self):
    r = dict()
-    for commit in self.project.bare_git.rev_list(
-        not_rev(self.base),
-        R_HEADS + self.name,
-        '--'):
+    for commit in self.project.bare_git.rev_list(not_rev(self.base),
+                                                 R_HEADS + self.name,
+                                                 '--'):
      r[commit[0:8]] = commit
    return r

  @property
  def date(self):
-    return self.project.bare_git.log(
-      '--pretty=format:%cd',
-      '-n', '1',
-      R_HEADS + self.name,
-      '--')
+    return self.project.bare_git.log('--pretty=format:%cd',
+                                     '-n', '1',
+                                     R_HEADS + self.name,
+                                     '--')

-  def UploadForReview(self, people, auto_topic=False, draft=False, dest_branch=None):
+  def UploadForReview(self, people,
+                      auto_topic=False,
+                      draft=False,
+                      dest_branch=None):
    self.project.UploadForReview(self.name,
                                 people,
                                 auto_topic=auto_topic,
@@ -173,8 +186,8 @@ class ReviewableBranch(object):
  def GetPublishedRefs(self):
    refs = {}
    output = self.project.bare_git.ls_remote(
-      self.branch.remote.SshReviewUrl(self.project.UserEmail),
-      'refs/changes/*')
+        self.branch.remote.SshReviewUrl(self.project.UserEmail),
+        'refs/changes/*')
    for line in output.split('\n'):
      try:
        (sha, ref) = line.split()
@@ -184,7 +197,9 @@ class ReviewableBranch(object):

    return refs

+
 class StatusColoring(Coloring):
+
  def __init__(self, config):
    Coloring.__init__(self, config, 'status')
    self.project = self.printer('header', attr='bold')
@@ -198,17 +213,22 @@ class StatusColoring(Coloring):


 class DiffColoring(Coloring):
+
  def __init__(self, config):
    Coloring.__init__(self, config, 'diff')
    self.project = self.printer('header', attr='bold')

+
 class _Annotation(object):
+
  def __init__(self, name, value, keep):
    self.name = name
    self.value = value
    self.keep = keep

+
 class _CopyFile(object):
+
  def __init__(self, src, dest, abssrc, absdest):
    self.src = src
    self.dest = dest
@@ -236,7 +256,9 @@ class _CopyFile(object):
      except IOError:
        _error('Cannot copy file %s to %s', src, dest)

+
 class _LinkFile(object):
+
  def __init__(self, git_worktree, src, dest, relsrc, absdest):
    self.git_worktree = git_worktree
    self.src = src
@@ -249,7 +271,7 @@ class _LinkFile(object):
    if not os.path.islink(absDest) or (os.readlink(absDest) != relSrc):
      try:
        # remove existing file first, since it might be read-only
-        if os.path.exists(absDest):
+        if os.path.lexists(absDest):
          os.remove(absDest)
        else:
          dest_dir = os.path.dirname(absDest)
@@ -275,7 +297,7 @@ class _LinkFile(object):
      absDestDir = self.abs_dest
      if os.path.exists(absDestDir) and not os.path.isdir(absDestDir):
        _error('Link error: src with wildcard, %s must be a directory',
-            absDestDir)
+               absDestDir)
      else:
        absSrcFiles = glob.glob(absSrc)
        for absSrcFile in absSrcFiles:
@@ -292,18 +314,24 @@ class _LinkFile(object):
          relSrc = os.path.join(relSrcDir, srcFile)
          self.__linkIt(relSrc, absDest)

+
 class RemoteSpec(object):
+
  def __init__(self,
               name,
               url=None,
               review=None,
-               revision=None):
+               revision=None,
+               orig_name=None):
    self.name = name
    self.url = url
    self.review = review
    self.revision = revision
+    self.orig_name = orig_name
+

 class RepoHook(object):
+
  """A RepoHook contains information about a script to run as a hook.

  Hooks are used to run a python script before running an upload (for instance,
@@ -316,10 +344,12 @@ class RepoHook(object):
  Hooks are always python.  When a hook is run, we will load the hook into the
  interpreter and execute its main() function.
  """
+
  def __init__(self,
               hook_type,
               hooks_project,
               topdir,
+               manifest_url,
               abort_if_user_denies=False):
    """RepoHook constructor.

@@ -333,11 +363,13 @@ class RepoHook(object):
      topdir: Repo's top directory (the one containing the .repo directory).
          Scripts will run with CWD as this directory.  If you have a manifest,
          this is manifest.topdir
+      manifest_url: The URL to the manifest git repo.
      abort_if_user_denies: If True, we'll throw a HookError() if the user
          doesn't allow us to run the hook.
    """
    self._hook_type = hook_type
    self._hooks_project = hooks_project
+    self._manifest_url = manifest_url
    self._topdir = topdir
    self._abort_if_user_denies = abort_if_user_denies

@@ -386,15 +418,38 @@ class RepoHook(object):
  def _CheckForHookApproval(self):
    """Check to see whether this hook has been approved.

-    We'll look at the hash of all of the hooks.  If this matches the hash that
-    the user last approved, we're done.  If it doesn't, we'll ask the user
-    about approval.
+    We'll accept approval of manifest URLs if they're using secure transports.
+    This way the user can say they trust the manifest hoster.  For insecure
+    hosts, we fall back to checking the hash of the hooks repo.

    Note that we ask permission for each individual hook even though we use
    the hash of all hooks when detecting changes.  We'd like the user to be
    able to approve / deny each hook individually.  We only use the hash of all
    hooks because there is no other easy way to detect changes to local imports.

+    Returns:
+      True if this hook is approved to run; False otherwise.
+
+    Raises:
+      HookError: Raised if the user doesn't approve and abort_if_user_denies
+          was passed to the consturctor.
+    """
+    if self._ManifestUrlHasSecureScheme():
+      return self._CheckForHookApprovalManifest()
+    else:
+      return self._CheckForHookApprovalHash()
+
+  def _CheckForHookApprovalHelper(self, subkey, new_val, main_prompt,
+                                  changed_prompt):
+    """Check for approval for a particular attribute and hook.
+
+    Args:
+      subkey: The git config key under [repo.hooks.<hook_type>] to store the
+          last approved string.
+      new_val: The new value to compare against the last approved one.
+      main_prompt: Message to display to the user to ask for approval.
+      changed_prompt: Message explaining why we're re-asking for approval.
+
    Returns:
      True if this hook is approved to run; False otherwise.

@@ -403,43 +458,34 @@ class RepoHook(object):
          was passed to the consturctor.
    """
    hooks_config = self._hooks_project.config
-    git_approval_key = 'repo.hooks.%s.approvedhash' % self._hook_type
+    git_approval_key = 'repo.hooks.%s.%s' % (self._hook_type, subkey)

-    # Get the last hash that the user approved for this hook; may be None.
-    old_hash = hooks_config.GetString(git_approval_key)
+    # Get the last value that the user approved for this hook; may be None.
+    old_val = hooks_config.GetString(git_approval_key)

-    # Get the current hash so we can tell if scripts changed since approval.
-    new_hash = self._GetHash()
-
-    if old_hash is not None:
+    if old_val is not None:
      # User previously approved hook and asked not to be prompted again.
-      if new_hash == old_hash:
+      if new_val == old_val:
        # Approval matched.  We're done.
        return True
      else:
        # Give the user a reason why we're prompting, since they last told
        # us to "never ask again".
-        prompt = 'WARNING: Scripts have changed since %s was allowed.\n\n' % (
-            self._hook_type)
+        prompt = 'WARNING: %s\n\n' % (changed_prompt,)
    else:
      prompt = ''

    # Prompt the user if we're not on a tty; on a tty we'll assume "no".
    if sys.stdout.isatty():
-      prompt += ('Repo %s run the script:\n'
-                 '  %s\n'
-                 '\n'
-                 'Do you want to allow this script to run '
-                 '(yes/yes-never-ask-again/NO)? ') % (
-                 self._GetMustVerb(), self._script_fullpath)
+      prompt += main_prompt + ' (yes/always/NO)? '
      response = input(prompt).lower()
      print()

      # User is doing a one-time approval.
      if response in ('y', 'yes'):
        return True
-      elif response == 'yes-never-ask-again':
-        hooks_config.SetString(git_approval_key, new_hash)
+      elif response == 'always':
+        hooks_config.SetString(git_approval_key, new_val)
        return True

    # For anything else, we'll assume no approval.
@@ -449,6 +495,40 @@ class RepoHook(object):

    return False

+  def _ManifestUrlHasSecureScheme(self):
+    """Check if the URI for the manifest is a secure transport."""
+    secure_schemes = ('file', 'https', 'ssh', 'persistent-https', 'sso', 'rpc')
+    parse_results = urllib.parse.urlparse(self._manifest_url)
+    return parse_results.scheme in secure_schemes
+
+  def _CheckForHookApprovalManifest(self):
+    """Check whether the user has approved this manifest host.
+
+    Returns:
+      True if this hook is approved to run; False otherwise.
+    """
+    return self._CheckForHookApprovalHelper(
+        'approvedmanifest',
+        self._manifest_url,
+        'Run hook scripts from %s' % (self._manifest_url,),
+        'Manifest URL has changed since %s was allowed.' % (self._hook_type,))
+
+  def _CheckForHookApprovalHash(self):
+    """Check whether the user has approved the hooks repo.
+
+    Returns:
+      True if this hook is approved to run; False otherwise.
+    """
+    prompt = ('Repo %s run the script:\n'
+              '  %s\n'
+              '\n'
+              'Do you want to allow this script to run')
+    return self._CheckForHookApprovalHelper(
+        'approvedhash',
+        self._GetHash(),
+        prompt % (self._GetMustVerb(), self._script_fullpath),
+        'Scripts have changed since %s was allowed.' % (self._hook_type,))
+
  def _ExecuteHook(self, **kwargs):
    """Actually execute the given hook.

@@ -475,19 +555,18 @@ class RepoHook(object):

      # Exec, storing global context in the context dict.  We catch exceptions
      # and  convert to a HookError w/ just the failing traceback.
-      context = {}
+      context = {'__file__': self._script_fullpath}
      try:
        exec(compile(open(self._script_fullpath).read(),
                     self._script_fullpath, 'exec'), context)
      except Exception:
-        raise HookError('%s\nFailed to import %s hook; see traceback above.' % (
-                        traceback.format_exc(), self._hook_type))
+        raise HookError('%s\nFailed to import %s hook; see traceback above.' %
+                        (traceback.format_exc(), self._hook_type))

      # Running the script should have defined a main() function.
      if 'main' not in context:
        raise HookError('Missing main() in: "%s"' % self._script_fullpath)

-
      # Add 'hook_should_take_kwargs' to the arguments to be passed to main.
      # We don't actually want hooks to define their main with this argument--
      # it's there to remind them that their hook should always take **kwargs.
@@ -505,8 +584,8 @@ class RepoHook(object):
        context['main'](**kwargs)
      except Exception:
        raise HookError('%s\nFailed to run main() for %s hook; see traceback '
-                        'above.' % (
-                        traceback.format_exc(), self._hook_type))
+                        'above.' % (traceback.format_exc(),
+                                    self._hook_type))
    finally:
      # Restore sys.path and CWD.
      sys.path = orig_syspath
@@ -530,8 +609,8 @@ class RepoHook(object):
          to run a required hook (from _CheckForHookApproval).
    """
    # No-op if there is no hooks project or if hook is disabled.
-    if ((not self._hooks_project) or
-        (self._hook_type not in self._hooks_project.enabled_repo_hooks)):
+    if ((not self._hooks_project) or (self._hook_type not in
+                                      self._hooks_project.enabled_repo_hooks)):
      return

    # Bail with a nice error if we can't find the hook.
@@ -553,6 +632,7 @@ class Project(object):
  # These objects can only be used by a single working tree.
  working_tree_files = ['config', 'packed-refs', 'shallow']
  working_tree_dirs = ['logs', 'refs']
+
  def __init__(self,
               manifest,
               name,
@@ -605,15 +685,15 @@ class Project(object):
    self.gitdir = gitdir.replace('\\', '/')
    self.objdir = objdir.replace('\\', '/')
    if worktree:
-      self.worktree = worktree.replace('\\', '/')
+      self.worktree = os.path.normpath(worktree.replace('\\', '/'))
    else:
      self.worktree = None
    self.relpath = relpath
    self.revisionExpr = revisionExpr

-    if   revisionId is None \
-     and revisionExpr \
-     and IsId(revisionExpr):
+    if revisionId is None \
+            and revisionExpr \
+            and IsId(revisionExpr):
      self.revisionId = revisionExpr
    else:
      self.revisionId = revisionId
@@ -633,9 +713,8 @@ class Project(object):
    self.copyfiles = []
    self.linkfiles = []
    self.annotations = []
-    self.config = GitConfig.ForRepository(
-                    gitdir=self.gitdir,
-                    defaults=self.manifest.globalConfig)
+    self.config = GitConfig.ForRepository(gitdir=self.gitdir,
+                                          defaults=self.manifest.globalConfig)

    if self.worktree:
      self.work_git = self._GitGetByExec(self, bare=False, gitdir=gitdir)
@@ -773,7 +852,7 @@ class Project(object):
    """
    expanded_manifest_groups = manifest_groups or ['default']
    expanded_project_groups = ['all'] + (self.groups or [])
-    if not 'notdefault' in expanded_project_groups:
+    if 'notdefault' not in expanded_project_groups:
      expanded_project_groups += ['default']

    matched = False
@@ -785,7 +864,7 @@ class Project(object):

    return matched

-## Status Display ##
+# Status Display ##
  def UncommitedFiles(self, get_all=True):
    """Returns a list of strings, uncommitted files in the git tree.

@@ -837,7 +916,7 @@ class Project(object):
      output: If specified, redirect the output to this object.
    """
    if not os.path.isdir(self.worktree):
-      if output_redir == None:
+      if output_redir is None:
        output_redir = sys.stdout
      print(file=output_redir)
      print('project %s/' % self.relpath, file=output_redir)
@@ -856,7 +935,7 @@ class Project(object):
      return 'CLEAN'

    out = StatusColoring(self.config)
-    if not output_redir == None:
+    if output_redir is not None:
      out.redirect(output_redir)
    out.project('project %-40s', self.relpath + '/ ')

@@ -899,7 +978,7 @@ class Project(object):

      if i and i.src_path:
        line = ' %s%s\t%s => %s (%s%%)' % (i_status, f_status,
-                                        i.src_path, p, i.level)
+                                           i.src_path, p, i.level)
      else:
        line = ' %s%s\t%s' % (i_status, f_status, p)

@@ -942,7 +1021,7 @@ class Project(object):
    p.Wait()


-## Publish / Upload ##
+# Publish / Upload ##

  def WasPublished(self, branch, all_refs=None):
    """Was the branch published (uploaded) for code review?
@@ -1085,7 +1164,7 @@ class Project(object):
                            message=msg)


-## Sync ##
+# Sync ##

  def _ExtractArchive(self, tarpath, path=None):
    """Extract the given tar on its current location
@@ -1103,14 +1182,15 @@ class Project(object):
    return False

  def Sync_NetworkHalf(self,
-      quiet=False,
-      is_new=None,
-      current_branch_only=False,
-      force_sync=False,
-      clone_bundle=True,
-      no_tags=False,
-      archive=False,
-      optimized_fetch=False):
+                       quiet=False,
+                       is_new=None,
+                       current_branch_only=False,
+                       force_sync=False,
+                       clone_bundle=True,
+                       no_tags=False,
+                       archive=False,
+                       optimized_fetch=False,
+                       prune=False):
    """Perform only the network IO portion of the sync process.
       Local working directory/branch state is not affected.
    """
@@ -1163,8 +1243,8 @@ class Project(object):
      alt_dir = None

    if clone_bundle \
-    and alt_dir is None \
-    and self._ApplyCloneBundle(initial=is_new, quiet=quiet):
+            and alt_dir is None \
+            and self._ApplyCloneBundle(initial=is_new, quiet=quiet):
      is_new = False

    if not current_branch_only:
@@ -1176,12 +1256,13 @@ class Project(object):
      elif self.manifest.default.sync_c:
        current_branch_only = True

-    need_to_fetch = not (optimized_fetch and \
-      (ID_RE.match(self.revisionExpr) and self._CheckForSha1()))
-    if (need_to_fetch
-        and not self._RemoteFetch(initial=is_new, quiet=quiet, alt_dir=alt_dir,
-                                  current_branch_only=current_branch_only,
-                                  no_tags=no_tags)):
+    need_to_fetch = not (optimized_fetch and
+                         (ID_RE.match(self.revisionExpr) and
+                          self._CheckForSha1()))
+    if (need_to_fetch and
+        not self._RemoteFetch(initial=is_new, quiet=quiet, alt_dir=alt_dir,
+                              current_branch_only=current_branch_only,
+                              no_tags=no_tags, prune=prune)):
      return False

    if self.worktree:
@@ -1218,9 +1299,8 @@ class Project(object):
    try:
      return self.bare_git.rev_list(self.revisionExpr, '-1')[0]
    except GitError:
-      raise ManifestInvalidRevisionError(
-        'revision %s in %s not found' % (self.revisionExpr,
-                                         self.name))
+      raise ManifestInvalidRevisionError('revision %s in %s not found' %
+                                         (self.revisionExpr, self.name))

  def GetRevisionId(self, all_refs=None):
    if self.revisionId:
@@ -1235,9 +1315,8 @@ class Project(object):
    try:
      return self.bare_git.rev_parse('--verify', '%s^0' % rev)
    except GitError:
-      raise ManifestInvalidRevisionError(
-        'revision %s in %s not found' % (self.revisionExpr,
-                                         self.name))
+      raise ManifestInvalidRevisionError('revision %s in %s not found' %
+                                         (self.revisionExpr, self.name))

  def Sync_LocalHalf(self, syncbuf, force_sync=False):
    """Perform only the local IO portion of the sync process.
@@ -1326,8 +1405,8 @@ class Project(object):
          # to rewrite the published commits so we punt.
          #
          syncbuf.fail(self,
-                       "branch %s is published (but not merged) and is now %d commits behind"
-                       % (branch.name, len(upstream_gain)))
+                       "branch %s is published (but not merged) and is now "
+                       "%d commits behind" % (branch.name, len(upstream_gain)))
        return
      elif pub == head:
        # All published commits are merged, and thus we are a
@@ -1421,7 +1500,7 @@ class Project(object):
    remote = self.GetRemote(self.remote.name)

    cmd = ['fetch', remote.name]
-    cmd.append('refs/changes/%2.2d/%d/%d' \
+    cmd.append('refs/changes/%2.2d/%d/%d'
               % (change_id % 100, change_id, patch_id))
    if GitCommand(self, cmd, bare=True).Wait() != 0:
      return None
@@ -1432,7 +1511,7 @@ class Project(object):
                            self.bare_git.rev_parse('FETCH_HEAD'))


-## Branch Management ##
+# Branch Management ##

  def StartBranch(self, name, branch_merge=''):
    """Create a new branch off the manifest's revision.
@@ -1582,8 +1661,6 @@ class Project(object):

    if kill:
      old = self.bare_git.GetHead()
-      if old is None:
-        old = 'refs/heads/please_never_use_this_as_a_branch_name'

      try:
        self.bare_git.DetachHead(rev)
@@ -1595,7 +1672,10 @@ class Project(object):
                       capture_stderr=True)
        b.Wait()
      finally:
-        self.bare_git.SetHead(old)
+        if ID_RE.match(old):
+          self.bare_git.DetachHead(old)
+        else:
+          self.bare_git.SetHead(old)
        left = self._allrefs

      for branch in kill:
@@ -1618,10 +1698,11 @@ class Project(object):
    return kept


-## Submodule Management ##
+# Submodule Management ##

  def GetRegisteredSubprojects(self):
    result = []
+
    def rec(subprojects):
      if not subprojects:
        return
@@ -1656,6 +1737,7 @@ class Project(object):

    re_path = re.compile(r'^submodule\.([^.]+)\.path=(.*)$')
    re_url = re.compile(r'^submodule\.([^.]+)\.url=(.*)$')
+
    def parse_gitmodules(gitdir, rev):
      cmd = ['cat-file', 'blob', '%s:.gitmodules' % rev]
      try:
@@ -1765,7 +1847,7 @@ class Project(object):
    return result


-## Direct Git Commands ##
+# Direct Git Commands ##
  def _CheckForSha1(self):
    try:
      # if revision (sha or tag) is not present then following function
@@ -1789,13 +1871,13 @@ class Project(object):
    if command.Wait() != 0:
      raise GitError('git archive %s: %s' % (self.name, command.stderr))

-
  def _RemoteFetch(self, name=None,
                   current_branch_only=False,
                   initial=False,
                   quiet=False,
                   alt_dir=None,
-                   no_tags=False):
+                   no_tags=False,
+                   prune=False):

    is_sha1 = False
    tag_name = None
@@ -1835,7 +1917,10 @@ class Project(object):
        #   will fail.
        # * otherwise, fetch all branches to make sure we end up with the
        #   specific commit.
-        current_branch_only = self.upstream and not ID_RE.match(self.upstream)
+        if self.upstream:
+          current_branch_only = not ID_RE.match(self.upstream)
+        else:
+          current_branch_only = False

    if not name:
      name = self.remote.name
@@ -1908,6 +1993,9 @@ class Project(object):
    else:
      cmd.append('--tags')

+    if prune:
+      cmd.append('--prune')
+
    spec = []
    if not current_branch_only:
      # Fetch whole repo
@@ -1949,9 +2037,9 @@ class Project(object):
          break
        continue
      elif current_branch_only and is_sha1 and ret == 128:
-        # Exit code 128 means "couldn't find the ref you asked for"; if we're in sha1
-        # mode, we just tried sync'ing from the upstream field; it doesn't exist, thus
-        # abort the optimization attempt and do a full sync.
+        # Exit code 128 means "couldn't find the ref you asked for"; if we're
+        # in sha1 mode, we just tried sync'ing from the upstream field; it
+        # doesn't exist, thus abort the optimization attempt and do a full sync.
        break
      elif ret < 0:
        # Git died with a signal, exit immediately
@@ -1978,20 +2066,24 @@ class Project(object):
                                   initial=False, quiet=quiet, alt_dir=alt_dir)
        if self.clone_depth:
          self.clone_depth = None
-          return self._RemoteFetch(name=name, current_branch_only=current_branch_only,
+          return self._RemoteFetch(name=name,
+                                   current_branch_only=current_branch_only,
                                   initial=False, quiet=quiet, alt_dir=alt_dir)

    return ok

  def _ApplyCloneBundle(self, initial=False, quiet=False):
-    if initial and (self.manifest.manifestProject.config.GetString('repo.depth') or self.clone_depth):
+    if initial and \
+        (self.manifest.manifestProject.config.GetString('repo.depth') or
+         self.clone_depth):
      return False

    remote = self.GetRemote(self.remote.name)
    bundle_url = remote.url + '/clone.bundle'
    bundle_url = GitConfig.ForUser().UrlInsteadOf(bundle_url)
-    if GetSchemeFromUrl(bundle_url) not in (
-        'http', 'https', 'persistent-http', 'persistent-https'):
+    if GetSchemeFromUrl(bundle_url) not in ('http', 'https',
+                                            'persistent-http',
+                                            'persistent-https'):
      return False

    bundle_dst = os.path.join(self.gitdir, 'clone.bundle')
@@ -2040,7 +2132,7 @@ class Project(object):
        os.remove(tmpPath)
    if 'http_proxy' in os.environ and 'darwin' == sys.platform:
      cmd += ['--proxy', os.environ['http_proxy']]
-    with GetUrlCookieFile(srcUrl, quiet) as (cookiefile, proxy):
+    with GetUrlCookieFile(srcUrl, quiet) as (cookiefile, _proxy):
      if cookiefile:
        cmd += ['--cookie', cookiefile, '--cookie-jar', cookiefile]
      if srcUrl.startswith('persistent-'):
@@ -2159,11 +2251,12 @@ class Project(object):
          self._CheckDirReference(self.objdir, self.gitdir, share_refs=False)
        except GitError as e:
          if force_sync:
-            print("Retrying clone after deleting %s" % self.gitdir, file=sys.stderr)
+            print("Retrying clone after deleting %s" %
+                  self.gitdir, file=sys.stderr)
            try:
              shutil.rmtree(os.path.realpath(self.gitdir))
-              if self.worktree and os.path.exists(
-                  os.path.realpath(self.worktree)):
+              if self.worktree and os.path.exists(os.path.realpath
+                                                  (self.worktree)):
                shutil.rmtree(os.path.realpath(self.worktree))
              return self._InitGitDir(mirror_git=mirror_git, force_sync=False)
            except:
@@ -2199,6 +2292,7 @@ class Project(object):
        for key in ['user.name', 'user.email']:
          if m.Has(key, include_defaults=False):
            self.config.SetString(key, m.GetString(key))
+        self.config.SetString('filter.lfs.smudge', 'git-lfs smudge --skip -- %f')
        if self.manifest.IsMirror:
          self.config.SetString('core.bare', 'true')
        else:
@@ -2222,7 +2316,7 @@ class Project(object):
      name = os.path.basename(stock_hook)

      if name in ('commit-msg',) and not self.remote.review \
-            and not self is self.manifest.manifestProject:
+              and self is not self.manifest.manifestProject:
        # Don't install a Gerrit Code Review hook if this
        # project does not appear to use it for reviews.
        #
@@ -2237,7 +2331,8 @@ class Project(object):
        if filecmp.cmp(stock_hook, dst, shallow=False):
          os.remove(dst)
        else:
-          _warn("%s: Not replacing locally modified %s hook", self.relpath, name)
+          _warn("%s: Not replacing locally modified %s hook",
+                self.relpath, name)
          continue
      try:
        os.symlink(os.path.relpath(stock_hook, os.path.dirname(dst)), dst)
@@ -2283,8 +2378,8 @@ class Project(object):
        self.bare_git.symbolic_ref('-m', msg, ref, dst)

  def _CheckDirReference(self, srcdir, destdir, share_refs):
-    symlink_files = self.shareable_files
-    symlink_dirs = self.shareable_dirs
+    symlink_files = self.shareable_files[:]
+    symlink_dirs = self.shareable_dirs[:]
    if share_refs:
      symlink_files += self.working_tree_files
      symlink_dirs += self.working_tree_dirs
@@ -2312,8 +2407,8 @@ class Project(object):
      copy_all: If true, copy all remaining files from |gitdir| -> |dotgit|.
          This saves you the effort of initializing |dotgit| yourself.
    """
-    symlink_files = self.shareable_files
-    symlink_dirs = self.shareable_dirs
+    symlink_files = self.shareable_files[:]
+    symlink_dirs = self.shareable_dirs[:]
    if share_refs:
      symlink_files += self.working_tree_files
      symlink_dirs += self.working_tree_dirs
@@ -2405,7 +2500,7 @@ class Project(object):
  def _allrefs(self):
    return self.bare_ref.all

-  def _getLogs(self, rev1, rev2, oneline=False, color=True):
+  def _getLogs(self, rev1, rev2, oneline=False, color=True, pretty_format=None):
    """Get logs between two revisions of this project."""
    comp = '..'
    if rev1:
@@ -2416,6 +2511,8 @@ class Project(object):
      out = DiffColoring(self.config)
      if out.is_on and color:
        cmd.append('--color')
+      if pretty_format is not None:
+        cmd.append('--pretty=format:%s' % pretty_format)
      if oneline:
        cmd.append('--oneline')

@@ -2432,17 +2529,21 @@ class Project(object):
          raise
    return None

-  def getAddedAndRemovedLogs(self, toProject, oneline=False, color=True):
+  def getAddedAndRemovedLogs(self, toProject, oneline=False, color=True,
+                             pretty_format=None):
    """Get the list of logs from this revision to given revisionId"""
    logs = {}
    selfId = self.GetRevisionId(self._allrefs)
    toId = toProject.GetRevisionId(toProject._allrefs)

-    logs['added'] = self._getLogs(selfId, toId, oneline=oneline, color=color)
-    logs['removed'] = self._getLogs(toId, selfId, oneline=oneline, color=color)
+    logs['added'] = self._getLogs(selfId, toId, oneline=oneline, color=color,
+                                  pretty_format=pretty_format)
+    logs['removed'] = self._getLogs(toId, selfId, oneline=oneline, color=color,
+                                    pretty_format=pretty_format)
    return logs

  class _GitGetByExec(object):
+
    def __init__(self, project, bare, gitdir):
      self._project = project
      self._bare = bare
@@ -2461,8 +2562,8 @@ class Project(object):
      if p.Wait() == 0:
        out = p.stdout
        if out:
+          # Backslash is not anomalous
          return out[:-1].split('\0')  # pylint: disable=W1401
-                                       # Backslash is not anomalous
      return []

    def DiffZ(self, name, *args):
@@ -2488,6 +2589,7 @@ class Project(object):
              break

            class _Info(object):
+
              def __init__(self, path, omode, nmode, oid, nid, state):
                self.path = path
                self.src_path = None
@@ -2590,10 +2692,8 @@ class Project(object):
          line = line[:-1]
        r.append(line)
      if p.Wait() != 0:
-        raise GitError('%s rev-list %s: %s' % (
-                       self._project.name,
-                       str(args),
-                       p.stderr))
+        raise GitError('%s rev-list %s: %s' %
+                       (self._project.name, str(args), p.stderr))
      return r

    def __getattr__(self, name):
@@ -2616,6 +2716,7 @@ class Project(object):
        A callable object that will try to call git with the named command.
      """
      name = name.replace('_', '-')
+
      def runner(*args, **kwargs):
        cmdv = []
        config = kwargs.pop('config', None)
@@ -2638,10 +2739,8 @@ class Project(object):
                       capture_stdout=True,
                       capture_stderr=True)
        if p.Wait() != 0:
-          raise GitError('%s %s: %s' % (
-                         self._project.name,
-                         name,
-                         p.stderr))
+          raise GitError('%s %s: %s' %
+                         (self._project.name, name, p.stderr))
        r = p.stdout
        try:
          r = r.decode('utf-8')
@@ -2654,14 +2753,19 @@ class Project(object):


 class _PriorSyncFailedError(Exception):
+
  def __str__(self):
    return 'prior sync failed; rebase still in progress'

+
 class _DirtyError(Exception):
+
  def __str__(self):
    return 'contains uncommitted changes'

+
 class _InfoMessage(object):
+
  def __init__(self, project, text):
    self.project = project
    self.text = text
@@ -2670,7 +2774,9 @@ class _InfoMessage(object):
    syncbuf.out.info('%s/: %s', self.project.relpath, self.text)
    syncbuf.out.nl()

+
 class _Failure(object):
+
  def __init__(self, project, why):
    self.project = project
    self.why = why
@@ -2681,7 +2787,9 @@ class _Failure(object):
                     str(self.why))
    syncbuf.out.nl()

+
 class _Later(object):
+
  def __init__(self, project, action):
    self.project = project
    self.action = action
@@ -2698,14 +2806,18 @@ class _Later(object):
      out.nl()
      return False

+
 class _SyncColoring(Coloring):
+
  def __init__(self, config):
    Coloring.__init__(self, config, 'reposync')
    self.project = self.printer('header', attr='bold')
    self.info = self.printer('info')
    self.fail = self.printer('fail', fg='red')

+
 class SyncBuffer(object):
+
  def __init__(self, config, detach_head=False):
    self._messages = []
    self._failures = []
@@ -2761,8 +2873,10 @@ class SyncBuffer(object):


 class MetaProject(Project):
+
  """A special project housed under .repo.
  """
+
  def __init__(self, manifest, name, gitdir, worktree):
    Project.__init__(self,
                     manifest=manifest,
@@ -2796,10 +2910,9 @@ class MetaProject(Project):
    syncbuf.Finish()

    return GitCommand(self,
-                        ['update-ref', '-d', 'refs/heads/default'],
-                        capture_stdout=True,
-                        capture_stderr=True).Wait() == 0
-
+                      ['update-ref', '-d', 'refs/heads/default'],
+                      capture_stdout=True,
+                      capture_stderr=True).Wait() == 0

  @property
  def LastFetch(self):
--- a/166
+++ b/166
@@ -1,8 +1,11 @@
 #!/usr/bin/env python

-## repo default configuration
-##
-REPO_URL = 'https://gerrit.googlesource.com/git-repo'
+# repo default configuration
+#
+import os
+REPO_URL = os.environ.get('REPO_URL', None)
+if not REPO_URL:
+  REPO_URL = 'https://gerrit.googlesource.com/git-repo'
 REPO_REV = 'stable'

 # Copyright (C) 2008 Google Inc.
@@ -20,7 +23,7 @@ REPO_REV = 'stable'
 # limitations under the License.

 # increment this whenever we make important changes to this script
-VERSION = (1, 21)
+VERSION = (1, 23)

 # increment this if the MAINTAINER_KEYS block is modified
 KEYRING_VERSION = (1, 2)
@@ -101,19 +104,19 @@ JuinEP+AwLAUZ1Bsx9ISC0Agpk2VeHXPL3FGhroEmoMvBzO0kTFGyoeT7PR/BfKv
 -----END PGP PUBLIC KEY BLOCK-----
 """

-GIT = 'git'                     # our git command
-MIN_GIT_VERSION = (1, 7, 2)     # minimum supported git version
-repodir = '.repo'               # name of repo's private directory
-S_repo = 'repo'                 # special repo repository
-S_manifests = 'manifests'       # special manifest repository
-REPO_MAIN = S_repo + '/main.py' # main script
-MIN_PYTHON_VERSION = (2, 6)     # minimum supported python version
+GIT = 'git'                      # our git command
+MIN_GIT_VERSION = (1, 7, 2)      # minimum supported git version
+repodir = '.repo'                # name of repo's private directory
+S_repo = 'repo'                  # special repo repository
+S_manifests = 'manifests'        # special manifest repository
+REPO_MAIN = S_repo + '/main.py'  # main script
+MIN_PYTHON_VERSION = (2, 6)      # minimum supported python version
 GITC_CONFIG_FILE = '/gitc/.config'
+GITC_FS_ROOT_DIR = '/gitc/manifest-rw/'


 import errno
 import optparse
-import os
 import re
 import shutil
 import stat
@@ -193,6 +196,9 @@ group.add_option('-p', '--platform',
                 help='restrict manifest projects to ones with a specified '
                      'platform group [auto|all|none|linux|darwin|...]',
                 metavar='PLATFORM')
+group.add_option('--no-clone-bundle',
+                 dest='no_clone_bundle', action='store_true',
+                 help='disable use of /clone.bundle on HTTP/HTTPS')


 # Tool
@@ -213,17 +219,20 @@ group.add_option('--config-name',
                 dest='config_name', action="store_true", default=False,
                 help='Always prompt for name/e-mail')

-def _GitcInitOptions(init_optparse):
-  init_optparse.set_usage("repo gitc-init -u url -c client [options]")
-  g = init_optparse.add_option_group('GITC options')
+
+def _GitcInitOptions(init_optparse_arg):
+  init_optparse_arg.set_usage("repo gitc-init -u url -c client [options]")
+  g = init_optparse_arg.add_option_group('GITC options')
  g.add_option('-f', '--manifest-file',
               dest='manifest_file',
               help='Optional manifest file to use for this GITC client.')
  g.add_option('-c', '--gitc-client',
               dest='gitc_client',
-               help='The name for the new gitc_client instance.')
+               help='The name of the gitc_client instance to create or modify.')

 _gitc_manifest_dir = None
+
+
 def get_gitc_manifest_dir():
  global _gitc_manifest_dir
  if _gitc_manifest_dir is None:
@@ -238,7 +247,32 @@ def get_gitc_manifest_dir():
      pass
  return _gitc_manifest_dir

+
+def gitc_parse_clientdir(gitc_fs_path):
+  """Parse a path in the GITC FS and return its client name.
+
+  @param gitc_fs_path: A subdirectory path within the GITC_FS_ROOT_DIR.
+
+  @returns: The GITC client name
+  """
+  if gitc_fs_path == GITC_FS_ROOT_DIR:
+    return None
+  if not gitc_fs_path.startswith(GITC_FS_ROOT_DIR):
+    manifest_dir = get_gitc_manifest_dir()
+    if manifest_dir == '':
+      return None
+    if manifest_dir[-1] != '/':
+      manifest_dir += '/'
+    if gitc_fs_path == manifest_dir:
+      return None
+    if not gitc_fs_path.startswith(manifest_dir):
+      return None
+    return gitc_fs_path.split(manifest_dir)[1].split('/')[0]
+  return gitc_fs_path.split(GITC_FS_ROOT_DIR)[1].split('/')[0]
+
+
 class CloneFailure(Exception):
+
  """Indicate the remote clone of repo itself failed.
  """

@@ -276,10 +310,13 @@ def _Init(args, gitc_init=False):
        _print('fatal: GITC filesystem is not available. Exiting...',
               file=sys.stderr)
        sys.exit(1)
-      if not opt.gitc_client:
+      gitc_client = opt.gitc_client
+      if not gitc_client:
+        gitc_client = gitc_parse_clientdir(os.getcwd())
+      if not gitc_client:
        _print('fatal: GITC client (-c) is required.', file=sys.stderr)
        sys.exit(1)
-      client_dir = os.path.join(gitc_manifest_dir, opt.gitc_client)
+      client_dir = os.path.join(gitc_manifest_dir, gitc_client)
      if not os.path.exists(client_dir):
        os.makedirs(client_dir)
      os.chdir(client_dir)
@@ -305,7 +342,7 @@ def _Init(args, gitc_init=False):
      can_verify = True

    dst = os.path.abspath(os.path.join(repodir, S_repo))
-    _Clone(url, dst, opt.quiet)
+    _Clone(url, dst, opt.quiet, not opt.no_clone_bundle)

    if can_verify and not opt.no_repo_verify:
      rev = _Verify(dst, branch, opt.quiet)
@@ -398,13 +435,16 @@ def SetupGnuPG(quiet):
      sys.exit(1)

  env = os.environ.copy()
-  env['GNUPGHOME'] = gpg_dir.encode()
+  try:
+    env['GNUPGHOME'] = gpg_dir
+  except UnicodeEncodeError:
+    env['GNUPGHOME'] = gpg_dir.encode()

  cmd = ['gpg', '--import']
  try:
    proc = subprocess.Popen(cmd,
-                            env = env,
-                            stdin = subprocess.PIPE)
+                            env=env,
+                            stdin=subprocess.PIPE)
  except OSError as e:
    if not quiet:
      _print('warning: gpg (GnuPG) is not available.', file=sys.stderr)
@@ -430,7 +470,7 @@ def _SetConfig(local, name, value):
  """Set a git configuration option to the specified value.
  """
  cmd = [GIT, 'config', name, value]
-  if subprocess.Popen(cmd, cwd = local).wait() != 0:
+  if subprocess.Popen(cmd, cwd=local).wait() != 0:
    raise CloneFailure()


@@ -443,9 +483,9 @@ def _InitHttp():
    n = netrc.netrc()
    for host in n.hosts:
      p = n.hosts[host]
-      mgr.add_password(p[1], 'http://%s/'  % host, p[0], p[2])
+      mgr.add_password(p[1], 'http://%s/' % host, p[0], p[2])
      mgr.add_password(p[1], 'https://%s/' % host, p[0], p[2])
-  except:
+  except:  # pylint: disable=bare-except
    pass
  handlers.append(urllib.request.HTTPBasicAuthHandler(mgr))
  handlers.append(urllib.request.HTTPDigestAuthHandler(mgr))
@@ -458,6 +498,7 @@ def _InitHttp():
    handlers.append(urllib.request.HTTPSHandler(debuglevel=1))
  urllib.request.install_opener(urllib.request.build_opener(*handlers))

+
 def _Fetch(url, local, src, quiet):
  if not quiet:
    _print('Get %s' % url, file=sys.stderr)
@@ -472,22 +513,23 @@ def _Fetch(url, local, src, quiet):
  cmd.append('+refs/heads/*:refs/remotes/origin/*')
  cmd.append('refs/tags/*:refs/tags/*')

-  proc = subprocess.Popen(cmd, cwd = local, stderr = err)
+  proc = subprocess.Popen(cmd, cwd=local, stderr=err)
  if err:
    proc.stderr.read()
    proc.stderr.close()
  if proc.wait() != 0:
    raise CloneFailure()

+
 def _DownloadBundle(url, local, quiet):
  if not url.endswith('/'):
    url += '/'
  url += 'clone.bundle'

  proc = subprocess.Popen(
-    [GIT, 'config', '--get-regexp', 'url.*.insteadof'],
-    cwd = local,
-    stdout = subprocess.PIPE)
+      [GIT, 'config', '--get-regexp', 'url.*.insteadof'],
+      cwd=local,
+      stdout=subprocess.PIPE)
  for line in proc.stdout:
    m = re.compile(r'^url\.(.*)\.insteadof (.*)$').match(line)
    if m:
@@ -507,7 +549,7 @@ def _DownloadBundle(url, local, quiet):
    try:
      r = urllib.request.urlopen(url)
    except urllib.error.HTTPError as e:
-      if e.code in [401, 403, 404]:
+      if e.code in [401, 403, 404, 501]:
        return False
      _print('fatal: Cannot get %s' % url, file=sys.stderr)
      _print('fatal: HTTP error %s' % e.code, file=sys.stderr)
@@ -529,6 +571,7 @@ def _DownloadBundle(url, local, quiet):
  finally:
    dest.close()

+
 def _ImportBundle(local):
  path = os.path.join(local, '.git', 'clone.bundle')
  try:
@@ -536,7 +579,8 @@ def _ImportBundle(local):
  finally:
    os.remove(path)

-def _Clone(url, local, quiet):
+
+def _Clone(url, local, quiet, clone_bundle):
  """Clones a git repository to a new subdirectory of repodir
  """
  try:
@@ -548,14 +592,14 @@ def _Clone(url, local, quiet):

  cmd = [GIT, 'init', '--quiet']
  try:
-    proc = subprocess.Popen(cmd, cwd = local)
+    proc = subprocess.Popen(cmd, cwd=local)
  except OSError as e:
    _print(file=sys.stderr)
    _print("fatal: '%s' is not available" % GIT, file=sys.stderr)
    _print('fatal: %s' % e, file=sys.stderr)
    _print(file=sys.stderr)
    _print('Please make sure %s is installed and in your path.' % GIT,
-          file=sys.stderr)
+           file=sys.stderr)
    raise CloneFailure()
  if proc.wait() != 0:
    _print('fatal: could not create %s' % local, file=sys.stderr)
@@ -563,12 +607,12 @@ def _Clone(url, local, quiet):

  _InitHttp()
  _SetConfig(local, 'remote.origin.url', url)
-  _SetConfig(local, 'remote.origin.fetch',
-                    '+refs/heads/*:refs/remotes/origin/*')
-  if _DownloadBundle(url, local, quiet):
+  _SetConfig(local,
+             'remote.origin.fetch',
+             '+refs/heads/*:refs/remotes/origin/*')
+  if clone_bundle and _DownloadBundle(url, local, quiet):
    _ImportBundle(local)
-  else:
-    _Fetch(url, local, 'origin', quiet)
+  _Fetch(url, local, 'origin', quiet)


 def _Verify(cwd, branch, quiet):
@@ -578,7 +622,7 @@ def _Verify(cwd, branch, quiet):
  proc = subprocess.Popen(cmd,
                          stdout=subprocess.PIPE,
                          stderr=subprocess.PIPE,
-                          cwd = cwd)
+                          cwd=cwd)
  cur = proc.stdout.read().strip()
  proc.stdout.close()

@@ -596,18 +640,21 @@ def _Verify(cwd, branch, quiet):
    if not quiet:
      _print(file=sys.stderr)
      _print("info: Ignoring branch '%s'; using tagged release '%s'"
-            % (branch, cur), file=sys.stderr)
+             % (branch, cur), file=sys.stderr)
      _print(file=sys.stderr)

  env = os.environ.copy()
-  env['GNUPGHOME'] = gpg_dir.encode()
+  try:
+    env['GNUPGHOME'] = gpg_dir
+  except UnicodeEncodeError:
+    env['GNUPGHOME'] = gpg_dir.encode()

  cmd = [GIT, 'tag', '-v', cur]
  proc = subprocess.Popen(cmd,
-                          stdout = subprocess.PIPE,
-                          stderr = subprocess.PIPE,
-                          cwd = cwd,
-                          env = env)
+                          stdout=subprocess.PIPE,
+                          stderr=subprocess.PIPE,
+                          cwd=cwd,
+                          env=env)
  out = proc.stdout.read()
  proc.stdout.close()

@@ -627,21 +674,21 @@ def _Checkout(cwd, branch, rev, quiet):
  """Checkout an upstream branch into the repository and track it.
  """
  cmd = [GIT, 'update-ref', 'refs/heads/default', rev]
-  if subprocess.Popen(cmd, cwd = cwd).wait() != 0:
+  if subprocess.Popen(cmd, cwd=cwd).wait() != 0:
    raise CloneFailure()

  _SetConfig(cwd, 'branch.default.remote', 'origin')
  _SetConfig(cwd, 'branch.default.merge', 'refs/heads/%s' % branch)

  cmd = [GIT, 'symbolic-ref', 'HEAD', 'refs/heads/default']
-  if subprocess.Popen(cmd, cwd = cwd).wait() != 0:
+  if subprocess.Popen(cmd, cwd=cwd).wait() != 0:
    raise CloneFailure()

  cmd = [GIT, 'read-tree', '--reset', '-u']
  if not quiet:
    cmd.append('-v')
  cmd.append('HEAD')
-  if subprocess.Popen(cmd, cwd = cwd).wait() != 0:
+  if subprocess.Popen(cmd, cwd=cwd).wait() != 0:
    raise CloneFailure()


@@ -653,8 +700,8 @@ def _FindRepo():

  olddir = None
  while curdir != '/' \
-    and curdir != olddir \
-    and not repo:
+          and curdir != olddir \
+          and not repo:
    repo = os.path.join(curdir, repodir, REPO_MAIN)
    if not os.path.isfile(repo):
      repo = None
@@ -663,7 +710,7 @@ def _FindRepo():
  return (repo, os.path.join(curdir, repodir))


-class _Options:
+class _Options(object):
  help = False


@@ -690,7 +737,7 @@ def _Usage():
    gitc_usage = "  gitc-init Initialize a GITC Client.\n"

  _print(
-"""usage: repo COMMAND [ARGS]
+      """usage: repo COMMAND [ARGS]

 repo is not yet installed.  Use "repo init" to install it here.

@@ -698,7 +745,7 @@ The most commonly used repo commands are:

  init      Install repo in the current working directory
 """ + gitc_usage +
-"""  help      Display detailed help on a command
+      """  help      Display detailed help on a command

 For access to the full online help, install repo ("repo init").
 """, file=sys.stderr)
@@ -759,8 +806,8 @@ def _SetDefaultsTo(gitdir):
                           '--git-dir=%s' % gitdir,
                           'symbolic-ref',
                           'HEAD'],
-                          stdout = subprocess.PIPE,
-                          stderr = subprocess.PIPE)
+                          stdout=subprocess.PIPE,
+                          stderr=subprocess.PIPE)
  REPO_REV = proc.stdout.read().strip()
  proc.stdout.close()

@@ -773,9 +820,13 @@ def _SetDefaultsTo(gitdir):


 def main(orig_args):
-  repo_main, rel_repo_dir = _FindRepo()
  cmd, opt, args = _ParseArguments(orig_args)

+  repo_main, rel_repo_dir = None, None
+  # Don't use the local repo copy, make sure to switch to the gitc client first.
+  if cmd != 'gitc-init':
+    repo_main, rel_repo_dir = _FindRepo()
+
  wrapper_path = os.path.abspath(__file__)
  my_main, my_git = _RunSelf(wrapper_path)

@@ -783,7 +834,8 @@ def main(orig_args):
  if get_gitc_manifest_dir() and cwd.startswith(get_gitc_manifest_dir()):
    _print('error: repo cannot be used in the GITC local manifest directory.'
           '\nIf you want to work on this GITC client please rerun this '
-           'command from the corresponding client under /gitc/', file=sys.stderr)
+           'command from the corresponding client under /gitc/',
+           file=sys.stderr)
    sys.exit(1)
  if not repo_main:
    if opt.help:
--- a/subcmds/diffmanifests.py
+++ b/subcmds/diffmanifests.py
@@ -71,6 +71,10 @@ synced and their revisions won't be found.
    p.add_option('--no-color',
                 dest='color', action='store_false', default=True,
                 help='does not display the diff in color.')
+    p.add_option('--pretty-format',
+                 dest='pretty_format', action='store',
+                 metavar='<FORMAT>',
+                 help='print the log using a custom git pretty format string')

  def _printRawDiff(self, diff):
    for project in diff['added']:
@@ -92,7 +96,7 @@ synced and their revisions won't be found.
                                     otherProject.revisionExpr))
      self.out.nl()

-  def _printDiff(self, diff, color=True):
+  def _printDiff(self, diff, color=True, pretty_format=None):
    if diff['added']:
      self.out.nl()
      self.printText('added projects : \n')
@@ -124,7 +128,8 @@ synced and their revisions won't be found.
        self.printText(' to ')
        self.printRevision(otherProject.revisionExpr)
        self.out.nl()
-        self._printLogs(project, otherProject, raw=False, color=color)
+        self._printLogs(project, otherProject, raw=False, color=color,
+                        pretty_format=pretty_format)
        self.out.nl()

    if diff['unreachable']:
@@ -139,9 +144,13 @@ synced and their revisions won't be found.
        self.printText(' not found')
        self.out.nl()

-  def _printLogs(self, project, otherProject, raw=False, color=True):
-    logs = project.getAddedAndRemovedLogs(otherProject, oneline=True,
-                                          color=color)
+  def _printLogs(self, project, otherProject, raw=False, color=True,
+                 pretty_format=None):
+
+    logs = project.getAddedAndRemovedLogs(otherProject,
+                                          oneline=(pretty_format is None),
+                                          color=color,
+                                          pretty_format=pretty_format)
    if logs['removed']:
      removedLogs = logs['removed'].split('\n')
      for log in removedLogs:
@@ -192,4 +201,4 @@ synced and their revisions won't be found.
    if opt.raw:
      self._printRawDiff(diff)
    else:
-      self._printDiff(diff, color=opt.color)
+      self._printDiff(diff, color=opt.color, pretty_format=opt.pretty_format)
--- a/subcmds/forall.py
+++ b/subcmds/forall.py
@@ -120,6 +120,9 @@ without iterating through the remaining projects.
    p.add_option('-r', '--regex',
                 dest='regex', action='store_true',
                 help="Execute the command only on projects matching regex or wildcard expression")
+    p.add_option('-i', '--inverse-regex',
+                 dest='inverse_regex', action='store_true',
+                 help="Execute the command only on projects not matching regex or wildcard expression")
    p.add_option('-g', '--groups',
                 dest='groups',
                 help="Execute the command only on projects matching the specified groups")
@@ -215,10 +218,12 @@ without iterating through the remaining projects.
    if os.path.isfile(smart_sync_manifest_path):
      self.manifest.Override(smart_sync_manifest_path)

-    if not opt.regex:
-      projects = self.GetProjects(args, groups=opt.groups)
-    else:
+    if opt.regex:
      projects = self.FindProjects(args)
+    elif opt.inverse_regex:
+      projects = self.FindProjects(args, inverse=True)
+    else:
+      projects = self.GetProjects(args, groups=opt.groups)

    os.environ['REPO_COUNT'] = str(len(projects))

@@ -240,7 +245,8 @@ without iterating through the remaining projects.
      rc = rc or errno.EINTR
    except Exception as e:
      # Catch any other exceptions raised
-      print('Got an error, terminating the pool: %r' % e,
+      print('Got an error, terminating the pool: %s: %s' %
+              (type(e).__name__, e),
            file=sys.stderr)
      pool.terminate()
      rc = rc or getattr(e, 'errno', 1)
@@ -254,7 +260,8 @@ without iterating through the remaining projects.
      try:
        project = self._SerializeProject(p)
      except Exception as e:
-        print('Project list error: %r' % e,
+        print('Project list error on project %s: %s: %s' %
+                (p.name, type(e).__name__, e),
              file=sys.stderr)
        return
      except KeyboardInterrupt:
--- a/subcmds/gitc_delete.py
+++ b/subcmds/gitc_delete.py
@@ -0,0 +1,55 @@
+#
+# Copyright (C) 2015 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import os
+import shutil
+import sys
+
+from command import Command, GitcClientCommand
+import gitc_utils
+
+from pyversion import is_python3
+if not is_python3():
+  # pylint:disable=W0622
+  input = raw_input
+  # pylint:enable=W0622
+
+class GitcDelete(Command, GitcClientCommand):
+  common = True
+  visible_everywhere = False
+  helpSummary = "Delete a GITC Client."
+  helpUsage = """
+%prog
+"""
+  helpDescription = """
+This subcommand deletes the current GITC client, deleting the GITC manifest
+and all locally downloaded sources.
+"""
+
+  def _Options(self, p):
+    p.add_option('-f', '--force',
+                 dest='force', action='store_true',
+                 help='Force the deletion (no prompt).')
+
+  def Execute(self, opt, args):
+    if not opt.force:
+      prompt = ('This will delete GITC client: %s\nAre you sure? (yes/no) ' %
+                self.gitc_manifest.gitc_client_name)
+      response = input(prompt).lower()
+      if not response == 'yes':
+        print('Response was not "yes"\n Exiting...')
+        sys.exit(1)
+    shutil.rmtree(self.gitc_manifest.gitc_client_dir)
--- a/subcmds/gitc_init.py
+++ b/subcmds/gitc_init.py
@@ -18,11 +18,13 @@ import os
 import sys

 import gitc_utils
-from command import RequiresGitcCommand
+from command import GitcAvailableCommand
+from manifest_xml import GitcManifest
 from subcmds import init
+import wrapper


-class GitcInit(init.Init, RequiresGitcCommand):
+class GitcInit(init.Init, GitcAvailableCommand):
  common = True
  helpSummary = "Initialize a GITC Client."
  helpUsage = """
@@ -34,7 +36,7 @@ with the GITC file system.

 This command will setup the client directory, initialize repo, just
 like repo init does, and then downloads the manifest collection
-and installs in in the .repo/directory of the GITC client.
+and installs it in the .repo/directory of the GITC client.

 Once this is done, a GITC manifest is generated by pulling the HEAD
 SHA for each project and generates the properly formatted XML file
@@ -54,18 +56,15 @@ use for this GITC client.
                 help='Optional manifest file to use for this GITC client.')
    g.add_option('-c', '--gitc-client',
                 dest='gitc_client',
-                 help='The name for the new gitc_client instance.')
+                 help='The name of the gitc_client instance to create or modify.')

  def Execute(self, opt, args):
-    if not opt.gitc_client:
-      print('fatal: gitc client (-c) is required', file=sys.stderr)
+    gitc_client = gitc_utils.parse_clientdir(os.getcwd())
+    if not gitc_client or (opt.gitc_client and gitc_client != opt.gitc_client):
+      print('fatal: Please update your repo command. See go/gitc for instructions.', file=sys.stderr)
      sys.exit(1)
    self.client_dir = os.path.join(gitc_utils.get_gitc_manifest_dir(),
-                                   opt.gitc_client)
-    if not os.path.exists(gitc_utils.get_gitc_manifest_dir()):
-      os.makedirs(gitc_utils.get_gitc_manifest_dir())
-    if not os.path.exists(self.client_dir):
-      os.mkdir(self.client_dir)
+                                   gitc_client)
    super(GitcInit, self).Execute(opt, args)

    manifest_file = self.manifest.manifestFile
@@ -75,6 +74,9 @@ use for this GITC client.
              opt.manifest_file)
        sys.exit(1)
      manifest_file = opt.manifest_file
-    gitc_utils.generate_gitc_manifest(self.repodir, opt.gitc_client, None, manifest_file)
+
+    manifest = GitcManifest(self.repodir, gitc_client)
+    manifest.Override(manifest_file)
+    gitc_utils.generate_gitc_manifest(None, manifest)
    print('Please run `cd %s` to view your GITC client.' %
-          os.path.join(gitc_utils.GITC_FS_ROOT_DIR, opt.gitc_client))
+          os.path.join(wrapper.Wrapper().GITC_FS_ROOT_DIR, gitc_client))
--- a/subcmds/help.py
+++ b/subcmds/help.py
@@ -19,7 +19,7 @@ import sys
 from formatter import AbstractFormatter, DumbWriter

 from color import Coloring
-from command import PagedCommand, MirrorSafeCommand, RequiresGitcCommand
+from command import PagedCommand, MirrorSafeCommand, GitcAvailableCommand, GitcClientCommand
 import gitc_utils

 class Help(PagedCommand, MirrorSafeCommand):
@@ -57,8 +57,12 @@ Displays detailed usage information about a command.
    print('The most commonly used repo commands are:')

    def gitc_supported(cmd):
-      if not isinstance(cmd, RequiresGitcCommand):
+      if not isinstance(cmd, GitcAvailableCommand) and not isinstance(cmd, GitcClientCommand):
        return True
+      if self.manifest.isGitcClient:
+        return True
+      if isinstance(cmd, GitcClientCommand):
+        return False
      if gitc_utils.get_gitc_manifest_dir():
        return True
      return False
--- a/subcmds/init.py
+++ b/subcmds/init.py
@@ -61,6 +61,11 @@ directory use as much data as possible from the local reference
 directory when fetching from the server. This will make the sync
 go a lot faster by reducing data traffic on the network.

+The --no-clone-bundle option disables any attempt to use
+$URL/clone.bundle to bootstrap a new Git repository from a
+resumeable bundle file on a content delivery network. This
+may be necessary if there are problems with the local Python
+HTTP client or proxy configuration, but the Git binary works.

 Switching Manifest Branches
 ---------------------------
@@ -113,6 +118,9 @@ to update the working directory files.
                 help='restrict manifest projects to ones with a specified '
                      'platform group [auto|all|none|linux|darwin|...]',
                 metavar='PLATFORM')
+    g.add_option('--no-clone-bundle',
+                 dest='no_clone_bundle', action='store_true',
+                 help='disable use of /clone.bundle on HTTP/HTTPS')

    # Tool
    g = p.add_option_group('repo Version options')
@@ -179,7 +187,7 @@ to update the working directory files.
      r.Save()

    groups = re.split(r'[,\s]+', opt.groups)
-    all_platforms = ['linux', 'darwin']
+    all_platforms = ['linux', 'darwin', 'windows']
    platformize = lambda x: 'platform-' + x
    if opt.platform == 'auto':
      if (not opt.mirror and
@@ -188,7 +196,7 @@ to update the working directory files.
    elif opt.platform == 'all':
      groups.extend(map(platformize, all_platforms))
    elif opt.platform in all_platforms:
-      groups.extend(platformize(opt.platform))
+      groups.append(platformize(opt.platform))
    elif opt.platform != 'none':
      print('fatal: invalid platform flag', file=sys.stderr)
      sys.exit(1)
@@ -222,7 +230,8 @@ to update the working directory files.
              'in another location.', file=sys.stderr)
        sys.exit(1)

-    if not m.Sync_NetworkHalf(is_new=is_new):
+    if not m.Sync_NetworkHalf(is_new=is_new, quiet=opt.quiet,
+        clone_bundle=not opt.no_clone_bundle):
      r = m.GetRemote(m.remote.name)
      print('fatal: cannot obtain manifest %s' % r.url, file=sys.stderr)

--- a/subcmds/rebase.py
+++ b/subcmds/rebase.py
@@ -54,6 +54,11 @@ branch but need to incorporate new upstream changes "underneath" them.
    p.add_option('--auto-stash',
                 dest='auto_stash', action='store_true',
                 help='Stash local modifications before starting')
+    p.add_option('-m', '--onto-manifest',
+                 dest='onto_manifest', action='store_true',
+                 help='Rebase onto the manifest version instead of upstream '
+                      'HEAD.  This helps to make sure the local tree stays '
+                      'consistent if you previously synced to a manifest.')

  def Execute(self, opt, args):
    all_projects = self.GetProjects(args)
@@ -106,6 +111,10 @@ branch but need to incorporate new upstream changes "underneath" them.
      if opt.interactive:
        args.append("-i")

+      if opt.onto_manifest:
+        args.append('--onto')
+        args.append(project.revisionExpr)
+
      args.append(upbranch.LocalMerge)

      print('# %s: rebasing %s -> %s'
--- a/subcmds/start.py
+++ b/subcmds/start.py
@@ -57,10 +57,15 @@ revision specified in the manifest.
        print("error: at least one project must be specified", file=sys.stderr)
        sys.exit(1)

+    all_projects = self.GetProjects(projects,
+                                    missing_ok=bool(self.gitc_manifest))
+
+    # This must happen after we find all_projects, since GetProjects may need
+    # the local directory, which will disappear once we save the GITC manifest.
    if self.gitc_manifest:
-      all_projects = self.GetProjects(projects, manifest=self.gitc_manifest,
-                                      missing_ok=True)
-      for project in all_projects:
+      gitc_projects = self.GetProjects(projects, manifest=self.gitc_manifest,
+                                       missing_ok=True)
+      for project in gitc_projects:
        if project.old_revision:
          project.already_synced = True
        else:
@@ -70,8 +75,10 @@ revision specified in the manifest.
      # Save the GITC manifest.
      gitc_utils.save_manifest(self.gitc_manifest)

-    all_projects = self.GetProjects(projects,
-                                    missing_ok=bool(self.gitc_manifest))
+      # Make sure we have a valid CWD
+      if not os.path.exists(os.getcwd()):
+        os.chdir(self.manifest.topdir)
+
    pm = Progress('Starting %s' % nb, len(all_projects))
    for project in all_projects:
      pm.update()
--- a/subcmds/sync.py
+++ b/subcmds/sync.py
@@ -75,6 +75,7 @@ from error import RepoChangedException, GitError, ManifestParseError
 from project import SyncBuffer
 from progress import Progress
 from wrapper import Wrapper
+from manifest_xml import GitcManifest

 _ONE_DAY_S = 24 * 60 * 60

@@ -150,6 +151,9 @@ The --optimized-fetch option can be used to only fetch projects that
 are fixed to a sha1 revision if the sha1 revision does not already
 exist locally.

+The --prune option can be used to remove any refs that no longer
+exist on the remote.
+
 SSH Connections
 ---------------

@@ -233,10 +237,12 @@ later is required to fix a server side protocol bug.
    p.add_option('--optimized-fetch',
                 dest='optimized_fetch', action='store_true',
                 help='only fetch projects fixed to sha1 if revision does not exist locally')
+    p.add_option('--prune', dest='prune', action='store_true',
+                 help='delete refs that no longer exist on the remote')
    if show_smart:
      p.add_option('-s', '--smart-sync',
                   dest='smart_sync', action='store_true',
-                   help='smart sync using manifest from a known good build')
+                   help='smart sync using manifest from the latest known good build')
      p.add_option('-t', '--smart-tag',
                   dest='smart_tag', action='store',
                   help='smart sync using manifest from a known tag')
@@ -304,7 +310,8 @@ later is required to fix a server side protocol bug.
          force_sync=opt.force_sync,
          clone_bundle=not opt.no_clone_bundle,
          no_tags=opt.no_tags, archive=self.manifest.IsArchive,
-          optimized_fetch=opt.optimized_fetch)
+          optimized_fetch=opt.optimized_fetch,
+          prune=opt.prune)
        self._fetch_times.Set(project, time.time() - start)

        # Lock around all the rest of the code, since printing, updating a set
@@ -313,6 +320,7 @@ later is required to fix a server side protocol bug.
        did_lock = True

        if not success:
+          err_event.set()
          print('error: Cannot fetch %s' % project.name, file=sys.stderr)
          if opt.force_broken:
            print('warn: --force-broken, continuing to sync',
@@ -323,7 +331,7 @@ later is required to fix a server side protocol bug.
        fetched.add(project.gitdir)
        pm.update()
      except _FetchError:
-        err_event.set()
+        pass
      except Exception as e:
        print('error: Cannot fetch %s (%s: %s)' \
            % (project.name, type(e).__name__, str(e)), file=sys.stderr)
@@ -687,10 +695,13 @@ later is required to fix a server side protocol bug.

      if gitc_projects != [] and not opt.local_only:
        print('Updating GITC client: %s' % self.gitc_manifest.gitc_client_name)
-        gitc_utils.generate_gitc_manifest(self.repodir,
-                                          self.gitc_manifest.gitc_client_name,
-                                          self.gitc_manifest,
-                                          self.manifest.manifestFile,
+        manifest = GitcManifest(self.repodir, self.gitc_manifest.gitc_client_name)
+        if manifest_name:
+          manifest.Override(manifest_name)
+        else:
+          manifest.Override(self.manifest.manifestFile)
+        gitc_utils.generate_gitc_manifest(self.gitc_manifest,
+                                          manifest,
                                          gitc_projects)
        print('GITC client successfully synced.')

@@ -912,6 +923,7 @@ class PersistentTransport(xmlrpc.client.Transport):
      # stripping those prefixes away.
      if cookiefile:
        tmpcookiefile = tempfile.NamedTemporaryFile()
+        tmpcookiefile.write("# HTTP Cookie File")
        try:
          with open(cookiefile) as f:
            for line in f:
@@ -921,7 +933,10 @@ class PersistentTransport(xmlrpc.client.Transport):
          tmpcookiefile.flush()

          cookiejar = cookielib.MozillaCookieJar(tmpcookiefile.name)
-          cookiejar.load()
+          try:
+            cookiejar.load()
+          except cookielib.LoadError:
+            cookiejar = cookielib.CookieJar()
        finally:
          tmpcookiefile.close()
      else:
--- a/subcmds/upload.py
+++ b/subcmds/upload.py
@@ -456,7 +456,9 @@ Gerrit Code Review:  http://code.google.com/p/gerrit/

    if pending and (not opt.bypass_hooks):
      hook = RepoHook('pre-upload', self.manifest.repo_hooks_project,
-                      self.manifest.topdir, abort_if_user_denies=True)
+                      self.manifest.topdir,
+                      self.manifest.manifestProject.GetRemote('origin').url,
+                      abort_if_user_denies=True)
      pending_proj_names = [project.name for (project, avail) in pending]
      pending_worktrees = [project.worktree for (project, avail) in pending]
      try:
--- a/tests/fixtures/gitc_config
+++ b/tests/fixtures/gitc_config
@@ -0,0 +1 @@
+gitc_dir=/test/usr/local/google/gitc
--- a/tests/test_wrapper.py
+++ b/tests/test_wrapper.py
@@ -0,0 +1,75 @@
+#
+# Copyright (C) 2015 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import unittest
+
+import wrapper
+
+def fixture(*paths):
+  """Return a path relative to tests/fixtures.
+  """
+  return os.path.join(os.path.dirname(__file__), 'fixtures', *paths)
+
+class RepoWrapperUnitTest(unittest.TestCase):
+  """Tests helper functions in the repo wrapper
+  """
+  def setUp(self):
+    """Load the wrapper module every time
+    """
+    wrapper._wrapper_module = None
+    self.wrapper = wrapper.Wrapper()
+
+  def test_get_gitc_manifest_dir_no_gitc(self):
+    """
+    Test reading a missing gitc config file
+    """
+    self.wrapper.GITC_CONFIG_FILE = fixture('missing_gitc_config')
+    val = self.wrapper.get_gitc_manifest_dir()
+    self.assertEqual(val, '')
+
+  def test_get_gitc_manifest_dir(self):
+    """
+    Test reading the gitc config file and parsing the directory
+    """
+    self.wrapper.GITC_CONFIG_FILE = fixture('gitc_config')
+    val = self.wrapper.get_gitc_manifest_dir()
+    self.assertEqual(val, '/test/usr/local/google/gitc')
+
+  def test_gitc_parse_clientdir_no_gitc(self):
+    """
+    Test parsing the gitc clientdir without gitc running
+    """
+    self.wrapper.GITC_CONFIG_FILE = fixture('missing_gitc_config')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/something'), None)
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/gitc/manifest-rw/test'), 'test')
+
+  def test_gitc_parse_clientdir(self):
+    """
+    Test parsing the gitc clientdir
+    """
+    self.wrapper.GITC_CONFIG_FILE = fixture('gitc_config')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/something'), None)
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/gitc/manifest-rw/test'), 'test')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/gitc/manifest-rw/test/'), 'test')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/gitc/manifest-rw/test/extra'), 'test')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/test'), 'test')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/test/'), 'test')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/test/extra'), 'test')
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/gitc/manifest-rw/'), None)
+    self.assertEqual(self.wrapper.gitc_parse_clientdir('/test/usr/local/google/gitc/'), None)
+
+if __name__ == '__main__':
+  unittest.main()