Compare commits

...

19 Commits
v2.18 ... v2.21

Author SHA1 Message Date
Mike Frysinger
8e91248655 project: mark gc.log as safe to discard when migrating .git/
This is just a log file that, while useful for humans when gc aborts,
doesn't contain any data, so it's safe to throw away.

Bug: https://crbug.com/gerrit/15619
Change-Id: Ia95e0e281f52260668f7a80b5d5f990e32a8597a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328999
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-01-26 16:46:03 +00:00
XD Trol
630876f9e4 init: add an option --enable-git-lfs-filter
It was reported that git-lfs did not work with git-repo. Specifically,
`git read-tree -u` run by `repo sync` would fail git-lfs's smudge
filter. See https://github.com/github/git-lfs/issues/1422.

In fact, by the time `git read-tree -u` is run, the repository is not
bare. It is just that, the working directory is not the same as the
.git directory. git-lfs's filter should work. No one seems to have
delved into that issue.

Today, with newer versions of git-repo and git-lfs, that issue will
not reproduce. Tested with
- git 2.33, git-lfs 2.13 on macOS
- git 2.17, git-lfs 2.3 on ubuntu

So, it seems fine to add an option --enable-git-lfs-filter, default to
false, and stat that it may not work with older versions of git and
git-lfs in the help doc.

Bug: https://crbug.com/gerrit/14516
Change-Id: I8d21854eeeea541e072f63d6b10ad1253b1a9826
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328359
Tested-by: XD Trol <milestonejxd@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-01-26 01:47:20 +00:00
Mike Frysinger
4aa8584ec6 init: make bad --repo-rev settings more clear
If the user passes a bad --repo-rev setting in a new checkout, add a
tip to the error message that their option is probably bad instead of
just saying "unable to resolve".

If the user has already initialized a checkout, we'd display a raw
traceback which would confuse them.  Swallow that and also include
the --repo-rev tip.

Bug: https://crbug.com/gerrit/15610
Change-Id: I5d72513c7b37bf9bb5d19862fcdfaf0d1f44e886
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328820
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-25 18:54:42 +00:00
Peter Kjellerstedt
b550501254 project: Ignore failure to remove the sample hooks
Removing the sample hooks is just clean up, so if repo cannot remove a
sample hook that should not cause it to fail.

Change-Id: I716b977da091c22b8f53e134f4fbc114116f9a65
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328635
Reviewed-by: Mike Frysinger <vapier@google.com>
2022-01-22 06:13:10 +00:00
LaMont Jones
a535ae4418 branches: Fix "not in" handling
If the branch is current, or present in less than half of the projects,
list which projects it is *in*.

Otherwise, correctly detect which projects (by relpath) it is not in.

Previously, the "not in" path would incorrectly list all projects.

Change-Id: Ia153856f577035a51f538b7bf5d3135b70c69d52
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328199
Tested-by: LaMont Jones <lamontjones@google.com>
Reviewed-by: Xin Li <delphij@google.com>
2022-01-21 17:20:21 +00:00
Mike Frysinger
67d6cdf2bc project: store objects in project-objects directly
In order to stop sharing objects/ directly between shared projects,
we have to fetch the remote objects into project-objects/ manually.
So instead of running git operations in the individual project dirs
and relying on .git/objects being symlinked to project-objects/,
tell git to store any objects it fetches in project-objects/.

We do this by leveraging the GIT_OBJECT_DIRECTORY override.  This
has been in git forever, or at least since v1.7.2 which is what we
already hard require.  This tells git to save new objects to the
specified path no matter where it's being run otherwise.

We still otherwise run git in the project-specific dir so that it
can find the right set of refs that it wants to compare against,
including local refs.  For that reason, we also have to leverage
GIT_ALTERNATE_OBJECT_DIRECTORIES to tell git where to find objects
that are not in the upstream remote.  This way git doesn't blow up
when it can't find objects only associated with local commits.

As it stands right now, the practical result is the same: since we
symlink the project objects/ dir to the project-objects/ tree, the
default objects dir, the one we set $GIT_OBJECT_DIRECTORY to, and
the one we set $GIT_ALTERNATE_OBJECT_DIRECTORIES to are actually
all the same.  So this commit by itself should be safe.  But in a
follow up commit, we can replace the symlink with a separate dir
and git will keep working.

Bug: https://crbug.com/gerrit/15553
Change-Id: Ie4e654aec3e1ee307eee925a54908a2db6a5869f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328100
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-19 17:24:51 +00:00
Mike Frysinger
152032cca2 project: move --reference handling to project-objects
When using --reference, the path is written to objects/info/alternates.
The path is accessed inconsistently -- sometimes through projects/ (via
self.gitdir) and sometimes through project-objects/ (via self.objdir).
This works because projects/.../objects is a symlink to the objects dir
under project-objects/.  Change all accesses to go through self.objdir.
This will allow us to stop symlinking projects/.../objects without the
reference dir logic breaking.  The projects/ path is going to use its
alternates file for its own needs.

Bug: https://crbug.com/gerrit/15553
Change-Id: I6b452ad1aaffec74ecb7ac1bb9baa3a3a52e076c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/328099
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jack Neus <jackneus@google.com>
2022-01-13 18:27:28 +00:00
Sebastian Wagner
a3ac816278 test_project: use os.readlink instead of Path.readlink
Path.readlink is only available on Python 3.9, breaking compatibility
with all python versions below. os.readlink is already used in other
places of this file, so use it here as well.

Change-Id: I5acf8f5334a3e7c8de9cea1939d7e2b9af5f30ae
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327844
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sebastian Wagner <sebix@sebix.at>
2022-01-11 20:16:42 +00:00
Mike Frysinger
98bb76577d project: prune sample hooks
These hooks are never used and often get stale, so just trim them.
Users rarely look in these dirs to begin with.

Change-Id: Ic785aa55fb7ec84a61376df101127d0018882030
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327538
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-10 17:41:45 +00:00
Mike Frysinger
d33dce0b77 project: drop support for symlinking internal .git files
Since we don't do this anymore, and there prob won't be a need to
bring it back, drop support for it.

Bug: https://crbug.com/gerrit/15460
Change-Id: I7d86706f108c797a5c7962cb1578693d49430367
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327537
Reviewed-by: Jack Neus <jackneus@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-10 17:41:40 +00:00
Mike Frysinger
89ed8acdbe project: abort a bit earlier before migrating .git/
Verify all the .git/ paths will be handled by the migration logic before
starting the migration.  This way we still abort & log an error, but the
user gets to see it before we put the tree into a state that they have to
manually recover.  Also add a few more known-safe-to-clobber paths.

Bug: https://crbug.com/gerrit/15273
Change-Id: If49d69b341bc960ddcafa30da333fb5ec7145b51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327557
Reviewed-by: Colin Cross <ccross@android.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-07 20:17:14 +00:00
Raman Tenneti
71e48b7672 Revert "sync: dropped "NOTICE: --use-superproject is in beta ..." message."
This reverts commit d53cb9549a. As long as
repo's reference docs treat this feature as a work in progress and don't
cover it well enough to allow all repo maintainers to easily support it,
it is inconsistent to report to users that it is no longer in beta.
Thanks for vapier@google.com for noticing.

https://crbug.com/gerrit/15527 tracks the required documentation changes
before we'd be ready to roll forward again.

Change-Id: Ic9bd951cfb3c1abf6e1bfa30dfe4afa1c9b7bec6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327337
Reviewed-by: Jonathan Nieder <jrn@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Jonathan Nieder <jrn@google.com>
2022-01-06 20:01:03 +00:00
Mike Frysinger
13576a8caf project: stop symlinking info dir under .git/
Unsharing this directory shouldn't be a problem.  The current repo code
treated it as a file, and while that's actually incorrect, files & dirs
are basically treated the same, so it's practically the same.

Let's enumerate each subpath since there aren't that many.

info/refs:
Only used when the project is exported over git dumb transports (i.e.
a http:// server).  Repo never does this, and it's extremely unlikely
any user has ever done this.  Plus, this proposal talks about unsharing
project refs, so this file should get unshared too.

info/grafts:
A user-configurable file that repo never touches.  Might be useful to
share across projects, but probably rarely (if ever) used by developers,
and forcing them to configure it for each project isn't that big of a
deal.

info/exclude:
info/attributes:
User-configurable files that repo never touches.  Doesn't seem like
most users ever touch these, and if they do, having them do it for
each shared project isn't a big deal.

info/sparse-checkout:
Repo doesn't use sparse checkouts, and it's extremely unlikely to even
work if a user tried doing something themselves.

Bug: https://crbug.com/gerrit/15460
Change-Id: I53e44d73a6d7a92da615b46600d8ea51cb46e3ac
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327519
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-06 08:31:45 +00:00
Mike Frysinger
2345906d04 project: stop symlinking description file under .git/
Nothing uses this path.  It’s only for exporting git dirs e.g. for
online gitweb use which probably no one does.  It is not the same
description file as exists on servers we cloned from.  Leaving it
as the default plain text file will simplify code.

We don't undo any existing symlinks if they exist since repo does
not care about them, and their existence doesn't hurt.

Bug: https://crbug.com/gerrit/15460
Change-Id: Ic34fe7c3cfb8f6da844de5be30158f59382b1cc8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327518
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-06 08:29:06 +00:00
Mike Frysinger
41289c62b4 project: stop symlinking svn under .git/
This path only matters to users of `git svn` who manually run it in
local projects after they get a full repo client checkout.  With svn
usage falling in general, and with the fact that the source checkout
now symlinks its .git/ state to the internal projects/ path, we don't
need to manage this anymore.

It means the path won't be shared among multiple local projects that
have the same remote, but so it goes.  It was an optimization only,
not functionality required for correctness.  We want to simplify the
internals to stop messing with git state, and this particular path
doesn't seem worth the effort to maintain.

We don't undo any existing svn symlinks if they exist since repo does
not care about them, and their existence doesn't hurt anything.

Bug: https://crbug.com/gerrit/15460
Change-Id: Ie8496b275bcc589771aa9f4ee874ed2ee6d5241d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327517
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2022-01-06 08:28:37 +00:00
Mike Frysinger
c72bd8486a project: clean up now unused code
Now that we symlink worktree .git/ paths to .repo/projects/, we never
set share_refs=True anywhere, which means all of this logic is dead
code.  Throw it all away.  Do it as a separate commit to make the
parent commit easier to review.

Bug: https://crbug.com/gerrit/15273
Change-Id: If496d39029d3d3bd523ba24c603ce47a63ad9b51
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/326817
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jack Neus <jackneus@google.com>
2022-01-06 04:08:05 +00:00
Raman Tenneti
d53cb9549a sync: dropped "NOTICE: --use-superproject is in beta ..." message.
Tested the code with the following commands.

$ ./run_tests -v

Bug: [google internal] b/209511230
Change-Id: Ia3c6de47709f5276e324a5bb608383aba3b2c562
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/327197
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-12-29 19:07:08 +00:00
Raman Tenneti
cf0ba48649 sync: With --mirror option, don't display no-use-superproject... message.
+ Display 'Defaulting to no-use-superproject because there is no working tree.'
  message if --use-superproject option is used and we are not using
  superproject because manifest is either a mirror or is an archive.

Tested the code with the following commands.

$ ./run_tests -v

Tested the sync code by using repo_dev alias and pointing to this CL.

$ repo init -u https://android.googlesource.com/mirror/manifest --mirror

$ repo_dev sync
Receiving objects: 100% (3/3), done.eiving objects:  33% (1/3)

$ repo_dev sync --use-superproject
Defaulting to no-use-superproject because there is no working tree.
Fetching:  0% (0/2158) warming up

Bug: https://crbug.com/gerrit/15368
Change-Id: I16b87ee9623315dbc3100b612b1decdaab7ac1dc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/325797
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
2021-12-07 16:46:41 +00:00
Mike Frysinger
2a089cfee4 project: migrate worktree .git/ dirs to symlinks
Historically we created a .git/ subdir in each source checkout and
symlinked individual files to the .repo/projects/ paths.  This layer
of indirection isn't actually needed: the .repo/projects/ paths are
guaranteed to only ever have a 1-to-1 mapping with the actual git
checkout.  So we don't need to worry about having files in .git/ be
isolated.

To that end, change how we manage the actual project checkouts from
a dir full of symlinks (and a few files) to a symlink to the internal
.repo/projects/ dir.  This makes the code simpler & faster.

The directory structure we have today is:
.repo/
  project-objects/chromiumos/third_party/kernel.git/
    <paths omitted as not relevant to this change>
  projects/src/third_party/kernel/
    v3.8.git/
      config
      description   -> …/project-objects/…/config
      FETCH_HEAD
      HEAD
      hooks/        -> …/project-objects/…/hooks/
      info/         -> …/project-objects/…/info/
      logs/
      objects/      -> …/project-objects/…/objects/
      packed-refs
      refs/
      rr-cache/     -> …/project-objects/…/rr-cache/
src/third_party/kernel/
  v3.8/
    .git/
      config        -> …/projects/…/v3.8.git/config
      description   -> …/project-objects/…/v3.8.git/description
      HEAD
      hooks/        -> …/project-objects/…/v3.8.git/hooks/
      index
      info/         -> …/project-objects/…/v3.8.git/info/
      logs/         -> …/projects/…/v3.8.git/logs/
      objects/      -> …/project-objects/…/v3.8.git/objects/
      packed-refs   -> …/projects/…/v3.8.git/packed-refs
      refs/         -> …/projects/…/v3.8.git/refs/
      rr-cache/     -> …/project-objects/…/v3.8.git/rr-cache/

The directory structure we have after this commit:
.repo/
  <nothing changes>
src/third_party/kernel/
  v3.8/
    .git            -> …/projects/…/v3.8.git

Bug: https://crbug.com/gerrit/15273
Change-Id: I9dd8def23fbfb2f4cb209a93f8b1b2b24002a444
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/323695
Reviewed-by: Mike Nichols <mikenichols@google.com>
Reviewed-by: Xin Li <delphij@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
2021-12-01 15:27:16 +00:00
9 changed files with 256 additions and 105 deletions

View File

@@ -163,6 +163,7 @@ User controlled settings are initialized when running `repo init`.
| repo.clonefilter | `--clone-filter` | Filter setting when using [partial git clones] |
| repo.depth | `--depth` | Create shallow checkouts when cloning |
| repo.dissociate | `--dissociate` | Dissociate from any reference/mirrors after initial clone |
| repo.git-lfs | `--git-lfs` | Enable [Git LFS] support |
| repo.mirror | `--mirror` | Checkout is a repo mirror |
| repo.partialclone | `--partial-clone` | Create [partial git clones] |
| repo.partialcloneexclude | `--partial-clone-exclude` | Comma-delimited list of project names (not paths) to exclude while using [partial git clones] |
@@ -254,6 +255,7 @@ Repo will create & maintain a few files in the user's home directory.
[git-config]: https://git-scm.com/docs/git-config
[Git LFS]: https://git-lfs.github.com/
[git worktree]: https://git-scm.com/docs/git-worktree
[gitsubmodules]: https://git-scm.com/docs/gitsubmodules
[manifest-format.md]: ./manifest-format.md

View File

@@ -169,7 +169,8 @@ class GitCommand(object):
disable_editor=False,
ssh_proxy=None,
cwd=None,
gitdir=None):
gitdir=None,
objdir=None):
env = self._GetBasicEnv()
if disable_editor:
@@ -194,13 +195,24 @@ class GitCommand(object):
cwd = project.worktree
if not gitdir:
gitdir = project.gitdir
# Git on Windows wants its paths only using / for reliability.
if platform_utils.isWindows():
if objdir:
objdir = objdir.replace('\\', '/')
if gitdir:
gitdir = gitdir.replace('\\', '/')
if objdir:
# Set to the place we want to save the objects.
env['GIT_OBJECT_DIRECTORY'] = objdir
if gitdir:
# Allow git to search the original place in case of local or unique refs
# that git will attempt to resolve even if we aren't fetching them.
env['GIT_ALTERNATE_OBJECT_DIRECTORIES'] = gitdir + '/objects'
command = [GIT]
if bare:
if gitdir:
# Git on Windows wants its paths only using / for reliability.
if platform_utils.isWindows():
gitdir = gitdir.replace('\\', '/')
env[GIT_DIR] = gitdir
cwd = None
command.append(cmdv[0])
@@ -234,6 +246,11 @@ class GitCommand(object):
dbg += ': export GIT_DIR=%s\n' % env[GIT_DIR]
LAST_GITDIR = env[GIT_DIR]
if 'GIT_OBJECT_DIRECTORY' in env:
dbg += ': export GIT_OBJECT_DIRECTORY=%s\n' % env['GIT_OBJECT_DIRECTORY']
if 'GIT_ALTERNATE_OBJECT_DIRECTORIES' in env:
dbg += ': export GIT_ALTERNATE_OBJECT_DIRECTORIES=%s\n' % env['GIT_ALTERNATE_OBJECT_DIRECTORIES']
dbg += ': '
dbg += ' '.join(command)
if stdin == subprocess.PIPE:

View File

@@ -666,6 +666,10 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
def HasSubmodules(self):
return self.manifestProject.config.GetBoolean('repo.submodules')
@property
def EnableGitLfs(self):
return self.manifestProject.config.GetBoolean('repo.git-lfs')
def GetDefaultGroupsStr(self):
"""Returns the default group string for the platform."""
return 'default,platform-' + platform.system().lower()

View File

@@ -457,11 +457,7 @@ class RemoteSpec(object):
class Project(object):
# These objects can be shared between several working trees.
shareable_files = ['description', 'info']
shareable_dirs = ['hooks', 'objects', 'rr-cache', 'svn']
# These objects can only be used by a single working tree.
working_tree_files = ['config', 'packed-refs', 'shallow']
working_tree_dirs = ['logs', 'refs']
shareable_dirs = ['hooks', 'objects', 'rr-cache']
def __init__(self,
manifest,
@@ -1124,7 +1120,7 @@ class Project(object):
self._InitRemote()
if is_new:
alt = os.path.join(self.gitdir, 'objects/info/alternates')
alt = os.path.join(self.objdir, 'objects/info/alternates')
try:
with open(alt) as fd:
# This works for both absolute and relative alternate directories.
@@ -1173,7 +1169,7 @@ class Project(object):
mp = self.manifest.manifestProject
dissociate = mp.config.GetBoolean('repo.dissociate')
if dissociate:
alternates_file = os.path.join(self.gitdir, 'objects/info/alternates')
alternates_file = os.path.join(self.objdir, 'objects/info/alternates')
if os.path.exists(alternates_file):
cmd = ['repack', '-a', '-d']
p = GitCommand(self, cmd, bare=True, capture_stdout=bool(output_redir),
@@ -2196,8 +2192,10 @@ class Project(object):
retry_cur_sleep = retry_sleep_initial_sec
ok = prune_tried = False
for try_n in range(retry_fetches):
gitcmd = GitCommand(self, cmd, bare=True, ssh_proxy=ssh_proxy,
merge_output=True, capture_stdout=quiet or bool(output_redir))
gitcmd = GitCommand(
self, cmd, bare=True, objdir=os.path.join(self.objdir, 'objects'),
ssh_proxy=ssh_proxy,
merge_output=True, capture_stdout=quiet or bool(output_redir))
if gitcmd.stdout and not quiet and output_redir:
output_redir.write(gitcmd.stdout)
ret = gitcmd.Wait()
@@ -2313,7 +2311,8 @@ class Project(object):
cmd.append(str(f))
cmd.append('+refs/tags/*:refs/tags/*')
ok = GitCommand(self, cmd, bare=True).Wait() == 0
ok = GitCommand(
self, cmd, bare=True, objdir=os.path.join(self.objdir, 'objects')).Wait() == 0
platform_utils.remove(bundle_dst, missing_ok=True)
platform_utils.remove(bundle_tmp, missing_ok=True)
return ok
@@ -2483,10 +2482,9 @@ class Project(object):
os.makedirs(self.gitdir)
if init_obj_dir or init_git_dir:
self._ReferenceGitDir(self.objdir, self.gitdir, share_refs=False,
copy_all=True)
self._ReferenceGitDir(self.objdir, self.gitdir, copy_all=True)
try:
self._CheckDirReference(self.objdir, self.gitdir, share_refs=False)
self._CheckDirReference(self.objdir, self.gitdir)
except GitError as e:
if force_sync:
print("Retrying clone after deleting %s" %
@@ -2509,8 +2507,8 @@ class Project(object):
if ref_dir or mirror_git:
if not mirror_git:
mirror_git = os.path.join(ref_dir, self.name + '.git')
repo_git = os.path.join(ref_dir, '.repo', 'projects',
self.relpath + '.git')
repo_git = os.path.join(ref_dir, '.repo', 'project-objects',
self.name + '.git')
worktrees_git = os.path.join(ref_dir, '.repo', 'worktrees',
self.name + '.git')
@@ -2528,15 +2526,16 @@ class Project(object):
# The alternate directory is relative to the object database.
ref_dir = os.path.relpath(ref_dir,
os.path.join(self.objdir, 'objects'))
_lwrite(os.path.join(self.gitdir, 'objects/info/alternates'),
_lwrite(os.path.join(self.objdir, 'objects/info/alternates'),
os.path.join(ref_dir, 'objects') + '\n')
m = self.manifest.manifestProject.config
for key in ['user.name', 'user.email']:
if m.Has(key, include_defaults=False):
self.config.SetString(key, m.GetString(key))
self.config.SetString('filter.lfs.smudge', 'git-lfs smudge --skip -- %f')
self.config.SetString('filter.lfs.process', 'git-lfs filter-process --skip')
if not self.manifest.EnableGitLfs:
self.config.SetString('filter.lfs.smudge', 'git-lfs smudge --skip -- %f')
self.config.SetString('filter.lfs.process', 'git-lfs filter-process --skip')
self.config.SetBoolean('core.bare', True if self.manifest.IsMirror else None)
except Exception:
if init_obj_dir and os.path.exists(self.objdir):
@@ -2553,6 +2552,14 @@ class Project(object):
hooks = platform_utils.realpath(os.path.join(self.objdir, 'hooks'))
if not os.path.exists(hooks):
os.makedirs(hooks)
# Delete sample hooks. They're noise.
for hook in glob.glob(os.path.join(hooks, '*.sample')):
try:
platform_utils.remove(hook, missing_ok=True)
except PermissionError:
pass
for stock_hook in _ProjectHooks():
name = os.path.basename(stock_hook)
@@ -2650,40 +2657,16 @@ class Project(object):
else:
active_git.symbolic_ref('-m', msg, ref, dst)
def _CheckDirReference(self, srcdir, destdir, share_refs):
def _CheckDirReference(self, srcdir, destdir):
# Git worktrees don't use symlinks to share at all.
if self.use_git_worktrees:
return
symlink_files = self.shareable_files[:]
symlink_dirs = self.shareable_dirs[:]
if share_refs:
symlink_files += self.working_tree_files
symlink_dirs += self.working_tree_dirs
to_symlink = symlink_files + symlink_dirs
for name in set(to_symlink):
for name in self.shareable_dirs:
# Try to self-heal a bit in simple cases.
dst_path = os.path.join(destdir, name)
src_path = os.path.join(srcdir, name)
if name in self.working_tree_dirs:
# If the dir is missing under .repo/projects/, create it.
if not os.path.exists(src_path):
os.makedirs(src_path)
elif name in self.working_tree_files:
# If it's a file under the checkout .git/ and the .repo/projects/ has
# nothing, move the file under the .repo/projects/ tree.
if not os.path.exists(src_path) and os.path.isfile(dst_path):
platform_utils.rename(dst_path, src_path)
# If the path exists under the .repo/projects/ and there's no symlink
# under the checkout .git/, recreate the symlink.
if name in self.working_tree_dirs or name in self.working_tree_files:
if os.path.exists(src_path) and not os.path.exists(dst_path):
platform_utils.symlink(
os.path.relpath(src_path, os.path.dirname(dst_path)), dst_path)
dst = platform_utils.realpath(dst_path)
if os.path.lexists(dst):
src = platform_utils.realpath(src_path)
@@ -2696,23 +2679,17 @@ class Project(object):
' use `repo sync --force-sync {0}` to '
'proceed.'.format(self.relpath))
def _ReferenceGitDir(self, gitdir, dotgit, share_refs, copy_all):
def _ReferenceGitDir(self, gitdir, dotgit, copy_all):
"""Update |dotgit| to reference |gitdir|, using symlinks where possible.
Args:
gitdir: The bare git repository. Must already be initialized.
dotgit: The repository you would like to initialize.
share_refs: If true, |dotgit| will store its refs under |gitdir|.
Only one work tree can store refs under a given |gitdir|.
copy_all: If true, copy all remaining files from |gitdir| -> |dotgit|.
This saves you the effort of initializing |dotgit| yourself.
"""
symlink_files = self.shareable_files[:]
symlink_dirs = self.shareable_dirs[:]
if share_refs:
symlink_files += self.working_tree_files
symlink_dirs += self.working_tree_dirs
to_symlink = symlink_files + symlink_dirs
to_symlink = symlink_dirs
to_copy = []
if copy_all:
@@ -2740,11 +2717,6 @@ class Project(object):
elif os.path.isfile(src):
shutil.copy(src, dst)
# If the source file doesn't exist, ensure the destination
# file doesn't either.
if name in symlink_files and not os.path.lexists(src):
platform_utils.remove(dst, missing_ok=True)
except OSError as e:
if e.errno == errno.EPERM:
raise DownloadError(self._get_symlink_error_message())
@@ -2781,50 +2753,112 @@ class Project(object):
self._InitMRef()
def _InitWorkTree(self, force_sync=False, submodules=False):
realdotgit = os.path.join(self.worktree, '.git')
tmpdotgit = realdotgit + '.tmp'
init_dotgit = not os.path.exists(realdotgit)
if init_dotgit:
if self.use_git_worktrees:
"""Setup the worktree .git path.
This is the user-visible path like src/foo/.git/.
With non-git-worktrees, this will be a symlink to the .repo/projects/ path.
With git-worktrees, this will be a .git file using "gitdir: ..." syntax.
Older checkouts had .git/ directories. If we see that, migrate it.
This also handles changes in the manifest. Maybe this project was backed
by "foo/bar" on the server, but now it's "new/foo/bar". We have to update
the path we point to under .repo/projects/ to match.
"""
dotgit = os.path.join(self.worktree, '.git')
# If using an old layout style (a directory), migrate it.
if not platform_utils.islink(dotgit) and platform_utils.isdir(dotgit):
self._MigrateOldWorkTreeGitDir(dotgit)
init_dotgit = not os.path.exists(dotgit)
if self.use_git_worktrees:
if init_dotgit:
self._InitGitWorktree()
self._CopyAndLinkFiles()
return
dotgit = tmpdotgit
platform_utils.rmtree(tmpdotgit, ignore_errors=True)
os.makedirs(tmpdotgit)
self._ReferenceGitDir(self.gitdir, tmpdotgit, share_refs=True,
copy_all=False)
else:
dotgit = realdotgit
if not init_dotgit:
# See if the project has changed.
if platform_utils.realpath(self.gitdir) != platform_utils.realpath(dotgit):
platform_utils.remove(dotgit)
try:
self._CheckDirReference(self.gitdir, dotgit, share_refs=True)
except GitError as e:
if force_sync and not init_dotgit:
try:
platform_utils.rmtree(dotgit)
return self._InitWorkTree(force_sync=False, submodules=submodules)
except Exception:
raise e
raise e
if init_dotgit or not os.path.exists(dotgit):
os.makedirs(self.worktree, exist_ok=True)
platform_utils.symlink(os.path.relpath(self.gitdir, self.worktree), dotgit)
if init_dotgit:
_lwrite(os.path.join(tmpdotgit, HEAD), '%s\n' % self.GetRevisionId())
if init_dotgit:
_lwrite(os.path.join(dotgit, HEAD), '%s\n' % self.GetRevisionId())
# Now that the .git dir is fully set up, move it to its final home.
platform_utils.rename(tmpdotgit, realdotgit)
# Finish checking out the worktree.
cmd = ['read-tree', '--reset', '-u', '-v', HEAD]
if GitCommand(self, cmd).Wait() != 0:
raise GitError('Cannot initialize work tree for ' + self.name)
# Finish checking out the worktree.
cmd = ['read-tree', '--reset', '-u']
cmd.append('-v')
cmd.append(HEAD)
if GitCommand(self, cmd).Wait() != 0:
raise GitError('Cannot initialize work tree for ' + self.name)
if submodules:
self._SyncSubmodules(quiet=True)
self._CopyAndLinkFiles()
if submodules:
self._SyncSubmodules(quiet=True)
self._CopyAndLinkFiles()
@classmethod
def _MigrateOldWorkTreeGitDir(cls, dotgit):
"""Migrate the old worktree .git/ dir style to a symlink.
This logic specifically only uses state from |dotgit| to figure out where to
move content and not |self|. This way if the backing project also changed
places, we only do the .git/ dir to .git symlink migration here. The path
updates will happen independently.
"""
# Figure out where in .repo/projects/ it's pointing to.
if not os.path.islink(os.path.join(dotgit, 'refs')):
raise GitError(f'{dotgit}: unsupported checkout state')
gitdir = os.path.dirname(os.path.realpath(os.path.join(dotgit, 'refs')))
# Remove known symlink paths that exist in .repo/projects/.
KNOWN_LINKS = {
'config', 'description', 'hooks', 'info', 'logs', 'objects',
'packed-refs', 'refs', 'rr-cache', 'shallow', 'svn',
}
# Paths that we know will be in both, but are safe to clobber in .repo/projects/.
SAFE_TO_CLOBBER = {
'COMMIT_EDITMSG', 'FETCH_HEAD', 'HEAD', 'gc.log', 'gitk.cache', 'index',
'ORIG_HEAD',
}
# First see if we'd succeed before starting the migration.
unknown_paths = []
for name in platform_utils.listdir(dotgit):
# Ignore all temporary/backup names. These are common with vim & emacs.
if name.endswith('~') or (name[0] == '#' and name[-1] == '#'):
continue
dotgit_path = os.path.join(dotgit, name)
if name in KNOWN_LINKS:
if not platform_utils.islink(dotgit_path):
unknown_paths.append(f'{dotgit_path}: should be a symlink')
else:
gitdir_path = os.path.join(gitdir, name)
if name not in SAFE_TO_CLOBBER and os.path.exists(gitdir_path):
unknown_paths.append(f'{dotgit_path}: unknown file; please file a bug')
if unknown_paths:
raise GitError('Aborting migration: ' + '\n'.join(unknown_paths))
# Now walk the paths and sync the .git/ to .repo/projects/.
for name in platform_utils.listdir(dotgit):
dotgit_path = os.path.join(dotgit, name)
# Ignore all temporary/backup names. These are common with vim & emacs.
if name.endswith('~') or (name[0] == '#' and name[-1] == '#'):
platform_utils.remove(dotgit_path)
elif name in KNOWN_LINKS:
platform_utils.remove(dotgit_path)
else:
gitdir_path = os.path.join(gitdir, name)
platform_utils.remove(gitdir_path, missing_ok=True)
platform_utils.rename(dotgit_path, gitdir_path)
# Now that the dir should be empty, clear it out, and symlink it over.
platform_utils.rmdir(dotgit)
platform_utils.symlink(os.path.relpath(gitdir, os.path.dirname(dotgit)), dotgit)
def _get_symlink_error_message(self):
if platform_utils.isWindows():

8
repo
View File

@@ -149,7 +149,7 @@ if not REPO_REV:
BUG_URL = 'https://bugs.chromium.org/p/gerrit/issues/entry?template=Repo+tool+issue'
# increment this whenever we make important changes to this script
VERSION = (2, 17)
VERSION = (2, 21)
# increment this if the MAINTAINER_KEYS block is modified
KEYRING_VERSION = (2, 3)
@@ -382,6 +382,11 @@ def InitParser(parser, gitc_init=False):
group.add_option('--no-clone-bundle',
dest='clone_bundle', action='store_false',
help='disable use of /clone.bundle on HTTP/HTTPS (default if --partial-clone)')
group.add_option('--git-lfs', action='store_true',
help='enable Git LFS support')
group.add_option('--no-git-lfs',
dest='git_lfs', action='store_false',
help='disable Git LFS support')
# Tool.
group = parser.add_option_group('repo Version options')
@@ -618,6 +623,7 @@ def _Init(args, gitc_init=False):
"REPO_URL set correctly?" % url, file=sys.stderr)
except CloneFailure:
print('fatal: double check your --repo-rev setting.', file=sys.stderr)
if opt.quiet:
print('fatal: repo init failed; run without --quiet to see why',
file=sys.stderr)

View File

@@ -151,7 +151,7 @@ is shown, then the branch appears in all projects.
fmt = out.write
paths = []
non_cur_paths = []
if i.IsSplitCurrent or (in_cnt < project_cnt - in_cnt):
if i.IsSplitCurrent or (in_cnt <= project_cnt - in_cnt):
in_type = 'in'
for b in i.projects:
if not i.IsSplitCurrent or b.current:
@@ -163,9 +163,9 @@ is shown, then the branch appears in all projects.
in_type = 'not in'
have = set()
for b in i.projects:
have.add(b.project)
have.add(b.project.relpath)
for p in projects:
if p not in have:
if p.relpath not in have:
paths.append(p.relpath)
s = ' %s %s' % (in_type, ', '.join(paths))

View File

@@ -291,6 +291,15 @@ to update the working directory files.
if opt.submodules:
m.config.SetBoolean('repo.submodules', opt.submodules)
if opt.git_lfs is not None:
if opt.git_lfs:
git_require((2, 17, 0), fail=True, msg='Git LFS support')
m.config.SetBoolean('repo.git-lfs', opt.git_lfs)
if not is_new:
print('warning: Changing --git-lfs settings will only affect new project checkouts.\n'
' Existing projects will require manual updates.\n', file=sys.stderr)
if opt.use_superproject is not None:
m.config.SetBoolean('repo.superproject', opt.use_superproject)
@@ -520,8 +529,12 @@ to update the working directory files.
# Handle new --repo-rev requests.
if opt.repo_rev:
wrapper = Wrapper()
remote_ref, rev = wrapper.check_repo_rev(
rp.gitdir, opt.repo_rev, repo_verify=opt.repo_verify, quiet=opt.quiet)
try:
remote_ref, rev = wrapper.check_repo_rev(
rp.gitdir, opt.repo_rev, repo_verify=opt.repo_verify, quiet=opt.quiet)
except wrapper.CloneFailure:
print('fatal: double check your --repo-rev setting.', file=sys.stderr)
sys.exit(1)
branch = rp.GetBranch('default')
branch.merge = remote_ref
rp.work_git.reset('--hard', rev)

View File

@@ -986,10 +986,11 @@ later is required to fix a server side protocol bug.
load_local_manifests = not self.manifest.HasLocalManifests
use_superproject = git_superproject.UseSuperproject(opt, self.manifest)
if self.manifest.IsMirror or self.manifest.IsArchive:
if use_superproject and (self.manifest.IsMirror or self.manifest.IsArchive):
# Don't use superproject, because we have no working tree.
use_superproject = False
print('Defaulting to no-use-superproject because there is no working tree.')
if opt.use_superproject is not None:
print('Defaulting to no-use-superproject because there is no working tree.')
superproject_logging_data = {
'superproject': use_superproject,
'haslocalmanifests': bool(self.manifest.HasLocalManifests),

View File

@@ -16,6 +16,7 @@
import contextlib
import os
from pathlib import Path
import shutil
import subprocess
import tempfile
@@ -335,3 +336,76 @@ class LinkFile(CopyLinkTestCase):
platform_utils.symlink(self.tempdir, dest)
lf._Link()
self.assertEqual(os.path.join('git-project', 'foo.txt'), os.readlink(dest))
class MigrateWorkTreeTests(unittest.TestCase):
"""Check _MigrateOldWorkTreeGitDir handling."""
_SYMLINKS = {
'config', 'description', 'hooks', 'info', 'logs', 'objects',
'packed-refs', 'refs', 'rr-cache', 'shallow', 'svn',
}
_FILES = {
'COMMIT_EDITMSG', 'FETCH_HEAD', 'HEAD', 'index', 'ORIG_HEAD',
'unknown-file-should-be-migrated',
}
_CLEAN_FILES = {
'a-vim-temp-file~', '#an-emacs-temp-file#',
}
@classmethod
@contextlib.contextmanager
def _simple_layout(cls):
"""Create a simple repo client checkout to test against."""
with tempfile.TemporaryDirectory() as tempdir:
tempdir = Path(tempdir)
gitdir = tempdir / '.repo/projects/src/test.git'
gitdir.mkdir(parents=True)
cmd = ['git', 'init', '--bare', str(gitdir)]
subprocess.check_call(cmd)
dotgit = tempdir / 'src/test/.git'
dotgit.mkdir(parents=True)
for name in cls._SYMLINKS:
(dotgit / name).symlink_to(f'../../../.repo/projects/src/test.git/{name}')
for name in cls._FILES | cls._CLEAN_FILES:
(dotgit / name).write_text(name)
yield tempdir
def test_standard(self):
"""Migrate a standard checkout that we expect."""
with self._simple_layout() as tempdir:
dotgit = tempdir / 'src/test/.git'
project.Project._MigrateOldWorkTreeGitDir(str(dotgit))
# Make sure the dir was transformed into a symlink.
self.assertTrue(dotgit.is_symlink())
self.assertEqual(os.readlink(dotgit), '../../.repo/projects/src/test.git')
# Make sure files were moved over.
gitdir = tempdir / '.repo/projects/src/test.git'
for name in self._FILES:
self.assertEqual(name, (gitdir / name).read_text())
# Make sure files were removed.
for name in self._CLEAN_FILES:
self.assertFalse((gitdir / name).exists())
def test_unknown(self):
"""A checkout with unknown files should abort."""
with self._simple_layout() as tempdir:
dotgit = tempdir / 'src/test/.git'
(tempdir / '.repo/projects/src/test.git/random-file').write_text('one')
(dotgit / 'random-file').write_text('two')
with self.assertRaises(error.GitError):
project.Project._MigrateOldWorkTreeGitDir(str(dotgit))
# Make sure no content was actually changed.
self.assertTrue(dotgit.is_dir())
for name in self._FILES:
self.assertTrue((dotgit / name).is_file())
for name in self._CLEAN_FILES:
self.assertTrue((dotgit / name).is_file())
for name in self._SYMLINKS:
self.assertTrue((dotgit / name).is_symlink())