By default, smart is invoked with --log-level=warning, which hides all
output from pre/post install scriptlets. That makes it hard to debug
scriptlet failure or why they get postponed to first-boot via
/etc/rpm-postinst.
The new ROOTFS_RPM_DEBUG variabled is expected to be set to an integer in
local.conf an incrementally adds more output:
0 = default, only warnings
1 = --log-level=info (includes information about executing scriptlets and their output)
2 = --log-level=debug
3 = --log-level=debug plus dumps of scriplet content and command invocation
The default behavior is not changed yet, but it seems that level 1 would
be a better default.
(From OE-Core rev: 5cb597a19dbfe825e5b26d828e18644c9ee58f86)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea behind the implementation of Yocto #8138 was that an
additional class can write additional files in the recipe directories,
for example by hooking into the functions of buildhistory.bbclass or
by implementing its own SSTATEPOSTINSTFUNCS function.
However, when these additional files get created before
buildhistory_emit_pkghistory(), they get removed again by that
function because it contains code which removes everything it does
not know about. The reason for that is that these unknown items
are probably obsolete.
This logic is the reason why the additional "kconfig" file from
buildhistory-extra.bbclass never showed up in the final build history.
To fix this, the hard-coded list of known files in
buildhistory_emit_pkghistory() must be turned into a variable which
derived classes can extend.
(From OE-Core rev: 97f77ea3313b0d79ae4a6090672a2a9344282262)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When image-vm.bbclass was introduced, it indirectly also introduced a
".hdddirect" image type based on boot-directdisk.bbclass. However, one
could only get that image when also enabling at least one of the
virtual machine images.
The .hdddirect images are useful by themselves. By registering
image-vm.bbclass as implementation of it, it becomes possible to
select them with:
IMAGE_FSTYPES = "hdddirect"
(From OE-Core rev: e3ff509091cbbfdef851f8a3c9e31c7b76d37e89)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes booting with initramfs and root=UUID on machines with IDE
disks, like "runqemu hdddirect", and kernels which still use the
deprecated CONFIG_IDE.
v2: Rebased against current master-next.
(From OE-Core rev: 3d27dfb7e78b8e17b76fcc1d8f8e2b29ca26b0df)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the "boot" parameter refers to a non-existent device, the only
visible output at normal log levels was a rather confusing:
ERROR: There's no '/dev' on rootfs.
That's because the actual error, not being able to find the root
device, was only a debug message, which gets ignored in the default
mode.
Promoting the "root '$bootparam_root' doesn't exist." message from
"debug" to "msg" gives sufficient context to understand the error. A
more intrusive change would be to change also the control flow.
(From OE-Core rev: 71d7803e5b13e26fd8001e87cfbac68114ddaa30)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Due to a missing $ before the variable name, all fatal errors ended up
invoking a shell, instead of only doing that when init_fatal_sh is set
as boot parameter.
(From OE-Core rev: 2a4335eae4b31ea9451a665dad2ba33ae4967670)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Giving anything with -image in it as bootparams or in qemuparams (for
example, an additional -drive parameter with an image file or an
"-initrd .../core-image-minimal-initramfs-qemux86.cpio.gz") caused
runqemu to treat these parameters as names of the rootfs image file.
Matching *-image) after checking the current argument for more
specific cases like bootparams and qemuparams avoids this
misinterpretation of the command line parameters.
(From OE-Core rev: 29e0aaa7345ca823bb4af2d4a870e98ac75e04e7)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the init_fatal_sh boot parameter is present (i.e. used without
value) and a fatal problem occurs inside the initramfs-module, a shell
will be started instead of looping forever.
Useful for debugging.
Interestingly enough, the code was already indented to support such an
if check...
(From OE-Core rev: 1d3dc681e809846dba7cae4f81566045a3f8c205)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It can be useful for debugging to override the default /sbin/init.
This is something typically done via the init boot parameter which
then gets interpreted by the kernel. But when using an initramfs, it
is the initramfs which must react to the option.
(From OE-Core rev: dfd6d4c765924f472ac2df724342547b5c15249a)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Default is to mount the rootfs read/write. "ro" can be used to turn
that into read-only, which is useful on systems where userspace does
an fsck before remounting read-write.
Giving both "ro" and "rw" will still mount read-only regardless of the
order, because the ordering information is not preserved by the
initramfs-framework's boot param support.
(From OE-Core rev: a09f10f9360862c16fb68972ac041d474d6e3a64)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These two parameters are supported by the kernel
(https://www.kernel.org/doc/Documentation/kernel-parameters.txt). When
an initramfs is used, the kernel does not mount the rootfs and the
initramfs needs to react to them.
The boot parameters can be set both by the image creator and
by users.
Supporting these two parameters is useful:
- rootflags is needed to ensure that the rootfs is already mounted as
intended in the time between starting init and init remounting
it (as systemd does); this is critical for IMA where iversion must be
active already when system starts writing files.
- setting it correctly up-front avoids messages from the kernel ("cannot
mount ... as ext2 because ...") when trying to guess the desired type.
For example, assuming that only one of ext4/ext3/ext2 is set,
rootfstype could be set in an image recipe with:
APPEND_append = "${@''.join([' rootfstype=' + i for i in ['ext4', 'ext3', 'ext2'] if i in d.getVar('IMAGE_FSTYPES', True).split()])}"
(From OE-Core rev: b8ea1c61b4b8071edf70f5d42119c54ea84de330)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 8ca8e2e5bf4a9f01dc48300149a8e1d71d715400.
Further testing showed the kernel does not support root=UUID; it is
something typically handled by the initramfs.
Because boot-directdisk.bbclass cannot know whether it is using a
suitable initramfs, root=UUID cannot be the default. Instead, it will
have to be set in image recipes on a case-by-case basis.
(From OE-Core rev: 8478024bd25651aa866d4582dcc193d51553554a)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This changes the default SYSLINUX_ROOTFS such that the rootfs is no
longer expected under a fixed device path. Instead, the UUID is used
to find it. This makes the resulting .hdddirect (and thus also the
vdi/vdmk/qcow2 images derived from that) more flexible.
(From OE-Core rev: 8ca8e2e5bf4a9f01dc48300149a8e1d71d715400)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As mentioned when introducing the VM images
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=7374), the
resulting images only work when the image is mounted as a disk that
results in the hard-coded path (/dev/sda in the current
default). Using the file system UUID to find the rootfs is more
flexible.
To enable this for boot-direct.bbclass and thus image-vm.bbclass (aka
FSTYPEs vdi/vmdk/qcow2), set SYSLINUX_ROOT =
"root=UUID=<<uuid-of-rootfs>>". The rootfs image must use an ext file
system.
The special string will get replaced in the APPEND line with the
actual UUID when the boot loader (grub-efi, syslinux or gummiboot)
writes the boot loader configuration files. At that time, the rootfs
image has already been created and its UUID can be extracted using
"tune2fs -l", which also should be available because the
e2fsprogs-native tools were needed to create the image in the first
place.
(From OE-Core rev: 1e29d77d0d33ee216b43022439876863f0db39bb)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set INITRD_IMAGE to automatically add a custom initramfs to the boot
partition.
(From OE-Core rev: 7fa76bd923fd643cf0984077321d6064d8ec3a2b)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some of the SYSLINUX defaults are the same as in the underlying
syslinux.bbclass. Let's not duplicate them, because that makes
changing the default harder.
(From OE-Core rev: 7418709450e428010545dc9943802dd64c9582e1)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Kernel parameters like "uvesafb.mode_option=640x480-32" were turned
into shell variables named "bootparam_uvesafb.mode_option", which
triggered errors from the shell because the name is not valid. Now
points get replaced with underscores, leading to
bootparam_uvesafb_mode_option in this example.
(From OE-Core rev: de81e804f0654092d20ccb8e6e40f4ff614c4b09)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some kernels, for example linux-yocto 3.19 for qemux86, fail to
execute /init in an initramfs unless there is already a /dev/console
char device in the initramfs. Booting then fails with:
Kernel panic - not syncing: /dev/console is missing or not a character device!
Please ensure your rootfs is properly configured
The panic itself comes from a linux-yocto specific patch to
kernel_init_freeable in init/main.c, but even without it, that
function will print an error when /dev/console is missing. The
kernel's Documentation/initrd.txt also mentions creating that device.
It remained unclear why this is not a problem on other machines. On
intel-corei7-64 from meta-intel, something (the kernel?) creates
/dev/console and /dev/[012] before transfering control to the init
script. In that case, creating /dev/console in advance is not
necessary, but does not cause any problem either.
(From OE-Core rev: 1b64664f0c388f41084f5db6e46e3e68c53fb6d9)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some kernels, for example linux-yocto 3.19 for qemux86, fail to
execute /init in an initramfs unless there is already a /dev/console
char device in the initramfs. Booting then fails with:
Kernel panic - not syncing: /dev/console is missing or not a character device!
Please ensure your rootfs is properly configured
The panic itself comes from a linux-yocto specific patch to
kernel_init_freeable in init/main.c, but even without it, that
function will print an error when /dev/console is missing. The
kernel's Documentation/initrd.txt also mentions creating that device.
It remained unclear why this is not a problem on other machines. On
intel-corei7-64 from meta-intel, something (the kernel?) creates
/dev/console and /dev/[012] before transfering control to the init
script. In that case, creating /dev/console in advance is not
necessary, but does not cause any problem either.
(From OE-Core rev: 0352841cd92f6316bcac092e2fff9d28c352b36b)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes it possible to boot images with multiple partitions (the
ones ending in .hddimg or .hdddirect) in several ways:
runqemu qemux86 core-image-minimal hddimg
runqemu tmp/deploy/images/qemux86/core-image-minimal-qemux86.hddimg
VM=tmp-glibc/deploy/images/qemux86/iot-os-image-qemux86.hddimg FSTYPE=hddimg runqemu
Same for hdddirect.
This is useful for testing initramfs scripts, secure boot (when
switching to UEFI), or boot loaders like syslinux. For testing the
content of the rootfs, the ext4 image is better because that approach
is faster (no need to create another large image during build, rootfs
can be read directly instead of reading boot.img through loop device).
When booting a live image, the kernel, initramfs (if any) and kernel
parameters are taken from the image by the virtual machine's BIOS, so any
additional kernel parameters given to runqemu are ignored. This can be
avoided (already without this change) in a slightly hacky runqemu setup:
ROOTFS=tmp/deploy/images/qemux86/core-image-minimal-qemux86.hddimg \
FSTYPE=ext4 \
KERNEL=tmp/deploy/images/qemux86/bzImage-initramfs-qemux86.bin \
MACHINE=qemux86 \
runqemu serial kvm nographic 'bootparams=root=/dev/ram0'
The additional bzImage-initramfs-qemux86.bin kernel here was created
by adding this to local.conf:
INITRAMFS_IMAGE = "core-image-minimal-initramfs"
INITRAMFS_IMAGE_BUNDLE = "1"
In the code, the new FSTYPE=hddimg resp. hdddirect behaves almost
exactly like the older vmdk FSTYPE. New types were chosen because it
seemed cleaner than using FSTYPE=vmdk when the actual image pointed to
by VM is not in that format. The downside is that several checks for
FSTYPE=vmdk had to be duplicated for FSTYPE=hddimg.
The VM variable now gets interpreted as "virtual machine disk image"
instead of "vmdk image".
(From OE-Core rev: 37741c539f5d3021e59828b49e968cd42b89a368)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The unconditional removal of -DWITHOUT_XATTR accidentally introduced a
compile-time dependency on "acl", because "sys/acl.h" gets
included. This caused random compile failures.
To fix this, we introduce a proper PACKAGECONFIG for the "xattr"
support, with the distro's "xattr" feature determining the
default. Setting the define must be done with custom code because
PACKAGECONFIG can only influence configure parameters.
The "acl" distro feature is not checked because although enabling
"xattr" support now triggers a build of acl, nothing from it will not
get included in the resulting target image. This seems more suitable
than not enabling xattr support in mtd-utils when acl support in the
distro is disabled.
(From OE-Core rev: 0c4498878df352b9dbaa118c3cc201454ac22154)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Usually, the host's tar command is sufficient. However, special cases
like archiving xattrs depend on a modern GNU tar version. The new
IMAGE_CMD_TAR makes that possible, with xattrs given as example.
(From OE-Core rev: d8db122114f4d0600b662f79b3673108b4b6c0b7)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Building tar-replacement-native as replacement of the host's tar in
the standard path was meant to be done manually by a user in
preparation for the regular bitbake run. Such a usage has been
superseeded by installing the pre-compiled buildutils and might have
been broken on hosts which need it by the sanity check for tar >=
1.26.
Therefore tar-replacement-native_1.28.bb can be removed in favor of
adapting the normal tar recipe such that it installs an opt-in binary
under a different path.
The special do_install logic is explicitly limited to class-target,
instead of making it the default and disabling it (which would be the
case for class-native and class-nativesdk).
(From OE-Core rev: e6fee3ddb5600fc564243a96d6232b4ae097df32)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
xattrs may be needed by some distros. Support that by compiling in the
necessary code, even if it is not used by default. Then .jffs2 images
including xattrs can be created with:
EXTRA_IMAGECMD_jffs2_append = " --with-xattr"
(From OE-Core rev: 24fde4d983cc8f056177de6c1ad308369f6279ad)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When reconfiguring the branch to something not already fetched,
action_pull fails with
error: pathspec '<new branch name>' did not match any file(s) known to git.
It is the "git checkout" which fails like that. To solve this,
try the faster "git checkout + git pull" first and only if that fails,
fall back to the slow "git fetch + git checkout".
In the conf.hard_reset case, do the checkout always after the git fetch.
(From OE-Core rev: ad4d3b1da190cf08c6ac5f9a94a2a1c4980a184d)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The git operations can fail, for example when the branch is unknown
or misconfigured.
Better move the info message and extend it such that it is printed
first and provides the necessary context, because otherwise the
CalledProcessError exception gets dumped without mentioning for which
component it occurred.
(From OE-Core rev: 00d01468a692faf4272894dd328e8c532bcf8b49)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When working specifically on source archiving, it is useful
to trigger that for all recipes required by something like an
image recipe, without actually having to build that.
"bitbake -c deploy_all_archives <target>" does that now.
(From OE-Core rev: 30b109729a81097cc6cfa65148e0e9ae1a564d2c)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a recipe called 'foobar-test' links against 'foobar' without
listing that in DEPENDS, the build-deps check misses that error
because it looks for 'foobar' in a package string containing (among
others) the 'foobar-test' name, leading to the incorrect conclusion
that the package is listed as dependency.
The 'packages' string needs to be split into individual package names
before the check. Doing that once directly after reading the value is
more efficient than splitting inside package_qa_check_rdepends() because
the caller also needs the individual components.
Also use a set to speed up the 'package in packages' check.
(From OE-Core rev: 9f5792088315ab42f77fe1a1af7d2225e7ad5418)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Checking the bbdebug() debug level parameter with a regular
test expression only works in bash. Using tr to filter out
digits and then checking whether anything is left achieves
the same result and is more portable.
(From OE-Core rev: 25cb71799e72d4e0c4fe39653d8b84280d087372)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous "clean up dest_dir checking" patch (f8cdbe7497) improved
handling of empty dest_dir but made handling of unset dest_dir worse:
instead showing the "Option dest_dir is not defined for component ..."
error, it fails with a Python exception.
Avoid that by providing a sane fallback for the unset case. With that
change, dest_dir is no longer strictly required, but the check for it
is kept to ensure that a combo-layer.conf also works with older
combo-layer versions.
[Yocto #7773]
(From OE-Core rev: d4bf858b2c15bef128fd6d606b08203a318e2d4c)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the head of a branch is a merge commit, combo-layer did not
record that commit as last_revision because it only considers applied
patches, and the merge commit never gets applied.
This causes problems when the merge commit leads to multiple patches
and the commit id that gets recorded only reaches some of these
patches. The next run then will try to re-apply the other patches.
This special case is now detected and dealt with by bumping
last_revision to the branch commit. The behavior where the head is a
normal commit is intentionally not changed, because some users might
prefer the traditional behavior.
(From OE-Core rev: 701bcd52c208f22a9a6c48a11a35bcf1c4e413df)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The main motivation for this class was the observation that
a) a core-image can hang under qemu when the kernel does not
have enough entropy to generate the ssh host key
b) ssh complains about changing ssh host key files when
rebooting the same machine with different images
For debugging it is okay to reuse an ssh host key generated on the device
before. There may be also similar use cases, so the class is generic enough to
also copy more than one file or directory, with dropbear_rsa_host_key given as
example.
The documentation and naming of the class makes it clear that it
should not be used for production images.
(From OE-Core rev: ae4d605d802edb4c8bdb2dbee1b0c36d7730c07f)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TEST_SUITES="auto" is useful to run all suitable tests without
having to hard-code the list. However, it did not take test
dependencies into account, which can be an issue for tests
which really depend on some other test to run first.
To fix this, modules get loaded in the order determined by
TESTS_SUITES, but then get re-ordered based on dependencies
derived from @skipUnlessPassed before running them. The original
order is used to break ties when there are no dependencies, so
reordering only occurs when really necessary.
@skipUnlessPassed gets extended such that it makes the test name
a method depends on available for inspection by the test loader
in oetest.py.
Unfortunately Python's unittest offers no API to inspect tests
in a TestSuite, so the code has to rely on implementation details
to find all tests. The worst that can happen when the implementation
changes is that tests are not found and reordering does not happen.
(From OE-Core rev: 6e4543a35836c572b23b9f8162b19d1e038d3ed2)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The SystemdServiceTests assume that avahi-daemon is installed,
which is not necessarily the case depending on the image being
tested.
Better check this dependency before starting the tests and skip
them if the service is not installed. This has to be done for
each test instead of for the entire module, because other
tests in the module can run without avavi.
(From OE-Core rev: ca23337d517cfdb7119e5fd8bd9a9a663ae135de)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Plain print has two drawbacks:
- the output is handled by different processes, causing the message
about found tests to appear randomly after the result of the initial
tests
- the output is not part of the bitbake console output
Affects image testing with testimage.bbclass.
(From OE-Core rev: 4b6f87a82ece230b7296f1577d4092ef559af1d4)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code detecting empty patches after removing files with
file_exclude failed for commits which were already empty before (like
the initial commit in some repos): such patches are completely empty
files, without a From line.
Detect that case and just let the normal empty patch detection deal
with it.
(From OE-Core rev: 3e310b8cd603539a2de115b2b73e1db35403b426)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"filterdiff -x ./some/file" does not remove changes for some/file.
We must be more careful about constructing the path name and
only add the prefix when it really means a directory.
While at it, also better normalize the path in copy_selected_files()
early on, to handle double slashes. Useful should the function ever
gets used for something other that dest_dir (which gets normalized in
sanity_check()).
(From OE-Core rev: 8ea63c6c920c39e5a4ba176223fe472f92e2632a)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Empty dest_dir is basically undocumented behavior. The sample conf
only mentions using just a dot for the current directory. In practice,
the empty string does not work because of code like this:
def action_splitpatch(conf, args):
...
if dest_dir != ".":
filerange_root = '%s -x "%s/*"' % (filerange_root, dest_dir)
However, the empty string was not explicitly checked for, leading to
strange errors when trying to apply patches:
[12:50:23] Applying: foobar: xyz
fatal: unable to stat newly created file '/foobar': No such file or directory
This patch turns the empty string into an alias for the dot. This seems
more user-friendly than throwing an error. This alias is intentionally
not document in the sample conf, because the dot is clearer and works also
with older copies of combo-layer.
Instead of checking for both all the time and normalizing the path when
needed (as done in some places), rewrite the value in sanity_check()
and then only check for '.'.
(From OE-Core rev: f8cdbe749755dc769150d3a6c2c54318c80e1562)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Inserting the "From rev" comment depended on having at least one
Signed-off-by line in the patch header. Some old repository commits in
openembedded-core and bitbake do not have those.
When inporting those, just insert at the end of the patch
header. While doing so, ensure that there's exactly one blank line
since the last non-blank line.
(From OE-Core rev: 58548081b15f37452b71c135d5aa8cbbda5b1b61)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Patching the subject line must be limited to the subject of
the main patch itself. In particular, git formatted patches embedded in the patch must not be changed.
Achieved by limiting the replacement to the lines until the first
subject in the patch, just as it is done for modifying the first
Signed-off-by.
(From OE-Core rev: 2737bd1c3f7c51dd6509a0f91337f695f9c69165)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing patch might already have the desired prefix, perhaps
even multiple times (due to some previous import error). Ensure
that after the replace, the prefix is present exactly once.
(From OE-Core rev: 03be3d350425783a5938a7e4c0b858cda634b71d)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new "since_revision" property can be used to cut off the imported
history at some point. This is useful to keep the resulting repository
smaller while still preserving enough history that "git annotate"
reports the right author and commit for most lines.
The initial, squashed import commit shows up with "unknown" as author
in the "git annotate" output. It has the repository name as prefix
in the subject line; importing that commit works best with a
layer hook which does not add the repository name again when
it is already present. Adding it here is useful for hooks
which do not extend the subject line.
(From OE-Core rev: 74f4c9e3bcdb3c4ca919623086e92a9379bd81ff)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of local graft entries rooting the imported branches in the
shared root commit, use replacement objects.
The advantage is that they get moved around by "git push" and "git
fetch", so everyone has the same, nicer view with everything starting
at the beginning of the combined repository.
If undesired, these objects can be removed with "git replace".
(From OE-Core rev: 220d816cda3c9b8d888288cc03eb74be5e71cc59)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new --history parameter enables a new mode in "combo-layer init"
where it copies the entire history of the components into the new
combined repository. This also imports merge commits.
Moving into a destination directory and applying commit hooks
is done via "git filter-branch" of the upstream branch. File
filtering uses the same code as before and just applies it
to that filtered branch to create the final commit which
then gets merged into the master branch of the new repository.
When multiple components are involved, they all get merged
into a single commit with an octopus merge. This depends
on a common ancestor, which is grafted onto the filtered
branches via .git/info/grafts.
These grafts are currently left in place. However, they do not get
pushed, so the local view on the entire history (all branches rooted
in the initial, empty commit, temporarily diverging and then
converging) is not the same as what others will see (branches starting
independently and converging). Perhaps "git replace" should be used
instead.
The final commit needs to be done manually, as before. A commit
message with some tracking information is ready for use as-is. This
information should be sufficient to implement also "combo-layer
update" using this approach, if desired. The advantage would be that
merge commits with conflict resolution would not longer break
the update.
(From OE-Core rev: 9e40cb1ab77029df7f2cf1e548a645ff6a62c919)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The Configuration class mirrors all properties in local hashes.
When updating the configuration, also update these properties
to remain consistent.
(From OE-Core rev: c575ff183ae5c35e0f7e8d90c222994574ce73c9)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some combined repos intentionally do not include certain files.
For example, Poky does not include bitbake's setup files and
OE-core's sample files under meta/conf.
When these files get modified in the upstream repository, applying the
patches fails and requires manual intervention. That is merely a
nuisance for someone familiar with the problem, but a real show
stopper when having the import run automatically or by someone less
experienced.
Therefore this change introduces "file_exclude", a new per-repo list
of file patterns which removes all matching files when initializing or
updating a combined repository. Because fnmatch is used under the hood
to match full path strings, removing entire directories must be done
with a pattern ending in a '/*' (in contrast to file_filter).
For Poky, the additional configuration looks like this:
[bitbake]
...
file_exclude = classes/base.bbclass
conf/bitbake.conf
.gitignore
MANIFEST.in
setup.py
TODO
[openembedded-core]
...
file_exclude = meta/conf/bblayers.conf.sample
meta/conf/local.conf.sample
meta/conf/local.conf.sample.extended
meta/conf/site.conf.sample
(From OE-Core rev: a51b37f4db6b144386d1bd5789ec91acc78a0bd8)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow the caller to specify a separate output stream. stderr is always
a temporary file opened by runcmd(), so read from that to capture
output for error reporting *and* the return value.
The reasoning for the latter is a) that this preserves the traditional
behavior when out=None and b) if the caller wants the content of
stdout, it can read from the stream itself, which is not possible for
the temporary stderr.
(From OE-Core rev: a084162a9dc4718ab453723f1f28aefc55100e2e)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It depends on the diligence of the person running the combo-layer tool
whether the Signed-off-by line added to each commit actually indicates
that the person was involved in validating the change.
When the import is purely automatic, it is better to not add the line,
because the history is more useful without it (searching for the person
really only lists changes he or she was involved with) and it would
be a false statement.
The 'signoff' property can be set per repository, like every
other property. But setting it in the special [DEFAULT] section
is more useful, so that is what the example shows.
(From OE-Core rev: dba34e5a6f0a4c726266654dc5f26e0f7f13db0f)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When updating a property (at the moment, only 'last_revision'),
first check whether the user already added it to the main config.
If so, update there even if there is a local config.
This way, 'last_revision' can be shared between developers
as part of the repository while still configuring per-developer
repo paths outside of the git repository in a local config.
An example of a repository which is set up like that is tizen-distro,
with instructions for such a collaborative maintenance found here:
https://review.tizen.org/git/?p=scm/bb/tizen-distro.git;a=blob;f=README;hb=refs/heads/tizen-ivi
(From OE-Core rev: a3b246ee928d2ec59c8b6a775d9309f5be33ecc7)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
os.walk() returns symlinks to directories in the "dirs" lists, but then never
enters them by default. As a result, the old code applied neither the
directory handling (because that is active once a directory gets entered) nor
the file handling, and thus never packaged such symlinks.
The fix is simple: find such special directory entries and move them to the
"files" list. However, one has to be careful about the undefined behavior of
modifying a list while iterating over it.
This fix was required for packaging a modified base-files that created
symlinks into /usr for /sbin /lib and /sbin.
(From OE-Core rev: 9c4ab427b6acbc3f708860adace65393562227a7)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The purpose of binconfig-disabled is to manipulate config scripts such that
using them causes errors. But that only works when the modified config script
really gets installed in the sysroot. That is not the case with the staging
code in binconfig.bbclass.
Only patched config files get staged. For that reason it seemed more
appropriate to change binconfig-disabled instead of binconfig.
The reason for the change was the observation that the swig recipe needs
pcre-config installed on the host system. Staging pcre-config removes that
host dependency. swig did not actually end up *using* the pcre-config from the
host, because later during do_compile the patched configure.ac is used to
re-generate configure.
(From OE-Core rev: 822df6d23c9c24e131c38fda9f0012c47ad7af46)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Returning a non-zero exit code is not enough to cause errors when configure
scripts call the patched config scripts: for example, swig's configure script
uses PCRE_LIBS=`$PCRE_CONFIG --libs` and does not abort on errors. Using empty
output may then succeed, for example when the required library is available
indirectly.
Returning some nonsense command line arguments covers such cases, because
using them will definitely lead to errors during compilation. The faked
arguments were chosen such that these errors can be linked back to the root
cause.
(From OE-Core rev: d1ff61a16a3fcfdd7cf882bc89fb1d164cb1603a)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Patrick Ohly