Updates the documentation for PROVIDES so that it recommends "+="
instead of "=".
(From yocto-docs rev: 39b2ca1e27592488d396d5f0d76965f0006515a1)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[ Chris Lamb ]
* Truncate jsondiff differences at 512 bytes lest they consume the
entire page.
* Wrap our external call to cmp(1) with a profile (to match the internal
profiling).
* Add a note regarding the specific ordering of the new
all_tools_are_listed test.
[ Dimitrios Apostolou ]
* Performance improvements:
- Improve speed of has_same_content by spawning cmp(1) less
frequently.
- Log whenever the external cmp(1) command is spawn.ed
- Avoid invoking external diff for identical, short outputs.
* Rework handling of temporary files:
- Clean up temporary directories as we go along, instead of at the
end.
- Delete FIFO files when the FIFO feeder's context manager exits.
[ Mattia Rizzolo ]
* Fix a number of potential crashes in --list-debian-substvars,
including explicitly listing lipo and otool as external tools.
- Remove redundant code and let object destructors clean up after
themselves.
[ Conrad Ratschan ]
* Add a comparator for Flattened Image Trees (FIT) files, a boot image
format used by U-Boot.
(From OE-Core rev: 65feaffe9857029fa230935cb0c3e8e6f2815776)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes the bug were long paths would break Unix domain socket clients
(for real this time; the previous attempt was missing os.path.basename).
Adds some tests to prevent regressions
(Bitbake rev: 77790e3656048eff5cb1a086c727d86d32773b68)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Restores a fix for unix domain socket path length limits when using the
synchronous hash equivalence client that was accidentally removed when
the async client was added.
Unfortunately, it's much more difficult to fix the same problem when
using the async client directly due to the interaction of chdir() and
async code, but this will at least restore the old behavior in the
synchronous case.
(Bitbake rev: 53e85022a8b1c8f407c9418260c59beffb96f0f9)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removes the keys() function from the MulticonfigCache. This appears to
be a leftover from before the class inherited from collections.Mapping,
is now unnecessary, and was outright incorrect.
(Bitbake rev: 5f37b6d2829fcac1f16602d9697f8bfbcb65ff62)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for an upstream server to be specified. The upstream server
will be queried for equivalent hashes whenever a miss is found in the
local server. If the server returns a match, it is merged into the
local database. In order to keep the get stream queries as fast as
possible since they are the critical path when bitbake is preparing the
run queue, missing tasks provided by the server are not immediately
pulled from the upstream server, but instead are put into a queue to be
backfilled by a worker task later.
(Bitbake rev: e6d6c0b39393e9bdf378c1eba141f815e26b724b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for create a client that operates using Python asynchronous
I/O.
(Bitbake rev: cf9bc0310b0092bf52b61057405aeb51c86ba137)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Running the weston compositor as the root user is an insecure default
behavior for OE-core. We can do much better, at least when using
systemd. Change the recipe to create a dedicated "weston" user and start
weston as this user. The systemd service and socket units are no longer
template units, as there were several inconsistencies in the templates.
Instead, there is now a global /run/wayland-0 socket that gets created,
and systemd will start weston on demand when a client connects to that
socket or when attempting to reach graphical.target, whichever comes
first. This also allows downstream users to easily change the behavior
so that weston *only* starts on demand by adding a drop file. Access to
the global socket is controlled by a "wayland" group; any user that is a
member of the group can use the socket to talk to the compositor. This
also satisfies another use case where another systemd service might
start a graphical application that needs to display with weston (e.g. a
single function device in kiosk mode). Finally, the udev rules for
starting weston with the existance of a DRM device have been removed.
Being WantedBy= a graphical target should eliminate the need for this
behavior, and having it present makes it difficult for downstream users
to start weston on demand (having to override the udev rules).
(From OE-Core rev: dd83fb40f76749c6689807afabc63b9d5c2a4065)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
systemd 245 introduced a regression in behavior where they removed
support for non-root users to chvt from a service file. This prevents
running compositors (e.g. weston) as any user other than root. The
intention is for polkit to be used to allow this (and in fact the
default polkit rules that ship with systemd allow this). However, polkit
is a huge dependency to bring in for an embedded system, and isn't
support by OE-core.
The patch has been proposed upstream to restore the previous behavior of
allowing a non-root user to chvt to unbreak the regression without
requiring polkit.
Upstream-Status: Submitted [https://github.com/systemd/systemd/pull/17494]
(From OE-Core rev: ef8cf581b680f8b8ee35d50c75891b4fd8b91f2f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a Pipfile so that Pipenv can be used to automatically install all
the dependencies required to build the documentation
(From yocto-docs rev: 31658ad30827d38dcd4ea83bcfb828441c7c3eca)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds documentation that describes how to use the --offset argument in a
kickstart file
(From yocto-docs rev: 0fbb2d71fe866b4ae7721f2f70d5b50dbc019030)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an example to SRC_URI that explains how to name sources in SRC_URI
and how to associate SRCREVs and checksums with the names
(From yocto-docs rev: 900af0addab7d6ea465922957f881a13012429ed)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds variables that can be used to allow a recipe to pass extra
arguments to `waf build` and `waf install`. In most cases, you want to
pass the same arguments to `build` and `install` (since install is a
superset of `build`), so by default setting EXTRA_OEWAF_BUILD also
affects `waf install`, but this can be overridded.
(From OE-Core rev: 493e17a2f5cbbbe3b1e435dadb281b007bca2cbf)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bumps the minimum python version to 3.5 to match bitbake and the test
matrix
(From OE-Core rev: df13c0f2348898023fb7ee1b229e9b5ccc893609)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Blocks SIGINT in the worker processes to prevent them from running the
parent process signal handler, which causes them to deadlock under
certain circumstances.
[YOCTO #14034]
(Bitbake rev: 9f4207f4b598f549cbd4159841c720276736f23b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a contrib/ script that repeatedly runs bitbake and interrupts
parsing to try and reproduce parsing deadlocks.
(Bitbake rev: 0457482e252f216618a6fccad0030fcd6c5a304f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffoscope changed the --exclude-directory-metadata option to require an
argument.
Add a test to validate that diffoscope is functioning as
expected to ensure that future upgrades do not unintentionally break
the reproducibility tests.
[YOCTO #14025]
(From OE-Core rev: ea8fbcb7978ce48d7a9a83143d09402329535f86)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Redirect stderr to stdout when running subcommands while doing the SDK
tests. The tests will show stdout when CalledProcessError is raised,
but any output to stderr was lost.
(From OE-Core rev: 7cb4e9ab8c1596281060e94a216966060103956e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch enables extra debugging for gpg to try and get more debugging
information when [YOCTO #14003] occurs
(From OE-Core rev: f24d7be1fb863006b5414fa8f9c9a37bb48d5a6f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allows the --offset argument to use the "s" or "S" suffix to specify
that it is reporting the number of 512 byte sectors.
This is required for some SoCs where the mask ROM looks for an item at a
sector that isn't aligned to a 1KB boundary.
(From OE-Core rev: 938595d1dc4abaf5f7f3a7900add3f0492b805d0)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under unknown conditions, Storable.pm will be placed in the base library
directory instead of under the arch specific one. This causes build
issues, see https://bugzilla.yoctoproject.org/show_bug.cgi?id=13946. To
try and catch this happening, add an explicit check for the bad file
existing and fail the build if it is found.
[YOCTO #13946]
(From OE-Core rev: 140ec7b71b71a23ebeae12c63ed1c1ac2c227097)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The check for a git HEAD still wasn't quite correct because it was using
the .git directory as the current working directory. Instead, it should
be passed as the --git-dir argument when running git. Running `git
rev-parse HEAD` in a .git directory with no HEAD reports 'HEAD' and
exits with success but then 'git log' will fail, which is not what we
want.
(From OE-Core rev: cdbd47dd7e1657b91b65a0940b7cbf119764240f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changes the findSigInfo command to accept a recipe specified with the
multiconfig prefix
(Bitbake rev: 379951b6417eacbafc92ac1413ae1358bafdddfb)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code was using TARGET_OS to try and detect what type of OS the
packages would be running on, but this is incorrect. TARGET_OS is the OS
for which the package will generate output (e.g. GCC output target or
GDB debugging target). The OS where the package will run is the HOST_OS.
Note that HOST_OS only differs from TARGET_OS when doing a
canadian cross compile, and even then in our case only when doing so for
a non-Linux host (e.g. MinGW).
Fix the code to use HOST_OS instead.
(From OE-Core rev: f284c5b2df220b520b025a59874e04ef4becd829)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes the class to use HOST_OS and HOST_ARCH to set the cmake SYSTEM
variables. The HOST variables should be used instead of
TARGET_OS/TARGET_ARCH because we want to control how cmake compiles for
where the package is going to run (the host), not what it will be
generating output for (the target). The distinction is only really
relevant when building canadian cross recipes.
Also fix up the host OS detection to account for MinGW by setting
CMAKE_SYSTEM_NAME to "Windows". This eliminates the need for meta-mingw
to patch this in all the cmake recipes it builds.
(From OE-Core rev: 01245db2893e39ffb5d4a00e4689f048d0698974)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes an error that occurs when attempting to get the timestamp of the
latest commit when there is no HEAD in the git repository. The easiest
way to trigger this condition is to use the 'subdir=' option when
specifying a 'git://' SRC_URI.
(From OE-Core rev: a64caca5b5dbe4a76acd0b5709b2c3e75b245863)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moves most of the python code used for dealing with the source date
epoch to library code.
(From OE-Core rev: a7ede90955bc0c8bec1cbb3cab498ef2583b2f4e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Improvements and bug fixes:
- Pass the absolute path when extracting members from SquashFS images as we
run the command with our working directory set to the temporary
directory. (Closes: #964365, reproducible-builds/diffoscope#189)
- Increase the minimum length of the output from strings(1) to 8 characters
to avoid unnecessary diff noise. (Re. reproducible-builds/diffoscope#148)
* Logging improvements:
- Fix the compare_files message when the file does not have a literal name.
- Reduce potential log noise by truncating the has_some_content messages.
* Codebase changes:
- Clarify use of a "null" diff in order to remember an exit code.
- Don't alias a variable when don't end up it; use "_" instead.
- Use a "NullChanges" file to represent missing data in the Debian package
comparator.
- Update some miscellaneous terms.
(From OE-Core rev: afbf68ddd20463ec7e584b481af6344482180ce0)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Providers of libgbm (including mesa) have move to the "virtual/libgbm"
name to provide this functionality. Update the documentation to prevent
confusion.
(From yocto-docs rev: afc290e85bc801549000a6f6d770e2975a313779)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The commit d4be264061 ("classes/archiver: run do_unpack_and_patch after
do_preconfigure") correctly moved do_unpack_and_patch to depend on
do_preconfigure, but left do_ar_patched as a dependency of do_configure
introduced from 8a7c779487 ("classes/archiver: Create patched archive
before configuring"). Having do_configure depend on do_ar_patched is
blatantly wrong. Firstly, doing so causes the taskhash of do_configure
to change which is undesirable. Secondly, the anon python that sets up
the tasks carefully skips GCC recipes that depend on gcc-source to
provide their source code since running do_unpack_and_patch in them
would delete the source code. Adding in the dependency effectively
bypasses this and makes these recipes delete the shared gcc-source
This fixes errors (for real this time) like:
cat: .../gcc/defaults.h: No such file or directory
when making certain configuration changes.
For posterity, you could easily reproduce gcc-source being erased by
running the commands (replacing aarch64 with your correct target arch):
bitbake -c cleanall gcc-source gcc-cross-aarch64
bitbake -c do_deploy_archives gcc-source
# The following would fail because
# gcc-cross-aarch64:do_unpack_and_patch erases shared source, so
# do_configure after that fails.
bitbake gcc-cross-aarch64
(From OE-Core rev: f02c1904330da47a25cf85f0077712f584f4ee5b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The commit 8a7c779487 ("classes/archiver: Create patched archive before
configuring") fixed a race condition when using the archiver, but
introduced a bug where the GCC source would occasionally be deleted due
to the archiving code running before do_preconfigure. Instead, make sure
the archiving code runs after do_preconfigure by making
do_unpack_and_patch depend on it. This makes more sense anyway since
do_preconfigure is effectively an extension of do_patch.
This fixes errors like:
cat: .../gcc/defaults.h: No such file or directory
when making certain configuration changes
(From OE-Core rev: d4be264061023ae0ea6e023d82a7b99605bfc22c)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code that handled exceptions from the hash equivalence client was
raising an exception itself because hashserv.client wasn't imported
(Bitbake rev: a76290dfc6f34ff9f6efdb13a6db74b6b4759daf)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The hash equivalence client and server can occasionally send messages
that are too large for the server to fit in the receive buffer (64 KB).
To prevent this, support is added to the protocol to "chunkify" the
stream and break it up into manageable pieces that the server can each
side can back together.
Ideally, this would be negotiated by the client and server, but it's
currently hard coded to 32 KB to prevent the round-trip delay.
(Bitbake rev: e27a28c1e40e886ee68ba4b99b537ffc9c3577d4)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The layout of the cache data hasn't changed, but the cache has now been
split into different files for multiconfig. If a user pulls in these
changes, it's possible that their base cache will still contain the
combined multiconfig cache entries, which are now unexpected and
generate errors like:
Unexpected multiconfig: foo
Bumping the version fixes this since the old cache data won't be
considered valid anymore.
(Bitbake rev: 1082188ce633ec6891c961d726f584b3f1259941)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_configure and do_preconfigure can modify source files, which causes
race conditions if these tasks run in parallel with do_ar_patched. Add
explicit task dependencies to ensure that do_ar_patched finishes before
these tasks start. Specifically, this fixes a race condition with
gcc-source where do_ar_patched races with do_preconfigure deleting
gcc/gengtype-lex.c
(From OE-Core rev: 8a7c7794870815030239e90b03e37ed302b7e885)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The error message was reporting the calculated offset instead of the
current offset, which made it confusing.
(From OE-Core rev: 2be775cfe1b49ce3889b5dc326e2b67a9667f18a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The virtualfn variable is not defined, the filename is what should be
shown instead.
(Bitbake rev: 1f9d2c21db3a1ad2ab13dfebd2f8e9a7c3682ee2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake now passes all the dataCaches to the taskhash API, so use this
to correctly filter mcdepends.
[YOCTO #13724]
(From OE-Core rev: 749731a420fb905b6af97ce2909f06b2bcd14fe2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a test to validate that mcdepends causes the dependent tasks to
build, and also that a change in the dependent task causes the dependee
task to re-execute.
(Bitbake rev: f2062c41693f9f684bdaf2df0a2a08b7f3871026)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Passing all the data caches to the task hashing functions allows them to
correctly account for mcdepends in task signatures. This allows tasks to
be correctly re-run when a mcdepends changes.
By default, the legacy behavior is maintained for derived signature
generators by passing a special proxy object that can either be used to
access all multiconfigs or the legacy behavior. If a derived signature
generator is updated, it can set the supports_multiconfig_datacaces
property to instruct bitbake it deals with multiconfigs properly.
[YOCTO #13724]
(Bitbake rev: 8ff9203de4fce9c104c2987d86980c9f34036b97)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
467f84e12b ("wic: Add --offset argument for partitions") broke the
--extra-space argument handling in wic. Fix the option and add a unit
test for the argument.
(From OE-Core rev: 87722a92c18f94917c8f70afc8cd0763462a5c25)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is expected that load_cachfile() returns an integer indicating how
many entries were loaded from the cache. In the event the cache needs to
be rebuilt, 0 must be returned to prevent python from attempting to add
an None and an integer together.
(Bitbake rev: 3459d98fbc280637ecb36961bda8436818ee51e5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All of the other multiconfig splitting functions are located in runqueue
so move the function to split a pn/fn there also so that its easier to
see them all together. Fixes a case where the findBestProvider() command
wasn't working for multiconfig because it was looking for a prefix of
"multiconfig:" instead of the newer "mc:"
(Bitbake rev: 325827af66434affc2da460cc8b9a5c460e38056)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a test to validate that multiconfigs can independently mask off
recipes by setting BBMASK. See the test description for further
information about how the test works.
(Bitbake rev: 513fc2dddf13d5e344162c26d89d2dde2fe85634)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that there is a cache object per multiconfig, it is not necessary
for each cache object to parse all other multiconfigs. Instead, each
cache now only parses the files for it's multiconfig.
(Bitbake rev: 3c5c7346adf4ca7ec761c08738f12401ba75b7c8)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improves the logging of Cache objects by prefixing the log messages with
the multiconfig name of the cache, so as to distinguish between multiple
instances of the class. Also adds a more log messages.
(Bitbake rev: 74fd10b33c66f4142d6eff6531200f7620a06ae0)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a helper logger adapter to add a prefix to all log messages. This
is useful to distinguish log messages between multiple instances of a
object.
(Bitbake rev: 5f363e4a9636b902229c257484ae0b479aedca65)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Creates a BBLoggingAdapter class that is monkey patched in place of the
logginer.LoggingAdapter. The new adapter is compatible with the BBLogger
class API, allowing adapters to be created for bitbake loggers. A new
BBLoggerMixin class is used to reduce code duplication between the BBLogger
and BBLoggerAdapter classes.
(Bitbake rev: 8f93d776fd6ce1a6d7094da9a9e00b5e9ee178f9)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Splits the parsing cache to maintain one cache per multiconfig instead
of one global cache. This is necessary now that the files and appends
can vary for each multiconfig. A bb.cache.MulticonfigCache
dictionary-like proxy object is created instead of a single
bb.cache.Cache object. This object will create and properly initialize
bb.cache.Cache object for each multiconfig, and each of these caches has
a dedicated cache file with a name based on the multiconfig.
(Bitbake rev: 5272f2489586479880ae8d046dfcdbe0963ee5bb)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Splits the cooker to track a collection per multiconfig instead of a
single collection for all multiconfigs. Practically speaking, this
allows each multiconfigs to each have different BBMASKs that apply to it
instead of each one using the mask specified in the base configuration.
(Bitbake rev: dd6d8eca2027f8d9be8a734a493227b440075e49)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for an --offset argument when defining a partition. Many
SoCs require that boot partitions be located at specific offsets. Prior
to this argument, most WKS files were using the --align attribute to
specify the location of these fixed partitions but this is not ideal
because in the event that the partition couldn't be placed in the
specified location, wic would move it to the next sector with that
alignment, often preventing the device from booting. Unlike the --align
argument, wic will fail if a partition cannot be placed at the exact
offset specified with --offset.
Changes in V2:
* Fixed a small typo that prevented test_fixed_size_error from passing
(From OE-Core rev: 467f84e12b96bc977d57575023517dd6f8ef7f29)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The commits 910ffaf5be ("pyelftools: Import from meta-python") and
a96f815c53 ("pycryptodome: Import from meta-python") moved recipes from
meta-python to oe-core. In order for this to be communicated with users,
bump the LAYERVERSION so that meta-python can key of it in its
LAYERDEPENDS.
(From OE-Core rev: 2d503b27e7c88cee9a37c79c4605c77b11f230b6)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As in other places in the file, skip layers that don't define a
collection when searching for a layer to resolve a dependency. Fixes
KeyError exceptions when attempting to access the layer collections
later
(From OE-Core rev: 26090a2861ebe21224aaf89d7be0c0a89ca58e48)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also splits apart the SRC_URI checksums to make automatic upgrades
easier
(From OE-Core rev: ae1f210546396b761ea86d9e32bf90c0867ff845)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Imports the pyelftools recipes from meta-python, as of 7c02c7d41
("gnome-themes-extra: correct the recipe name").
This recipe is commonly used by other layers, so moving it into
OE-core helps to cut down on layer dependencies.
(From OE-Core rev: 910ffaf5beed42936588c95b0c7c1b1ad67f99d3)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Imports the pycryptodome recipes from meta-python, as of 7c02c7d41
("gnome-themes-extra: correct the recipe name").
These recipes are commonly used by other layers, so moving them into
OE-core helps to cut down on layer dependencies.
(From OE-Core rev: a96f815c53364b119b5743b8b7100eb5588d5cf5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If reproducible binaries are requested, generate the kernel modules
tarball in a reproducible way. Namely, sort the tarball contents, clamp
the mtime to the source date epoch, compress the archive with a
consistent compression level (9), and don't include the name and
timestamp in the compressed file.
[YOCTO #12719]
(From OE-Core rev: 5c6a472c82a9b3c122fc54d3ee7171508c643a8b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevents git commands run in a recipe from moving up past ${WORKDIR}
when searching for a .git directory, and thus prevents them from
detecting the parent OE-core .git directory. Fixes several
reproducibility issues where recipes would use the OE-core version as
the recipe version due to git walking up the tree.
(From OE-Core rev: e496605ea197414b58025293e73dec099a8976c6)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Backports a patch from upstream to fix a reproducibility problem where
paths would be encoded in the binary.
Drops an obsolete patch that conflicted with the backport
(From OE-Core rev: 303bf23048cb326dd8a8079a7cf773c3fd8f21ff)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Spilts out the inc file for u-boot-tools so that BSPs can include it for
their forked version of the tools, just like u-boot.inc.
As an example, this recipe [1] builds mkimage u-boot as part of the
u-boot recipe itself. This is incorrect because u-boot is a target
recipe, which means that the uninative loader transformations do not get
applied when the recipe is restored from sstate, breaking the mkimage
command breaks. The correct solution is to have a u-boot-tools-imx
recipe that pulls in their custom u-boot source; this inc file makes
that much easier to maintain.
[1]: https://github.com/Freescale/meta-freescale/blob/aa27fc62095cfd6c147c8f6865296ddc29bf0fa6/recipes-bsp/u-boot/u-boot-imx_2019.04.bb
(From OE-Core rev: 2f3bd198d9b1e85f17f7a587b1690ab196ab0300)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The logging module provides a shutdown() function that does the same
thing in a much better way
(Bitbake rev: 970cd2fc4f0bbc93069dee5a15a608dd76081c67)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The BitBake.verbconsole needs to be treated like a console output logger
(meaning that the TerminalFilter attaches an InteractConsoleLogFilter to
it), even if it's not directly attached to the root 'BitBake' logger.
First, assign a special "is_console" property to the relevant handlers,
then look for the property in the handlers from the configuration object
return by bb.msg.setLoggingConfig(). Finally, pass the list of all
handlers to the TerminalFilter object; it doesn't care about the
difference between console and errconsole, so pass all the relevant
handlers as a list.
This fixes cases where the console output was corrupted when messages
were sent to the 'BitBake.verbconsole' handler.
(Bitbake rev: 2010be588c74a99256df7b565a309c84c2973546)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Returns the configuration object from setLoggingConfig(). This object
has a config dictionary that contains all of the created handlers,
filters and loggers, which makes it much easier to pull out items with
specific names.
(Bitbake rev: 93c98a85cb10d3795b3cebc9cd97214d0f4748e5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updates hash equivalence logging so that the interesting VERBOSE
messages are always logged to the consolelog file so that issues in
individual user builds can be diagnosed. The autobuilder logging config
then updates this so that they also are shown on stdout, since the
consolelog file is not capture there.
In order to facilitate this, 2 new logging handlers were added,
"BitBake.verbconsole" and "BitBake.verbconsolelog". Neither of these
handlers are attached to anything by default, but they will log any
messages that wouldn't otherwise be logged by the normal console or
consolelog handlers. Users can attach whatever loggers the desire to
this handler to get them to appear on the console or in the consolelog,
as demonstrated by the autobuilderlog.json file.
(Bitbake rev: 766587c0baaaeb5cda3e9454395edbb70e33f756)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a configuration file that the autobuilder can use to capture
interesting logging domains above the ones that show up for normal users
on stdout/stderr.
(Bitbake rev: 2259b5172b37442a4e0420a16a7bde9e21ffa086)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Lowers the level of the log message when setscene tasks have completed.
This message can occur multiple times when hash equivalence is enabled,
since the runqueue switches between executing setscene tasks and normal
tasks. Since this is primarily of use when debugging hash equivalence,
use the hash equivalence logger at VERBOSE level.
[YOCTO #13813]
(Bitbake rev: 7dd5b3900622008ff34ec70d71c6e994f460a46f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a function that can be used by UI front ends to merge logging
configuration fragments.
(Bitbake rev: e860d93b76c142fdae90226bc68e4fe6e7b6a9db)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds code to close all loggers when bitbake exits. This prevents
unclosed file ResourceWarnings. A form of this closing existed
previously, but was removed in the new logging code.
(Bitbake rev: b3f3779adf63c0d970462a558a6205da1d30c0ed)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds documentation describing how to use the BB_LOGCONFIG variable to
enable custom logging.
(Bitbake rev: e22565968828c86983162e67f52ebb106242ca76)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Lowers the level at which hash equivalence messages are logged so as to
not annoy the majority of users. The autobuilder can use a custom
logging configuration to log these to a file for debugging (see
contrib/autobuilderlog.json)
[YOCTO #13813]
(Bitbake rev: 2ddb649ea31afe052f08e3969e36abf6fb515bc2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switches the hash equivalence logging to use a different logger so that
it can be easily filtered out with python's structured logging.
(Bitbake rev: 20bb29ef973e9c5483eb50a74550ea207637367b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding the default log filter here is unnecessary because there are no
defined logging domains when it is called, which means it does no actual
filtering.
(Bitbake rev: dcdb8f2c14f09ce34d0a1facc33a441570912c05)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sets up logging in knotty to use python's structured logging config and
the bb.msg.setLoggingConfig() helper to setup logging. This allows the
user to specify additional logging mechanism in a config file with
BB_LOGCONFIG
(Bitbake rev: 646a68a49364b50a42168b4b16308f7217eec0dc)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Passing around the log formatter variable was unnecessary since the log
levels of interest can be accesses as class members of
bb.msg.BBLogFormatter. Switching to do this will make using the
structured python logging much easier, since it can be difficult to
extract out the formatter for a specific handler.
(Bitbake rev: c1c867df24b4ef204027d485acac7c75c63f2bc0)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a helper function to setup the structure logging information in
bitbake. The helper function takes a default configuration and an
optional path to a user config file that can be merged into the default.
(Bitbake rev: 14c98d36b74c1599b4649078170b6e72df79ba2a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that the filter are described using the python logging structure,
these classes are no longer needed.
(Bitbake rev: 2cb16e0c61609f3fb8a86530ddedf8ad0e69428e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds generic filter utilities that can be used to filter when the log
level is greater than or equal to a defined value, or below a defined
value, as well as a generic function to translate a string to a logging
level (or bitbake logging level)
(Bitbake rev: 8ebe8a7662200fad9b88be8b08376262a61c85c4)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a comment section that can be easily uncommented to enable dumping
the logging tree. This module is extremely useful for debugging issued
with logging configuration
(Bitbake rev: 30461310915f911b80f92e03df694af7c1eb1f46)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a __repr__ function for BBLogFormatter. This allows it to get a
human readable string when printed using the logging_tree module
(Bitbake rev: 5ff962dca71f4ef16a3aa11b41a23f2227fe8c21)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Passes around the actual logging level as the default log level variable
instead of the debug count. This makes it easier to deal with logging
levels since the conversion from debug count and verbose flag only has
to occur once when logging is initialized and after that actual log
levels can be used
(Bitbake rev: 41bd155faf7f65cb0727fcce972715769b26ca89)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Handles the log messages from the bitbake server with the specific
logger that the event originated from. This allows hierarchical logging
configurations to work as expected.
(Bitbake rev: 9624d42133e024fd044d0d089c7017ed53eed874)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Converts the default domain variable to a dictionary where the keys are
the logging domains and the values are the logging level (instead of the
debug count). This makes it easier to deal with the logging domains and
the awkward conversion from a list to a dictionary only needs to be done
once when logging is initialized. Finally, other code has been written
that already assumes this variable is a dictionary, see:
f04cd93109 ("bitbake: lib/bb: Optimise out debug messages from cooker")
(Bitbake rev: f32a8bc7ff7a0b0750b6934a96f5d48391b1383a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an option to the show-recipes subcommand that allows the user to
specify which multiconfig should be shown.
(Bitbake rev: 83256115c7b1fdf3fa5129cfba6b9e7cba2ae0da)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for the Tinfoil cache adaptor to be bound to a specific
multiconfig and invoke the appropriate commands for that multiconfig
instead of the default. The cooker adapter now creates a cache adapter
for each multiconfig specified in BBMULTICONFIG so that each multiconfig
is present.
(Bitbake rev: d515481681dca4a0bc733c6ad8a8498a82e3d9b9)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cooker had a multiconfig parameter for the findProviders() and
findBestProviders() API, but it was being ignored.
(Bitbake rev: ea0b68ac2b77676ed1c63f0ee1ae5d300f2b4696)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a multiconfig selection parameter to the findProviders command.
This allows a client to find the providers for a specific multiconfig
instead of the base configuration.
(Bitbake rev: 95bb446137734865f3fe04fe97681d90250deb11)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
legitimize_package_name wants the actual value of KERNEL_REVISION, so
use d.getVar() to fetch it as is done elsewhere in the file. Failing to
do so can result it weird errors at parsing time.
(From OE-Core rev: 9bff4d14693c1890fc181ec68c9f883dc4e4accf)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Applies a patch to fix strndup being detected as present incorrectly
when building for MinGW.
(From OE-Core rev: ffa91ff19f73c3ed4060a6ecd3b6da7f46732cb3)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Includes jquery in the output directory to make the diffoscope output
easier to navigate
(From OE-Core rev: ff7c386455848e497ef40f9fee2be87d5171e203)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a recipe that packages jQuery in a manner borrowed from debian. The
primary purpose is to make the diffoscope output from the autobuilder
easier to navigate.
(From OE-Core rev: 9fdfff92e5f6b56657802c06c698ef37a95fd449)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are differing packages and they are being saved for review,
automatically run diffoscope on them and include the output in the saved
output. The output is currently done in HTML format since these are
typically published on a webpage by the autobuilder.
(From OE-Core rev: 6e1d5e8b58f0940ba6dfd99536159dd974e6f24c)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Joshua Watt