mirror of
https://git.yoctoproject.org/poky
synced 2026-05-31 00:39:46 +00:00
overview-manual, ref-manual: Moved "Shared State Cache" to overview manual
Fixes [YOCTO #12370] The section on shared state cache needed to be in the overview manual and not in the ref-manual. I moved it. Some links were affected, which I fixed. (From yocto-docs rev: 1c4e5207bdde19d4b48ef42b1de81390d8a02d64) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Scott Rifenbark
committed by
Richard Purdie
Richard Purdie
parent
937b66e9d1
commit
00f87f8416
@@ -7346,7 +7346,7 @@ Some notes from Cal:
|
|||||||
|
|
||||||
<para>
|
<para>
|
||||||
Because the OpenEmbedded build system uses
|
Because the OpenEmbedded build system uses
|
||||||
"<ulink url='&YOCTO_DOCS_REF_URL;#checksums'>signatures</ulink>",
|
"<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#overview-checksums'>signatures</ulink>",
|
||||||
which are unique to a given build, the build system
|
which are unique to a given build, the build system
|
||||||
knows when to rebuild packages.
|
knows when to rebuild packages.
|
||||||
All the inputs into a given task are represented by a
|
All the inputs into a given task are represented by a
|
||||||
@@ -7450,8 +7450,8 @@ Some notes from Cal:
|
|||||||
|
|
||||||
<para>
|
<para>
|
||||||
For more information on shared state, see the
|
For more information on shared state, see the
|
||||||
"<ulink url='&YOCTO_DOCS_REF_URL;#shared-state-cache'>Shared State Cache</ulink>"
|
"<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>Shared State Cache</ulink>"
|
||||||
section in the Yocto Project Reference Manual.
|
section in the Yocto Project Overview Manual.
|
||||||
</para>
|
</para>
|
||||||
</note>
|
</note>
|
||||||
</section>
|
</section>
|
||||||
|
|||||||
@@ -471,6 +471,641 @@
|
|||||||
</note>
|
</note>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<section id="shared-state-cache">
|
||||||
|
<title>Shared State Cache</title>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
By design, the OpenEmbedded build system builds everything from
|
||||||
|
scratch unless BitBake can determine that parts do not need to be
|
||||||
|
rebuilt.
|
||||||
|
Fundamentally, building from scratch is attractive as it means all
|
||||||
|
parts are built fresh and there is no possibility of stale data
|
||||||
|
causing problems.
|
||||||
|
When developers hit problems, they typically default back to
|
||||||
|
building from scratch so they know the state of things from the
|
||||||
|
start.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
Building an image from scratch is both an advantage and a
|
||||||
|
disadvantage to the process.
|
||||||
|
As mentioned in the previous paragraph, building from scratch
|
||||||
|
ensures that everything is current and starts from a known state.
|
||||||
|
However, building from scratch also takes much longer as it
|
||||||
|
generally means rebuilding things that do not necessarily need
|
||||||
|
to be rebuilt.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The Yocto Project implements shared state code that supports
|
||||||
|
incremental builds.
|
||||||
|
The implementation of the shared state code answers the following
|
||||||
|
questions that were fundamental roadblocks within the OpenEmbedded
|
||||||
|
incremental build support system:
|
||||||
|
<itemizedlist>
|
||||||
|
<listitem><para>
|
||||||
|
What pieces of the system have changed and what pieces have
|
||||||
|
not changed?
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
How are changed pieces of software removed and replaced?
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
How are pre-built components that do not need to be rebuilt
|
||||||
|
from scratch used when they are available?
|
||||||
|
</para></listitem>
|
||||||
|
</itemizedlist>
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
For the first question, the build system detects changes in the
|
||||||
|
"inputs" to a given task by creating a checksum (or signature) of
|
||||||
|
the task's inputs.
|
||||||
|
If the checksum changes, the system assumes the inputs have changed
|
||||||
|
and the task needs to be rerun.
|
||||||
|
For the second question, the shared state (sstate) code tracks
|
||||||
|
which tasks add which output to the build process.
|
||||||
|
This means the output from a given task can be removed, upgraded
|
||||||
|
or otherwise manipulated.
|
||||||
|
The third question is partly addressed by the solution for the
|
||||||
|
second question assuming the build system can fetch the sstate
|
||||||
|
objects from remote locations and install them if they are deemed
|
||||||
|
to be valid.
|
||||||
|
<note>
|
||||||
|
The OpenEmbedded build system does not maintain
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
|
||||||
|
information as part of the shared state packages.
|
||||||
|
Consequently, considerations exist that affect maintaining
|
||||||
|
shared state feeds.
|
||||||
|
For information on how the OpenEmbedded build system
|
||||||
|
works with packages and can track incrementing
|
||||||
|
<filename>PR</filename> information, see the
|
||||||
|
"<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
|
||||||
|
section in the Yocto Project Development Tasks Manual.
|
||||||
|
</note>
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The rest of this section goes into detail about the overall
|
||||||
|
incremental build architecture, the checksums (signatures), shared
|
||||||
|
state, and some tips and tricks.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<section id='overall-architecture'>
|
||||||
|
<title>Overall Architecture</title>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
When determining what parts of the system need to be built,
|
||||||
|
BitBake works on a per-task basis rather than a per-recipe
|
||||||
|
basis.
|
||||||
|
You might wonder why using a per-task basis is preferred over
|
||||||
|
a per-recipe basis.
|
||||||
|
To help explain, consider having the IPK packaging backend
|
||||||
|
enabled and then switching to DEB.
|
||||||
|
In this case, the
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
|
||||||
|
and
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
|
||||||
|
task outputs are still valid.
|
||||||
|
However, with a per-recipe approach, the build would not
|
||||||
|
include the <filename>.deb</filename> files.
|
||||||
|
Consequently, you would have to invalidate the whole build and
|
||||||
|
rerun it.
|
||||||
|
Rerunning everything is not the best solution.
|
||||||
|
Also, in this case, the core must be "taught" much about
|
||||||
|
specific tasks.
|
||||||
|
This methodology does not scale well and does not allow users
|
||||||
|
to easily add new tasks in layers or as external recipes
|
||||||
|
without touching the packaged-staging core.
|
||||||
|
</para>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section id='overview-checksums'>
|
||||||
|
<title>Checksums (Signatures)</title>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The shared state code uses a checksum, which is a unique
|
||||||
|
signature of a task's inputs, to determine if a task needs to
|
||||||
|
be run again.
|
||||||
|
Because it is a change in a task's inputs that triggers a
|
||||||
|
rerun, the process needs to detect all the inputs to a given
|
||||||
|
task.
|
||||||
|
For shell tasks, this turns out to be fairly easy because
|
||||||
|
the build process generates a "run" shell script for each task
|
||||||
|
and it is possible to create a checksum that gives you a good
|
||||||
|
idea of when the task's data changes.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
To complicate the problem, there are things that should not be
|
||||||
|
included in the checksum.
|
||||||
|
First, there is the actual specific build path of a given
|
||||||
|
task - the
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
|
||||||
|
It does not matter if the work directory changes because it
|
||||||
|
should not affect the output for target packages.
|
||||||
|
Also, the build process has the objective of making native
|
||||||
|
or cross packages relocatable.
|
||||||
|
<note>
|
||||||
|
Both native and cross packages run on the build host.
|
||||||
|
However, cross packages generate output for the target
|
||||||
|
architecture.
|
||||||
|
</note>
|
||||||
|
The checksum therefore needs to exclude
|
||||||
|
<filename>WORKDIR</filename>.
|
||||||
|
The simplistic approach for excluding the work directory is to
|
||||||
|
set <filename>WORKDIR</filename> to some fixed value and
|
||||||
|
create the checksum for the "run" script.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
Another problem results from the "run" scripts containing
|
||||||
|
functions that might or might not get called.
|
||||||
|
The incremental build solution contains code that figures out
|
||||||
|
dependencies between shell functions.
|
||||||
|
This code is used to prune the "run" scripts down to the
|
||||||
|
minimum set, thereby alleviating this problem and making the
|
||||||
|
"run" scripts much more readable as a bonus.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
So far we have solutions for shell scripts.
|
||||||
|
What about Python tasks?
|
||||||
|
The same approach applies even though these tasks are more
|
||||||
|
difficult.
|
||||||
|
The process needs to figure out what variables a Python
|
||||||
|
function accesses and what functions it calls.
|
||||||
|
Again, the incremental build solution contains code that first
|
||||||
|
figures out the variable and function dependencies, and then
|
||||||
|
creates a checksum for the data used as the input to the task.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
Like the <filename>WORKDIR</filename> case, situations exist
|
||||||
|
where dependencies should be ignored.
|
||||||
|
For these cases, you can instruct the build process to
|
||||||
|
ignore a dependency by using a line like the following:
|
||||||
|
<literallayout class='monospaced'>
|
||||||
|
PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
|
||||||
|
</literallayout>
|
||||||
|
This example ensures that the
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink>
|
||||||
|
variable does not depend on the value of
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>,
|
||||||
|
even if it does reference it.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
Equally, there are cases where we need to add dependencies
|
||||||
|
BitBake is not able to find.
|
||||||
|
You can accomplish this by using a line like the following:
|
||||||
|
<literallayout class='monospaced'>
|
||||||
|
PACKAGE_ARCHS[vardeps] = "MACHINE"
|
||||||
|
</literallayout>
|
||||||
|
This example explicitly adds the <filename>MACHINE</filename>
|
||||||
|
variable as a dependency for
|
||||||
|
<filename>PACKAGE_ARCHS</filename>.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
Consider a case with in-line Python, for example, where
|
||||||
|
BitBake is not able to figure out dependencies.
|
||||||
|
When running in debug mode (i.e. using
|
||||||
|
<filename>-DDD</filename>), BitBake produces output when it
|
||||||
|
discovers something for which it cannot figure out dependencies.
|
||||||
|
The Yocto Project team has currently not managed to cover
|
||||||
|
those dependencies in detail and is aware of the need to fix
|
||||||
|
this situation.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
Thus far, this section has limited discussion to the direct
|
||||||
|
inputs into a task.
|
||||||
|
Information based on direct inputs is referred to as the
|
||||||
|
"basehash" in the code.
|
||||||
|
However, there is still the question of a task's indirect
|
||||||
|
inputs - the things that were already built and present in the
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
|
||||||
|
The checksum (or signature) for a particular task needs to add
|
||||||
|
the hashes of all the tasks on which the particular task
|
||||||
|
depends.
|
||||||
|
Choosing which dependencies to add is a policy decision.
|
||||||
|
However, the effect is to generate a master checksum that
|
||||||
|
combines the basehash and the hashes of the task's
|
||||||
|
dependencies.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
At the code level, there are a variety of ways both the
|
||||||
|
basehash and the dependent task hashes can be influenced.
|
||||||
|
Within the BitBake configuration file, we can give BitBake
|
||||||
|
some extra information to help it construct the basehash.
|
||||||
|
The following statement effectively results in a list of
|
||||||
|
global variable dependency excludes - variables never
|
||||||
|
included in any checksum:
|
||||||
|
<literallayout class='monospaced'>
|
||||||
|
BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
|
||||||
|
SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
|
||||||
|
USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
|
||||||
|
PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
|
||||||
|
CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
|
||||||
|
</literallayout>
|
||||||
|
The previous example excludes
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>
|
||||||
|
since that variable is actually constructed as a path within
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>,
|
||||||
|
which is on the whitelist.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The rules for deciding which hashes of dependent tasks to
|
||||||
|
include through dependency chains are more complex and are
|
||||||
|
generally accomplished with a Python function.
|
||||||
|
The code in <filename>meta/lib/oe/sstatesig.py</filename> shows
|
||||||
|
two examples of this and also illustrates how you can insert
|
||||||
|
your own policy into the system if so desired.
|
||||||
|
This file defines the two basic signature generators
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink>
|
||||||
|
uses: "OEBasic" and "OEBasicHash".
|
||||||
|
By default, there is a dummy "noop" signature handler enabled
|
||||||
|
in BitBake.
|
||||||
|
This means that behavior is unchanged from previous versions.
|
||||||
|
OE-Core uses the "OEBasicHash" signature handler by default
|
||||||
|
through this setting in the <filename>bitbake.conf</filename>
|
||||||
|
file:
|
||||||
|
<literallayout class='monospaced'>
|
||||||
|
BB_SIGNATURE_HANDLER ?= "OEBasicHash"
|
||||||
|
</literallayout>
|
||||||
|
The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename>
|
||||||
|
is the same as the "OEBasic" version but adds the task hash to
|
||||||
|
the stamp files.
|
||||||
|
This results in any
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>
|
||||||
|
change that changes the task hash, automatically
|
||||||
|
causing the task to be run again.
|
||||||
|
This removes the need to bump
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
|
||||||
|
values, and changes to Metadata automatically ripple across
|
||||||
|
the build.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
It is also worth noting that the end result of these
|
||||||
|
signature generators is to make some dependency and hash
|
||||||
|
information available to the build.
|
||||||
|
This information includes:
|
||||||
|
<itemizedlist>
|
||||||
|
<listitem><para>
|
||||||
|
<filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
|
||||||
|
The base hashes for each task in the recipe.
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
<filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
|
||||||
|
The base hashes for each dependent task.
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
<filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
|
||||||
|
The task dependencies for each task.
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
<filename>BB_TASKHASH</filename>:
|
||||||
|
The hash of the currently running task.
|
||||||
|
</para></listitem>
|
||||||
|
</itemizedlist>
|
||||||
|
</para>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section id='shared-state'>
|
||||||
|
<title>Shared State</title>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
Checksums and dependencies, as discussed in the previous
|
||||||
|
section, solve half the problem of supporting a shared state.
|
||||||
|
The other part of the problem is being able to use checksum
|
||||||
|
information during the build and being able to reuse or rebuild
|
||||||
|
specific components.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
|
||||||
|
class is a relatively generic implementation of how to
|
||||||
|
"capture" a snapshot of a given task.
|
||||||
|
The idea is that the build process does not care about the
|
||||||
|
source of a task's output.
|
||||||
|
Output could be freshly built or it could be downloaded and
|
||||||
|
unpacked from somewhere - the build process does not need to
|
||||||
|
worry about its origin.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
There are two types of output, one is just about creating a
|
||||||
|
directory in
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
|
||||||
|
A good example is the output of either
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
|
||||||
|
or
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>.
|
||||||
|
The other type of output occurs when a set of data is merged
|
||||||
|
into a shared directory tree such as the sysroot.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The Yocto Project team has tried to keep the details of the
|
||||||
|
implementation hidden in <filename>sstate</filename> class.
|
||||||
|
From a user's perspective, adding shared state wrapping to a task
|
||||||
|
is as simple as this
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
|
||||||
|
example taken from the
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink>
|
||||||
|
class:
|
||||||
|
<literallayout class='monospaced'>
|
||||||
|
DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
|
||||||
|
SSTATETASKS += "do_deploy"
|
||||||
|
do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
|
||||||
|
do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
|
||||||
|
|
||||||
|
python do_deploy_setscene () {
|
||||||
|
sstate_setscene(d)
|
||||||
|
}
|
||||||
|
addtask do_deploy_setscene
|
||||||
|
do_deploy[dirs] = "${DEPLOYDIR} ${B}"
|
||||||
|
</literallayout>
|
||||||
|
The following list explains the previous example:
|
||||||
|
<itemizedlist>
|
||||||
|
<listitem><para>
|
||||||
|
Adding "do_deploy" to <filename>SSTATETASKS</filename>
|
||||||
|
adds some required sstate-related processing, which is
|
||||||
|
implemented in the
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
|
||||||
|
class, to before and after the
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
|
||||||
|
task.
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
The
|
||||||
|
<filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
|
||||||
|
declares that <filename>do_deploy</filename> places its
|
||||||
|
output in <filename>${DEPLOYDIR}</filename> when run
|
||||||
|
normally (i.e. when not using the sstate cache).
|
||||||
|
This output becomes the input to the shared state cache.
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
The
|
||||||
|
<filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
|
||||||
|
line causes the contents of the shared state cache to be
|
||||||
|
copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
|
||||||
|
<note>
|
||||||
|
If <filename>do_deploy</filename> is not already in
|
||||||
|
the shared state cache or if its input checksum
|
||||||
|
(signature) has changed from when the output was
|
||||||
|
cached, the task will be run to populate the shared
|
||||||
|
state cache, after which the contents of the shared
|
||||||
|
state cache is copied to
|
||||||
|
<filename>${DEPLOY_DIR_IMAGE}</filename>.
|
||||||
|
If <filename>do_deploy</filename> is in the shared
|
||||||
|
state cache and its signature indicates that the
|
||||||
|
cached output is still valid (i.e. if no
|
||||||
|
relevant task inputs have changed), then the
|
||||||
|
contents of the shared state cache will be copied
|
||||||
|
directly to
|
||||||
|
<filename>${DEPLOY_DIR_IMAGE}</filename> by the
|
||||||
|
<filename>do_deploy_setscene</filename> task
|
||||||
|
instead, skipping the
|
||||||
|
<filename>do_deploy</filename> task.
|
||||||
|
</note>
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
The following task definition is glue logic needed to
|
||||||
|
make the previous settings effective:
|
||||||
|
<literallayout class='monospaced'>
|
||||||
|
python do_deploy_setscene () {
|
||||||
|
sstate_setscene(d)
|
||||||
|
}
|
||||||
|
addtask do_deploy_setscene
|
||||||
|
</literallayout>
|
||||||
|
<filename>sstate_setscene()</filename> takes the flags
|
||||||
|
above as input and accelerates the
|
||||||
|
<filename>do_deploy</filename> task through the
|
||||||
|
shared state cache if possible.
|
||||||
|
If the task was accelerated,
|
||||||
|
<filename>sstate_setscene()</filename> returns True.
|
||||||
|
Otherwise, it returns False, and the normal
|
||||||
|
<filename>do_deploy</filename> task runs.
|
||||||
|
For more information, see the
|
||||||
|
"<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
|
||||||
|
section in the BitBake User Manual.
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
|
||||||
|
line creates <filename>${DEPLOYDIR}</filename> and
|
||||||
|
<filename>${B}</filename> before the
|
||||||
|
<filename>do_deploy</filename> task runs, and also sets
|
||||||
|
the current working directory of
|
||||||
|
<filename>do_deploy</filename> to
|
||||||
|
<filename>${B}</filename>.
|
||||||
|
For more information, see the
|
||||||
|
"<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
|
||||||
|
section in the BitBake User Manual.
|
||||||
|
<note>
|
||||||
|
In cases where
|
||||||
|
<filename>sstate-inputdirs</filename> and
|
||||||
|
<filename>sstate-outputdirs</filename> would be the
|
||||||
|
same, you can use
|
||||||
|
<filename>sstate-plaindirs</filename>.
|
||||||
|
For example, to preserve the
|
||||||
|
<filename>${PKGD}</filename> and
|
||||||
|
<filename>${PKGDEST}</filename> output from the
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
|
||||||
|
task, use the following:
|
||||||
|
<literallayout class='monospaced'>
|
||||||
|
do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
|
||||||
|
</literallayout>
|
||||||
|
</note>
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
<filename>sstate-inputdirs</filename> and
|
||||||
|
<filename>sstate-outputdirs</filename> can also be used
|
||||||
|
with multiple directories.
|
||||||
|
For example, the following declares
|
||||||
|
<filename>PKGDESTWORK</filename> and
|
||||||
|
<filename>SHLIBWORK</filename> as shared state
|
||||||
|
input directories, which populates the shared state
|
||||||
|
cache, and <filename>PKGDATA_DIR</filename> and
|
||||||
|
<filename>SHLIBSDIR</filename> as the corresponding
|
||||||
|
shared state output directories:
|
||||||
|
<literallayout class='monospaced'>
|
||||||
|
do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
|
||||||
|
do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
|
||||||
|
</literallayout>
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
These methods also include the ability to take a
|
||||||
|
lockfile when manipulating shared state directory
|
||||||
|
structures, for cases where file additions or removals
|
||||||
|
are sensitive:
|
||||||
|
<literallayout class='monospaced'>
|
||||||
|
do_package[sstate-lockfile] = "${PACKAGELOCK}"
|
||||||
|
</literallayout>
|
||||||
|
</para></listitem>
|
||||||
|
</itemizedlist>
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
Behind the scenes, the shared state code works by looking in
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
|
||||||
|
and
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
|
||||||
|
for shared state files.
|
||||||
|
Here is an example:
|
||||||
|
<literallayout class='monospaced'>
|
||||||
|
SSTATE_MIRRORS ?= "\
|
||||||
|
file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
|
||||||
|
file://.* file:///some/local/dir/sstate/PATH"
|
||||||
|
</literallayout>
|
||||||
|
<note>
|
||||||
|
The shared state directory
|
||||||
|
(<filename>SSTATE_DIR</filename>) is organized into
|
||||||
|
two-character subdirectories, where the subdirectory
|
||||||
|
names are based on the first two characters of the hash.
|
||||||
|
If the shared state directory structure for a mirror has the
|
||||||
|
same structure as <filename>SSTATE_DIR</filename>, you must
|
||||||
|
specify "PATH" as part of the URI to enable the build system
|
||||||
|
to map to the appropriate subdirectory.
|
||||||
|
</note>
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The shared state package validity can be detected just by
|
||||||
|
looking at the filename since the filename contains the task
|
||||||
|
checksum (or signature) as described earlier in this section.
|
||||||
|
If a valid shared state package is found, the build process
|
||||||
|
downloads it and uses it to accelerate the task.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The build processes use the <filename>*_setscene</filename>
|
||||||
|
tasks for the task acceleration phase.
|
||||||
|
BitBake goes through this phase before the main execution
|
||||||
|
code and tries to accelerate any tasks for which it can find
|
||||||
|
shared state packages.
|
||||||
|
If a shared state package for a task is available, the
|
||||||
|
shared state package is used.
|
||||||
|
This means the task and any tasks on which it is dependent
|
||||||
|
are not executed.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
As a real world example, the aim is when building an IPK-based
|
||||||
|
image, only the
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>
|
||||||
|
tasks would have their shared state packages fetched and
|
||||||
|
extracted.
|
||||||
|
Since the sysroot is not used, it would never get extracted.
|
||||||
|
This is another reason why a task-based approach is preferred
|
||||||
|
over a recipe-based approach, which would have to install the
|
||||||
|
output from every task.
|
||||||
|
</para>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section id='tips-and-tricks'>
|
||||||
|
<title>Tips and Tricks</title>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The code in the build system that supports incremental builds
|
||||||
|
is not simple code.
|
||||||
|
This section presents some tips and tricks that help you work
|
||||||
|
around issues related to shared state code.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<section id='overview-debugging'>
|
||||||
|
<title>Debugging</title>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
Seeing what metadata went into creating the input signature
|
||||||
|
of a shared state (sstate) task can be a useful debugging
|
||||||
|
aid.
|
||||||
|
This information is available in signature information
|
||||||
|
(<filename>siginfo</filename>) files in
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>.
|
||||||
|
For information on how to view and interpret information in
|
||||||
|
<filename>siginfo</filename> files, see the
|
||||||
|
"<ulink url='&YOCTO_DOCS_REF_URL;#usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>"
|
||||||
|
section in the Yocto Project Reference Manual.
|
||||||
|
</para>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section id='invalidating-shared-state'>
|
||||||
|
<title>Invalidating Shared State</title>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The OpenEmbedded build system uses checksums and shared
|
||||||
|
state cache to avoid unnecessarily rebuilding tasks.
|
||||||
|
Collectively, this scheme is known as "shared state code."
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
As with all schemes, this one has some drawbacks.
|
||||||
|
It is possible that you could make implicit changes to your
|
||||||
|
code that the checksum calculations do not take into
|
||||||
|
account.
|
||||||
|
These implicit changes affect a task's output but do not
|
||||||
|
trigger the shared state code into rebuilding a recipe.
|
||||||
|
Consider an example during which a tool changes its output.
|
||||||
|
Assume that the output of <filename>rpmdeps</filename>
|
||||||
|
changes.
|
||||||
|
The result of the change should be that all the
|
||||||
|
<filename>package</filename> and
|
||||||
|
<filename>package_write_rpm</filename> shared state cache
|
||||||
|
items become invalid.
|
||||||
|
However, because the change to the output is
|
||||||
|
external to the code and therefore implicit,
|
||||||
|
the associated shared state cache items do not become
|
||||||
|
invalidated.
|
||||||
|
In this case, the build process uses the cached items
|
||||||
|
rather than running the task again.
|
||||||
|
Obviously, these types of implicit changes can cause
|
||||||
|
problems.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
To avoid these problems during the build, you need to
|
||||||
|
understand the effects of any changes you make.
|
||||||
|
Realize that changes you make directly to a function
|
||||||
|
are automatically factored into the checksum calculation.
|
||||||
|
Thus, these explicit changes invalidate the associated
|
||||||
|
area of shared state cache.
|
||||||
|
However, you need to be aware of any implicit changes that
|
||||||
|
are not obvious changes to the code and could affect
|
||||||
|
the output of a given task.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
When you identify an implicit change, you can easily
|
||||||
|
take steps to invalidate the cache and force the tasks
|
||||||
|
to run.
|
||||||
|
The steps you can take are as simple as changing a
|
||||||
|
function's comments in the source code.
|
||||||
|
For example, to invalidate package shared state files,
|
||||||
|
change the comment statements of
|
||||||
|
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
|
||||||
|
or the comments of one of the functions it calls.
|
||||||
|
Even though the change is purely cosmetic, it causes the
|
||||||
|
checksum to be recalculated and forces the OpenEmbedded
|
||||||
|
build system to run the task again.
|
||||||
|
<note>
|
||||||
|
For an example of a commit that makes a cosmetic
|
||||||
|
change to invalidate shared state, see this
|
||||||
|
<ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
|
||||||
|
</note>
|
||||||
|
</para>
|
||||||
|
</section>
|
||||||
|
</section>
|
||||||
|
</section>
|
||||||
|
|
||||||
<section id='x32'>
|
<section id='x32'>
|
||||||
<title>x32 psABI</title>
|
<title>x32 psABI</title>
|
||||||
|
|
||||||
|
|||||||
@@ -2490,7 +2490,7 @@
|
|||||||
<ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink>
|
<ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink>
|
||||||
variable, and the end of the name consists of the task's name
|
variable, and the end of the name consists of the task's name
|
||||||
and current
|
and current
|
||||||
<ulink url='&YOCTO_DOCS_BB_URL;#checksums'>input checksum</ulink>.
|
<link linkend='overview-checksums'>input checksum</link>.
|
||||||
<note>
|
<note>
|
||||||
This naming scheme assumes that
|
This naming scheme assumes that
|
||||||
<ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
|
<ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
|
||||||
|
|||||||
@@ -348,8 +348,10 @@
|
|||||||
If doing so results in unnecessary rebuilds of tasks, you can whitelist the
|
If doing so results in unnecessary rebuilds of tasks, you can whitelist the
|
||||||
variable so that the shared state code ignores the dependency when it creates
|
variable so that the shared state code ignores the dependency when it creates
|
||||||
checksums.
|
checksums.
|
||||||
For information on this process, see the <filename>BB_HASHBASE_WHITELIST</filename>
|
For information on this process, see the
|
||||||
example in the "<link linkend='checksums'>Checksums (Signatures)</link>" section.
|
<filename>BB_HASHBASE_WHITELIST</filename> example in the
|
||||||
|
"<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#overview-checksums'>Checksums (Signatures)</ulink>"
|
||||||
|
section in the Yocto Project Overview Manual.
|
||||||
</note>
|
</note>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
|||||||
@@ -3182,8 +3182,8 @@ This check was removed for YP 2.3 release
|
|||||||
|
|
||||||
<para>
|
<para>
|
||||||
For more information on sstate, see the
|
For more information on sstate, see the
|
||||||
"<link linkend='shared-state-cache'>Shared State Cache</link>"
|
"<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>Shared State Cache</ulink>"
|
||||||
section.
|
section in the Yocto Project Overview Manual.
|
||||||
</para>
|
</para>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
|||||||
@@ -628,8 +628,8 @@
|
|||||||
|
|
||||||
<para>
|
<para>
|
||||||
Running this task does not remove the
|
Running this task does not remove the
|
||||||
<link linkend='shared-state-cache'>sstate</link>) cache
|
<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>
|
||||||
files.
|
cache files.
|
||||||
Consequently, if no changes have been made and the recipe is
|
Consequently, if no changes have been made and the recipe is
|
||||||
rebuilt after cleaning, output files are simply restored from the
|
rebuilt after cleaning, output files are simply restored from the
|
||||||
sstate cache.
|
sstate cache.
|
||||||
@@ -645,8 +645,9 @@
|
|||||||
|
|
||||||
<para>
|
<para>
|
||||||
Removes all output files, shared state
|
Removes all output files, shared state
|
||||||
(<link linkend='shared-state-cache'>sstate</link>) cache, and
|
(<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>)
|
||||||
downloaded source files for a target (i.e. the contents of
|
cache, and downloaded source files for a target (i.e. the contents
|
||||||
|
of
|
||||||
<link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>).
|
<link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>).
|
||||||
Essentially, the <filename>do_cleanall</filename> task is
|
Essentially, the <filename>do_cleanall</filename> task is
|
||||||
identical to the
|
identical to the
|
||||||
@@ -675,13 +676,14 @@
|
|||||||
|
|
||||||
<para>
|
<para>
|
||||||
Removes all output files and shared state
|
Removes all output files and shared state
|
||||||
(<link linkend='shared-state-cache'>sstate</link>)
|
(<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>)
|
||||||
cache for a target.
|
cache for a target.
|
||||||
Essentially, the <filename>do_cleansstate</filename> task is
|
Essentially, the <filename>do_cleansstate</filename> task is
|
||||||
identical to the
|
identical to the
|
||||||
<link linkend='ref-tasks-clean'><filename>do_clean</filename></link>
|
<link linkend='ref-tasks-clean'><filename>do_clean</filename></link>
|
||||||
task with the added removal of shared state
|
task with the added removal of shared state
|
||||||
(<link linkend='shared-state-cache'>sstate</link>) cache.
|
(<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>)
|
||||||
|
cache.
|
||||||
</para>
|
</para>
|
||||||
|
|
||||||
<para>
|
<para>
|
||||||
|
|||||||
@@ -10654,11 +10654,11 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
|
|||||||
<filename>PR</filename> to know when to rebuild a
|
<filename>PR</filename> to know when to rebuild a
|
||||||
recipe.
|
recipe.
|
||||||
The build system uses the task
|
The build system uses the task
|
||||||
<ulink url='&YOCTO_DOCS_BB_URL;#checksums'>input checksums</ulink>
|
<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#overview-checksums'>input checksums</ulink>
|
||||||
along with the
|
along with the
|
||||||
<link linkend='structure-build-tmp-stamps'>stamp</link>
|
<link linkend='structure-build-tmp-stamps'>stamp</link>
|
||||||
and
|
and
|
||||||
<link linkend='shared-state-cache'>shared state cache</link>
|
<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>shared state cache</ulink>
|
||||||
mechanisms.
|
mechanisms.
|
||||||
</note>
|
</note>
|
||||||
The <filename>PR</filename> variable primarily becomes
|
The <filename>PR</filename> variable primarily becomes
|
||||||
|
|||||||
@@ -13,609 +13,6 @@
|
|||||||
x32, Wayland support, and Licenses.
|
x32, Wayland support, and Licenses.
|
||||||
</para>
|
</para>
|
||||||
|
|
||||||
<section id="shared-state-cache">
|
|
||||||
<title>Shared State Cache</title>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
By design, the OpenEmbedded build system builds everything from scratch unless
|
|
||||||
BitBake can determine that parts do not need to be rebuilt.
|
|
||||||
Fundamentally, building from scratch is attractive as it means all parts are
|
|
||||||
built fresh and there is no possibility of stale data causing problems.
|
|
||||||
When developers hit problems, they typically default back to building from scratch
|
|
||||||
so they know the state of things from the start.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
Building an image from scratch is both an advantage and a disadvantage to the process.
|
|
||||||
As mentioned in the previous paragraph, building from scratch ensures that
|
|
||||||
everything is current and starts from a known state.
|
|
||||||
However, building from scratch also takes much longer as it generally means
|
|
||||||
rebuilding things that do not necessarily need to be rebuilt.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
The Yocto Project implements shared state code that supports incremental builds.
|
|
||||||
The implementation of the shared state code answers the following questions that
|
|
||||||
were fundamental roadblocks within the OpenEmbedded incremental build support system:
|
|
||||||
<itemizedlist>
|
|
||||||
<listitem><para>What pieces of the system have changed and what pieces have
|
|
||||||
not changed?</para></listitem>
|
|
||||||
<listitem><para>How are changed pieces of software removed and replaced?</para></listitem>
|
|
||||||
<listitem><para>How are pre-built components that do not need to be rebuilt from scratch
|
|
||||||
used when they are available?</para></listitem>
|
|
||||||
</itemizedlist>
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
For the first question, the build system detects changes in the "inputs" to a given task by
|
|
||||||
creating a checksum (or signature) of the task's inputs.
|
|
||||||
If the checksum changes, the system assumes the inputs have changed and the task needs to be
|
|
||||||
rerun.
|
|
||||||
For the second question, the shared state (sstate) code tracks which tasks add which output
|
|
||||||
to the build process.
|
|
||||||
This means the output from a given task can be removed, upgraded or otherwise manipulated.
|
|
||||||
The third question is partly addressed by the solution for the second question
|
|
||||||
assuming the build system can fetch the sstate objects from remote locations and
|
|
||||||
install them if they are deemed to be valid.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<note>
|
|
||||||
The OpenEmbedded build system does not maintain
|
|
||||||
<link linkend='var-PR'><filename>PR</filename></link> information
|
|
||||||
as part of the shared state packages.
|
|
||||||
Consequently, considerations exist that affect maintaining shared
|
|
||||||
state feeds.
|
|
||||||
For information on how the OpenEmbedded build system
|
|
||||||
works with packages and can
|
|
||||||
track incrementing <filename>PR</filename> information, see the
|
|
||||||
"<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
|
|
||||||
section in the Yocto Project Development Tasks Manual.
|
|
||||||
</note>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
The rest of this section goes into detail about the overall incremental build
|
|
||||||
architecture, the checksums (signatures), shared state, and some tips and tricks.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<section id='overall-architecture'>
|
|
||||||
<title>Overall Architecture</title>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
When determining what parts of the system need to be built, BitBake
|
|
||||||
works on a per-task basis rather than a per-recipe basis.
|
|
||||||
You might wonder why using a per-task basis is preferred over a per-recipe basis.
|
|
||||||
To help explain, consider having the IPK packaging backend enabled and then switching to DEB.
|
|
||||||
In this case, the
|
|
||||||
<link linkend='ref-tasks-install'><filename>do_install</filename></link>
|
|
||||||
and
|
|
||||||
<link linkend='ref-tasks-package'><filename>do_package</filename></link>
|
|
||||||
task outputs are still valid.
|
|
||||||
However, with a per-recipe approach, the build would not include the
|
|
||||||
<filename>.deb</filename> files.
|
|
||||||
Consequently, you would have to invalidate the whole build and rerun it.
|
|
||||||
Rerunning everything is not the best solution.
|
|
||||||
Also, in this case, the core must be "taught" much about specific tasks.
|
|
||||||
This methodology does not scale well and does not allow users to easily add new tasks
|
|
||||||
in layers or as external recipes without touching the packaged-staging core.
|
|
||||||
</para>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<section id='checksums'>
|
|
||||||
<title>Checksums (Signatures)</title>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
The shared state code uses a checksum, which is a unique signature of a task's
|
|
||||||
inputs, to determine if a task needs to be run again.
|
|
||||||
Because it is a change in a task's inputs that triggers a rerun, the process
|
|
||||||
needs to detect all the inputs to a given task.
|
|
||||||
For shell tasks, this turns out to be fairly easy because
|
|
||||||
the build process generates a "run" shell script for each task and
|
|
||||||
it is possible to create a checksum that gives you a good idea of when
|
|
||||||
the task's data changes.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
To complicate the problem, there are things that should not be
|
|
||||||
included in the checksum.
|
|
||||||
First, there is the actual specific build path of a given task -
|
|
||||||
the <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.
|
|
||||||
It does not matter if the work directory changes because it should
|
|
||||||
not affect the output for target packages.
|
|
||||||
Also, the build process has the objective of making native
|
|
||||||
or cross packages relocatable.
|
|
||||||
<note>
|
|
||||||
Both native and cross packages run on the build host.
|
|
||||||
However, cross packages generate output for the target
|
|
||||||
architecture.
|
|
||||||
</note>
|
|
||||||
The checksum therefore needs to exclude
|
|
||||||
<filename>WORKDIR</filename>.
|
|
||||||
The simplistic approach for excluding the work directory is to set
|
|
||||||
<filename>WORKDIR</filename> to some fixed value and create the
|
|
||||||
checksum for the "run" script.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
Another problem results from the "run" scripts containing functions that
|
|
||||||
might or might not get called.
|
|
||||||
The incremental build solution contains code that figures out dependencies
|
|
||||||
between shell functions.
|
|
||||||
This code is used to prune the "run" scripts down to the minimum set,
|
|
||||||
thereby alleviating this problem and making the "run" scripts much more
|
|
||||||
readable as a bonus.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
So far we have solutions for shell scripts.
|
|
||||||
What about Python tasks?
|
|
||||||
The same approach applies even though these tasks are more difficult.
|
|
||||||
The process needs to figure out what variables a Python function accesses
|
|
||||||
and what functions it calls.
|
|
||||||
Again, the incremental build solution contains code that first figures out
|
|
||||||
the variable and function dependencies, and then creates a checksum for the data
|
|
||||||
used as the input to the task.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
Like the <filename>WORKDIR</filename> case, situations exist where dependencies
|
|
||||||
should be ignored.
|
|
||||||
For these cases, you can instruct the build process to ignore a dependency
|
|
||||||
by using a line like the following:
|
|
||||||
<literallayout class='monospaced'>
|
|
||||||
PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
|
|
||||||
</literallayout>
|
|
||||||
This example ensures that the
|
|
||||||
<link linkend='var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></link>
|
|
||||||
variable does not
|
|
||||||
depend on the value of
|
|
||||||
<link linkend='var-MACHINE'><filename>MACHINE</filename></link>,
|
|
||||||
even if it does reference it.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
Equally, there are cases where we need to add dependencies BitBake is not able to find.
|
|
||||||
You can accomplish this by using a line like the following:
|
|
||||||
<literallayout class='monospaced'>
|
|
||||||
PACKAGE_ARCHS[vardeps] = "MACHINE"
|
|
||||||
</literallayout>
|
|
||||||
This example explicitly adds the <filename>MACHINE</filename> variable as a
|
|
||||||
dependency for <filename>PACKAGE_ARCHS</filename>.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
Consider a case with in-line Python, for example, where BitBake is not
|
|
||||||
able to figure out dependencies.
|
|
||||||
When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
|
|
||||||
produces output when it discovers something for which it cannot figure out
|
|
||||||
dependencies.
|
|
||||||
The Yocto Project team has currently not managed to cover those dependencies
|
|
||||||
in detail and is aware of the need to fix this situation.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
Thus far, this section has limited discussion to the direct inputs into a task.
|
|
||||||
Information based on direct inputs is referred to as the "basehash" in the
|
|
||||||
code.
|
|
||||||
However, there is still the question of a task's indirect inputs - the
|
|
||||||
things that were already built and present in the
|
|
||||||
<link linkend='build-directory'>Build Directory</link>.
|
|
||||||
The checksum (or signature) for a particular task needs to add the hashes
|
|
||||||
of all the tasks on which the particular task depends.
|
|
||||||
Choosing which dependencies to add is a policy decision.
|
|
||||||
However, the effect is to generate a master checksum that combines the basehash
|
|
||||||
and the hashes of the task's dependencies.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
At the code level, there are a variety of ways both the basehash and the
|
|
||||||
dependent task hashes can be influenced.
|
|
||||||
Within the BitBake configuration file, we can give BitBake some extra information
|
|
||||||
to help it construct the basehash.
|
|
||||||
The following statement effectively results in a list of global variable
|
|
||||||
dependency excludes - variables never included in any checksum:
|
|
||||||
<literallayout class='monospaced'>
|
|
||||||
BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
|
|
||||||
SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
|
|
||||||
USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
|
|
||||||
PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
|
|
||||||
CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
|
|
||||||
</literallayout>
|
|
||||||
The previous example excludes
|
|
||||||
<link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>
|
|
||||||
since that variable is actually constructed as a path within
|
|
||||||
<link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>, which is on
|
|
||||||
the whitelist.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
The rules for deciding which hashes of dependent tasks to include through
|
|
||||||
dependency chains are more complex and are generally accomplished with a
|
|
||||||
Python function.
|
|
||||||
The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
|
|
||||||
of this and also illustrates how you can insert your own policy into the system
|
|
||||||
if so desired.
|
|
||||||
This file defines the two basic signature generators
|
|
||||||
<link linkend='oe-core'>OE-Core</link> uses: "OEBasic" and
|
|
||||||
"OEBasicHash".
|
|
||||||
By default, there is a dummy "noop" signature handler enabled in BitBake.
|
|
||||||
This means that behavior is unchanged from previous versions.
|
|
||||||
OE-Core uses the "OEBasicHash" signature handler by default
|
|
||||||
through this setting in the <filename>bitbake.conf</filename> file:
|
|
||||||
<literallayout class='monospaced'>
|
|
||||||
BB_SIGNATURE_HANDLER ?= "OEBasicHash"
|
|
||||||
</literallayout>
|
|
||||||
The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
|
|
||||||
"OEBasic" version but adds the task hash to the stamp files.
|
|
||||||
This results in any
|
|
||||||
<link linkend='metadata'>Metadata</link>
|
|
||||||
change that changes the task hash, automatically
|
|
||||||
causing the task to be run again.
|
|
||||||
This removes the need to bump <link linkend='var-PR'><filename>PR</filename></link>
|
|
||||||
values, and changes to Metadata automatically ripple across the build.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
It is also worth noting that the end result of these signature generators is to
|
|
||||||
make some dependency and hash information available to the build.
|
|
||||||
This information includes:
|
|
||||||
<itemizedlist>
|
|
||||||
<listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
|
|
||||||
The base hashes for each task in the recipe.
|
|
||||||
</para></listitem>
|
|
||||||
<listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
|
|
||||||
The base hashes for each dependent task.
|
|
||||||
</para></listitem>
|
|
||||||
<listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
|
|
||||||
The task dependencies for each task.
|
|
||||||
</para></listitem>
|
|
||||||
<listitem><para><filename>BB_TASKHASH</filename>:
|
|
||||||
The hash of the currently running task.
|
|
||||||
</para></listitem>
|
|
||||||
</itemizedlist>
|
|
||||||
</para>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<section id='shared-state'>
|
|
||||||
<title>Shared State</title>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
Checksums and dependencies, as discussed in the previous section, solve half the
|
|
||||||
problem of supporting a shared state.
|
|
||||||
The other part of the problem is being able to use checksum information during the build
|
|
||||||
and being able to reuse or rebuild specific components.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
The
|
|
||||||
<link linkend='ref-classes-sstate'><filename>sstate</filename></link>
|
|
||||||
class is a relatively generic implementation of how to "capture"
|
|
||||||
a snapshot of a given task.
|
|
||||||
The idea is that the build process does not care about the source of a task's output.
|
|
||||||
Output could be freshly built or it could be downloaded and unpacked from
|
|
||||||
somewhere - the build process does not need to worry about its origin.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
There are two types of output, one is just about creating a directory
|
|
||||||
in <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.
|
|
||||||
A good example is the output of either
|
|
||||||
<link linkend='ref-tasks-install'><filename>do_install</filename></link>
|
|
||||||
or
|
|
||||||
<link linkend='ref-tasks-package'><filename>do_package</filename></link>.
|
|
||||||
The other type of output occurs when a set of data is merged into a shared directory
|
|
||||||
tree such as the sysroot.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
The Yocto Project team has tried to keep the details of the
|
|
||||||
implementation hidden in <filename>sstate</filename> class.
|
|
||||||
From a user's perspective, adding shared state wrapping to a task
|
|
||||||
is as simple as this
|
|
||||||
<link linkend='ref-tasks-deploy'><filename>do_deploy</filename></link>
|
|
||||||
example taken from the
|
|
||||||
<link linkend='ref-classes-deploy'><filename>deploy</filename></link>
|
|
||||||
class:
|
|
||||||
<literallayout class='monospaced'>
|
|
||||||
DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
|
|
||||||
SSTATETASKS += "do_deploy"
|
|
||||||
do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
|
|
||||||
do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
|
|
||||||
|
|
||||||
python do_deploy_setscene () {
|
|
||||||
sstate_setscene(d)
|
|
||||||
}
|
|
||||||
addtask do_deploy_setscene
|
|
||||||
do_deploy[dirs] = "${DEPLOYDIR} ${B}"
|
|
||||||
</literallayout>
|
|
||||||
The following list explains the previous example:
|
|
||||||
<itemizedlist>
|
|
||||||
<listitem><para>
|
|
||||||
Adding "do_deploy" to <filename>SSTATETASKS</filename>
|
|
||||||
adds some required sstate-related processing, which is
|
|
||||||
implemented in the
|
|
||||||
<link linkend='ref-classes-sstate'><filename>sstate</filename></link>
|
|
||||||
class, to before and after the
|
|
||||||
<link linkend='ref-tasks-deploy'><filename>do_deploy</filename></link>
|
|
||||||
task.
|
|
||||||
</para></listitem>
|
|
||||||
<listitem><para>
|
|
||||||
The
|
|
||||||
<filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
|
|
||||||
declares that <filename>do_deploy</filename> places its
|
|
||||||
output in <filename>${DEPLOYDIR}</filename> when run
|
|
||||||
normally (i.e. when not using the sstate cache).
|
|
||||||
This output becomes the input to the shared state cache.
|
|
||||||
</para></listitem>
|
|
||||||
<listitem><para>
|
|
||||||
The
|
|
||||||
<filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
|
|
||||||
line causes the contents of the shared state cache to be
|
|
||||||
copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
|
|
||||||
<note>
|
|
||||||
If <filename>do_deploy</filename> is not already in
|
|
||||||
the shared state cache or if its input checksum
|
|
||||||
(signature) has changed from when the output was
|
|
||||||
cached, the task will be run to populate the shared
|
|
||||||
state cache, after which the contents of the shared
|
|
||||||
state cache is copied to
|
|
||||||
<filename>${DEPLOY_DIR_IMAGE}</filename>.
|
|
||||||
If <filename>do_deploy</filename> is in the shared
|
|
||||||
state cache and its signature indicates that the
|
|
||||||
cached output is still valid (i.e. if no
|
|
||||||
relevant task inputs have changed), then the contents
|
|
||||||
of the shared state cache will be copied directly to
|
|
||||||
<filename>${DEPLOY_DIR_IMAGE}</filename> by the
|
|
||||||
<filename>do_deploy_setscene</filename> task instead,
|
|
||||||
skipping the <filename>do_deploy</filename> task.
|
|
||||||
</note>
|
|
||||||
</para></listitem>
|
|
||||||
<listitem><para>
|
|
||||||
The following task definition is glue logic needed to make
|
|
||||||
the previous settings effective:
|
|
||||||
<literallayout class='monospaced'>
|
|
||||||
python do_deploy_setscene () {
|
|
||||||
sstate_setscene(d)
|
|
||||||
}
|
|
||||||
addtask do_deploy_setscene
|
|
||||||
</literallayout>
|
|
||||||
<filename>sstate_setscene()</filename> takes the flags
|
|
||||||
above as input and accelerates the
|
|
||||||
<filename>do_deploy</filename> task through the
|
|
||||||
shared state cache if possible.
|
|
||||||
If the task was accelerated,
|
|
||||||
<filename>sstate_setscene()</filename> returns True.
|
|
||||||
Otherwise, it returns False, and the normal
|
|
||||||
<filename>do_deploy</filename> task runs.
|
|
||||||
For more information, see the
|
|
||||||
"<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
|
|
||||||
section in the BitBake User Manual.
|
|
||||||
</para></listitem>
|
|
||||||
<listitem><para>
|
|
||||||
The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
|
|
||||||
line creates <filename>${DEPLOYDIR}</filename> and
|
|
||||||
<filename>${B}</filename> before the
|
|
||||||
<filename>do_deploy</filename> task runs, and also sets
|
|
||||||
the current working directory of
|
|
||||||
<filename>do_deploy</filename> to
|
|
||||||
<filename>${B}</filename>.
|
|
||||||
For more information, see the
|
|
||||||
"<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
|
|
||||||
section in the BitBake User Manual.
|
|
||||||
<note>
|
|
||||||
In cases where
|
|
||||||
<filename>sstate-inputdirs</filename> and
|
|
||||||
<filename>sstate-outputdirs</filename> would be the
|
|
||||||
same, you can use
|
|
||||||
<filename>sstate-plaindirs</filename>.
|
|
||||||
For example, to preserve the
|
|
||||||
<filename>${PKGD}</filename> and
|
|
||||||
<filename>${PKGDEST}</filename> output from the
|
|
||||||
<link linkend='ref-tasks-package'><filename>do_package</filename></link>
|
|
||||||
task, use the following:
|
|
||||||
<literallayout class='monospaced'>
|
|
||||||
do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
|
|
||||||
</literallayout>
|
|
||||||
</note>
|
|
||||||
</para></listitem>
|
|
||||||
<listitem><para>
|
|
||||||
<filename>sstate-inputdirs</filename> and
|
|
||||||
<filename>sstate-outputdirs</filename> can also be used
|
|
||||||
with multiple directories.
|
|
||||||
For example, the following declares
|
|
||||||
<filename>PKGDESTWORK</filename> and
|
|
||||||
<filename>SHLIBWORK</filename> as shared state
|
|
||||||
input directories, which populates the shared state
|
|
||||||
cache, and <filename>PKGDATA_DIR</filename> and
|
|
||||||
<filename>SHLIBSDIR</filename> as the corresponding
|
|
||||||
shared state output directories:
|
|
||||||
<literallayout class='monospaced'>
|
|
||||||
do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
|
|
||||||
do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
|
|
||||||
</literallayout>
|
|
||||||
</para></listitem>
|
|
||||||
<listitem><para>
|
|
||||||
These methods also include the ability to take a lockfile
|
|
||||||
when manipulating shared state directory structures,
|
|
||||||
for cases where file additions or removals are sensitive:
|
|
||||||
<literallayout class='monospaced'>
|
|
||||||
do_package[sstate-lockfile] = "${PACKAGELOCK}"
|
|
||||||
</literallayout>
|
|
||||||
</para></listitem>
|
|
||||||
</itemizedlist>
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<!--
|
|
||||||
<para>
|
|
||||||
In this example, we add some extra flags to the task, a name field ("deploy"), an
|
|
||||||
input directory where the task sends data, and the output
|
|
||||||
directory where the data from the task should eventually be copied.
|
|
||||||
We also add a <filename>_setscene</filename> variant of the task and add the task
|
|
||||||
name to the <filename>SSTATETASKS</filename> list.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
If you have a directory whose contents you need to preserve, you can do this with
|
|
||||||
a line like the following:
|
|
||||||
<literallayout class='monospaced'>
|
|
||||||
do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
|
|
||||||
</literallayout>
|
|
||||||
This method, as well as the following example, also works for multiple directories.
|
|
||||||
<literallayout class='monospaced'>
|
|
||||||
do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
|
|
||||||
do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
|
|
||||||
do_package[sstate-lockfile] = "${PACKAGELOCK}"
|
|
||||||
</literallayout>
|
|
||||||
These methods also include the ability to take a lockfile when manipulating
|
|
||||||
shared state directory structures since some cases are sensitive to file
|
|
||||||
additions or removals.
|
|
||||||
</para>
|
|
||||||
-->
|
|
||||||
|
|
||||||
<para>
|
|
||||||
Behind the scenes, the shared state code works by looking in
|
|
||||||
<link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link> and
|
|
||||||
<link linkend='var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></link>
|
|
||||||
for shared state files.
|
|
||||||
Here is an example:
|
|
||||||
<literallayout class='monospaced'>
|
|
||||||
SSTATE_MIRRORS ?= "\
|
|
||||||
file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
|
|
||||||
file://.* file:///some/local/dir/sstate/PATH"
|
|
||||||
</literallayout>
|
|
||||||
<note>
|
|
||||||
The shared state directory (<filename>SSTATE_DIR</filename>) is
|
|
||||||
organized into two-character subdirectories, where the subdirectory
|
|
||||||
names are based on the first two characters of the hash.
|
|
||||||
If the shared state directory structure for a mirror has the
|
|
||||||
same structure as <filename>SSTATE_DIR</filename>, you must
|
|
||||||
specify "PATH" as part of the URI to enable the build system
|
|
||||||
to map to the appropriate subdirectory.
|
|
||||||
</note>
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
The shared state package validity can be detected just by looking at the
|
|
||||||
filename since the filename contains the task checksum (or signature) as
|
|
||||||
described earlier in this section.
|
|
||||||
If a valid shared state package is found, the build process downloads it
|
|
||||||
and uses it to accelerate the task.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
The build processes use the <filename>*_setscene</filename> tasks
|
|
||||||
for the task acceleration phase.
|
|
||||||
BitBake goes through this phase before the main execution code and tries
|
|
||||||
to accelerate any tasks for which it can find shared state packages.
|
|
||||||
If a shared state package for a task is available, the shared state
|
|
||||||
package is used.
|
|
||||||
This means the task and any tasks on which it is dependent are not
|
|
||||||
executed.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
As a real world example, the aim is when building an IPK-based image,
|
|
||||||
only the
|
|
||||||
<link linkend='ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></link>
|
|
||||||
tasks would have their
|
|
||||||
shared state packages fetched and extracted.
|
|
||||||
Since the sysroot is not used, it would never get extracted.
|
|
||||||
This is another reason why a task-based approach is preferred over a
|
|
||||||
recipe-based approach, which would have to install the output from every task.
|
|
||||||
</para>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<section id='tips-and-tricks'>
|
|
||||||
<title>Tips and Tricks</title>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
The code in the build system that supports incremental builds is not
|
|
||||||
simple code.
|
|
||||||
This section presents some tips and tricks that help you work around
|
|
||||||
issues related to shared state code.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<section id='debugging'>
|
|
||||||
<title>Debugging</title>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
Seeing what metadata went into creating the input signature
|
|
||||||
of a shared state (sstate) task can be a useful debugging aid.
|
|
||||||
This information is available in signature information
|
|
||||||
(<filename>siginfo</filename>) files in
|
|
||||||
<link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>.
|
|
||||||
For information on how to view and interpret information in
|
|
||||||
<filename>siginfo</filename> files, see the
|
|
||||||
"<link linkend='usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</link>"
|
|
||||||
section.
|
|
||||||
</para>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<section id='invalidating-shared-state'>
|
|
||||||
<title>Invalidating Shared State</title>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
The OpenEmbedded build system uses checksums and shared state
|
|
||||||
cache to avoid unnecessarily rebuilding tasks.
|
|
||||||
Collectively, this scheme is known as "shared state code."
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
As with all schemes, this one has some drawbacks.
|
|
||||||
It is possible that you could make implicit changes to your
|
|
||||||
code that the checksum calculations do not take into
|
|
||||||
account.
|
|
||||||
These implicit changes affect a task's output but do not trigger
|
|
||||||
the shared state code into rebuilding a recipe.
|
|
||||||
Consider an example during which a tool changes its output.
|
|
||||||
Assume that the output of <filename>rpmdeps</filename> changes.
|
|
||||||
The result of the change should be that all the
|
|
||||||
<filename>package</filename> and
|
|
||||||
<filename>package_write_rpm</filename> shared state cache
|
|
||||||
items become invalid.
|
|
||||||
However, because the change to the output is
|
|
||||||
external to the code and therefore implicit,
|
|
||||||
the associated shared state cache items do not become
|
|
||||||
invalidated.
|
|
||||||
In this case, the build process uses the cached items rather
|
|
||||||
than running the task again.
|
|
||||||
Obviously, these types of implicit changes can cause problems.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
To avoid these problems during the build, you need to
|
|
||||||
understand the effects of any changes you make.
|
|
||||||
Realize that changes you make directly to a function
|
|
||||||
are automatically factored into the checksum calculation.
|
|
||||||
Thus, these explicit changes invalidate the associated area of
|
|
||||||
shared state cache.
|
|
||||||
However, you need to be aware of any implicit changes that
|
|
||||||
are not obvious changes to the code and could affect the output
|
|
||||||
of a given task.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<para>
|
|
||||||
When you identify an implicit change, you can easily take steps
|
|
||||||
to invalidate the cache and force the tasks to run.
|
|
||||||
The steps you can take are as simple as changing a function's
|
|
||||||
comments in the source code.
|
|
||||||
For example, to invalidate package shared state files, change
|
|
||||||
the comment statements of
|
|
||||||
<link linkend='ref-tasks-package'><filename>do_package</filename></link>
|
|
||||||
or the comments of one of the functions it calls.
|
|
||||||
Even though the change is purely cosmetic, it causes the
|
|
||||||
checksum to be recalculated and forces the OpenEmbedded build
|
|
||||||
system to run the task again.
|
|
||||||
</para>
|
|
||||||
|
|
||||||
<note>
|
|
||||||
For an example of a commit that makes a cosmetic change to
|
|
||||||
invalidate shared state, see this
|
|
||||||
<ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
|
|
||||||
</note>
|
|
||||||
</section>
|
|
||||||
</section>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<section id='automatically-added-runtime-dependencies'>
|
<section id='automatically-added-runtime-dependencies'>
|
||||||
<title>Automatically Added Runtime Dependencies</title>
|
<title>Automatically Added Runtime Dependencies</title>
|
||||||
|
|
||||||
|
|||||||
@@ -535,7 +535,7 @@
|
|||||||
${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
|
${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
|
||||||
</literallayout>
|
</literallayout>
|
||||||
For tasks that are accelerated through the shared state
|
For tasks that are accelerated through the shared state
|
||||||
(<link linkend='shared-state-cache'>sstate</link>)
|
(<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>)
|
||||||
cache, an additional <filename>siginfo</filename> file is
|
cache, an additional <filename>siginfo</filename> file is
|
||||||
written into
|
written into
|
||||||
<link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
|
<link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
|
||||||
|
|||||||
Reference in New Issue
Block a user