mirror of
https://git.yoctoproject.org/poky
synced 2026-05-30 00:20:08 +00:00
overview-manual, ref-manual: Moved "Shared State Cache" to overview manual
Fixes [YOCTO #12370] The section on shared state cache needed to be in the overview manual and not in the ref-manual. I moved it. Some links were affected, which I fixed. (From yocto-docs rev: 1c4e5207bdde19d4b48ef42b1de81390d8a02d64) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Scott Rifenbark
committed by
Richard Purdie
Richard Purdie
parent
937b66e9d1
commit
00f87f8416
@@ -471,6 +471,641 @@
|
||||
</note>
|
||||
</section>
|
||||
|
||||
|
||||
|
||||
|
||||
<section id="shared-state-cache">
|
||||
<title>Shared State Cache</title>
|
||||
|
||||
<para>
|
||||
By design, the OpenEmbedded build system builds everything from
|
||||
scratch unless BitBake can determine that parts do not need to be
|
||||
rebuilt.
|
||||
Fundamentally, building from scratch is attractive as it means all
|
||||
parts are built fresh and there is no possibility of stale data
|
||||
causing problems.
|
||||
When developers hit problems, they typically default back to
|
||||
building from scratch so they know the state of things from the
|
||||
start.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Building an image from scratch is both an advantage and a
|
||||
disadvantage to the process.
|
||||
As mentioned in the previous paragraph, building from scratch
|
||||
ensures that everything is current and starts from a known state.
|
||||
However, building from scratch also takes much longer as it
|
||||
generally means rebuilding things that do not necessarily need
|
||||
to be rebuilt.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The Yocto Project implements shared state code that supports
|
||||
incremental builds.
|
||||
The implementation of the shared state code answers the following
|
||||
questions that were fundamental roadblocks within the OpenEmbedded
|
||||
incremental build support system:
|
||||
<itemizedlist>
|
||||
<listitem><para>
|
||||
What pieces of the system have changed and what pieces have
|
||||
not changed?
|
||||
</para></listitem>
|
||||
<listitem><para>
|
||||
How are changed pieces of software removed and replaced?
|
||||
</para></listitem>
|
||||
<listitem><para>
|
||||
How are pre-built components that do not need to be rebuilt
|
||||
from scratch used when they are available?
|
||||
</para></listitem>
|
||||
</itemizedlist>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
For the first question, the build system detects changes in the
|
||||
"inputs" to a given task by creating a checksum (or signature) of
|
||||
the task's inputs.
|
||||
If the checksum changes, the system assumes the inputs have changed
|
||||
and the task needs to be rerun.
|
||||
For the second question, the shared state (sstate) code tracks
|
||||
which tasks add which output to the build process.
|
||||
This means the output from a given task can be removed, upgraded
|
||||
or otherwise manipulated.
|
||||
The third question is partly addressed by the solution for the
|
||||
second question assuming the build system can fetch the sstate
|
||||
objects from remote locations and install them if they are deemed
|
||||
to be valid.
|
||||
<note>
|
||||
The OpenEmbedded build system does not maintain
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
|
||||
information as part of the shared state packages.
|
||||
Consequently, considerations exist that affect maintaining
|
||||
shared state feeds.
|
||||
For information on how the OpenEmbedded build system
|
||||
works with packages and can track incrementing
|
||||
<filename>PR</filename> information, see the
|
||||
"<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
|
||||
section in the Yocto Project Development Tasks Manual.
|
||||
</note>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The rest of this section goes into detail about the overall
|
||||
incremental build architecture, the checksums (signatures), shared
|
||||
state, and some tips and tricks.
|
||||
</para>
|
||||
|
||||
<section id='overall-architecture'>
|
||||
<title>Overall Architecture</title>
|
||||
|
||||
<para>
|
||||
When determining what parts of the system need to be built,
|
||||
BitBake works on a per-task basis rather than a per-recipe
|
||||
basis.
|
||||
You might wonder why using a per-task basis is preferred over
|
||||
a per-recipe basis.
|
||||
To help explain, consider having the IPK packaging backend
|
||||
enabled and then switching to DEB.
|
||||
In this case, the
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
|
||||
and
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
|
||||
task outputs are still valid.
|
||||
However, with a per-recipe approach, the build would not
|
||||
include the <filename>.deb</filename> files.
|
||||
Consequently, you would have to invalidate the whole build and
|
||||
rerun it.
|
||||
Rerunning everything is not the best solution.
|
||||
Also, in this case, the core must be "taught" much about
|
||||
specific tasks.
|
||||
This methodology does not scale well and does not allow users
|
||||
to easily add new tasks in layers or as external recipes
|
||||
without touching the packaged-staging core.
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='overview-checksums'>
|
||||
<title>Checksums (Signatures)</title>
|
||||
|
||||
<para>
|
||||
The shared state code uses a checksum, which is a unique
|
||||
signature of a task's inputs, to determine if a task needs to
|
||||
be run again.
|
||||
Because it is a change in a task's inputs that triggers a
|
||||
rerun, the process needs to detect all the inputs to a given
|
||||
task.
|
||||
For shell tasks, this turns out to be fairly easy because
|
||||
the build process generates a "run" shell script for each task
|
||||
and it is possible to create a checksum that gives you a good
|
||||
idea of when the task's data changes.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
To complicate the problem, there are things that should not be
|
||||
included in the checksum.
|
||||
First, there is the actual specific build path of a given
|
||||
task - the
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
|
||||
It does not matter if the work directory changes because it
|
||||
should not affect the output for target packages.
|
||||
Also, the build process has the objective of making native
|
||||
or cross packages relocatable.
|
||||
<note>
|
||||
Both native and cross packages run on the build host.
|
||||
However, cross packages generate output for the target
|
||||
architecture.
|
||||
</note>
|
||||
The checksum therefore needs to exclude
|
||||
<filename>WORKDIR</filename>.
|
||||
The simplistic approach for excluding the work directory is to
|
||||
set <filename>WORKDIR</filename> to some fixed value and
|
||||
create the checksum for the "run" script.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Another problem results from the "run" scripts containing
|
||||
functions that might or might not get called.
|
||||
The incremental build solution contains code that figures out
|
||||
dependencies between shell functions.
|
||||
This code is used to prune the "run" scripts down to the
|
||||
minimum set, thereby alleviating this problem and making the
|
||||
"run" scripts much more readable as a bonus.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
So far we have solutions for shell scripts.
|
||||
What about Python tasks?
|
||||
The same approach applies even though these tasks are more
|
||||
difficult.
|
||||
The process needs to figure out what variables a Python
|
||||
function accesses and what functions it calls.
|
||||
Again, the incremental build solution contains code that first
|
||||
figures out the variable and function dependencies, and then
|
||||
creates a checksum for the data used as the input to the task.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Like the <filename>WORKDIR</filename> case, situations exist
|
||||
where dependencies should be ignored.
|
||||
For these cases, you can instruct the build process to
|
||||
ignore a dependency by using a line like the following:
|
||||
<literallayout class='monospaced'>
|
||||
PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
|
||||
</literallayout>
|
||||
This example ensures that the
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink>
|
||||
variable does not depend on the value of
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>,
|
||||
even if it does reference it.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Equally, there are cases where we need to add dependencies
|
||||
BitBake is not able to find.
|
||||
You can accomplish this by using a line like the following:
|
||||
<literallayout class='monospaced'>
|
||||
PACKAGE_ARCHS[vardeps] = "MACHINE"
|
||||
</literallayout>
|
||||
This example explicitly adds the <filename>MACHINE</filename>
|
||||
variable as a dependency for
|
||||
<filename>PACKAGE_ARCHS</filename>.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Consider a case with in-line Python, for example, where
|
||||
BitBake is not able to figure out dependencies.
|
||||
When running in debug mode (i.e. using
|
||||
<filename>-DDD</filename>), BitBake produces output when it
|
||||
discovers something for which it cannot figure out dependencies.
|
||||
The Yocto Project team has currently not managed to cover
|
||||
those dependencies in detail and is aware of the need to fix
|
||||
this situation.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Thus far, this section has limited discussion to the direct
|
||||
inputs into a task.
|
||||
Information based on direct inputs is referred to as the
|
||||
"basehash" in the code.
|
||||
However, there is still the question of a task's indirect
|
||||
inputs - the things that were already built and present in the
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
|
||||
The checksum (or signature) for a particular task needs to add
|
||||
the hashes of all the tasks on which the particular task
|
||||
depends.
|
||||
Choosing which dependencies to add is a policy decision.
|
||||
However, the effect is to generate a master checksum that
|
||||
combines the basehash and the hashes of the task's
|
||||
dependencies.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
At the code level, there are a variety of ways both the
|
||||
basehash and the dependent task hashes can be influenced.
|
||||
Within the BitBake configuration file, we can give BitBake
|
||||
some extra information to help it construct the basehash.
|
||||
The following statement effectively results in a list of
|
||||
global variable dependency excludes - variables never
|
||||
included in any checksum:
|
||||
<literallayout class='monospaced'>
|
||||
BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
|
||||
SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
|
||||
USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
|
||||
PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
|
||||
CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
|
||||
</literallayout>
|
||||
The previous example excludes
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>
|
||||
since that variable is actually constructed as a path within
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>,
|
||||
which is on the whitelist.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The rules for deciding which hashes of dependent tasks to
|
||||
include through dependency chains are more complex and are
|
||||
generally accomplished with a Python function.
|
||||
The code in <filename>meta/lib/oe/sstatesig.py</filename> shows
|
||||
two examples of this and also illustrates how you can insert
|
||||
your own policy into the system if so desired.
|
||||
This file defines the two basic signature generators
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink>
|
||||
uses: "OEBasic" and "OEBasicHash".
|
||||
By default, there is a dummy "noop" signature handler enabled
|
||||
in BitBake.
|
||||
This means that behavior is unchanged from previous versions.
|
||||
OE-Core uses the "OEBasicHash" signature handler by default
|
||||
through this setting in the <filename>bitbake.conf</filename>
|
||||
file:
|
||||
<literallayout class='monospaced'>
|
||||
BB_SIGNATURE_HANDLER ?= "OEBasicHash"
|
||||
</literallayout>
|
||||
The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename>
|
||||
is the same as the "OEBasic" version but adds the task hash to
|
||||
the stamp files.
|
||||
This results in any
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>
|
||||
change that changes the task hash, automatically
|
||||
causing the task to be run again.
|
||||
This removes the need to bump
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
|
||||
values, and changes to Metadata automatically ripple across
|
||||
the build.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
It is also worth noting that the end result of these
|
||||
signature generators is to make some dependency and hash
|
||||
information available to the build.
|
||||
This information includes:
|
||||
<itemizedlist>
|
||||
<listitem><para>
|
||||
<filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
|
||||
The base hashes for each task in the recipe.
|
||||
</para></listitem>
|
||||
<listitem><para>
|
||||
<filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
|
||||
The base hashes for each dependent task.
|
||||
</para></listitem>
|
||||
<listitem><para>
|
||||
<filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
|
||||
The task dependencies for each task.
|
||||
</para></listitem>
|
||||
<listitem><para>
|
||||
<filename>BB_TASKHASH</filename>:
|
||||
The hash of the currently running task.
|
||||
</para></listitem>
|
||||
</itemizedlist>
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='shared-state'>
|
||||
<title>Shared State</title>
|
||||
|
||||
<para>
|
||||
Checksums and dependencies, as discussed in the previous
|
||||
section, solve half the problem of supporting a shared state.
|
||||
The other part of the problem is being able to use checksum
|
||||
information during the build and being able to reuse or rebuild
|
||||
specific components.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
|
||||
class is a relatively generic implementation of how to
|
||||
"capture" a snapshot of a given task.
|
||||
The idea is that the build process does not care about the
|
||||
source of a task's output.
|
||||
Output could be freshly built or it could be downloaded and
|
||||
unpacked from somewhere - the build process does not need to
|
||||
worry about its origin.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
There are two types of output, one is just about creating a
|
||||
directory in
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
|
||||
A good example is the output of either
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
|
||||
or
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>.
|
||||
The other type of output occurs when a set of data is merged
|
||||
into a shared directory tree such as the sysroot.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The Yocto Project team has tried to keep the details of the
|
||||
implementation hidden in <filename>sstate</filename> class.
|
||||
From a user's perspective, adding shared state wrapping to a task
|
||||
is as simple as this
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
|
||||
example taken from the
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink>
|
||||
class:
|
||||
<literallayout class='monospaced'>
|
||||
DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
|
||||
SSTATETASKS += "do_deploy"
|
||||
do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
|
||||
do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
|
||||
|
||||
python do_deploy_setscene () {
|
||||
sstate_setscene(d)
|
||||
}
|
||||
addtask do_deploy_setscene
|
||||
do_deploy[dirs] = "${DEPLOYDIR} ${B}"
|
||||
</literallayout>
|
||||
The following list explains the previous example:
|
||||
<itemizedlist>
|
||||
<listitem><para>
|
||||
Adding "do_deploy" to <filename>SSTATETASKS</filename>
|
||||
adds some required sstate-related processing, which is
|
||||
implemented in the
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
|
||||
class, to before and after the
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
|
||||
task.
|
||||
</para></listitem>
|
||||
<listitem><para>
|
||||
The
|
||||
<filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
|
||||
declares that <filename>do_deploy</filename> places its
|
||||
output in <filename>${DEPLOYDIR}</filename> when run
|
||||
normally (i.e. when not using the sstate cache).
|
||||
This output becomes the input to the shared state cache.
|
||||
</para></listitem>
|
||||
<listitem><para>
|
||||
The
|
||||
<filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
|
||||
line causes the contents of the shared state cache to be
|
||||
copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
|
||||
<note>
|
||||
If <filename>do_deploy</filename> is not already in
|
||||
the shared state cache or if its input checksum
|
||||
(signature) has changed from when the output was
|
||||
cached, the task will be run to populate the shared
|
||||
state cache, after which the contents of the shared
|
||||
state cache is copied to
|
||||
<filename>${DEPLOY_DIR_IMAGE}</filename>.
|
||||
If <filename>do_deploy</filename> is in the shared
|
||||
state cache and its signature indicates that the
|
||||
cached output is still valid (i.e. if no
|
||||
relevant task inputs have changed), then the
|
||||
contents of the shared state cache will be copied
|
||||
directly to
|
||||
<filename>${DEPLOY_DIR_IMAGE}</filename> by the
|
||||
<filename>do_deploy_setscene</filename> task
|
||||
instead, skipping the
|
||||
<filename>do_deploy</filename> task.
|
||||
</note>
|
||||
</para></listitem>
|
||||
<listitem><para>
|
||||
The following task definition is glue logic needed to
|
||||
make the previous settings effective:
|
||||
<literallayout class='monospaced'>
|
||||
python do_deploy_setscene () {
|
||||
sstate_setscene(d)
|
||||
}
|
||||
addtask do_deploy_setscene
|
||||
</literallayout>
|
||||
<filename>sstate_setscene()</filename> takes the flags
|
||||
above as input and accelerates the
|
||||
<filename>do_deploy</filename> task through the
|
||||
shared state cache if possible.
|
||||
If the task was accelerated,
|
||||
<filename>sstate_setscene()</filename> returns True.
|
||||
Otherwise, it returns False, and the normal
|
||||
<filename>do_deploy</filename> task runs.
|
||||
For more information, see the
|
||||
"<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
|
||||
section in the BitBake User Manual.
|
||||
</para></listitem>
|
||||
<listitem><para>
|
||||
The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
|
||||
line creates <filename>${DEPLOYDIR}</filename> and
|
||||
<filename>${B}</filename> before the
|
||||
<filename>do_deploy</filename> task runs, and also sets
|
||||
the current working directory of
|
||||
<filename>do_deploy</filename> to
|
||||
<filename>${B}</filename>.
|
||||
For more information, see the
|
||||
"<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
|
||||
section in the BitBake User Manual.
|
||||
<note>
|
||||
In cases where
|
||||
<filename>sstate-inputdirs</filename> and
|
||||
<filename>sstate-outputdirs</filename> would be the
|
||||
same, you can use
|
||||
<filename>sstate-plaindirs</filename>.
|
||||
For example, to preserve the
|
||||
<filename>${PKGD}</filename> and
|
||||
<filename>${PKGDEST}</filename> output from the
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
|
||||
task, use the following:
|
||||
<literallayout class='monospaced'>
|
||||
do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
|
||||
</literallayout>
|
||||
</note>
|
||||
</para></listitem>
|
||||
<listitem><para>
|
||||
<filename>sstate-inputdirs</filename> and
|
||||
<filename>sstate-outputdirs</filename> can also be used
|
||||
with multiple directories.
|
||||
For example, the following declares
|
||||
<filename>PKGDESTWORK</filename> and
|
||||
<filename>SHLIBWORK</filename> as shared state
|
||||
input directories, which populates the shared state
|
||||
cache, and <filename>PKGDATA_DIR</filename> and
|
||||
<filename>SHLIBSDIR</filename> as the corresponding
|
||||
shared state output directories:
|
||||
<literallayout class='monospaced'>
|
||||
do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
|
||||
do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
|
||||
</literallayout>
|
||||
</para></listitem>
|
||||
<listitem><para>
|
||||
These methods also include the ability to take a
|
||||
lockfile when manipulating shared state directory
|
||||
structures, for cases where file additions or removals
|
||||
are sensitive:
|
||||
<literallayout class='monospaced'>
|
||||
do_package[sstate-lockfile] = "${PACKAGELOCK}"
|
||||
</literallayout>
|
||||
</para></listitem>
|
||||
</itemizedlist>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Behind the scenes, the shared state code works by looking in
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
|
||||
and
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
|
||||
for shared state files.
|
||||
Here is an example:
|
||||
<literallayout class='monospaced'>
|
||||
SSTATE_MIRRORS ?= "\
|
||||
file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
|
||||
file://.* file:///some/local/dir/sstate/PATH"
|
||||
</literallayout>
|
||||
<note>
|
||||
The shared state directory
|
||||
(<filename>SSTATE_DIR</filename>) is organized into
|
||||
two-character subdirectories, where the subdirectory
|
||||
names are based on the first two characters of the hash.
|
||||
If the shared state directory structure for a mirror has the
|
||||
same structure as <filename>SSTATE_DIR</filename>, you must
|
||||
specify "PATH" as part of the URI to enable the build system
|
||||
to map to the appropriate subdirectory.
|
||||
</note>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The shared state package validity can be detected just by
|
||||
looking at the filename since the filename contains the task
|
||||
checksum (or signature) as described earlier in this section.
|
||||
If a valid shared state package is found, the build process
|
||||
downloads it and uses it to accelerate the task.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The build processes use the <filename>*_setscene</filename>
|
||||
tasks for the task acceleration phase.
|
||||
BitBake goes through this phase before the main execution
|
||||
code and tries to accelerate any tasks for which it can find
|
||||
shared state packages.
|
||||
If a shared state package for a task is available, the
|
||||
shared state package is used.
|
||||
This means the task and any tasks on which it is dependent
|
||||
are not executed.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
As a real world example, the aim is when building an IPK-based
|
||||
image, only the
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>
|
||||
tasks would have their shared state packages fetched and
|
||||
extracted.
|
||||
Since the sysroot is not used, it would never get extracted.
|
||||
This is another reason why a task-based approach is preferred
|
||||
over a recipe-based approach, which would have to install the
|
||||
output from every task.
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='tips-and-tricks'>
|
||||
<title>Tips and Tricks</title>
|
||||
|
||||
<para>
|
||||
The code in the build system that supports incremental builds
|
||||
is not simple code.
|
||||
This section presents some tips and tricks that help you work
|
||||
around issues related to shared state code.
|
||||
</para>
|
||||
|
||||
<section id='overview-debugging'>
|
||||
<title>Debugging</title>
|
||||
|
||||
<para>
|
||||
Seeing what metadata went into creating the input signature
|
||||
of a shared state (sstate) task can be a useful debugging
|
||||
aid.
|
||||
This information is available in signature information
|
||||
(<filename>siginfo</filename>) files in
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>.
|
||||
For information on how to view and interpret information in
|
||||
<filename>siginfo</filename> files, see the
|
||||
"<ulink url='&YOCTO_DOCS_REF_URL;#usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>"
|
||||
section in the Yocto Project Reference Manual.
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='invalidating-shared-state'>
|
||||
<title>Invalidating Shared State</title>
|
||||
|
||||
<para>
|
||||
The OpenEmbedded build system uses checksums and shared
|
||||
state cache to avoid unnecessarily rebuilding tasks.
|
||||
Collectively, this scheme is known as "shared state code."
|
||||
</para>
|
||||
|
||||
<para>
|
||||
As with all schemes, this one has some drawbacks.
|
||||
It is possible that you could make implicit changes to your
|
||||
code that the checksum calculations do not take into
|
||||
account.
|
||||
These implicit changes affect a task's output but do not
|
||||
trigger the shared state code into rebuilding a recipe.
|
||||
Consider an example during which a tool changes its output.
|
||||
Assume that the output of <filename>rpmdeps</filename>
|
||||
changes.
|
||||
The result of the change should be that all the
|
||||
<filename>package</filename> and
|
||||
<filename>package_write_rpm</filename> shared state cache
|
||||
items become invalid.
|
||||
However, because the change to the output is
|
||||
external to the code and therefore implicit,
|
||||
the associated shared state cache items do not become
|
||||
invalidated.
|
||||
In this case, the build process uses the cached items
|
||||
rather than running the task again.
|
||||
Obviously, these types of implicit changes can cause
|
||||
problems.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
To avoid these problems during the build, you need to
|
||||
understand the effects of any changes you make.
|
||||
Realize that changes you make directly to a function
|
||||
are automatically factored into the checksum calculation.
|
||||
Thus, these explicit changes invalidate the associated
|
||||
area of shared state cache.
|
||||
However, you need to be aware of any implicit changes that
|
||||
are not obvious changes to the code and could affect
|
||||
the output of a given task.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
When you identify an implicit change, you can easily
|
||||
take steps to invalidate the cache and force the tasks
|
||||
to run.
|
||||
The steps you can take are as simple as changing a
|
||||
function's comments in the source code.
|
||||
For example, to invalidate package shared state files,
|
||||
change the comment statements of
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
|
||||
or the comments of one of the functions it calls.
|
||||
Even though the change is purely cosmetic, it causes the
|
||||
checksum to be recalculated and forces the OpenEmbedded
|
||||
build system to run the task again.
|
||||
<note>
|
||||
For an example of a commit that makes a cosmetic
|
||||
change to invalidate shared state, see this
|
||||
<ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
|
||||
</note>
|
||||
</para>
|
||||
</section>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
<section id='x32'>
|
||||
<title>x32 psABI</title>
|
||||
|
||||
|
||||
@@ -2490,7 +2490,7 @@
|
||||
<ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink>
|
||||
variable, and the end of the name consists of the task's name
|
||||
and current
|
||||
<ulink url='&YOCTO_DOCS_BB_URL;#checksums'>input checksum</ulink>.
|
||||
<link linkend='overview-checksums'>input checksum</link>.
|
||||
<note>
|
||||
This naming scheme assumes that
|
||||
<ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
|
||||
|
||||
Reference in New Issue
Block a user