mirrors/poky
1
0
Fork 0
mirror of https://git.yoctoproject.org/poky synced 2026-06-02 13:29:49 +00:00
Code Issues Projects Releases Wiki Activity

sphinx: remove test-manual

Browse Source 
This is a new manual in YP 3.2, let's remove it from the 3.1 Sphinx backport.

(From yocto-docs rev: baf4410abad44b7f1fc6e4f1c445060b87c21c32)

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Nicolas Dechesne
2020-11-13 01:25:13 +01:00
committed by Richard Purdie
parent 8408c03763
commit 6f248db9a2
15 changed files with 1 additions and 3166 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/index.rst
-1
View File
@@ -33,7 +33,6 @@ Welcome to The Yocto Project's documentation!
Profile and Tracing Manual <profile-manual/profile-manual>
Application Development and the Extensible SDK (eSDK) <sdk-manual/sdk-manual>
Toaster Manual <toaster-manual/toaster-manual>
Test Environment Manual <test-manual/test-manual>
Bitbake User Manual <https://docs.yoctoproject.org/bitbake>
.. toctree::
documentation/overview-manual/overview-manual-yp-intro.rst
+1 -1
View File
@@ -401,7 +401,7 @@ activities using the Yocto Project:
benefit of the development community.
You can learn more about the AutoBuilder used by the Yocto Project
Autobuilder :doc:`here <../test-manual/test-manual-understand-autobuilder>`.
Autobuilder `here <&YOCTO_AB_URL;>`__.
- *Cross-Prelink:* Prelinking is the process of pre-computing the load
addresses and link tables generated by the dynamic linker as compared
documentation/test-manual/figures/ab-test-cluster.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 18 KiB

documentation/test-manual/figures/test-manual-title.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 15 KiB

documentation/test-manual/history.rst
-16
View File
@@ -1,16 +0,0 @@
.. SPDX-License-Identifier: CC-BY-2.0-UK
***********************
Manual Revision History
***********************
.. list-table::
:widths: 10 15 40
:header-rows: 1
* - Revision
- Date
- Note
* - 3.2
- October 2020
- The initial document released with the Yocto Project 3.2 Release
documentation/test-manual/test-manual-customization.xsl
-29
View File
@@ -1,29 +0,0 @@
<?xml version='1.0'?>
<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" xmlns:fo="http://www.w3.org/1999/XSL/Format" version="1.0">
<xsl:import href="http://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-xsl-1.76.1/xhtml/docbook.xsl" />
<!--
<xsl:import href="../template/1.76.1/docbook-xsl-1.76.1/xhtml/docbook.xsl" />
<xsl:import href="http://docbook.sourceforge.net/release/xsl/1.76.1/xhtml/docbook.xsl" />
-->
<xsl:include href="../template/permalinks.xsl"/>
<xsl:include href="../template/section.title.xsl"/>
<xsl:include href="../template/component.title.xsl"/>
<xsl:include href="../template/division.title.xsl"/>
<xsl:include href="../template/formal.object.heading.xsl"/>
<xsl:param name="html.stylesheet" select="'test-manual-style.css'" />
<xsl:param name="chapter.autolabel" select="1" />
<xsl:param name="appendix.autolabel" select="A" />
<xsl:param name="section.autolabel" select="1" />
<xsl:param name="section.label.includes.component.label" select="1" />
<xsl:param name="generate.id.attributes" select="1" />
</xsl:stylesheet>
documentation/test-manual/test-manual-intro.rst
-550
View File
@@ -1,550 +0,0 @@
.. SPDX-License-Identifier: CC-BY-2.0-UK
*****************************************
The Yocto Project Test Environment Manual
*****************************************
.. _test-welcome:
Welcome
=======
Welcome to the Yocto Project Test Environment Manual! This manual is a
work in progress. The manual contains information about the testing
environment used by the Yocto Project to make sure each major and minor
release works as intended. All the project's testing infrastructure and
processes are publicly visible and available so that the community can
see what testing is being performed, how it's being done and the current
status of the tests and the project at any given time. It is intended
that Other organizations can leverage off the process and testing
environment used by the Yocto Project to create their own automated,
production test environment, building upon the foundations from the
project core.
Currently, the Yocto Project Test Environment Manual has no projected
release date. This manual is a work-in-progress and is being initially
loaded with information from the README files and notes from key
engineers:
- *yocto-autobuilder2:* This
:yocto_git:`README.md </cgit.cgi/yocto-autobuilder2/tree/README.md>`
is the main README which detials how to set up the Yocto Project
Autobuilder. The ``yocto-autobuilder2`` repository represents the
Yocto Project's console UI plugin to Buildbot and the configuration
necessary to configure Buildbot to perform the testing the project
requires.
- *yocto-autobuilder-helper:* This :yocto_git:`README </cgit.cgi/yocto-autobuilder-helper/tree/README/>`
and repository contains Yocto Project Autobuilder Helper scripts and
configuration. The ``yocto-autobuilder-helper`` repository contains
the "glue" logic that defines which tests to run and how to run them.
As a result, it can be used by any Continuous Improvement (CI) system
to run builds, support getting the correct code revisions, configure
builds and layers, run builds, and collect results. The code is
independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__,
Jenkins, or others. This repository has a branch per release of the
project defining the tests to run on a per release basis.
.. _test-yocto-project-autobuilder-overview:
Yocto Project Autobuilder Overview
==================================
The Yocto Project Autobuilder collectively refers to the software,
tools, scripts, and procedures used by the Yocto Project to test
released software across supported hardware in an automated and regular
fashion. Basically, during the development of a Yocto Project release,
the Autobuilder tests if things work. The Autobuilder builds all test
targets and runs all the tests.
The Yocto Project uses now uses standard upstream
`Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__ (version 9) to
drive its integration and testing. Buildbot Nine has a plug-in interface
that the Yocto Project customizes using code from the
``yocto-autobuilder2`` repository, adding its own console UI plugin. The
resulting UI plug-in allows you to visualize builds in a way suited to
the project's needs.
A ``helper`` layer provides configuration and job management through
scripts found in the ``yocto-autobuilder-helper`` repository. The
``helper`` layer contains the bulk of the build configuration
information and is release-specific, which makes it highly customizable
on a per-project basis. The layer is CI system-agnostic and contains a
number of Helper scripts that can generate build configurations from
simple JSON files.
.. note::
The project uses Buildbot for historical reasons but also because
many of the project developers have knowledge of python. It is
possible to use the outer layers from another Continuous Integration
(CI) system such as
`Jenkins <https://en.wikipedia.org/wiki/Jenkins_(software)>`__
instead of Buildbot.
The following figure shows the Yocto Project Autobuilder stack with a
topology that includes a controller and a cluster of workers:
.. image:: figures/ab-test-cluster.png
:align: center
.. _test-project-tests:
Yocto Project Tests - Types of Testing Overview
===============================================
The Autobuilder tests different elements of the project by using
thefollowing types of tests:
- *Build Testing:* Tests whether specific configurations build by
varying :term:`MACHINE`,
:term:`DISTRO`, other configuration
options, and the specific target images being built (or world). Used
to trigger builds of all the different test configurations on the
Autobuilder. Builds usually cover many different targets for
different architectures, machines, and distributions, as well as
different configurations, such as different init systems. The
Autobuilder tests literally hundreds of configurations and targets.
- *Sanity Checks During the Build Process:* Tests initiated through
the :ref:`insane <ref-classes-insane>`
class. These checks ensure the output of the builds are correct.
For example, does the ELF architecture in the generated binaries
match the target system? ARM binaries would not work in a MIPS
system!
- *Build Performance Testing:* Tests whether or not commonly used steps
during builds work efficiently and avoid regressions. Tests to time
commonly used usage scenarios are run through ``oe-build-perf-test``.
These tests are run on isolated machines so that the time
measurements of the tests are accurate and no other processes
interfere with the timing results. The project currently tests
performance on two different distributions, Fedora and Ubuntu, to
ensure we have no single point of failure and can ensure the
different distros work effectively.
- *eSDK Testing:* Image tests initiated through the following command::
$ bitbake image -c testsdkext
The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task.
- *Feature Testing:* Various scenario-based tests are run through the
:ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions
we support.
- *Image Testing:* Image tests initiated through the following command::
$ bitbake image -c testimage
The tests utilize the :ref:`testimage* <ref-classes-testimage*>`
classes and the :ref:`ref-tasks-testimage` task.
- *Layer Testing:* The Autobuilder has the possibility to test whether
specific layers work with the test of the system. The layers tested
may be selected by members of the project. Some key community layers
are also tested periodically.
- *Package Testing:* A Package Test (ptest) runs tests against packages
built by the OpenEmbedded build system on the target machine. See the
:ref:`Testing Packages With
ptest <dev-manual/dev-manual-common-tasks:Testing Packages With ptest>` section
in the Yocto Project Development Tasks Manual and the
":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more
information on Ptest.
- *SDK Testing:* Image tests initiated through the following command::
$ bitbake image -c testsdk
The tests utilize the :ref:`testsdk <ref-classes-testsdk>` class and
the ``do_testsdk`` task.
- *Unit Testing:* Unit tests on various components of the system run
through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>` and
:ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>`.
- *Automatic Upgrade Helper:* This target tests whether new versions of
software are available and whether we can automatically upgrade to
those new versions. If so, this target emails the maintainers with a
patch to let them know this is possible.
.. _test-test-mapping:
How Tests Map to Areas of Code
==============================
Tests map into the codebase as follows:
- *bitbake-selftest:*
These tests are self-contained and test BitBake as well as its APIs,
which include the fetchers. The tests are located in
``bitbake/lib/*/tests``.
From within the BitBake repository, run the following::
$ bitbake-selftest
To skip tests that access the Internet, use the ``BB_SKIP_NETTEST``
variable when running "bitbake-selftest" as follows::
$ BB_SKIP_NETTEST=yes bitbake-selftest
The default output is quiet and just prints a summary of what was
run. To see more information, there is a verbose option::
$ bitbake-selftest -v
Use this option when you wish to skip tests that access the network,
which are mostly necessary to test the fetcher modules. To specify
individual test modules to run, append the test module name to the
"bitbake-selftest" command. For example, to specify the tests for the
bb.data.module, run::
$ bitbake-selftest bb.test.data.module
You can also specify individual tests by defining the full name and module
plus the class path of the test, for example::
$ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
The tests are based on `Python
unittest <https://docs.python.org/3/library/unittest.html>`__.
- *oe-selftest:*
- These tests use OE to test the workflows, which include testing
specific features, behaviors of tasks, and API unit tests.
- The tests can take advantage of parallelism through the "-j"
option, which can specify a number of threads to spread the tests
across. Note that all tests from a given class of tests will run
in the same thread. To parallelize large numbers of tests you can
split the class into multiple units.
- The tests are based on Python unittest.
- The code for the tests resides in
``meta/lib/oeqa/selftest/cases/``.
- To run all the tests, enter the following command::
$ oe-selftest -a
- To run a specific test, use the following command form where
testname is the name of the specific test::
$ oe-selftest -r <testname>
For example, the following command would run the tinfoil
getVar API test::
$ oe-selftest -r tinfoil.TinfoilTests.test_getvar
It is also possible to run a set
of tests. For example the following command will run all of the
tinfoil tests::
$ oe-selftest -r tinfoil
- *testimage:*
- These tests build an image, boot it, and run tests against the
image's content.
- The code for these tests resides in ``meta/lib/oeqa/runtime/cases/``.
- You need to set the :term:`IMAGE_CLASSES` variable as follows::
IMAGE_CLASSES += "testimage"
- Run the tests using the following command form::
$ bitbake image -c testimage
- *testsdk:*
- These tests build an SDK, install it, and then run tests against
that SDK.
- The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``.
- Run the test using the following command form::
$ bitbake image -c testsdk
- *testsdk_ext:*
- These tests build an extended SDK (eSDK), install that eSDK, and
run tests against the eSDK.
- The code for these tests resides in ``meta/lib/oeqa/esdk``.
- To run the tests, use the following command form::
$ bitbake image -c testsdkext
- *oe-build-perf-test:*
- These tests run through commonly used usage scenarios and measure
the performance times.
- The code for these tests resides in ``meta/lib/oeqa/buildperf``.
- To run the tests, use the following command form::
$ oe-build-perf-test <options>
The command takes a number of options,
such as where to place the test results. The Autobuilder Helper
Scripts include the ``build-perf-test-wrapper`` script with
examples of how to use the oe-build-perf-test from the command
line.
Use the ``oe-git-archive`` command to store test results into a
Git repository.
Use the ``oe-build-perf-report`` command to generate text reports
and HTML reports with graphs of the performance data. For
examples, see
:yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html`
and
:yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`.
- The tests are contained in ``lib/oeqa/buildperf/test_basic.py``.
Test Examples
=============
This section provides example tests for each of the tests listed in the
:ref:`test-manual/test-manual-intro:How Tests Map to Areas of Code` section.
For oeqa tests, testcases for each area reside in the main test
directory at ``meta/lib/oeqa/selftest/cases`` directory.
For oe-selftest. bitbake testcases reside in the ``lib/bb/tests/``
directory.
.. _bitbake-selftest-example:
``bitbake-selftest``
--------------------
A simple test example from ``lib/bb/tests/data.py`` is::
class DataExpansions(unittest.TestCase):
def setUp(self):
self.d = bb.data.init()
self.d["foo"] = "value_of_foo"
self.d["bar"] = "value_of_bar"
self.d["value_of_foo"] = "value_of_'value_of_foo'"
def test_one_var(self):
val = self.d.expand("${foo}")
self.assertEqual(str(val), "value_of_foo")
In this example, a ``DataExpansions`` class of tests is created,
derived from standard python unittest. The class has a common ``setUp``
function which is shared by all the tests in the class. A simple test is
then added to test that when a variable is expanded, the correct value
is found.
Bitbake selftests are straightforward python unittest. Refer to the
Python unittest documentation for additional information on writing
these tests at: https://docs.python.org/3/library/unittest.html.
.. _oe-selftest-example:
``oe-selftest``
---------------
These tests are more complex due to the setup required behind the scenes
for full builds. Rather than directly using Python's unittest, the code
wraps most of the standard objects. The tests can be simple, such as
testing a command from within the OE build environment using the
following example::
class BitbakeLayers(OESelftestTestCase):
def test_bitbakelayers_showcrossdepends(self):
result = runCmd('bitbake-layers show-cross-depends')
self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output)
This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``,
creates a testcase from the ``OESelftestTestCase`` class, derived
from ``unittest.TestCase``, which runs the ``bitbake-layers`` command
and checks the output to ensure it contains something we know should be
here.
The ``oeqa.utils.commands`` module contains Helpers which can assist
with common tasks, including:
- *Obtaining the value of a bitbake variable:* Use
``oeqa.utils.commands.get_bb_var()`` or use
``oeqa.utils.commands.get_bb_vars()`` for more than one variable
- *Running a bitbake invocation for a build:* Use
``oeqa.utils.commands.bitbake()``
- *Running a command:* Use ``oeqa.utils.commandsrunCmd()``
There is also a ``oeqa.utils.commands.runqemu()`` function for launching
the ``runqemu`` command for testing things within a running, virtualized
image.
You can run these tests in parallel. Parallelism works per test class,
so tests within a given test class should always run in the same build,
while tests in different classes or modules may be split into different
builds. There is no data store available for these tests since the tests
launch the ``bitbake`` command and exist outside of its context. As a
result, common bitbake library functions (bb.\*) are also unavailable.
.. _testimage-example:
``testimage``
-------------
These tests are run once an image is up and running, either on target
hardware or under QEMU. As a result, they are assumed to be running in a
target image environment, as opposed to a host build environment. A
simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
the following::
class PythonTest(OERuntimeTestCase):
@OETestDepends(['ssh.SSHTest.test_ssh'])
@OEHasPackage(['python3-core'])
def test_python3(self):
cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
status, output = self.target.run(cmd)
msg = 'Exit status was not 0. Output: %s' % output
self.assertEqual(status, 0, msg=msg)
In this example, the ``OERuntimeTestCase`` class wraps
``unittest.TestCase``. Within the test, ``self.target`` represents the
target system, where commands can be run on it using the ``run()``
method.
To ensure certain test or package dependencies are met, you can use the
``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test
in this example would only make sense if python3-core is installed in
the image.
.. _testsdk_ext-example:
``testsdk_ext``
---------------
These tests are run against built extensible SDKs (eSDKs). The tests can
assume that the eSDK environment has already been setup. An example from
``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
class DevtoolTest(OESDKExtTestCase):
@classmethod def setUpClass(cls):
myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
shutil.copytree(myapp_src, cls.myapp_dst)
subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
@classmethod
def tearDownClass(cls):
shutil.rmtree(cls.myapp_dst)
def _test_devtool_build(self, directory):
self._run('devtool add myapp %s' % directory)
try:
self._run('devtool build myapp')
finally:
self._run('devtool reset myapp')
def test_devtool_build_make(self):
self._test_devtool_build(self.myapp_dst)
In this example, the ``devtool``
command is tested to see whether a sample application can be built with
the ``devtool build`` command within the eSDK.
.. _testsdk-example:
``testsdk``
-----------
These tests are run against built SDKs. The tests can assume that an SDK
has already been extracted and its environment file has been sourced. A
simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the
following::
class Python3Test(OESDKTestCase):
def setUp(self):
if not (self.tc.hasHostPackage("nativesdk-python3-core") or
self.tc.hasHostPackage("python3-core-native")):
raise unittest.SkipTest("No python3 package in the SDK")
def test_python3(self):
cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
output = self._run(cmd)
self.assertEqual(output, "Hello, world\n")
In this example, if nativesdk-python3-core has been installed into the SDK, the code runs
the python3 interpreter with a basic command to check it is working
correctly. The test would only run if python3 is installed in the SDK.
.. _oe-build-perf-test-example:
``oe-build-perf-test``
----------------------
The performance tests usually measure how long operations take and the
resource utilisation as that happens. An example from
``meta/lib/oeqa/buildperf/test_basic.py`` contains the following::
class Test3(BuildPerfTestCase):
def test3(self):
"""Bitbake parsing (bitbake -p)"""
# Drop all caches and parse
self.rm_cache()
oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
'bitbake -p (no caches)')
# Drop tmp/cache
oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
'bitbake -p (no tmp/cache)')
# Parse with fully cached data
self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
'bitbake -p (cached)')
This example shows how three specific parsing timings are
measured, with and without various caches, to show how BitBake's parsing
performance trends over time.
.. _test-writing-considerations:
Considerations When Writing Tests
=================================
When writing good tests, there are several things to keep in mind. Since
things running on the Autobuilder are accessed concurrently by multiple
workers, consider the following:
**Running "cleanall" is not permitted.**
This can delete files from DL_DIR which would potentially break other
builds running in parallel. If this is required, DL_DIR must be set to
an isolated directory.
**Running "cleansstate" is not permitted.**
This can delete files from SSTATE_DIR which would potentially break
other builds running in parallel. If this is required, SSTATE_DIR must
be set to an isolated directory. Alternatively, you can use the "-f"
option with the ``bitbake`` command to "taint" tasks by changing the
sstate checksums to ensure sstate cache items will not be reused.
**Tests should not change the metadata.**
This is particularly true for oe-selftests since these can run in
parallel and changing metadata leads to changing checksums, which
confuses BitBake while running in parallel. If this is necessary, copy
layers to a temporary location and modify them. Some tests need to
change metadata, such as the devtool tests. To prevent the metadate from
changes, set up temporary copies of that data first.
documentation/test-manual/test-manual-intro.xml
-624
View File
@@ -1,624 +0,0 @@
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
<chapter id='test-manual-intro'>
<title>The Yocto Project Test Environment Manual</title>
<section id='test-welcome'>
<title>Welcome</title>
<para> Welcome to the Yocto Project Test Environment Manual! This manual is a work in
progress. The manual contains information about the testing environment used by the
Yocto Project to make sure each major and minor release works as intended. All the
project's testing infrastructure and processes are publicly visible and available so
that the community can see what testing is being performed, how it's being done and the
current status of the tests and the project at any given time. It is intended that Other
organizations can leverage off the process and testing environment used by the Yocto
Project to create their own automated, production test environment, building upon the
foundations from the project core. </para>
<para> Currently, the Yocto Project Test Environment Manual has no projected release date.
This manual is a work-in-progress and is being initially loaded with information from
the <ulink url="">README</ulink> files and notes from key engineers: <itemizedlist>
<listitem>
<para>
<emphasis><filename>yocto-autobuilder2</filename>:</emphasis> This <ulink
url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md"
><filename>README.md</filename></ulink> is the main README which
detials how to set up the Yocto Project Autobuilder. The
<filename>yocto-autobuilder2</filename> repository represents the Yocto
Project's console UI plugin to Buildbot and the configuration necessary to
configure Buildbot to perform the testing the project requires. </para>
</listitem>
<listitem>
<para>
<emphasis><filename>yocto-autobuilder-helper</filename>:</emphasis> This
<ulink
url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README"
><filename>README</filename></ulink> and repository contains Yocto
Project Autobuilder Helper scripts and configuration. The
<filename>yocto-autobuilder-helper</filename> repository contains the
"glue" logic that defines which tests to run and how to run them. As a
result, it can be used by any Continuous Improvement (CI) system to run
builds, support getting the correct code revisions, configure builds and
layers, run builds, and collect results. The code is independent of any CI
system, which means the code can work Buildbot, Jenkins, or others. This
repository has a branch per release of the project defining the tests to run
on a per release basis.</para>
</listitem>
</itemizedlist>
</para>
</section>
<section id='test-yocto-project-autobuilder-overview'>
<title>Yocto Project Autobuilder Overview</title>
<para>The Yocto Project Autobuilder collectively refers to the software, tools, scripts, and
procedures used by the Yocto Project to test released software across supported hardware
in an automated and regular fashion. Basically, during the development of a Yocto
Project release, the Autobuilder tests if things work. The Autobuilder builds all test
targets and runs all the tests. </para>
<para>The Yocto Project uses now uses standard upstream <ulink
url="https://docs.buildbot.net/0.9.15.post1/">Buildbot</ulink> (version 9) to drive
its integration and testing. Buildbot Nine has a plug-in interface that the Yocto
Project customizes using code from the <filename>yocto-autobuilder2</filename>
repository, adding its own console UI plugin. The resulting UI plug-in allows you to
visualize builds in a way suited to the project's needs.</para>
<para>A <filename>helper</filename> layer provides configuration and job management through
scripts found in the <filename>yocto-autobuilder-helper</filename> repository. The
<filename>helper</filename> layer contains the bulk of the build configuration
information and is release-specific, which makes it highly customizable on a per-project
basis. The layer is CI system-agnostic and contains a number of Helper scripts that can
generate build configurations from simple JSON files. <note>
<para>The project uses Buildbot for historical reasons but also because many of the
project developers have knowledge of python. It is possible to use the outer
layers from another Continuous Integration (CI) system such as <ulink
url="https://en.wikipedia.org/wiki/Jenkins_(software)">Jenkins</ulink>
instead of Buildbot. </para>
</note>
</para>
<para> The following figure shows the Yocto Project Autobuilder stack with a topology that
includes a controller and a cluster of workers: <imagedata
fileref="figures/ab-test-cluster.png" width="4.6in" depth="4.35in" align="center"
scalefit="1"/>
</para>
</section>
<section id='test-project-tests'>
<title>Yocto Project Tests - Types of Testing Overview</title>
<para>The Autobuilder tests different elements of the project by using thefollowing types of
tests: <itemizedlist>
<listitem>
<para>
<emphasis>Build Testing:</emphasis> Tests whether specific configurations
build by varying <ulink url="&YOCTO_DOCS_REF_URL;#var-MACHINE"
><filename>MACHINE</filename></ulink>, <ulink
url="&YOCTO_DOCS_REF_URL;#var-DISTRO"
><filename>DISTRO</filename></ulink>, other configuration options, and
the specific target images being built (or world). Used to trigger builds of
all the different test configurations on the Autobuilder. Builds usually
cover many different targets for different architectures, machines, and
distributions, as well as different configurations, such as different init
systems. The Autobuilder tests literally hundreds of configurations and
targets. <itemizedlist>
<listitem>
<para>
<emphasis>Sanity Checks During the Build Process:</emphasis>
Tests initiated through the <ulink
url="&YOCTO_DOCS_REF_URL;#ref-classes-insane"
><filename>insane</filename></ulink> class. These checks
ensure the output of the builds are correct. For example, does
the ELF architecture in the generated binaries match the target
system? ARM binaries would not work in a MIPS system! </para>
</listitem>
</itemizedlist></para>
</listitem>
<listitem>
<para>
<emphasis>Build Performance Testing:</emphasis> Tests whether or not
commonly used steps during builds work efficiently and avoid regressions.
Tests to time commonly used usage scenarios are run through
<filename>oe-build-perf-test</filename>. These tests are run on isolated
machines so that the time measurements of the tests are accurate and no
other processes interfere with the timing results. The project currently
tests performance on two different distributions, Fedora and Ubuntu, to
ensure we have no single point of failure and can ensure the different
distros work effectively. </para>
</listitem>
<listitem>
<para>
<emphasis>eSDK Testing:</emphasis> Image tests initiated through the
following command:
<literallayout class="monospaced">
$ bitbake <replaceable>image</replaceable> -c testsdkext
</literallayout>
The tests utilize the <filename>testsdkext</filename> class and the
<filename>do_testsdkext</filename> task. </para>
</listitem>
<listitem>
<para>
<emphasis>Feature Testing:</emphasis> Various scenario-based tests are run
through the <ulink url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
>OpenEmbedded Self-Test</ulink> (oe-selftest). We test oe-selftest on
each of the main distrubutions we support. </para>
</listitem>
<listitem>
<para>
<emphasis>Image Testing:</emphasis> Image tests initiated through the
following command:
<literallayout class="monospaced">
$ bitbake <replaceable>image</replaceable> -c testimage
</literallayout>
The tests utilize the <ulink
url="&YOCTO_DOCS_REF_URL;#ref-classes-testimage*"
><filename>testimage*</filename></ulink> classes and the <ulink
url="&YOCTO_DOCS_REF_URL;#ref-tasks-testimage"
><filename>do_testimage</filename></ulink> task. </para>
</listitem>
<listitem>
<para>
<emphasis>Layer Testing:</emphasis> The Autobuilder has the possibility to
test whether specific layers work with the test of the system. The layers
tested may be selected by members of the project. Some key community layers
are also tested periodically.</para>
</listitem>
<listitem>
<para>
<emphasis>Package Testing:</emphasis> A Package Test (ptest) runs tests
against packages built by the OpenEmbedded build system on the target
machine. See the "<ulink
url="&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest">Testing Packages
With ptest</ulink>" section in the Yocto Project Development Tasks
Manual and the "<ulink url="&YOCTO_WIKI_URL;/wiki/Ptest">Ptest</ulink>" Wiki
page for more information on Ptest. </para>
</listitem>
<listitem>
<para>
<emphasis>SDK Testing:</emphasis> Image tests initiated through the
following command:
<literallayout class="monospaced">
$ bitbake <replaceable>image</replaceable> -c testsdk
</literallayout>
The tests utilize the <ulink url="&YOCTO_DOCS_REF_URL;#ref-classes-testsdk"
><filename>testsdk</filename></ulink> class and the
<filename>do_testsdk</filename> task. </para>
</listitem>
<listitem>
<para>
<emphasis>Unit Testing:</emphasis> Unit tests on various components of the
system run through <filename>oe-selftest</filename> and <ulink
url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
><filename>bitbake-selftest</filename></ulink>. </para>
</listitem>
<listitem>
<para>
<emphasis>Automatic Upgrade Helper:</emphasis> This target tests whether new
versions of software are available and whether we can automatically upgrade
to those new versions. If so, this target emails the maintainers with a
patch to let them know this is possible.</para>
</listitem>
</itemizedlist>
</para>
</section>
<section id='test-test-mapping'>
<title>How Tests Map to Areas of Code</title>
<para>
Tests map into the codebase as follows:
<itemizedlist>
<listitem><para>
<emphasis>bitbake-selftest</emphasis>: </para>
<para>These tests are self-contained and test BitBake as well as its APIs, which
include the fetchers. The tests are located in
<filename>bitbake/lib/*/tests</filename>. </para>
<para>From within the BitBake repository, run the following:
<literallayout class="monospaced">
$ bitbake-selftest
</literallayout>
</para>
<para>To skip tests that access the Internet, use the
<filename>BB_SKIP_NETTEST</filename> variable when running
"bitbake-selftest" as follows:
<literallayout class="monospaced">
$ BB_SKIP_NETTEST=yes bitbake-selftest
</literallayout></para>
<para>The default output is quiet and just prints a summary of what was run. To
see more information, there is a verbose
option:<literallayout class="monospaced">
$ bitbake-selftest -v
</literallayout></para>
<para>Use this option when you wish to skip tests that access the network, which
are mostly necessary to test the fetcher modules. To specify individual test
modules to run, append the test module name to the "bitbake-selftest"
command. For example, to specify the tests for the bb.data.module, run:
<literallayout class="monospaced">
$ bitbake-selftest bb.test.data.module
</literallayout>You
can also specify individual tests by defining the full name and module plus
the class path of the test, for example:
<literallayout class="monospaced">
$ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
</literallayout></para>
<para>The tests are based on <ulink
url="https://docs.python.org/3/library/unittest.html">Python
unittest</ulink>. </para></listitem>
<listitem><para>
<emphasis>oe-selftest</emphasis>: <itemizedlist>
<listitem>
<para>These tests use OE to test the workflows, which include
testing specific features, behaviors of tasks, and API unit
tests. </para>
</listitem>
<listitem>
<para>The tests can take advantage of parallelism through the "-j"
option, which can specify a number of threads to spread the
tests across. Note that all tests from a given class of tests
will run in the same thread. To parallelize large numbers of
tests you can split the class into multiple units.</para>
</listitem>
<listitem>
<para>The tests are based on Python unittest. </para>
</listitem>
<listitem>
<para>The code for the tests resides in
<filename>meta/lib/oeqa/selftest/cases/</filename>. </para>
</listitem>
<listitem>
<para>To run all the tests, enter the following command:
<literallayout class="monospaced">
$ oe-selftest -a
</literallayout>
</para>
</listitem>
<listitem>
<para>To run a specific test, use the following command form where
<replaceable>testname</replaceable> is the name of the
specific test:
<literallayout class="monospaced">
$ oe-selftest -r <replaceable>testname</replaceable>
</literallayout>
For example, the following command would run the tinfoil getVar
API
test:<literallayout class="monospaced">
$ oe-selftest -r tinfoil.TinfoilTests.test_getvar
</literallayout>It
is also possible to run a set of tests. For example the
following command will run all of the tinfoil
tests:<literallayout class="monospaced">
$ oe-selftest -r tinfoil
</literallayout></para>
</listitem>
</itemizedlist>
</para></listitem>
<listitem><para>
<emphasis>testimage:</emphasis>
<itemizedlist>
<listitem><para>
These tests build an image, boot it, and run tests
against the image's content.
</para></listitem>
<listitem><para> The code for these tests resides in <filename>meta/lib/oeqa/runtime/cases/</filename>. </para></listitem>
<listitem><para>
You need to set the
<ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_CLASSES'><filename>IMAGE_CLASSES</filename></ulink>
variable as follows:
<literallayout class='monospaced'>
IMAGE_CLASSES += "testimage"
</literallayout>
</para></listitem>
<listitem><para>
Run the tests using the following command form:
<literallayout class='monospaced'>
$ bitbake <replaceable>image</replaceable> -c testimage
</literallayout>
</para></listitem>
</itemizedlist>
</para></listitem>
<listitem><para>
<emphasis>testsdk:</emphasis>
<itemizedlist>
<listitem><para>These tests build an SDK, install it, and then run tests against that SDK. </para></listitem>
<listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/sdk/cases/</filename>. </para></listitem>
<listitem><para>Run the test using the following command form:
<literallayout class="monospaced">
$ bitbake <replaceable>image</replaceable> -c testsdk
</literallayout>
</para></listitem>
</itemizedlist>
</para></listitem>
<listitem><para>
<emphasis>testsdk_ext:</emphasis>
<itemizedlist>
<listitem><para>These tests build an extended SDK (eSDK), install that eSDK, and run tests against the eSDK. </para></listitem>
<listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/esdk</filename>. </para></listitem>
<listitem><para>To run the tests, use the following command form:
<literallayout class="monospaced">
$ bitbake <replaceable>image</replaceable> -c testsdkext
</literallayout>
</para></listitem>
</itemizedlist>
</para></listitem>
<listitem><para>
<emphasis>oe-build-perf-test:</emphasis>
<itemizedlist>
<listitem><para>These tests run through commonly used usage scenarios and measure the performance times. </para></listitem>
<listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/buildperf</filename>. </para></listitem>
<listitem><para>To run the tests, use the following command form:
<literallayout class="monospaced">
$ oe-build-perf-test <replaceable>options</replaceable>
</literallayout>The
command takes a number of options, such as where to place the
test results. The Autobuilder Helper Scripts include the
<filename>build-perf-test-wrapper</filename> script with
examples of how to use the oe-build-perf-test from the command
line.</para>
<para>Use the <filename>oe-git-archive</filename> command to store
test results into a Git repository. </para>
<para>Use the <filename>oe-build-perf-report</filename> command to
generate text reports and HTML reports with graphs of the
performance data. For examples, see <link linkend=""
>http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html</link>
and <link linkend=""
>http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt</link>.</para></listitem>
<listitem>
<para>The tests are contained in
<filename>lib/oeqa/buildperf/test_basic.py</filename>.</para>
</listitem>
</itemizedlist>
</para></listitem>
</itemizedlist>
</para>
</section>
<section id='test-examples'>
<title>Test Examples</title>
<para>This section provides example tests for each of the tests listed in the <link
linkend="test-test-mapping">How Tests Map to Areas of Code</link> section. </para>
<para>For oeqa tests, testcases for each area reside in the main test directory at
<filename>meta/lib/oeqa/selftest/cases</filename> directory.</para>
<para>For oe-selftest. bitbake testcases reside in the <filename>lib/bb/tests/</filename>
directory. </para>
<section id='bitbake-selftest-example'>
<title><filename>bitbake-selftest</filename></title>
<para>A simple test example from <filename>lib/bb/tests/data.py</filename> is:
<literallayout class="monospaced">
class DataExpansions(unittest.TestCase):
def setUp(self):
self.d = bb.data.init()
self.d["foo"] = "value_of_foo"
self.d["bar"] = "value_of_bar"
self.d["value_of_foo"] = "value_of_'value_of_foo'"
def test_one_var(self):
val = self.d.expand("${foo}")
self.assertEqual(str(val), "value_of_foo")
</literallayout>
</para>
<para>In this example, a <ulink url=""><filename>DataExpansions</filename></ulink> class
of tests is created, derived from standard python unittest. The class has a common
<filename>setUp</filename> function which is shared by all the tests in the
class. A simple test is then added to test that when a variable is expanded, the
correct value is found.</para>
<para>Bitbake selftests are straightforward python unittest. Refer to the Python
unittest documentation for additional information on writing these tests at: <link
linkend="">https://docs.python.org/3/library/unittest.html</link>.</para>
</section>
<section id='oe-selftest-example'>
<title><filename>oe-selftest</filename></title>
<para>These tests are more complex due to the setup required behind the scenes for full
builds. Rather than directly using Python's unittest, the code wraps most of the
standard objects. The tests can be simple, such as testing a command from within the
OE build environment using the following
example:<literallayout class="monospaced">
class BitbakeLayers(OESelftestTestCase):
def test_bitbakelayers_showcrossdepends(self):
result = runCmd('bitbake-layers show-cross-depends')
self.assertTrue('aspell' in result.output, msg = "No dependencies
were shown. bitbake-layers show-cross-depends output:
%s"% result.output)
</literallayout></para>
<para>This example, taken from
<filename>meta/lib/oeqa/selftest/cases/bblayers.py</filename>, creates a
testcase from the <ulink url=""><filename>OESelftestTestCase</filename></ulink>
class, derived from <filename>unittest.TestCase</filename>, which runs the
<filename>bitbake-layers</filename> command and checks the output to ensure it
contains something we know should be here.</para>
<para>The <filename>oeqa.utils.commands</filename> module contains Helpers which can
assist with common tasks, including:<itemizedlist>
<listitem>
<para><emphasis>Obtaining the value of a bitbake variable:</emphasis> Use
<filename>oeqa.utils.commands.get_bb_var()</filename> or use
<filename>oeqa.utils.commands.get_bb_vars()</filename> for more than
one variable</para>
</listitem>
<listitem>
<para><emphasis>Running a bitbake invocation for a build:</emphasis> Use
<filename>oeqa.utils.commands.bitbake()</filename></para>
</listitem>
<listitem>
<para><emphasis>Running a command:</emphasis> Use
<filename>oeqa.utils.commandsrunCmd()</filename></para>
</listitem>
</itemizedlist></para>
<para>There is also a <filename>oeqa.utils.commands.runqemu()</filename> function for
launching the <filename>runqemu</filename> command for testing things within a
running, virtualized image.</para>
<para>You can run these tests in parallel. Parallelism works per test class, so tests
within a given test class should always run in the same build, while tests in
different classes or modules may be split into different builds. There is no data
store available for these tests since the tests launch the
<filename>bitbake</filename> command and exist outside of its context. As a
result, common bitbake library functions (bb.*) are also unavailable.</para>
</section>
<section id='testimage-example'>
<title><filename>testimage</filename></title>
<para>These tests are run once an image is up and running, either on target hardware or
under QEMU. As a result, they are assumed to be running in a target image
environment, as opposed to a host build environment. A simple example from
<filename>meta/lib/oeqa/runtime/cases/python.py</filename> contains the
following:<literallayout class="monospaced">
class PythonTest(OERuntimeTestCase):
@OETestDepends(['ssh.SSHTest.test_ssh'])
@OEHasPackage(['python3-core'])
def test_python3(self):
cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb,
jbeyq', 'rot13'))\""
status, output = self.target.run(cmd)
msg = 'Exit status was not 0. Output: %s' % output
self.assertEqual(status, 0, msg=msg)
</literallayout></para>
<para>In this example, the <ulink url=""><filename>OERuntimeTestCase</filename></ulink>
class wraps <filename>unittest.TestCase</filename>. Within the test,
<filename>self.target</filename> represents the target system, where commands
can be run on it using the <filename>run()</filename> method. </para>
<para>To ensure certain test or package dependencies are met, you can use the
<filename>OETestDepends</filename> and <filename>OEHasPackage</filename>
decorators. For example, the test in this example would only make sense if
python3-core is installed in the image.</para>
</section>
<section id='testsdk_ext-example'>
<title><filename>testsdk_ext</filename></title>
<para>These tests are run against built extensible SDKs (eSDKs). The tests can assume
that the eSDK environment has already been setup. An example from
<filename>meta/lib/oeqa/sdk/cases/devtool.py</filename> contains the
following:<literallayout class="monospaced">
class DevtoolTest(OESDKExtTestCase):
@classmethod
def setUpClass(cls):
myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
shutil.copytree(myapp_src, cls.myapp_dst)
subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
@classmethod
def tearDownClass(cls):
shutil.rmtree(cls.myapp_dst)
def _test_devtool_build(self, directory):
self._run('devtool add myapp %s' % directory)
try:
self._run('devtool build myapp')
finally:
self._run('devtool reset myapp')
def test_devtool_build_make(self):
self._test_devtool_build(self.myapp_dst)
</literallayout>In
this example, the <filename>devtool</filename> command is tested to see whether a
sample application can be built with the <filename>devtool build</filename> command
within the eSDK.</para>
</section>
<section id='testsdk-example'>
<title><filename>testsdk</filename></title>
<para>These tests are run against built SDKs. The tests can assume that an SDK has
already been extracted and its environment file has been sourced. A simple example
from <filename>meta/lib/oeqa/sdk/cases/python2.py</filename> contains the
following:<literallayout class="monospaced">
class Python3Test(OESDKTestCase):
def setUp(self):
if not (self.tc.hasHostPackage("nativesdk-python3-core") or
self.tc.hasHostPackage("python3-core-native")):
raise unittest.SkipTest("No python3 package in the SDK")
def test_python3(self):
cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
output = self._run(cmd)
self.assertEqual(output, "Hello, world\n")
</literallayout>In
this example, if nativesdk-python3-core has been installed into the SDK, the code
runs the python3 interpreter with a basic command to check it is working correctly.
The test would only run if python3 is installed in the SDK.</para>
</section>
<section id='oe-build-perf-test-example'>
<title><filename>oe-build-perf-test</filename></title>
<para>The performance tests usually measure how long operations take and the resource
utilisation as that happens. An example from
<filename>meta/lib/oeqa/buildperf/test_basic.py</filename> contains the
following:<literallayout class="monospaced">
class Test3(BuildPerfTestCase):
def test3(self):
"""Bitbake parsing (bitbake -p)"""
# Drop all caches and parse
self.rm_cache()
oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
'bitbake -p (no caches)')
# Drop tmp/cache
oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
'bitbake -p (no tmp/cache)')
# Parse with fully cached data
self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
'bitbake -p (cached)')
</literallayout>This
example shows how three specific parsing timings are measured, with and without
various caches, to show how BitBake's parsing performance trends over time.</para>
</section>
</section>
<section id='test-writing-considerations'>
<title>Considerations When Writing Tests</title>
<para>When writing good tests, there are several things to keep in mind. Since things
running on the Autobuilder are accessed concurrently by multiple workers, consider the
following:</para>
<formalpara>
<title>Running "cleanall" is not permitted</title>
<para>This can delete files from DL_DIR which would potentially break other builds
running in parallel. If this is required, DL_DIR must be set to an isolated
directory.</para>
</formalpara>
<formalpara>
<title>Running "cleansstate" is not permitted</title>
<para>This can delete files from SSTATE_DIR which would potentially break other builds
running in parallel. If this is required, SSTATE_DIR must be set to an isolated
directory. Alternatively, you can use the "-f" option with the
<filename>bitbake</filename> command to "taint" tasks by changing the sstate
checksums to ensure sstate cache items will not be reused.</para>
</formalpara>
<formalpara>
<title>Tests should not change the metadata</title>
<para>This is particularly true for oe-selftests since these can run in parallel and
changing metadata leads to changing checksums, which confuses BitBake while running
in parallel. If this is necessary, copy layers to a temporary location and modify
them. Some tests need to change metadata, such as the devtool tests. To prevent the
metadate from changes, set up temporary copies of that data first.</para>
</formalpara>
</section>
</chapter>
<!--
vim: expandtab tw=80 ts=4
-->
documentation/test-manual/test-manual-style.css
-991
View File
@@ -1,991 +0,0 @@
/*
Generic XHTML / DocBook XHTML CSS Stylesheet.
SPDX-License-Identifier: CC-BY-2.0-UK
Browser wrangling and typographic design by
Oyvind Kolas / pippin@gimp.org
Customised for Poky by
Matthew Allum / mallum@o-hand.com
Thanks to:
Liam R. E. Quin
William Skaggs
Jakub Steiner
Structure
---------
The stylesheet is divided into the following sections:
Positioning
Margins, paddings, width, font-size, clearing.
Decorations
Borders, style
Colors
Colors
Graphics
Graphical backgrounds
Nasty IE tweaks
Workarounds needed to make it work in internet explorer,
currently makes the stylesheet non validating, but up until
this point it is validating.
Mozilla extensions
Transparency for footer
Rounded corners on boxes
*/
/*************** /
/ Positioning /
/ ***************/
body {
font-family: Verdana, Sans, sans-serif;
min-width: 640px;
width: 80%;
margin: 0em auto;
padding: 2em 5em 5em 5em;
color: #333;
}
h1,h2,h3,h4,h5,h6,h7 {
font-family: Arial, Sans;
color: #00557D;
clear: both;
}
h1 {
font-size: 2em;
text-align: left;
padding: 0em 0em 0em 0em;
margin: 2em 0em 0em 0em;
}
h2.subtitle {
margin: 0.10em 0em 3.0em 0em;
padding: 0em 0em 0em 0em;
font-size: 1.8em;
padding-left: 20%;
font-weight: normal;
font-style: italic;
}
h2 {
margin: 2em 0em 0.66em 0em;
padding: 0.5em 0em 0em 0em;
font-size: 1.5em;
font-weight: bold;
}
h3.subtitle {
margin: 0em 0em 1em 0em;
padding: 0em 0em 0em 0em;
font-size: 142.14%;
text-align: right;
}
h3 {
margin: 1em 0em 0.5em 0em;
padding: 1em 0em 0em 0em;
font-size: 140%;
font-weight: bold;
}
h4 {
margin: 1em 0em 0.5em 0em;
padding: 1em 0em 0em 0em;
font-size: 120%;
font-weight: bold;
}
h5 {
margin: 1em 0em 0.5em 0em;
padding: 1em 0em 0em 0em;
font-size: 110%;
font-weight: bold;
}
h6 {
margin: 1em 0em 0em 0em;
padding: 1em 0em 0em 0em;
font-size: 110%;
font-weight: bold;
}
.authorgroup {
background-color: transparent;
background-repeat: no-repeat;
padding-top: 256px;
background-image: url("figures/test-manual-title.png");
background-position: left top;
margin-top: -256px;
padding-right: 50px;
margin-left: 0px;
text-align: right;
width: 740px;
}
h3.author {
margin: 0em 0me 0em 0em;
padding: 0em 0em 0em 0em;
font-weight: normal;
font-size: 100%;
color: #333;
clear: both;
}
.author tt.email {
font-size: 66%;
}
.titlepage hr {
width: 0em;
clear: both;
}
.revhistory {
padding-top: 2em;
clear: both;
}
.toc,
.list-of-tables,
.list-of-examples,
.list-of-figures {
padding: 1.33em 0em 2.5em 0em;
color: #00557D;
}
.toc p,
.list-of-tables p,
.list-of-figures p,
.list-of-examples p {
padding: 0em 0em 0em 0em;
padding: 0em 0em 0.3em;
margin: 1.5em 0em 0em 0em;
}
.toc p b,
.list-of-tables p b,
.list-of-figures p b,
.list-of-examples p b{
font-size: 100.0%;
font-weight: bold;
}
.toc dl,
.list-of-tables dl,
.list-of-figures dl,
.list-of-examples dl {
margin: 0em 0em 0.5em 0em;
padding: 0em 0em 0em 0em;
}
.toc dt {
margin: 0em 0em 0em 0em;
padding: 0em 0em 0em 0em;
}
.toc dd {
margin: 0em 0em 0em 2.6em;
padding: 0em 0em 0em 0em;
}
div.glossary dl,
div.variablelist dl {
}
.glossary dl dt,
.variablelist dl dt,
.variablelist dl dt span.term {
font-weight: normal;
width: 20em;
text-align: right;
}
.variablelist dl dt {
margin-top: 0.5em;
}
.glossary dl dd,
.variablelist dl dd {
margin-top: 0em;
margin-left: 25.5em;
}
.glossary dd p,
.variablelist dd p {
margin-top: 0em;
margin-bottom: 1em;
}
div.calloutlist table td {
padding: 0em 0em 0em 0em;
margin: 0em 0em 0em 0em;
}
div.calloutlist table td p {
margin-top: 0em;
margin-bottom: 1em;
}
div p.copyright {
text-align: left;
}
div.legalnotice p.legalnotice-title {
margin-bottom: 0em;
}
p {
line-height: 1.5em;
margin-top: 0em;
}
dl {
padding-top: 0em;
}
hr {
border: solid 1px;
}
.mediaobject,
.mediaobjectco {
text-align: center;
}
img {
border: none;
}
ul {
padding: 0em 0em 0em 1.5em;
}
ul li {
padding: 0em 0em 0em 0em;
}
ul li p {
text-align: left;
}
table {
width :100%;
}
th {
padding: 0.25em;
text-align: left;
font-weight: normal;
vertical-align: top;
}
td {
padding: 0.25em;
vertical-align: top;
}
p a[id] {
margin: 0px;
padding: 0px;
display: inline;
background-image: none;
}
a {
text-decoration: underline;
color: #444;
}
pre {
overflow: auto;
}
a:hover {
text-decoration: underline;
/*font-weight: bold;*/
}
/* This style defines how the permalink character
appears by itself and when hovered over with
the mouse. */
[alt='Permalink'] { color: #eee; }
[alt='Permalink']:hover { color: black; }
div.informalfigure,
div.informalexample,
div.informaltable,
div.figure,
div.table,
div.example {
margin: 1em 0em;
padding: 1em;
page-break-inside: avoid;
}
div.informalfigure p.title b,
div.informalexample p.title b,
div.informaltable p.title b,
div.figure p.title b,
div.example p.title b,
div.table p.title b{
padding-top: 0em;
margin-top: 0em;
font-size: 100%;
font-weight: normal;
}
.mediaobject .caption,
.mediaobject .caption p {
text-align: center;
font-size: 80%;
padding-top: 0.5em;
padding-bottom: 0.5em;
}
.epigraph {
padding-left: 55%;
margin-bottom: 1em;
}
.epigraph p {
text-align: left;
}
.epigraph .quote {
font-style: italic;
}
.epigraph .attribution {
font-style: normal;
text-align: right;
}
span.application {
font-style: italic;
}
.programlisting {
font-family: monospace;
font-size: 80%;
white-space: pre;
margin: 1.33em 0em;
padding: 1.33em;
}
.tip,
.warning,
.caution,
.note {
margin-top: 1em;
margin-bottom: 1em;
}
/* force full width of table within div */
.tip table,
.warning table,
.caution table,
.note table {
border: none;
width: 100%;
}
.tip table th,
.warning table th,
.caution table th,
.note table th {
padding: 0.8em 0.0em 0.0em 0.0em;
margin : 0em 0em 0em 0em;
}
.tip p,
.warning p,
.caution p,
.note p {
margin-top: 0.5em;
margin-bottom: 0.5em;
padding-right: 1em;
text-align: left;
}
.acronym {
text-transform: uppercase;
}
b.keycap,
.keycap {
padding: 0.09em 0.3em;
margin: 0em;
}
.itemizedlist li {
clear: none;
}
.filename {
font-size: medium;
font-family: Courier, monospace;
}
div.navheader, div.heading{
position: absolute;
left: 0em;
top: 0em;
width: 100%;
background-color: #cdf;
width: 100%;
}
div.navfooter, div.footing{
position: fixed;
left: 0em;
bottom: 0em;
background-color: #eee;
width: 100%;
}
div.navheader td,
div.navfooter td {
font-size: 66%;
}
div.navheader table th {
/*font-family: Georgia, Times, serif;*/
/*font-size: x-large;*/
font-size: 80%;
}
div.navheader table {
border-left: 0em;
border-right: 0em;
border-top: 0em;
width: 100%;
}
div.navfooter table {
border-left: 0em;
border-right: 0em;
border-bottom: 0em;
width: 100%;
}
div.navheader table td a,
div.navfooter table td a {
color: #777;
text-decoration: none;
}
/* normal text in the footer */
div.navfooter table td {
color: black;
}
div.navheader table td a:visited,
div.navfooter table td a:visited {
color: #444;
}
/* links in header and footer */
div.navheader table td a:hover,
div.navfooter table td a:hover {
text-decoration: underline;
background-color: transparent;
color: #33a;
}
div.navheader hr,
div.navfooter hr {
display: none;
}
.qandaset tr.question td p {
margin: 0em 0em 1em 0em;
padding: 0em 0em 0em 0em;
}
.qandaset tr.answer td p {
margin: 0em 0em 1em 0em;
padding: 0em 0em 0em 0em;
}
.answer td {
padding-bottom: 1.5em;
}
.emphasis {
font-weight: bold;
}
/************* /
/ decorations /
/ *************/
.titlepage {
}
.part .title {
}
.subtitle {
border: none;
}
/*
h1 {
border: none;
}
h2 {
border-top: solid 0.2em;
border-bottom: solid 0.06em;
}
h3 {
border-top: 0em;
border-bottom: solid 0.06em;
}
h4 {
border: 0em;
border-bottom: solid 0.06em;
}
h5 {
border: 0em;
}
*/
.programlisting {
border: solid 1px;
}
div.figure,
div.table,
div.informalfigure,
div.informaltable,
div.informalexample,
div.example {
border: 1px solid;
}
.tip,
.warning,
.caution,
.note {
border: 1px solid;
}
.tip table th,
.warning table th,
.caution table th,
.note table th {
border-bottom: 1px solid;
}
.question td {
border-top: 1px solid black;
}
.answer {
}
b.keycap,
.keycap {
border: 1px solid;
}
div.navheader, div.heading{
border-bottom: 1px solid;
}
div.navfooter, div.footing{
border-top: 1px solid;
}
/********* /
/ colors /
/ *********/
body {
color: #333;
background: white;
}
a {
background: transparent;
}
a:hover {
background-color: #dedede;
}
h1,
h2,
h3,
h4,
h5,
h6,
h7,
h8 {
background-color: transparent;
}
hr {
border-color: #aaa;
}
.tip, .warning, .caution, .note {
border-color: #fff;
}
.tip table th,
.warning table th,
.caution table th,
.note table th {
border-bottom-color: #fff;
}
.warning {
background-color: #f0f0f2;
}
.caution {
background-color: #f0f0f2;
}
.tip {
background-color: #f0f0f2;
}
.note {
background-color: #f0f0f2;
}
.glossary dl dt,
.variablelist dl dt,
.variablelist dl dt span.term {
color: #044;
}
div.figure,
div.table,
div.example,
div.informalfigure,
div.informaltable,
div.informalexample {
border-color: #aaa;
}
pre.programlisting {
color: black;
background-color: #fff;
border-color: #aaa;
border-width: 2px;
}
.guimenu,
.guilabel,
.guimenuitem {
background-color: #eee;
}
b.keycap,
.keycap {
background-color: #eee;
border-color: #999;
}
div.navheader {
border-color: black;
}
div.navfooter {
border-color: black;
}
.writernotes {
color: red;
}
/*********** /
/ graphics /
/ ***********/
/*
body {
background-image: url("images/body_bg.jpg");
background-attachment: fixed;
}
.navheader,
.note,
.tip {
background-image: url("images/note_bg.jpg");
background-attachment: fixed;
}
.warning,
.caution {
background-image: url("images/warning_bg.jpg");
background-attachment: fixed;
}
.figure,
.informalfigure,
.example,
.informalexample,
.table,
.informaltable {
background-image: url("images/figure_bg.jpg");
background-attachment: fixed;
}
*/
h1,
h2,
h3,
h4,
h5,
h6,
h7{
}
/*
Example of how to stick an image as part of the title.
div.article .titlepage .title
{
background-image: url("figures/white-on-black.png");
background-position: center;
background-repeat: repeat-x;
}
*/
div.preface .titlepage .title,
div.colophon .title,
div.chapter .titlepage .title,
div.article .titlepage .title
{
}
div.section div.section .titlepage .title,
div.sect2 .titlepage .title {
background: none;
}
h1.title {
background-color: transparent;
background-image: url("figures/test-title.png");
background-repeat: no-repeat;
height: 256px;
text-indent: -9000px;
overflow:hidden;
}
h2.subtitle {
background-color: transparent;
text-indent: -9000px;
overflow:hidden;
width: 0px;
display: none;
}
/*************************************** /
/ pippin.gimp.org specific alterations /
/ ***************************************/
/*
div.heading, div.navheader {
color: #777;
font-size: 80%;
padding: 0;
margin: 0;
text-align: left;
position: absolute;
top: 0px;
left: 0px;
width: 100%;
height: 50px;
background: url('/gfx/heading_bg.png') transparent;
background-repeat: repeat-x;
background-attachment: fixed;
border: none;
}
div.heading a {
color: #444;
}
div.footing, div.navfooter {
border: none;
color: #ddd;
font-size: 80%;
text-align:right;
width: 100%;
padding-top: 10px;
position: absolute;
bottom: 0px;
left: 0px;
background: url('/gfx/footing_bg.png') transparent;
}
*/
/****************** /
/ nasty ie tweaks /
/ ******************/
/*
div.heading, div.navheader {
width:expression(document.body.clientWidth + "px");
}
div.footing, div.navfooter {
width:expression(document.body.clientWidth + "px");
margin-left:expression("-5em");
}
body {
padding:expression("4em 5em 0em 5em");
}
*/
/**************************************** /
/ mozilla vendor specific css extensions /
/ ****************************************/
/*
div.navfooter, div.footing{
-moz-opacity: 0.8em;
}
div.figure,
div.table,
div.informalfigure,
div.informaltable,
div.informalexample,
div.example,
.tip,
.warning,
.caution,
.note {
-moz-border-radius: 0.5em;
}
b.keycap,
.keycap {
-moz-border-radius: 0.3em;
}
*/
table tr td table tr td {
display: none;
}
hr {
display: none;
}
table {
border: 0em;
}
.photo {
float: right;
margin-left: 1.5em;
margin-bottom: 1.5em;
margin-top: 0em;
max-width: 17em;
border: 1px solid gray;
padding: 3px;
background: white;
}
.seperator {
padding-top: 2em;
clear: both;
}
#validators {
margin-top: 5em;
text-align: right;
color: #777;
}
@media print {
body {
font-size: 8pt;
}
.noprint {
display: none;
}
}
.tip,
.note {
background: #f0f0f2;
color: #333;
padding: 20px;
margin: 20px;
}
.tip h3,
.note h3 {
padding: 0em;
margin: 0em;
font-size: 2em;
font-weight: bold;
color: #333;
}
.tip a,
.note a {
color: #333;
text-decoration: underline;
}
.footnote {
font-size: small;
color: #333;
}
/* Changes the announcement text */
.tip h3,
.warning h3,
.caution h3,
.note h3 {
font-size:large;
color: #00557D;
}
documentation/test-manual/test-manual-test-process.rst
-103
View File
@@ -1,103 +0,0 @@
.. SPDX-License-Identifier: CC-BY-2.0-UK
***********************************
Project Testing and Release Process
***********************************
.. _test-daily-devel:
Day to Day Development
======================
This section details how the project tests changes, through automation
on the Autobuilder or with the assistance of QA teams, through to making
releases.
The project aims to test changes against our test matrix before those
changes are merged into the master branch. As such, changes are queued
up in batches either in the ``master-next`` branch in the main trees, or
in user trees such as ``ross/mut`` in ``poky-contrib`` (Ross Burton
helps review and test patches and this is his testing tree).
We have two broad categories of test builds, including "full" and
"quick". On the Autobuilder, these can be seen as "a-quick" and
"a-full", simply for ease of sorting in the UI. Use our Autobuilder
console view to see where me manage most test-related items, available
at: :yocto_ab:`/typhoon/#/console`.
Builds are triggered manually when the test branches are ready. The
builds are monitored by the SWAT team. For additional information, see
:yocto_wiki:`/wiki/Yocto_Build_Failure_Swat_Team`.
If successful, the changes would usually be merged to the ``master``
branch. If not successful, someone would respond to the changes on the
mailing list explaining that there was a failure in testing. The choice
of quick or full would depend on the type of changes and the speed with
which the result was required.
The Autobuilder does build the ``master`` branch once daily for several
reasons, in particular, to ensure the current ``master`` branch does
build, but also to keep ``yocto-testresults``
(:yocto_git:`/cgit.cgi/yocto-testresults/`),
buildhistory
(:yocto_git:`/cgit.cgi/poky-buildhistory/`), and
our sstate up to date. On the weekend, there is a master-next build
instead to ensure the test results are updated for the less frequently
run targets.
Performance builds (buildperf-\* targets in the console) are triggered
separately every six hours and automatically push their results to the
buildstats repository at:
:yocto_git:`/cgit.cgi/yocto-buildstats/`.
The 'quick' targets have been selected to be the ones which catch the
most failures or give the most valuable data. We run 'fast' ptests in
this case for example but not the ones which take a long time. The quick
target doesn't include \*-lsb builds for all architectures, some world
builds and doesn't trigger performance tests or ltp testing. The full
build includes all these things and is slower but more comprehensive.
Release Builds
==============
The project typically has two major releases a year with a six month
cadence in April and October. Between these there would be a number of
milestone releases (usually four) with the final one being stablization
only along with point releases of our stable branches.
The build and release process for these project releases is similar to
that in `Day to Day Development <#test-daily-devel>`__, in that the
a-full target of the Autobuilder is used but in addition the form is
configured to generate and publish artefacts and the milestone number,
version, release candidate number and other information is entered. The
box to "generate an email to QA"is also checked.
When the build completes, an email is sent out using the send-qa-email
script in the ``yocto-autobuilder-helper`` repository to the list of
people configured for that release. Release builds are placed into a
directory in https://autobuilder.yocto.io/pub/releases on the
Autobuilder which is included in the email. The process from here is
more manual and control is effectively passed to release engineering.
The next steps include:
- QA teams respond to the email saying which tests they plan to run and
when the results will be available.
- QA teams run their tests and share their results in the yocto-
testresults-contrib repository, along with a summary of their
findings.
- Release engineering prepare the release as per their process.
- Test results from the QA teams are included into the release in
separate directories and also uploaded to the yocto-testresults
repository alongside the other test results for the given revision.
- The QA report in the final release is regenerated using resulttool to
include the new test results and the test summaries from the teams
(as headers to the generated report).
- The release is checked against the release checklist and release
readiness criteria.
- A final decision on whether to release is made by the YP TSC who have
final oversight on release readiness.
documentation/test-manual/test-manual-test-process.xml
-110
View File
@@ -1,110 +0,0 @@
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
<chapter id='test-manual-test-process'>
<title>Project Testing and Release Process</title>
<section id='test-daily-devel'>
<title>Day to Day Development</title>
<para>This section details how the project tests changes, through automation on the
Autobuilder or with the assistance of QA teams, through to making releases.</para>
<para>The project aims to test changes against our test matrix before those changes are
merged into the master branch. As such, changes are queued up in batches either in the
<filename>master-next</filename> branch in the main trees, or in user trees such as
<filename>ross/mut</filename> in <filename>poky-contrib</filename> (Ross Burton
helps review and test patches and this is his testing tree).</para>
<para>We have two broad categories of test builds, including "full" and "quick". On the
Autobuilder, these can be seen as "a-quick" and "a-full", simply for ease of sorting in
the UI. Use our Autobuilder console view to see where me manage most test-related items,
available at: <link linkend=""
>https://autobuilder.yoctoproject.org/typhoon/#/console</link>.</para>
<para>Builds are triggered manually when the test branches are ready. The builds are
monitored by the SWAT team. For additional information, see <link linkend=""
>https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team</link>. If
successful, the changes would usually be merged to the <filename>master</filename>
branch. If not successful, someone would respond to the changes on the mailing list
explaining that there was a failure in testing. The choice of quick or full would depend
on the type of changes and the speed with which the result was required.</para>
<para>The Autobuilder does build the <filename>master</filename> branch once daily for
several reasons, in particular, to ensure the current <filename>master</filename> branch
does build, but also to keep <filename>yocto-testresults</filename> (<link linkend=""
>http://git.yoctoproject.org/cgit.cgi/yocto-testresults/</link>), buildhistory
(<link linkend="">http://git.yoctoproject.org/cgit.cgi/poky-buildhistory/</link>),
and our sstate up to date. On the weekend, there is a master-next build instead to
ensure the test results are updated for the less frequently run targets.</para>
<para>Performance builds (buildperf-* targets in the console) are triggered separately every
six hours and automatically push their results to the buildstats repository at: <link
linkend="">http://git.yoctoproject.org/cgit.cgi/yocto-buildstats/</link>. </para>
<para>The 'quick' targets have been selected to be the ones which catch the most failures or
give the most valuable data. We run 'fast' ptests in this case for example but not the
ones which take a long time. The quick target doesn't include *-lsb builds for all
architectures, some world builds and doesn't trigger performance tests or ltp testing.
The full build includes all these things and is slower but more comprehensive.</para>
</section>
<section id='test-yocto-project-autobuilder-overview'>
<title>Release Builds</title>
<para>The project typically has two major releases a year with a six month cadence in April
and October. Between these there would be a number of milestone releases (usually four)
with the final one being stablization only along with point releases of our stable
branches.</para>
<para>The build and release process for these project releases is similar to that in <link
linkend="test-daily-devel">Day to Day Development</link>, in that the a-full target
of the Autobuilder is used but in addition the form is configured to generate and
publish artefacts and the milestone number, version, release candidate number and other
information is entered. The box to "generate an email to QA"is also checked.</para>
<para>When the build completes, an email is sent out using the send-qa-email script in the
<filename>yocto-autobuilder-helper</filename> repository to the list of people
configured for that release. Release builds are placed into a directory in <link
linkend="">https://autobuilder.yocto.io/pub/releases</link> on the Autobuilder which
is included in the email. The process from here is more manual and control is
effectively passed to release engineering. The next steps include:<itemizedlist>
<listitem>
<para>QA teams respond to the email saying which tests they plan to run and when
the results will be available.</para>
</listitem>
<listitem>
<para>QA teams run their tests and share their results in the yocto-
testresults-contrib repository, along with a summary of their findings.
</para>
</listitem>
<listitem>
<para>Release engineering prepare the release as per their process. </para>
</listitem>
<listitem>
<para>Test results from the QA teams are included into the release in separate
directories and also uploaded to the yocto-testresults repository alongside
the other test results for the given revision.</para>
</listitem>
<listitem>
<para>The QA report in the final release is regenerated using resulttool to
include the new test results and the test summaries from the teams (as
headers to the generated report).</para>
</listitem>
<listitem>
<para>The release is checked against the release checklist and release readiness
criteria.</para>
</listitem>
<listitem>
<para>A final decision on whether to release is made by the YP TSC who have
final oversight on release readiness.</para>
</listitem>
</itemizedlist></para>
</section>
</chapter>
<!--
vim: expandtab tw=80 ts=4
-->
documentation/test-manual/test-manual-understand-autobuilder.rst
-305
View File
@@ -1,305 +0,0 @@
.. SPDX-License-Identifier: CC-BY-2.0-UK
*******************************************
Understanding the Yocto Project Autobuilder
*******************************************
Execution Flow within the Autobuilder
=====================================
The "a-full" and "a-quick" targets are the usual entry points into the
Autobuilder and it makes sense to follow the process through the system
starting there. This is best visualised from the Autobuilder Console
view (:yocto_ab:`/typhoon/#/console`).
Each item along the top of that view represents some "target build" and
these targets are all run in parallel. The 'full' build will trigger the
majority of them, the "quick" build will trigger some subset of them.
The Autobuilder effectively runs whichever configuration is defined for
each of those targets on a seperate buildbot worker. To understand the
configuration, you need to look at the entry on ``config.json`` file
within the ``yocto-autobuilder-helper`` repository. The targets are
defined in the ‘overrides' section, a quick example could be qemux86-64
which looks like::
"qemux86-64" : {
"MACHINE" : "qemux86-64",
"TEMPLATE" : "arch-qemu",
"step1" : {
"extravars" : [
"IMAGE_FSTYPES_append = ' wic wic.bmap'"
]
}
},
And to expand that, you need the "arch-qemu" entry from
the "templates" section, which looks like::
"arch-qemu" : {
"BUILDINFO" : true,
"BUILDHISTORY" : true,
"step1" : {
"BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
"SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
},
"step2" : {
"SDKMACHINE" : "x86_64",
"BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",
"SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
},
"step3" : {
"BUILDHISTORY" : false,
"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
"ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
}
},
Combining these two entries you can see that "qemux86-64" is a three step build where the
``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for
``MACHINE="qemx86-64"`` but with differing SDKMACHINE settings. In step
1 an extra variable is added to the ``auto.conf`` file to enable wic
image generation.
While not every detail of this is covered here, you can see how the
template mechanism allows quite complex configurations to be built up
yet allows duplication and repetition to be kept to a minimum.
The different build targets are designed to allow for parallelisation,
so different machines are usually built in parallel, operations using
the same machine and metadata are built sequentially, with the aim of
trying to optimise build efficiency as much as possible.
The ``config.json`` file is processed by the scripts in the Helper
repository in the ``scripts`` directory. The following section details
how this works.
.. _test-autobuilder-target-exec-overview:
Autobuilder Target Execution Overview
=====================================
For each given target in a build, the Autobuilder executes several
steps. These are configured in ``yocto-autobuilder2/builders.py`` and
roughly consist of:
#. *Run clobberdir*.
This cleans out any previous build. Old builds are left around to
allow easier debugging of failed builds. For additional information,
see :ref:`test-manual/test-manual-understand-autobuilder:clobberdir`.
#. *Obtain yocto-autobuilder-helper*
This step clones the ``yocto-autobuilder-helper`` git repository.
This is necessary to prevent the requirement to maintain all the
release or project-specific code within Buildbot. The branch chosen
matches the release being built so we can support older releases and
still make changes in newer ones.
#. *Write layerinfo.json*
This transfers data in the Buildbot UI when the build was configured
to the Helper.
#. *Call scripts/shared-repo-unpack*
This is a call into the Helper scripts to set up a checkout of all
the pieces this build might need. It might clone the BitBake
repository and the OpenEmbedded-Core repository. It may clone the
Poky repository, as well as additional layers. It will use the data
from the ``layerinfo.json`` file to help understand the
configuration. It will also use a local cache of repositories to
speed up the clone checkouts. For additional information, see
:ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Clone Cache`.
This step has two possible modes of operation. If the build is part
of a parent build, its possible that all the repositories needed may
already be available, ready in a pre-prepared directory. An "a-quick"
or "a-full" build would prepare this before starting the other
sub-target builds. This is done for two reasons:
- the upstream may change during a build, for example, from a forced
push and this ensures we have matching content for the whole build
- if 15 Workers all tried to pull the same data from the same repos,
we can hit resource limits on upstream servers as they can think
they are under some kind of network attack
This pre-prepared directory is shared among the Workers over NFS. If
the build is an individual build and there is no "shared" directory
available, it would clone from the cache and the upstreams as
necessary. This is considered the fallback mode.
#. *Call scripts/run-config*
This is another call into the Helper scripts where its expected that
the main functionality of this target will be executed.
.. _test-autobuilder-tech:
Autobuilder Technology
======================
The Autobuilder has Yocto Project-specific functionality to allow builds
to operate with increased efficiency and speed.
.. _test-clobberdir:
clobberdir
----------
When deleting files, the Autobuilder uses ``clobberdir``, which is a
special script that moves files to a special location, rather than
deleting them. Files in this location are deleted by an ``rm`` command,
which is run under ``ionice -c 3``. For example, the deletion only
happens when there is idle IO capacity on the Worker. The Autobuilder
Worker Janitor runs this deletion. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`.
.. _test-autobuilder-clone-cache:
Autobuilder Clone Cache
-----------------------
Cloning repositories from scratch each time they are required was slow
on the Autobuilder. We therefore have a stash of commonly used
repositories pre-cloned on the Workers. Data is fetched from these
during clones first, then "topped up" with later revisions from any
upstream when necesary. The cache is maintained by the Autobuilder
Worker Janitor. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`.
.. _test-autobuilder-worker-janitor:
Autobuilder Worker Janitor
--------------------------
This is a process running on each Worker that performs two basic
operations, including background file deletion at IO idle (see :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and
maintainenance of a cache of cloned repositories to improve the speed
the system can checkout repositories.
.. _test-shared-dl-dir:
Shared DL_DIR
-------------
The Workers are all connected over NFS which allows DL_DIR to be shared
between them. This reduces network accesses from the system and allows
the build to be sped up. Usage of the directory within the build system
is designed to be able to be shared over NFS.
.. _test-shared-sstate-cache:
Shared SSTATE_DIR
-----------------
The Workers are all connected over NFS which allows the ``sstate``
directory to be shared between them. This means once a Worker has built
an artifact, all the others can benefit from it. Usage of the directory
within the directory is designed for sharing over NFS.
.. _test-resulttool:
Resulttool
----------
All of the different tests run as part of the build generate output into
``testresults.json`` files. This allows us to determine which tests ran
in a given build and their status. Additional information, such as
failure logs or the time taken to run the tests, may also be included.
Resulttool is part of OpenEmbedded-Core and is used to manipulate these
json results files. It has the ability to merge files together, display
reports of the test results and compare different result files.
For details, see :yocto_wiki:`/wiki/Resulttool`.
.. _test-run-config-tgt-execution:
run-config Target Execution
===========================
The ``scripts/run-config`` execution is where most of the work within
the Autobuilder happens. It runs through a number of steps; the first
are general setup steps that are run once and include:
#. Set up any ``buildtools-tarball`` if configured.
#. Call "buildhistory-init" if buildhistory is configured.
For each step that is configured in ``config.json``, it will perform the
following:
#. Add any layers that are specified using the
``bitbake-layers add-layer`` command (logging as stepXa)
#. Call the ``scripts/setup-config`` script to generate the necessary
``auto.conf`` configuration file for the build
#. Run the ``bitbake BBTARGETS`` command (logging as stepXb)
#. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc)
#. Run the ``EXTRACMDS`` command, which are run within the BitBake build
environment (logging as stepXd)
#. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the
BitBake build environment (logging as stepXd)
#. Remove any layers added in step
1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)
Once the execution steps above complete, ``run-config`` executes a set
of post-build steps, including:
#. Call ``scripts/publish-artifacts`` to collect any output which is to
be saved from the build.
#. Call ``scripts/collect-results`` to collect any test results to be
saved from the build.
#. Call ``scripts/upload-error-reports`` to send any error reports
generated to the remote server.
#. Cleanup the build directory using
:ref:`test-manual/test-manual-understand-autobuilder:clobberdir` if the build was successful,
else rename it to "build-renamed" for potential future debugging.
.. _test-deploying-yp-autobuilder:
Deploying Yocto Autobuilder
===========================
The most up to date information about how to setup and deploy your own
Autbuilder can be found in README.md in the ``yocto-autobuilder2``
repository.
We hope that people can use the ``yocto-autobuilder2`` code directly but
it is inevitable that users will end up needing to heavily customise the
``yocto-autobuilder-helper`` repository, particularly the
``config.json`` file as they will want to define their own test matrix.
The Autobuilder supports wo customization options:
- variable substitution
- overlaying configuration files
The standard ``config.json`` minimally attempts to allow substitution of
the paths. The Helper script repository includes a
``local-example.json`` file to show how you could override these from a
separate configuration file. Pass the following into the environment of
the Autobuilder::
$ ABHELPER_JSON="config.json local-example.json"
As another example, you could also pass the following into the
environment::
$ ABHELPER_JSON="config.json /some/location/local.json"
One issue users often run into is validation of the ``config.json`` files. A
tip for minimizing issues from invalid json files is to use a Git
``pre-commit-hook.sh`` script to verify the JSON file before committing
it. Create a symbolic link as follows::
$ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
documentation/test-manual/test-manual-understand-autobuilder.xml
-314
View File
@@ -1,314 +0,0 @@
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
<chapter id='test-manual-understand-autobuilder'>
<title>Understanding the Yocto Project Autobuilder</title>
<section>
<title>Execution Flow within the Autobuilder</title>
<para>The "a-full" and "a-quick" targets are the usual entry points into the Autobuilder and
it makes sense to follow the process through the system starting there. This is best
visualised from the Autobuilder Console view (<link linkend=""
>https://autobuilder.yoctoproject.org/typhoon/#/console</link>). </para>
<para>Each item along the top of that view represents some "target build" and these targets
are all run in parallel. The 'full' build will trigger the majority of them, the "quick"
build will trigger some subset of them. The Autobuilder effectively runs whichever
configuration is defined for each of those targets on a seperate buildbot worker. To
understand the configuration, you need to look at the entry on
<filename>config.json</filename> file within the
<filename>yocto-autobuilder-helper</filename> repository. The targets are defined in
the ‘overrides' section, a quick example could be qemux86-64 which looks
like:<literallayout class="monospaced">
"qemux86-64" : {
"MACHINE" : "qemux86-64",
"TEMPLATE" : "arch-qemu",
"step1" : {
"extravars" : [
"IMAGE_FSTYPES_append = ' wic wic.bmap'"
]
}
},
</literallayout>And
to expand that, you need the "arch-qemu" entry from the "templates" section, which looks
like:<literallayout class="monospaced">
"arch-qemu" : {
"BUILDINFO" : true,
"BUILDHISTORY" : true,
"step1" : {
"BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
"SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
},
"step2" : {
"SDKMACHINE" : "x86_64",
"BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",
"SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
},
"step3" : {
"BUILDHISTORY" : false,
"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
"ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
}
},
</literallayout>Combining
these two entries you can see that "qemux86-64" is a three step build where the
<filename>bitbake BBTARGETS</filename> would be run, then <filename>bitbake
SANITYTARGETS</filename> for each step; all for
<filename>MACHINE="qemx86-64"</filename> but with differing SDKMACHINE settings. In
step 1 an extra variable is added to the <filename>auto.conf</filename> file to enable
wic image generation.</para>
<para>While not every detail of this is covered here, you can see how the templating
mechanism allows quite complex configurations to be built up yet allows duplication and
repetition to be kept to a minimum.</para>
<para>The different build targets are designed to allow for parallelisation, so different
machines are usually built in parallel, operations using the same machine and metadata
are built sequentially, with the aim of trying to optimise build efficiency as much as
possible.</para>
<para>The <filename>config.json</filename> file is processed by the scripts in the Helper
repository in the <filename>scripts</filename> directory. The following section details
how this works.</para>
</section>
<section id='test-autobuilder-target-exec-overview'>
<title>Autobuilder Target Execution Overview</title>
<para>For each given target in a build, the Autobuilder executes several steps. These are
configured in <filename>yocto-autobuilder2/builders.py</filename> and roughly consist
of: <orderedlist>
<listitem id='test-list-tgt-exec-clobberdir'>
<para><emphasis>Run <filename>clobberdir</filename></emphasis></para>
<para>This cleans out any previous build. Old builds are left around to allow
easier debugging of failed builds. For additional information, see <link
linkend="test-clobberdir"><filename>clobberdir</filename></link>.</para>
</listitem>
<listitem>
<para><emphasis>Obtain yocto-autobuilder-helper</emphasis></para>
<para>This step clones the <filename>yocto-autobuilder-helper</filename> git
repository. This is necessary to prevent the requirement to maintain all the
release or project-specific code within Buildbot. The branch chosen matches
the release being built so we can support older releases and still make
changes in newer ones.</para>
</listitem>
<listitem>
<para><emphasis>Write layerinfo.json</emphasis></para>
<para>This transfers data in the Buildbot UI when the build was configured to
the Helper.</para>
</listitem>
<listitem>
<para><emphasis>Call scripts/shared-repo-unpack</emphasis></para>
<para>This is a call into the Helper scripts to set up a checkout of all the
pieces this build might need. It might clone the BitBake repository and the
OpenEmbedded-Core repository. It may clone the Poky repository, as well as
additional layers. It will use the data from the
<filename>layerinfo.json</filename> file to help understand the
configuration. It will also use a local cache of repositories to speed up
the clone checkouts. For additional information, see <link
linkend="test-autobuilder-clone-cache">Autobuilder Clone
Cache</link>.</para>
<para>This step has two possible modes of operation. If the build is part of a
parent build, its possible that all the repositories needed may already be
available, ready in a pre-prepared directory. An "a-quick" or "a-full" build
would prepare this before starting the other sub-target builds. This is done
for two reasons:<itemizedlist>
<listitem>
<para>the upstream may change during a build, for example, from a
forced push and this ensures we have matching content for the
whole build</para>
</listitem>
<listitem>
<para>if 15 Workers all tried to pull the same data from the same
repos, we can hit resource limits on upstream servers as they
can think they are under some kind of network attack</para>
</listitem>
</itemizedlist>This pre-prepared directory is shared among the Workers over
NFS. If the build is an individual build and there is no "shared" directory
available, it would clone from the cache and the upstreams as necessary.
This is considered the fallback mode.</para>
</listitem>
<listitem>
<para><emphasis>Call scripts/run-config</emphasis></para>
<para>This is another call into the Helper scripts where its expected that the
main functionality of this target will be executed.</para>
</listitem>
</orderedlist></para>
</section>
<section id='test-autobuilder-tech'>
<title>Autobuilder Technology</title>
<para>The Autobuilder has Yocto Project-specific functionality to allow builds to operate
with increased efficiency and speed.</para>
<section id='test-clobberdir'>
<title>clobberdir</title>
<para>When deleting files, the Autobuilder uses <filename>clobberdir</filename>, which
is a special script that moves files to a special location, rather than deleting
them. Files in this location are deleted by an <filename>rm</filename> command,
which is run under <filename>ionice -c 3</filename>. For example, the deletion only
happens when there is idle IO capacity on the Worker. The Autobuilder Worker Janitor
runs this deletion. See <link linkend="test-autobuilder-worker-janitor">Autobuilder
Worker Janitor</link>.</para>
</section>
<section id='test-autobuilder-clone-cache'>
<title>Autobuilder Clone Cache</title>
<para>Cloning repositories from scratch each time they are required was slow on the
Autobuilder. We therefore have a stash of commonly used repositories pre-cloned on
the Workers. Data is fetched from these during clones first, then "topped up" with
later revisions from any upstream when necesary. The cache is maintained by the
Autobuilder Worker Janitor. See <link linkend="test-autobuilder-worker-janitor"
>Autobuilder Worker Janitor</link>.</para>
</section>
<section id='test-autobuilder-worker-janitor'>
<title>Autobuilder Worker Janitor</title>
<para>This is a process running on each Worker that performs two basic operations,
including background file deletion at IO idle (see <link
linkend="test-list-tgt-exec-clobberdir">Target Execution: clobberdir</link>) and
maintainenance of a cache of cloned repositories to improve the speed the system can
checkout repositories.</para>
</section>
<section id='test-shared-dl-dir'>
<title>Shared DL_DIR</title>
<para>The Workers are all connected over NFS which allows DL_DIR to be shared between
them. This reduces network accesses from the system and allows the build to be sped
up. Usage of the directory within the build system is designed to be able to be
shared over NFS.</para>
</section>
<section id='test-shared-sstate-cache'>
<title>Shared SSTATE_DIR</title>
<para>The Workers are all connected over NFS which allows the
<filename>sstate</filename> directory to be shared between them. This means once
a Worker has built an artefact, all the others can benefit from it. Usage of the
directory within the directory is designed for sharing over NFS.</para>
</section>
<section id='test-resulttool'>
<title>Resulttool</title>
<para>All of the different tests run as part of the build generate output into
<filename>testresults.json</filename> files. This allows us to determine which
tests ran in a given build and their status. Additional information, such as failure
logs or the time taken to run the tests, may also be included.</para>
<para>Resulttool is part of OpenEmbedded-Core and is used to manipulate these json
results files. It has the ability to merge files together, display reports of the
test results and compare different result files.</para>
<para>For details, see <link linkend=""
>https://wiki.yoctoproject.org/wiki/Resulttool</link>.</para>
</section>
</section>
<section id='test-run-config-tgt-execution'>
<title>run-config Target Execution</title>
<para>The <filename>scripts/run-config</filename> execution is where most of the work within
the Autobuilder happens. It runs through a number of steps; the first are general setup
steps that are run once and include:<orderedlist>
<listitem>
<para>Set up any <filename>buildtools-tarball</filename> if configured.</para>
</listitem>
<listitem>
<para>Call "buildhistory-init" if buildhistory is configured.</para>
</listitem>
</orderedlist></para>
<para>For each step that is configured in <filename>config.json</filename>, it will perform
the following:</para>
<para>
<remark>## WRITER's question: What does "logging in as stepXa" and others refer to
below? ##</remark>
<orderedlist>
<listitem id="test-run-config-add-layers-step">
<para dir="ltr">Add any layers that are specified using the
<filename>bitbake-layers add-layer</filename> command (logging as
stepXa)</para>
</listitem>
<listitem>
<para dir="ltr">Call the <filename>scripts/setup-config</filename> script to
generate the necessary <filename>auto.conf</filename> configuration file for
the build</para>
</listitem>
<listitem>
<para dir="ltr">Run the <filename>bitbake BBTARGETS</filename> command (logging
as stepXb)</para>
</listitem>
<listitem>
<para dir="ltr">Run the <filename>bitbake SANITYTARGETS</filename> command
(logging as stepXc)</para>
</listitem>
<listitem>
<para dir="ltr">Run the <filename>EXTRACMDS</filename> command, which are run
within the BitBake build environment (logging as stepXd)</para>
</listitem>
<listitem>
<para dir="ltr">Run the <filename>EXTRAPLAINCMDS</filename> command(s), which
are run outside the BitBake build environment (logging as stepXd)</para>
</listitem>
<listitem>
<para dir="ltr">Remove any layers added in <link
linkend="test-run-config-add-layers-step">step 1</link> using the
<filename>bitbake-layers remove-layer</filename> command (logging as
stepXa)</para>
</listitem>
</orderedlist>
</para>
<para>Once the execution steps above complete, <filename>run-config</filename> executes a
set of post-build steps, including:<orderedlist>
<listitem>
<para dir="ltr">Call <filename>scripts/publish-artifacts</filename> to collect
any output which is to be saved from the build.</para>
</listitem>
<listitem>
<para dir="ltr">Call <filename>scripts/collect-results</filename> to collect any
test results to be saved from the build.</para>
</listitem>
<listitem>
<para dir="ltr">Call <filename>scripts/upload-error-reports</filename> to send
any error reports generated to the remote server.</para>
</listitem>
<listitem>
<para dir="ltr">Cleanup the build directory using <link
linkend="test-clobberdir"><filename>clobberdir</filename></link> if the
build was successful, else rename it to "build-renamed" for potential future
debugging.</para>
</listitem>
</orderedlist></para>
</section>
<section id='test-deploying-yp-autobuilder'>
<title>Deploying Yocto Autobuilder</title>
<para>The most up to date information about how to setup and deploy your own Autbuilder can
be found in README.md in the <filename>yocto-autobuilder2</filename> repository.</para>
<para>We hope that people can use the <filename>yocto-autobuilder2</filename> code directly
but it is inevitable that users will end up needing to heavily customise the
<filename>yocto-autobuilder-helper</filename> repository, particularly the
<filename>config.json</filename> file as they will want to define their own test
matrix.</para>
<para>The Autobuilder supports wo customization options: <itemizedlist>
<listitem>
<para>variable substitution</para>
</listitem>
<listitem>
<para>overlaying configuration files</para>
</listitem>
</itemizedlist>The standard <filename>config.json</filename> minimally attempts to allow
substitution of the paths. The Helper script repository includes a
<filename>local-example.json</filename> file to show how you could override these
from a separate configuration file. Pass the following into the environment of the
Autobuilder:<literallayout class="monospaced">
$ ABHELPER_JSON="config.json local-example.json"
</literallayout>As
another example, you could also pass the following into the
environment:<literallayout class="monospaced">
$ ABHELPER_JSON="config.json <replaceable>/some/location/</replaceable>local.json"
</literallayout>One
issue users often run into is validation of the <filename>config.json</filename> files.
A tip for minimizing issues from invalid json files is to use a Git
<filename>pre-commit-hook.sh</filename> script to verify the JSON file before
committing it. Create a symbolic link as
follows:<literallayout class="monospaced">
$ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
</literallayout></para>
</section>
</chapter>
<!--
vim: expandtab tw=80 ts=4
-->
documentation/test-manual/test-manual.rst
-18
View File
@@ -1,18 +0,0 @@
.. SPDX-License-Identifier: CC-BY-2.0-UK
=====================================
Yocto Project Test Environment Manual
=====================================
|
.. toctree::
:caption: Table of Contents
:numbered:
test-manual-intro
test-manual-test-process
test-manual-understand-autobuilder
history
.. include:: /boilerplate.rst
documentation/test-manual/test-manual.xml
-104
View File
@@ -1,104 +0,0 @@
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
<book id='test-manual' lang='en'
xmlns:xi="http://www.w3.org/2003/XInclude"
xmlns="http://docbook.org/ns/docbook"
>
<bookinfo>
<mediaobject>
<imageobject>
<imagedata fileref='figures/test-manual-title.png'
format='SVG'
align='left' scalefit='1' width='100%'/>
</imageobject>
</mediaobject>
<title>
Yocto Project Test Environment Manual
</title>
<authorgroup>
<author>
<affiliation>
<orgname>&ORGNAME;</orgname>
</affiliation>
<email>&ORGEMAIL;</email>
</author>
</authorgroup>
<revhistory>
<revision>
<revnumber>3.1.1</revnumber>
<date>TBD</date>
<revremark>DRAFT - Work-in-Progress - posted June 16, 2020</revremark>
</revision>
</revhistory>
<copyright>
<year>&COPYRIGHT_YEAR;</year>
<holder>Linux Foundation</holder>
</copyright>
<legalnotice>
<para>
Permission is granted to copy, distribute and/or modify this document under
the terms of the <ulink type="http" url="http://creativecommons.org/licenses/by-sa/2.0/uk/">
Creative Commons Attribution-Share Alike 2.0 UK: England &amp; Wales</ulink> as published by
Creative Commons.
</para>
<note><title>Manual Notes</title>
<itemizedlist>
<listitem><para>
This version of the
<emphasis>Yocto Project Test Environment Manual</emphasis>
is for the &YOCTO_DOC_VERSION; release of the
Yocto Project.
To be sure you have the latest version of the manual
for this release, go to the
<ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
and select the manual from that site.
Manuals from the site are more up-to-date than manuals
derived from the Yocto Project released TAR files.
</para></listitem>
<listitem><para>
If you located this manual through a web search, the
version of the manual might not be the one you want
(e.g. the search might have returned a manual much
older than the Yocto Project version with which you
are working).
You can see all Yocto Project major releases by
visiting the
<ulink url='&YOCTO_WIKI_URL;/wiki/Releases'>Releases</ulink>
page.
If you need a version of this manual for a different
Yocto Project release, visit the
<ulink url='&YOCTO_HOME_URL;/documentation'>Yocto Project documentation page</ulink>
and select the manual set by using the
"ACTIVE RELEASES DOCUMENTATION" or "DOCUMENTS ARCHIVE"
pull-down menus.
</para></listitem>
<listitem><para>
To report any inaccuracies or problems with this
manual, send an email to the Yocto Project
discussion group at
<filename>yocto@yoctoproject.com</filename> or log into
the freenode <filename>#yocto</filename> channel.
</para></listitem>
</itemizedlist>
</note>
</legalnotice>
</bookinfo>
<xi:include href="test-manual-intro.xml"/>
<xi:include href="test-manual-test-process.xml"/>
<xi:include href="test-manual-understand-autobuilder.xml"/>
</book>
<!--
vim: expandtab tw=80 ts=4
-->