diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml index 7f59c39f2d..2dbc3d14b8 100644 --- a/documentation/dev-manual/dev-manual-common-tasks.xml +++ b/documentation/dev-manual/dev-manual-common-tasks.xml @@ -4252,6 +4252,420 @@ +
+ Running and Writing Tests for a QEMU Image + + + The OpenEmbedded build system makes available a series of automated + tests for QEMU images. + These tests are commands that run on the target system over + ssh and are written in Python, thus making + use of the unittest module. + + + + This section describes how you set up the environment to use these + tests, run available tests, and write and add your own tests. + You can the same information in a different form and with example + log files of test runs on the + Image Tests + Wiki page. + + +
+ Enabling Tests + + + In order to run tests, you need to do the following: + + Ensure you run the test + automatically with no interaction and under + sudo: + To accomplish this, you must do one of the + following: + + Add + NOPASSWD for your user + in /etc/sudoers either for + ALL commands or just for + runqemu-ifup. + You must provide the full path as that can + change if you are using multiple clones of the + source repository. + + On some distributions, you also need to + comment out "Defaults requiretty" in + /etc/sudoers. + + Manually configure a tap interface + for your system. + Run as root the script in + scripts/runqemu-gen-tapdev, + which should generate a list of tap devices. + List generation is usually done in environments + similar to AutoBuilder. + + Set up the + DISPLAY variable: + You need to set this variable so that you have an X + server available (e.g. start + vncserver for a headless machine). + + Be sure your host's firewall + accepts incoming connections from + 192.168.7.0/24: + Some of the tests (in particular smart tests) start a + HTTP server on a random high number port, which is + used to serve files to the target. + The smart module serves + ${DEPLOY_DIR}/rpm so it can run + smart channel commands. That means your host's firewall + must accept incoming connections from 192.168.7.0/24, + which is the default class used for + tap0 devices by + runqemu. + Have your QEMU image built: + The QEMU image on which you want to run tests needs to + be built. + For example, the following command would build the + core-image-sato image when + MACHINE is set to "qemu". + + bitbake core-image-sato + + Globally inherit + testimage.class: + Edit your local.conf file as + follows: + + INHERIT += "testimage" + + + + + + If you built your QEMU image using rm_work, + most of the tests will fail with errors because they rely on + ${WORKDIR}/installed_pkgs.txt. + +
+ +
+ Running Tests + + + After setting up to run the tests, use BitBake to start + the standard suite of tests: + + bitbake core-image-sato -c testimage + + + + + Once you start the tests, the following happens: + + A copy of the root filesystem is written + to ${WORKDIR}/testimage. + + The image is booted under QEMU using the + standard runqemu script. + + A default timeout of 500 seconds occurs + to allow for the boot process to reach the login prompt. + You can change the timeout period by setting + TEST_QEMUBOOT_TIMEOUT + in the local.conf file. + + Once the boot process is reached and the + login prompt appears, the tests run. + The full boot log is written to + ${WORKDIR}/testimage/qemu_boot_log. + + Each test module loads in the order found + in TEST_SUITES. + You can find the full output of the commands run over + ssh in + ${WORKDIR}/testimgage/ssh_target_log. + + If no failures occur, the task running the + tests ends successfully. + You can find the output from the + unittest in the task log at + ${WORKDIR}/temp/log.do_testimage. + + + + + + All test files reside in + meta/lib/oeqa/runtime in the + Source Directory. + Tests (also referred to as modules) have a one-to-one + relationship with the filenames in runtime. + Modules can have multiple classes and test methods and are + usually grouped together by the area tested (e.g tests for + systemd reside in + meta/lib/oeqa/runtime/systemd.py). + + + + You can add tests to any layer provided you place them in the + proper area and you extend + BBPATH + in the local.conf file. + Be sure that tests reside in + <layer>/lib/oeqa/runtime. + + Be sure that module names do not collide with module names + provided by the Yocto Project. + + + + + You can change the set of tests run by appending or overriding + TEST_SUITES + variable in local.conf. + Each name in TEST_SUITES represents a + required test for the image. + Module skipping is not allowed even if a test is not suitable + for an image (e.g. running the rpm tests on + an image without rpm). + Appending "auto" to TEST_SUITES causes the + build system to try to run all tests that are suitable for the + image (i.e. the build system decides whether to run each test). + + + + The order you list tests in TEST_SUITES + is important. + The order influences test dependencies. + Consequently, tests that depend on other tests should be added + after the test on which they depend. + For example, since ssh depends on the + ping test, ssh + needs to come after ping in the list. + The test class provides no re-ordering or dependency handling. + + Each module can have multiple classes with multiple test + methods. + And, Python unittest rules apply. + + + + + The following list summarizes how to run the tests: + + Run the default set of tests simply by + running the following: + + bitbake <qemu_image> -c testimage + + The default tests for the image are defined + as: + + DEFAULT_TEST_SUITES_pn-<qemu_image> = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg" + + Add your own test to the list of the + by using the following: + + TEST_SUITES_append = " mytest" + + Run a specific list of tests as follows: + + TEST_SUITES = "test1 test2 test3" + + Remember, order is important. + Be sure to place a test that is dependent on another test + later in the order. + + +
+ +
+ Writing New Tests + + + As mentioned previously, all new test files need to be in the + proper place for the build system to find them. + New tests can go in the + Source Directory at + meta/lib/oeqa/runtime. + Alternatively, a layer can add its own tests in + <layer>/lib/oeqa/runtime as long + as you extend BBPATH. + Just remember that filenames need to map directory to test + (module) names and you not use module names that collide with + existing core tests. + + + + To create a new test, start by copying an existing module + (e.g. syslog.py or + gcc.py are good ones to use). + Test modules can use code from + meta/lib/oeqa/utils, which are helper + classes. + + + + Structure shell commands such that you rely on them and they + return a single code for success. + Be aware that sometimes you will need to parse the output. + See the df.py and + date.py modules for examples. + + + + You will notice that all test classes inherit + oeRuntimeTest, which is found in + meta/lib/oetest.py. + This base class offers some helper attributes, which are + described in the following sections: + + +
+ Class Methods + + + Class methods are as follows: + + hasPackage(pkg): + Returns "True" if pkg is in the + installed package list of the image, which is based + on + ${WORKDIR}/installed_pkgs.txt + that is generated during the + do.rootfs task. + + hasFeature(feature): + Returns "True" if the feature is in + IMAGE_FEATURES + or + DISTRO_FEATURES. + + restartTarget(params): + Restarts the QEMU image optionally passing + params to the + runqemu script's + qemuparams list (e.g "-m 1024" for + more memory). + + +
+ +
+ Class Attributes + + + Class attributes are as follows: + + pscmd: + Equals "ps -ef" if procps is + installed in the image. + Otherwise, pscmd equals + "ps" (busybox). + + tc: + The called text context, which gives access to the + following attributes: + + d: + The BitBake data store, which allows you to + use stuff such as + oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager"). + + testslist and testsrequired: + Used internally. + The tests do not need these. + + filesdir: + The absolute path to + meta/lib/oeqa/runtime/files, + which contains helper files for tests meant + for copying on the target such as small + files written in C for compilation. + + qemu: + Provides access to the + QemuRunner object, + which is the class that boots the image. + The qemu attribute + provides the following useful attributes: + + ip: + The machine's IP address. + + host_ip: + The host IP address, which is only + used by smart tests. + + + target: + The SSHControl object, + which is used for running the following + commands on the image: + + host: + Used internally. + The tests do not use this command. + + timeout: + A global timeout for commands run on + the target for the instance of a + test. + The default is 300 seconds. + + run(cmd, timeout=None): + The single, most used method. + This command is a wrapper for: + ssh root@host "cmd". + The command returns a tuple: + (status, output), which are what + their names imply - the return code + of 'cmd' and whatever output + it produces. + The optional timeout argument + represents the number of seconds the + test should wait for 'cmd' to + return. + If the argument is "None", the + test uses the default instance's + timeout period, which is 300 + seconds. + If the argument is "0", the test + runs until the command returns. + + copy_to(localpath, remotepath): + scp localpath root@ip:remotepath. + + copy_from(remotepath, localpath): + scp root@host:remotepath localpath. + + + + + +
+ +
+ Instance Attributes + + + A single instance attribute exists, which is + target. + The target instance attribute is + identical to the class attribute of the same name, which + is described in the previous section. + This attribute exists as both an instance and class + attribute so tests can use + self.target.run(cmd) in instance + methods instead of + oeRuntimeTest.tc.target.run(cmd). + +
+
+
+
Debugging With the GNU Project Debugger (GDB) Remotely