1
0
mirror of https://git.yoctoproject.org/poky synced 2026-07-17 04:07:06 +00:00

553 Commits

Author SHA1 Message Date
Richard Purdie 5ddf7fff99 devtool: Split tests into multiple classes
This allows better parallelism between the different tests as currently
this block takes the longest time to execute. devtool tests are still
all grouped into the "devtool" module for ease of exection.

This also makes it easier to execute some subset of devtool tests for
testing devtool changes.

(From OE-Core rev: 84f19e78d9b1f3d634cf1d46ce48f24670199d0b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Richard Purdie 9a8d6d9fd2 oeqa: Default to buffer mode for tests
Currently some tests run in buffer mode and some don't. Those that don't can
corrupt stdout/stderr. Switch to using buffer mode everywhere so we're consistent.

If there is useful output on stdout/stderr, it will be displayed if the test
fails.

(From OE-Core rev: 85c1b6fb516aae58240330a0aca659bfafcd3883)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Richard Purdie 65681147b2 oeqa/selftest/devtool: Ensure dbus is built befoe running test
If dbus isn't build first the dbus dependency of dbus-wait can't be detected
through pkgconfig and the test fails:

AssertionError: {'DEPENDS': {'dbus'}} != {}
- {'DEPENDS': {'dbus'}}
+ {} : Some expected variables not found in recipe: {'DEPENDS': {'dbus'}}

Ensure dbus is built and present in the sysroot.

(From OE-Core rev: af7ba26a603a12de0aed35e786674c92049c2bee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Chen Qi 8c7ca97dd6 logging: use warning instead warn
The warn method is deprecated. We should use the documented warning instead.

Quoting from the python's official doc:
"""
Note: There is an obsolete method warn which is functionally identical to warning.
      As warn is deprecated, please do not use it - use warning instead.
"""

(From OE-Core rev: cc771aa4b74f222f1bea38b0b50196b2fbc97ab4)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Chen Qi 56ddd6b3f4 pkgdata.py: avoid target-sdk-dummy-provides to mess things up
Sometimes we meet the following failure for the test_lookup_recipe
test case.

  AssertionError: 'zlib\nbusybox is in the RPROVIDES of target-sdk-provid[32 chars]ummy' != 'zlib\nbusybox'
    zlib
  + busybox- busybox is in the RPROVIDES of target-sdk-provides-dummy:
  - target-sdk-provides-dummy

This is because target-sdk-provides-dummy rprovides busybox.

So clean things up to avoid failure.

(From OE-Core rev: 5ae49260180adc0070287db01c01a0820ce99ad3)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Richard Purdie c7eb843d7c resulttool: Improvements to allow integration to the autobuilder
This is a combined patch of the various tweaks and improvements I
made to resulttool:

* Avoid subprocess.run() as its a python 3.6 feature and we
  have autobuilder workers with 3.5.

* Avoid python keywords as variable names

* Simplify dict accesses using .get()

* Rename resultsutils -> resultutils to match the resultstool ->
  resulttool rename

* Formalised the handling of "file_name" to "TESTSERIES" which the code
  will now add into the json configuration data if its not present, based
  on the directory name.

* When we don't have failed test cases, print something saying so
  instead of an empty table

* Tweak the table headers in the report to be more readable (reference
  "Test Series" instead if file_id and ID instead of results_id)

* Improve/simplify the max string length handling

* Merge the counts and percentage data into one table in the report
  since printing two reports of the same data confuses the user

* Removed the confusing header in the regression report

* Show matches, then regressions, then unmatched runs in the regression
  report, also remove chatting unneeded output

* Try harder to "pair" up matching configurations to reduce noise in
  the regressions report

* Abstracted the "mapping" table concept used to pairing in the
  regression code to general code in resultutils

* Created multiple mappings for results analysis, results storage and
  'flattening' results data in a merge

* Simplify the merge command to take a source and a destination,
  letting the destination be a directory or a file, removing the need for
  an output directory parameter

* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression
  mappings

* Have the store command place the testresults files in a layout from
  the mapping, making commits into the git repo for results storage more
  useful for simple comparison purposes

* Set the oe-git-archive tag format appropriately for oeqa results
  storage (and simplify the commit messages closer to their defaults)

* Fix oe-git-archive to use the commit/branch data from the results file

* Cleaned up the command option help to match other changes

* Follow the model of git branch/tag processing used by oe-build-perf-report
  and use that to read the data using git show to avoid branch change

* Add ptest summary to the report command

* Update the tests to match the above changes

(From OE-Core rev: e4195565d2a50046d4378c97f7a593c41bed51bd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Yeoh Ee Peng d038d97d48 resulttool: enable merge, store, report and regression analysis
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.

These scripts were developed as a test result tools to manage
these testresults.json file.

Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.

Using the "merge" operation, user can merge multiple testresults.json
files to a target file.

Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.

Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.

These resulttool operations expect the testresults.json file to use
the json format below.
{
    "<testresult_1>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
    ...
    "<testresult_n>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
}

To use these scripts, first source oe environment, then run the
entry point script to look for help.
    $ resulttool

To store test result from oeqa automated tests, execute the below
    $ resulttool store <source_dir> <git_branch>

To merge multiple testresults.json files, execute the below
    $ resulttool merge <base_result_file> <target_result_file>

To report test report, execute the below
    $ resulttool report <source_dir>

To perform regression file analysis, execute the below
    $ resulttool regression-file <base_result_file> <target_result_file>

To perform regression dir analysis, execute the below
    $ resulttool regression-dir <base_result_dir> <target_result_dir>

To perform regression git analysis, execute the below
    $ resulttool regression-git <source_dir> <base_branch> <target_branch>

[YOCTO# 13012]
[YOCTO# 12654]

(From OE-Core rev: bb0bc6368bb51ac0be77d13fe931601d493951ea)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Peter Kjellerstedt c4681cc02e oe-selftest: devtool: Support meta being a symbolic link
oe-selftest's devtool tests have been broken since commit 2457cd57
(oe-selftest: devtool: avoid parallel races by using temporary copy of
core) if meta is a symbolic link.

(From OE-Core rev: daba6c5a991b370709d17e51305334f55a3858ec)

(From OE-Core rev: 3eb59559ecd2e93fb590a330b47de1db0750fc0b)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Paul Eggleton e5eadb501b oe-selftest: devtool: avoid parallel races by using temporary copy of core
Some of the devtool tests make changes to files under meta/ -
legitimately since we want these tests to be working with real recipes
and associated files. Unfortunately with the new oe-selftest
parallelisation this can break other tests if files go missing at the
wrong time (among other scenarios). To avoid this issue, simply take a
copy of the core repository and use that for these tests. (We copy the
entire repository since changing the path of meta/ influences COREBASE
and thus we need to have things like scripts/ alongside as well).

(From OE-Core rev: 2457cd57b4195924ef127f497efa2f34f411e660)

(From OE-Core rev: 9cb8353a4f0137823d6ed3e467db9dd7ead7b3de)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Robert Yang 0bda550627 oeqa/selftest/runqemu: Enable kvm when QEMU_USE_KVM is set
(From OE-Core rev: 564de3681353fe8e203425388e8be9703a89d2da)

(From OE-Core rev: 1207949c986cc9f6b3940a99e87bc4d8cfb86d5b)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Alexander Kanavin c172c8fdc6 oeqa: replace Alex Kanavin's @intel email address with a personal one
As I will be leaving Intel, this address will no longer be valid,
so swap it for my personal one for now.

(From OE-Core rev: ea58ff101e28dfda3410de66d775df3d8a1e5a96)

(From OE-Core rev: 61bca5a7f6b3fc03cd88cbf8867708d28e0a3522)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Richard Purdie fa5ebe62e2 testimage/testsdk/selftest: Avoid platform.distro_identifier deprecation warnings
Use our own lsb function instead as used elsewhere by the codebase.

(From OE-Core rev: acac45a6fd604d28ef7c23d67482af3d7e8bcfe3)

(From OE-Core rev: 570256a64af5a3fa994a20a5cc4c74d59ffc361f)

(From OE-Core rev: d58fe9d352ae7de857e7f55b88f6e7d35b2cd706)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Richard Purdie 6512ffb090 oeqa/selftest/esdk: Fix typo causing test failure
2018-12-06 23:19:24,564 - oe-selftest - INFO - Traceback (most recent call last):
  File "/media/build1/poky-sumo/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/media/build1/poky-sumo/meta/lib/oeqa/selftest/cases/eSDK.py", line 78, in setUpClass
    cls.tmpdir_eSDKQA = cls.tempdirobj.name
AttributeError: type object 'oeSDKExtSelfTest' has no attribute 'tempdirobj'

(From OE-Core rev: 2c60908a2039d333a9fe2651622750ff6ed4cce1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie 766a95ecb2 oeqa/selftest/esdk: Ensure parent directory exists
INFO - ======================================================================
INFO - ERROR: setUpClass (eSDK.oeSDKExtSelfTest)
INFO - ----------------------------------------------------------------------
INFO - Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/eSDK.py", line 76, in setUpClass
    cls.tmpdirobj = tempfile.TemporaryDirectory(prefix="selftest-esdk-", dir=bb_vars["WORKDIR"])
  File "/usr/lib/python3.5/tempfile.py", line 929, in __init__
    self.name = mkdtemp(suffix, prefix, dir)
  File "/usr/lib/python3.5/tempfile.py", line 507, in mkdtemp
    _os.mkdir(file, 0o700)
FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-debian/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-q7ln84gc'

(From OE-Core rev: 170a601a99836d13b69e5287bee0d3e71983dd46)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie 5df29d420a image-buildinfo,oeqa/selftest/containerimage: Ensure image-buildinfo doesn't break tests
Having image-buildinfo enabled causes containerimage.ContainerImageTests.test_expected_files
to fail due to the presence of an unexpected file:
  ['./',
   './etc/',
-  './etc/build',
   './etc/default/',
   './etc/default/postinst',

Tweak the class to allow it to be disabled and disable it from the test just in
case it was enabled.

(From OE-Core rev: af67bf422a4df5b7e07894512ff73a5f493682ab)

(From OE-Core rev: f49ab8b1610c045acaed7b964d12f07f969df856)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Yeoh Ee Peng 152a347f8d oeqa/selftest: Standardize json logging output directory
Currently sdk & sdkext will output json file to LOG_DIR, while
selftest will output json file to TOPDIR/log.

Standardize selftest json output file to LOG_DIR.

(From OE-Core rev: 9f2e39684cbbe9f87eeef6a81961e6db783439e3)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie f50977c3b5 oeqa/selftest: Improvements to the json logging
Tweak the preceeding commit to:

* Add STARTTIME to the identifier to make it unique
* Use LOG_DIR
* Store the layer config in a more natural json format
* Drop '_' function prefixes

(From OE-Core rev: 173f59acf9722e2ef27fdd49c20f7d3d664917eb)

(From OE-Core rev: 3b69099edc7db99c11bfb41eab2af50bd0e3d4f2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Yeoh Ee Peng 5a99880992 oeqa/selftest/context: write testresult to json files
As part of the solution to replace Testopia to store testresult,
OEQA selftest need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.

By default, json testresult file will be written to "oeqa"
directory under the oe-selftest log directory.

To configure multiple instances of bitbake to write json testresult
to a single testresult file at custom directory, user will define
the variable "OEQA_JSON_RESULT_DIR" with the custom directory for
json testresult.

(From OE-Core rev: a95218525a4c8228fff9908ffbda85c6b85e101c)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie fa12ad627b oeqa/selftest/context: Improve log file handling
The existing logfile is simply placed in the current directory. Since the test
changes cwd to BUILDDIR, the symlink to the log can be placed in an invalid
directory. We also see trackbacks if the symlink is invalid.

Improve things by:

* Placing logs in LOG_DIR (or BUILDDIR if unset).
* Using a full path to the log meaning the log and link are placed in the same directory.
* Using lexists instead of exists so invalid symlinks are handled correctly.

(From OE-Core rev: 750ece11bed0e62a11e0003d1d16a81f7c219761)

(From OE-Core rev: 9c4c3c876dd5d224133571fcad1095af1098ae1d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie ea1539b872 oeqa/selftest/context: Replace deprecated imp module usage
Avoid the warning:

meta/lib/oeqa/selftest/context.py:8: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp

In this case importlib is a direct replacement.

(From OE-Core rev: db7a60c36a2d3eefc61ae6e1ede01680dc932035)

(From OE-Core rev: 351a7b75959593922909d7e1929a6429a2bf94a7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Ross Burton 5f74f3c639 oeqa/selftest/esdk: run selftest inside workdir not /tmp
We've seen issues with rootfs size calculations and we've seen systems
like opensuse which have btrfs mounted on /tmp causing selftest failures.

(From OE-Core rev: 61be3cd748d1b7321a1fc4cfe84efa9b26a6aee0)

(From OE-Core rev: d936faabfb29ea377d74e77332a2a91603747ac7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie 9bd0b59f73 oeqa/selftest/runqemu: Improve testcase failure handling
assertTrue doesn't give good debug information when things fail. Update
several to use assertIn which gives information upon failure, for the
others print the log information upon failure.

(From OE-Core rev: c29cb75d5ce6b0873a934f4709b0c8824f7164d3)

(From OE-Core rev: 6f2bb2c12289422396deb793ae4b4e99cfa88c7b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Ross Burton 765f90b383 oeqa/oelib/path: don't leak temporary directories
setUp() is used to populate a directory of temporary files, and deleted in
__del__.  However setUp() is called once *per test* so __del__ would only be
able to remove the last directory created.

Fix the code by using the natural counterpart to setUp, tearDown(), to clean up.

(From OE-Core rev: 68b4723e6fb11d171869185bccf28f32f6284c18)

(From OE-Core rev: edd2ecbc86d8b02a0467491451306f67e81d9ead)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie 202097fe5b oeqa/selftest/buildoptions: Ensure diskmon tests run consistently
Heartbeat events default to once a second and we need to ensure we have
enough time in the task to see them.

Add a nostamp delay task 5s long so we can have a consistently timed
task which doesn't need cleanup or have unneeded dependencies. This
ensures we should deterministically see the disk moinitor events
regardless of the state of the build. This is done in a way which
doesn't corrupt build state or need cleanup and is efficient.

(From OE-Core rev: ecc49ee8986929e2429d948000a0ca588fe63959)

(From OE-Core rev: d5ce38168238181423c9dcd1d258253e3515d0a5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie 26f65c2612 oeqa/selftest/buildoptions: Improve ccache test
This test occisionally fails as m4 doesn't recompile, meaning the logfile test
then doesn't find mention of ccache.

To ensure m4 does recompile, clean m4 before force compiling it.

(Reading the test is confusing due to the test cleanup also involving a clean)

(From OE-Core rev: 6e0b9214a0d57ed45a5df0ba5c9887a9045b89b1)

(From OE-Core rev: b729a46820cce38b3d500271df19bdaf31973140)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie 029c414cd5 oeqa/selftest/buildoptions: Improve ccache test failure output
The current failure mode doesn't show us what the logs actually looked like
and later cleans can lose them. Show the whole log in case of failure
to aid debugging intermittent problems on the autobuilder.

(From OE-Core rev: 7c3a0dc5978cea898b1ca51decf4d6e7cf9d519f)

(From OE-Core rev: 60b58e976dcc2a45a036e18cfcc87933a8859c9f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie c2fdfb7981 oeqa/selftest/case: Use bb.utils.remove() instead of shutil.remove()
This avoids problems where shutil.remove will error with:

  File "/usr/lib/python3.5/shutil.py", line 436, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'S.gpg-agent.extra'

when there are races over file deletion (gpg agent may be slow to exit).

We already worked around speed and race issues in bb.utils.

(From OE-Core rev: 00a8fd5b93a5c19ce0b7498e2bc653ce8ad58aaf)

(From OE-Core rev: e7c0d29e063d1a47004acdc07a63996be98c74ef)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:26 +00:00
Richard Purdie de5ed54deb oeqa/selftest/signing: Use do_populate_lic target instead of do_package
This should speed the test up signficiantly without any loss of functionality
for the purposes of the test.

(From OE-Core rev: 3dde0b749643575878bfbca2f8d2d9ec30bad166)

(From OE-Core rev: 51989db2d9eebd9f190994109c2932dac3f3034b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:26 +00:00
Richard Purdie d030fa6043 oeqa/selftest/signing: Allow tests not to need gpg on the host
We ideally don't want to use gpg from the host. This is straightforward for package
management but not for sstate.

For sstate, create a second build directory to run the test in using gnupg-native
from the original build directory.

(From OE-Core rev: 10afa94c3f0d7eb7524a26deda86949073d55fde)

(From OE-Core rev: c5d68a24b0c6df0a16f50075a690b3aab0e273ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:26 +00:00
Richard Purdie ac34287384 oeqa/selftest/signing: Skip tests if gpg isn't found
Raising an assertionError in the class setup isn't a particuarly good way to
indicate gpg isn't installed. Instead skip the tests if the required binary
isn't present. For the signing tests we do require it to be present and can't
use a prebuilt one.

(From OE-Core rev: 2d486af97e51b9daa9c40482c31d637c9ab4ae79)

(From OE-Core rev: 65403bb556f7e3132722288a62ef36631af0b557)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:26 +00:00
Mohamad Noor Alim Hussin 0995109f36 oeqa/selftest/recipetool: Fix problems from changing upstream source
The upstream source tarball checksums changed. Use the copy from our source
mirror to avoid failures.

[YOCTO #12979]

(From OE-Core rev: e97a31e6bbaec5cb56d4750bf5171dbba510ee33)

(From OE-Core rev: d637eea4af8b538fb45056f0022975d7c41cdf5f)

Signed-off-by: Mohamad Noor Alim Hussin <mohamad.noor.alim.hussin@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-24 21:45:58 +00:00
Richard Purdie 0b96af3645 selftest/wic: Improve error message for test_fixed_size
Currently this can fail with a message like 127 != 0 which is unhelpful.

If we remove the ignore_status=False, the debugging from runCmd is much
more helpful printing status.output.

Also remove the now unneeded exit code check.

(From OE-Core rev: 1aa7471b11aedc68de5116c461fe73152e3985fd)

(From OE-Core rev: d9ad083ee5a1bc8723b01b31a0010128e26375fd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-24 21:45:58 +00:00
Richard Purdie 200366f34a oeqa/selftest/wic: Ensure initramfs exists for test_iso_image
AssertionError: Command 'wic create mkhybridiso --image-name core-image-minimal -o /var/tmp/wic.oe-selftest/' returned non-zero exit status 1:

ERROR: _exec_cmd: gzip -f -9 -c /var/tmp/wic.oe-selftest/tmp.wic.drhn6edm/initrd.cpio > /var/tmp/wic.oe-selftest/tmp.wic.drhn6edm/initrd.cpio.gz returned '1' instead of 0
output: gzip: /var/tmp/wic.oe-selftest/tmp.wic.drhn6edm/initrd.cpio: No such file or directory

This is because in a clean build directory, the initramfs may not be rebuilt.
Add a call to ensure it is built to avoid the error.

(From OE-Core rev: 2a80fa234d31992691a157425e8990db30158fd1)

(From OE-Core rev: cdc226b8d4114ef4ff51d6f13ceb09f8d264bf76)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-24 21:45:58 +00:00
Richard Purdie 8799bc3171 oeqa/selftest/wic: Use a subdir of builddir, not /var/
Using /var/ leave wic open to races with other processes on the system, use
a subdir of builddir instead to avoid this.

(From OE-Core rev: e07ec908ce7f26143a7bdf0a07a1230c0fd6ac87)

(From OE-Core rev: 17223b0045896c9f342e9079d2345b730a3048cc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-24 21:45:58 +00:00
Richard Purdie eebbc00b25 oeqa/selftest/runtime_test: Ensure we build/use gnupg-native
Without this, we see errors if gpg is missing from the host system
for "oe-selftest -r runtime_test.TestImage.test_testimage_dnf".

(From OE-Core rev: e91838b63b506e2969582b2b8511fd3724d6aa3f)

(From OE-Core rev: 40e0b6244c0c6f276de501765daff660d7a44363)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-16 16:33:09 +00:00
Richard Purdie 91aa78a8cd oeqa/selftest/runqemu: qemu nfs testing not ready for deployment
This test shouldn't have merged yet since we don't run portmap/rpcbind
on the autobuilder infrastructure and the test therefore cannot succeed.

We need to document this, set it up, then enable the test. The test itself
is fine and good to have so its left in the code but disabled for now.

(From OE-Core rev: 9640af873d490c5d22b70e32d918c2db37371d21)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-20 08:12:17 +01:00
Yeoh Ee Peng a0edc4917a oe-selftest: runqemu: add tests for qemu boot and shutdown
QA team were testing qemu boot image and shutdown on each
qemu architecture manually. Add automated test to test qemu boot on
ext4 and nfs, finally check that it can shutdown properly.

(From OE-Core rev: 1df5f2dff832528905ff6fcf1d324619fb3d307f)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-20 08:12:17 +01:00
Richard Purdie ff282528be devtool: Ensure added layer sets LAYERSERIES_COMPAT
Now that we see warnings if LAYERSERIES_COMPAT is unset, the auto generated
code from devtool needs to set this to avoid warnings which break
various tests.

(From OE-Core rev: f65ebfeda0bfbac78e4a2a6609ba654ca38a8b0e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-07 22:33:19 +01:00
Alexander Kanavin 6827053895 oe-selftest: add a test for failing package post-installation scriptlets
The test runs a scriptlet that has an intentionally failing command in the middle
and checks for two things:
1) that bitbake does warn the user about the failure
2) that scriptlet execution stops at that point.

The test is run for all three package types: rpm, deb, ipk.

(From OE-Core rev: 865fafb0dff19d27bd417c28c95fb8fdf0326a2b)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-15 06:27:18 -07:00
California Sullivan 7e9b658196 selftests: Add test case for booting a generic EFI boot partition image
Simple test case that adds 'efi' to MACHINE_FEATURES, sets WKS_FILE to
"efi-bootdisk.wks.in", installed required boot items, and attempts to
boot the wic image.

Quick check to make sure that the feature actually works.

(From OE-Core rev: 192c8738f4a8d0f82848a440acf24a1892f2ce93)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-06 06:35:42 -08:00
Ross Burton 1bcbe159aa sstatetests: pass the right files when comparing sigfiles
(From OE-Core rev: 65cc20690e7827df3d84ebea6357eebabb668f50)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-24 10:31:48 +00:00
Alexander Kanavin 3bbf10c364 oe-selftest: add a test for recipes without maintainers
'bitbake -c checkpkg world' is moved to class initializer to avoid
it being run twice in a row.

The no-maintainers test checks only oe-core recipes, as other layers
may be be configured, and assigning maintainership to specific people via
maintainers.inc is known to be used only in oe-core.

(From OE-Core rev: ac6a72aab19e04adb2fa2b9932a9427d1b6fbb46)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-24 10:31:45 +00:00
Alexander Kanavin 6bb3595ec5 oe-selftest: fix tests to reflect man->man-db transition
Rather than apply a patch twice, an incorrect patch is applied
(as the new man-db recipe does not have any patches yet).

(From OE-Core rev: a8bf4da183824caa5fceaf7546db52daa05f8662)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-24 10:31:45 +00:00
Yeoh Ee Peng 3f6eb216ef oe-selftest: meta_ide: add tests for meta-ide-support
QA team were testing meta-ide-support manually. Add automated
tests to test that bibtake meta-ide-support will create the
toolchain and environment setup script. Also test that after
using environment setup script, one can compile c program
and build cpio project.

(From OE-Core rev: db40eba68f51d02677526dfa4bc21343d9c27958)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-15 11:23:11 +00:00
Saul Wold 58dd6fe372 imagefeatures: disable f2fs from test_image_features by default
Since the primary f2fs utilities are provided by the meta-openembedded
meta-filesystems layer, we disable the testing of that functionality
here.

(From OE-Core rev: f691ed2572e54fa2af04c1569746c36ce04463ee)

Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-23 23:43:46 +00:00
Robert Yang 598a4e9e85 selftest/cases/devtool.py: fix workspace layer checking
Fixed:
$ oe-selftest -r devtool.DevtoolTests.test_create_workspace
[snip]
2017-12-20 18:28:59,404 - oe-selftest - INFO - Traceback (most recent call last):
  File "/workspace2/lyang1/poky/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/workspace2/lyang1/poky/meta/lib/oeqa/selftest/cases/devtool.py", line 177, in test_create_workspace
    self.assertTrue('/workspace' not in result.output, 'This test cannot be run with a workspace layer in bblayers.conf')
AssertionError: False is not true : This test cannot be run with a workspace layer in bblayers.conf
[snip]

$ bitbake-layers show-layers
NOTE: Starting bitbake server...
layer                 path                                      priority
==========================================================================
meta                  /workspace2/lyang1/poky/meta              5
meta-poky             /workspace2/lyang1/poky/meta-poky         5
meta-yocto-bsp        /workspace2/lyang1/poky/meta-yocto-bsp    5
meta-selftest         /workspace2/lyang1/poky/meta-selftest     5

There is no workspace layer, but I'm in /workspace2, this patch can fix the problem.

[YOCTO #12442]

(From OE-Core rev: 695b234ea4f034d428f8cffacceabc2b8f00bc74)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-14 09:11:57 +00:00
Richard Purdie 3ab6e17ffe bitbake.conf: Set AUTOREV to have a vardepvalue
If you have a recipe which does not include SRCPV in PV but does set
SRCREV = "${AUTOREV}" and you run do_fetch, then change the repo to a
new commit then run do_unpack, do_unpack will fail since the new commit
doesn't exist in the repo that was fetched.

The problem is the revision chosen is not represented in the do_fetch
task hash. It if were, the fetch would rerun first and the commit would be
present. It works when PV includes SRCPV since that does contain the chosen
commit from the AUTOREV.

The solution is to include the SRCPV value into the representation of AUTOREV
used for checksum calculation purposes.

Add a selftest for this issue.

(From OE-Core rev: 7b8ee9285a197784d51e339f1603240f49435846)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-06 22:32:22 +00:00
Paul Eggleton a77b6ee767 oe-selftest: devtool: switch away from mkelfimage
mkelfimage is about to be removed, so test_devtool_modify_git needs to
use a different recipe. psplash is a reasonable choice given it uses a
git repository (and probably will forever), and doesn't have too many
dependencies, so change the test to use that recipe instead.

(From OE-Core rev: daf583efc87faa058684dfe34df596d088caa8ef)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-02 17:24:37 +00:00
Alexander Kanavin 0ac5ca3fcf rpm: update to 4.14.0
Previously oe-core had a development snapshot of rpm, it's better to update
to something more stable.

Removed patches:
0001-Add-PYTHON_ABI-when-searching-for-python-libraries.patch
(upstream is using pkg-config)
0001-When-nice-value-cannot-be-reset-issue-a-notice-inste.patch
(functionality has been moved to a plugin, we disable plugins
for rpm-native)
0012-Use-conditional-to-access-_docdir-in-macros.in.patch
(merged upstream)

Changed patches:
0001-Fix-build-with-musl-C-library.patch
(one previous musl issue has been resolved upstream; another has been added)

Rest of the patches are trivial rebases.

Update the signing oe-selftest so that the reference output matches
the upstream changes.

(From OE-Core rev: b4613b6ce07c295c5d6de6861acf19315acaccb2)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-02 17:24:36 +00:00
Joshua Lock a2fee27d0a oeqa/selftest/cases/wic: update OETestID decorator for test_qemu
Test case ID 1422 is not correct for this issue, the correct test
case ID for test_qemu is 1424.

(From OE-Core rev: d214b1dc9339c4873156934f5291b9373a11471f)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-21 10:21:00 +00:00