mirror of
https://git.yoctoproject.org/poky
synced 2026-07-17 04:07:06 +00:00
Yeoh Ee Peng
661fa1335b
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.
These scripts were developed as a test result tools to manage
these testresults.json file.
Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.
Using the "merge" operation, user can merge multiple testresults.json
files to a target file.
Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.
Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.
These resulttool operations expect the testresults.json file to use
the json format below.
{
"<testresult_1>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
...
"<testresult_n>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
}
To use these scripts, first source oe environment, then run the
entry point script to look for help.
$ resulttool
To store test result from oeqa automated tests, execute the below
$ resulttool store <source_dir> <git_branch>
To merge multiple testresults.json files, execute the below
$ resulttool merge <base_result_file> <target_result_file>
To report test report, execute the below
$ resulttool report <source_dir>
To perform regression file analysis, execute the below
$ resulttool regression-file <base_result_file> <target_result_file>
To perform regression dir analysis, execute the below
$ resulttool regression-dir <base_result_dir> <target_result_dir>
To perform regression git analysis, execute the below
$ resulttool regression-git <source_dir> <base_branch> <target_branch>
[YOCTO# 13012]
[YOCTO# 12654]
(From OE-Core rev: 39cda3498ec68b91a671ead256231324d74f7d4c)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
40 lines
1.4 KiB
JSON
40 lines
1.4 KiB
JSON
{
|
|
"runtime_core-image-minimal_qemuarm_20181225195701": {
|
|
"configuration": {
|
|
"DISTRO": "poky",
|
|
"HOST_DISTRO": "ubuntu-16.04",
|
|
"IMAGE_BASENAME": "core-image-minimal",
|
|
"IMAGE_PKGTYPE": "rpm",
|
|
"LAYERS": {
|
|
"meta": {
|
|
"branch": "master",
|
|
"commit": "801745d918e83f976c706f29669779f5b292ade3",
|
|
"commit_count": 52782
|
|
},
|
|
"meta-poky": {
|
|
"branch": "master",
|
|
"commit": "801745d918e83f976c706f29669779f5b292ade3",
|
|
"commit_count": 52782
|
|
},
|
|
"meta-yocto-bsp": {
|
|
"branch": "master",
|
|
"commit": "801745d918e83f976c706f29669779f5b292ade3",
|
|
"commit_count": 52782
|
|
}
|
|
},
|
|
"MACHINE": "qemuarm",
|
|
"STARTTIME": "20181225195701",
|
|
"TEST_TYPE": "runtime"
|
|
},
|
|
"result": {
|
|
"apt.AptRepoTest.test_apt_install_from_repo": {
|
|
"log": "Test requires apt to be installed",
|
|
"status": "PASSED"
|
|
},
|
|
"buildcpio.BuildCpioTest.test_cpio": {
|
|
"log": "Test requires autoconf to be installed",
|
|
"status": "ERROR"
|
|
}
|
|
}
|
|
}
|
|
} |