Use OEPTestResultTestCase to collect results and add logfile collection
and compression.
(From OE-Core rev: b75aced558013a459f29617b53cf477cb9b387ac)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the OEPTestResultTestCase class as a mix-in class to provide helper
functions for interacting with ptestresults within the extraresults
object generated by the test case.
This class also provides default compression of log text and log files.
Also add support to resulttool for decoding/decompressing log files
embedded in the test results.
(From OE-Core rev: 06cba9883a5964320969301fd05eeb6bec3e786d)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Process binaries within the build directory before stashing to be
relocatable with ORIGIN relative rpaths. This corrects issues with
rpaths being invalid when trying to use the binaries from an unstashed
build directory (e.g. gcc-runtime).
(From OE-Core rev: 34d9f60a8c2e98fdacbb799af11ec015bc5700f4)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Handle multiple results series having ptestresults content. The contents
are merged on a per-result basis where duplicates are ignored (with a
warning message printed). The 'ptestresults.sections' collection is also
merged on a per-suite basis.
(From OE-Core rev: 47edd51970ed0c33edbe04fd72abd1cfc6ecd3d1)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Supplemental to commit fb17b46e2. The libevent "regress" test
outputs its own pass/fail results, e.g.
"2/300 TESTS FAILED. (31 skipped)", which will be miscounted
as an extra test fail in the ptest log. Fixed this to ignore
the libevent results line when counting actual pass/fail
results.
Also removed the for loop in run-ptest and targeted only the
libevent "regress" test, as the other tests being run were
related to performance and did not provide a relevant pass/fail
output.
(From OE-Core rev: 86b8a1d534bfcd70775c6e2b59eabe10de29f526)
Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the break_hardlinks kwarg to break hardlinks when modifying files.
This uses the bb.utils.break_hardlinks function to break hardlinks. The
default is to maintain existing behaviour and leave hardlinks in place.
(From OE-Core rev: 7628f6bdb5704c018d83e284364994b72557eaa5)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the 'append' action of argparse instead of nargs. This changes the
behaviour of the option from "-t foo bar -r" to "-t foo -t bar -r".
Additionally rename the long form options to be consistent with
behaviour, such that they specifying a single tag at a time.
(From OE-Core rev: ffe9e4303fa9799d2e8af9188853a262e15af226)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Nowadays bitbake applies overrides dynamically, not at a single
specific point in time during parsing.
(Bitbake rev: 218431b0f7c97764cb2c0b79a3aadfe2007f490b)
Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The effects of _append/_prepend/_remove are applied when a variable
is expanded, not after parsing has completed.
(Bitbake rev: f9b67433cb4fe5132ab2cf4a9c6bc078b42e1960)
Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When removed all it did was
inherit gnomebase gtk-icon-cache gconf mime
which would also be the most trivial replacement.
Most of the time not all of these classes were needed,
and it is recommended to use only the ones actually required.
(From yocto-docs rev: a0a3a8318a1488ad2fb00b29f1cf26a9d6701be2)
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Debian 10 is a supported distribution now, so add it to the documentation.
(From yocto-docs rev: 77ff109cf7dbe1858e9959c4b9f1225d0a1c5e32)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bluez4 was removed even from meta-oe 2 years ago,
which made made the bluez5 feature for selecting between
bluez4 and bluez5 mandatory for using the bluetooth feature.
The backfilled bluez5 feature has been removed,
including the bluetooth class that helped recipes
for selecting between bluez4/bluez5.
Recipes can replace ${BLUEZ} with bluez5.
(From yocto-docs rev: 025e2b5797b987b0260d7410e2e6d7283f5a7e91)
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This matches the other toolchain tests.
(From OE-Core rev: d41606244c170fd547496e5df9e3d28ce2d2af68)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rework the glibc execution into a common base class. Additionally tag
the tests with "toolchain-user" and "toolchain-system".
(From OE-Core rev: 94bf24268108774e022ad247c647e48a781debbb)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Split the gcc selftest cases into multiple classes one for each test.
This is done in order to make it easy to execute multiple gcc tests in
parallel when using oe-selftest with the '-j' arg.
Additionally tag the user tests with "toolchain-user" and the system
emulation (qemu system) tests with "toolchain-system".
(From OE-Core rev: 7b2f03eff9fc9b4ce48d5ea7e54faa114a6cdcae)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the -t/-T args to be optional so that they can be used together
with the existing -r/-a/... args to run a more flexible filtering of
test tags.
(From OE-Core rev: 55ee27bb07113a45da18711b5509764f62be4d75)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to take advantage of multiprocess execution of tests the
extraresults must be passed through the TestResult. With changes to how
oeqa/core handles test cases the extraresults attribute of the testcase
is passed to the TestResult, with passing across process boundaries
handled automatically.
(From OE-Core rev: 6a1b0c2003a0b4a1983f9494440e6ea02dc25585)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Suffix the ptestresults suite with "-user" for tests that are executing
against usermode qemu.
(From OE-Core rev: 0becf9c1fabb080a2481ebdacef6221f52301621)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Do not fail the running selftest test case if the test suite has a
failed test case. Currently toolchain tests suites (binutils, gcc,
glibc) fail but this does not indicate failure to execute the tests.
Also remove the logging of each test that failed.
(From OE-Core rev: 073575ff9c06b2791cc2bd88063d815d2220f038)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running the system emulation test case, do not include python3 or
pexpect in the image. The test cases that use these also need gdb (with
python configured).
(From OE-Core rev: 7e5be0803ea0cbfd8e5b052e43b54e16ab3230ed)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a oeqa selftest test case to execute the glibc test suite and
report the results. The results are populated into the extraresults
variable of the test case which are written to testresults.json for
resulttool to analyse.
An additional subclass is created to separate the execution with qemu
linux-user and qemu system. The GlibcSelfTestSystemEmulated test case
handles setup of the target image, setup of and NFS server as well as
execution with runqemu.
(From OE-Core rev: 730832ebcca305477e1c13248cd35eea095b35c6)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a oeqa selftest test case to execute the gcc test suites and
report the results. The results are populated into the extraresults
variable of the test case which are written to testresults.json for
resulttool to analyse.
An additional subclass is created to separate the execution with qemu
linux-user and qemu system. The GccSelfTestSystemEmulated test case
handles setup of the target image as well as execution with runqemu.
(From OE-Core rev: 2c86a25f8992243311e7fa1a8654b41f12b749de)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a oeqa selftest test case to execute the binutils test suites and
report the results. The results are populated into the extraresults
variable of the test case which are written to testresults.json for
resulttool to analyse.
(From OE-Core rev: e5629aa4bd939072208f6eb5b30a98e17eb6a8ae)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These test cases are run by the autobuilder on a machine specific basis.
Add tags to these classes so they can be controlled by the metadata rather
than hardcoded in the autobuilder config.
(From OE-Core rev: de0b761b550d591f301ee5e9c232e0d5bd1342f2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the __oeqa_testtags attribute added by OETestTag and display no, one
or more tags separated by a comma. Also change the formatting of the
output so that the list of tests is formatted as "<test> (<tags>)" and
remove the table header for "id" (it is no longer available).
(From OE-Core rev: d62e577a4e1a5f6accbce9f7bff7317a1162d72d)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add '--run-only-tags' for running tests which match any of the provided
tags, and '--run-exclude-tags' for running all tests except those that
have any of the provided tags.
(From OE-Core rev: 0dc3caf21c4519ef16c2ac99b93c03e23aab61d9)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rework OETestTag so that it does not rely on the existing decorator code
base and instead inserts the tags into an attribute on the decorated
target (e.g. class/type or method). This allows the use of OETestTag on
classes and method.
In order to filter tagged tests rework the loaders filtering code,
removing the generic-ness (with validation and attributes/etc.) and
replace it with a "tags_filter" parameter which is a function that
filters a test based on the tags it has. This allows the loader user to
filter on tags in more specific ways (e.g. include all untagged tests
and any tests tagged with foo). Plumb all this through the context code
and testing code.
Update the associated tests to pass correctly with the changes.
(From OE-Core rev: b8a4a4c2de68110d74607cb9807c9e741ca9441c)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implement handling of extra result (e.g. ptestresult) collection with
the addition of a "extraresults" extraction function in OETestResult. In
order to be able to serialize and deserialize the extraresults data,
allow OETestResult add* calls to take a details kwarg. The subunit
module can handle cross-process transfer of binary data for the details
kwarg. With a TestResult proxy class to sit inbetween to encode and
decode to and from json.
(From OE-Core rev: b0831d43606415807af80e2aa1d0566d0b8c209c)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Thanks to wildcards in bbappend filenames, it's possible to have
multiple bbappends that apply to the same recipe in the same directory.
In order to get sstate hits between different workspaces, we want to
apply those bbappend files in a consistent order. Since readdir()
returns files in a non-deterministic order between workspaces (based on
inode number and/or time of creation), we'll need to sort its result in
order to have any consistency.
(Bitbake rev: 94c0c7f15c7a6244a8576ed948ffc21afb96ba82)
Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The 5.2 reference kernels replace the EOL 5.0 series. So we drop
the named recipes.
(From OE-Core rev: 868e84c57e6725591fa1e93f5ea0a308d49e13f1)
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The libevent ptest used to report only a global pass or a fail result.
Count individual PASS, FAIL, SKIP results. The SKIP results now
include tests that are disabled in the libevent code.
libevent's ptest output did not comply with the automake-style output
"result: testname", and reported a FAIL status at the end of the test
run if any of the libevent tests failed. This patch makes the log
consistent with the automake style:
PASS: http/cancel_by_host_no_ns
PASS: http/cancel_inactive_server
PASS: http/cancel_by_host_no_ns_inactive_server
SKIPPED: http/cancel_by_host_server_timeout
SKIPPED: http/cancel_server_timeout
and provides a summary as follows:
=== Test Summary ===
TOTAL: 316
PASSED: 300
FAILED: 0
SKIPPED: 16
DURATION: 87
END: /usr/lib/libevent/ptest
(From OE-Core rev: fb17b46e202cc08277d3eeb34872067c73a6bfbc)
Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added perl to the run-time dependency of the recipe for diffutils since
it is required by the test "large-subpot".
The test "strip-trailing-cr" is skipped since it requires valgrind to
work, but valgrind is considered too heavy-weight for diffutils package.
(From OE-Core rev: 802c67c7c27011812d690c41347b38481cdab2e9)
Signed-off-by: Peiran Hong <peiran.hong@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is provided by runit which another init system like systemd,
sysvinit, this lets oeqa run on systems which are using runit
(From OE-Core rev: e4eb3ca113985ab2be123eb5b5ea76764761df88)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
LICENSE file checksum changed because few contributors were mentioned
(see @edbdbbc50beced9c723e7405334583c60a702796).
(From OE-Core rev: 06eb03ad1c0357731496531b290a0a2fdfd10f48)
Signed-off-by: Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since TUNE_FEATURES now either contains a CPU or an architecture (but
not both) we can't rely on finding the architecture in TUNE_FEATURES.
Use architecture specific over-rides instead.
(From OE-Core rev: 805dd4807d322dc70cef97edd68fdb3142b60fb1)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix build with usrmerge enabled:
WARNING: iputils-s20190709-r0 do_package: iputils: alternative target (/usr/bin/ping or /usr/bin/ping.iputils) does not exist, skipping...
WARNING: iputils-s20190709-r0 do_package: iputils: NOT adding alternative provide /usr/bin/ping: /usr/bin/ping.iputils does not exist
ERROR: iputils-s20190709-r0 do_package: QA Issue: iputils: Files/directories were installed but not shipped in any package:
/bin/tftpd
/bin/tracepath
/bin/arping
/bin/clockdiff
/bin/ping
/bin/traceroute6
/sbin/rarpd
/sbin/ninfod
/sbin/rdisc
Please set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install.
iputils: 9 installed and not shipped files. [installed-vs-shipped]
WARNING: iputils-s20190709-r0 do_package: iputils: alt_link == alt_target: /usr/bin/ping == /usr/bin/ping
(From OE-Core rev: 4b2322e90f6a3a4d2c44145aa85b688f28edddaa)
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update sysstat from 12.1.3 to 12.1.6.
* make sa_lib_dir refer to ${libexecdir}/sa to fix conflictions when
multilib is enabled
(From OE-Core rev: 8862f21eeb814c64b8b9e662b093ed39474cd55b)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Inherit multilib_script to fix file confliction when mutlilib enabled.
| Error: Transaction check error:
| file /usr/bin/texi2any conflicts between attempted installs of
lib32-texinfo-6.5-r0.core2_32 and texinfo-6.5-r0.core2_64
(From OE-Core rev: 08b6a12e674f7a202bb29fbb465997e8303ccb5d)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Redefine CUPS_SERVERBIN to "$libexecdir/cups" for cups which solves file
confliction when multilib is enabled.
| Error: Transaction check error:
| file /lib/systemd/system/org.cups.cups-lpd@.service conflicts between
attempted installs of cups-2.2.11-r0.core2_64 and lib32-cups-2.2.11-r0.core2_32
(From OE-Core rev: 274bed042b9c2b50a8bdd11b42f1a62405fb5b11)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
