Add test cases to test “exclude” and “installroot“ options, also modify
the logic of filtering packages on the feed to have all the packages
needed by the tests.
[YOCTO #10744]
(From OE-Core rev: 1121806603c6f621d084b692216f3f616a0768dc)
(From OE-Core rev: e1b050f53ece2a31cd6866d2d737d7c67a44cea4)
Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows the ptest results from ptest-runner, run in an image to be
transferred over to the resulting json results output.
Each test is given a pass/skip/fail so individual results can be monitored
and the raw log output from the ptest-runner is also dumped into the
results json file as this means after the fact debugging becomes much easier.
Currently the log output is not split up per test but that would make a good
future enhancement.
I attempted to implement this as python subTests however it failed as the
output was too confusing, subTests don't support any kind of log
output handling, subTest successes aren't logged and it was making things
far more complex than they needed to be.
We mark ptest-runner as "EXPECTEDFAILURE" since its unlikely every ptest
will pass currently and we don't want that to fail the whole image test run.
Its assumed there would be later analysis of the json output to determine
regressions. We do have to change the test runner code so that
'unexpectedsuccess' is not a failure.
Also, the test names are manipuated to remove spaces and brackets with
"_" used as a replacement and any duplicate occurrences truncated.
(From OE-Core rev: a13e088942e2a3c3521e98954a394e61a15234e8)
(From OE-Core rev: 526ceab9d0e43f73635bb92e8dd7763ef75ad33b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The test_file_connman should be executed only when 'lib32-connman' is
installed and 'connman' is not installed.
When lib32-connman and connman are both installed, the /usr/sbin/connmand
could be from connman or lib32-connman, depending on the installation
order. What we want to check is the connmand command from lib32-connman,
so we need to make sure that connman is not there to cause chaos.
(From OE-Core rev: bc6839394c06bb695b92b2183337e7381da1e86c)
(From OE-Core rev: 55eed10368da6b3179e3ad1ab9322cff32fa99cf)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
1) The test cases use 'readelf' command to do the check. This command
is from binutils. So skip the test if the needed binutils package is
not installed.
The related error message in log.do_testimage is like below.
Output: sh: readelf: not found
2) The test case tests /lib/libc.so.6 from lib32-libc6. So skip the test
if lib32-libc6 is not installed.
The related error message in log.do_testimage is like below.
Output: readelf: Error: 'lib/libc.so.6': No such file
(From OE-Core rev: eae929a5c224f5c3468d6a0466d1bbb3f678a5a1)
(From OE-Core rev: cb80105bbd03d8d9cc10139ed6f39b193e79c1b8)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This test case should only run when rpm package is installed.
So skip it if rpm package is not installed. This fixes:
RESULTS - rpm.RpmBasicTest.test_rpm_help - Testcase 1059: FAILED
(From OE-Core rev: bb909a60c04248d015d988e4454f0a11b1c287da)
(From OE-Core rev: 520c375d49df29453213c3cd4e2155e80e27a0ca)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add runtime test for stap to test basic SystemTap
operations: can compile very basic module and run on
target device.
Note we disable (-DSTP_NO_VERREL_CHECK) SystemTap
additional kernel release check since during OE testing
mismatching kernel-devsrc and kernels are used.
(From OE-Core rev: 659d19fcddb7edaca8f5221148d479e73304b430)
Signed-off-by: Victor Kamensky <kamensky@cisco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Kernels which use tools/objtool can now fail when building external modules
due to objtool being missing, the generated files can also cause problems
for kernel-devsrc.
Ensure objtool is generated in make-mod-scripts by also calling
"make prepare".
For devsrc, delete the generated binaries since they'd be native
binaries and unsuitable for the target.
The oeqa kernel module tests also need to have the additional "make prepare"
step added.
(From OE-Core rev: 52fd2993784b4218f5df4f343e7da45d964df305)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add runtime test for opkg to test that it can install ipk
package from remote source.
[YOCTO# 11488]
(From OE-Core rev: 9dd4af2b70f58540b2799823957aff3413068126)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add runtime test for apt to test that it can install deb
package from remote source.
[YOCTO# 11488]
(From OE-Core rev: f380fa77d69051212fdf7dff97da611e884d05d2)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We see occasional network glitches which break this test. Use our own
mirror (which has a .gz instead of .bz2) to avoid the errors, we're not
trying to test network connectivity.
(From OE-Core rev: 038d7270aef0cccf87d50a117160c58261beb9b9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
That's the whole point isn't it? Previously this testcase succeeded
even if some of the underlying on-target tests failed; the only way
to find out if anything was wrong was to manually inspect the logs.
(From OE-Core rev: 21b27d1e9d54d4aab412facff22cd5d3d77827a8)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously the test would execute only when all available ptests
for packages in the image were installed; some of those tests may
be broken, never finish, take a very long time or simply irrelevant
to the user who wants to check ptests of only a few specific packages,
and does so by listing them explicitly via IMAGE_INSTALL_append or similar.
Presence of ptest-runner means there is at least one ptest package installed
as they pull it in via a class dependency; ptest-runner is not generally
installed otherwise.
(From OE-Core rev: e07a2b9c2b08a465baeaaca86461e07817f84a52)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Make it work with current oeqa
* Skip the test if ptest is not in DISTRO_FEATURES
* Skip the test if ptest-pkgs is not in IMAGE_FEATURES
* The logs are saved to:
testimage/ptest_log -> testimage/ptest_log.<datetime>
* This provides data that could be used to detect regressions in ptest results
[YOCTO #11547]
(From OE-Core rev: f1dfb59495db9e79441c8aa623ede7ef20045a20)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
THe newner kernels have ope firmware support enabled which leads to warnings
during boot on mips. These aren't interesting and we should ignore them.
(From OE-Core rev: 4ba9a6ffb4e7f6c7eee47aa13252fd981cfe5618)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Buildiptables test cases are conflicting with images built with “musl”
as standard C library, in order to avoid those issues lzip package was
selected to be used on the tests as this does not have any "musl"
dependency.
This patch is applicable for testimage tests
[YOCTO # 11713]
(From OE-Core rev: 41683e0ab316049e28b1f4ceaf39f0fe17722d92)
Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This error is seen on the Braswell RVP platform we have been using for
testing. The error is caused by atkbd.c keyboard driver, which is only
for AT and PS/2 keyboards. I tested a PS/2 keyboard with the board,
which worked fine, and the board does not have a separate AT connector,
so this error won't cause any functional issues.
[YOCTO #10110].
(From OE-Core rev: c9f83639242313ca04ec36b49602a8464e10dae8)
Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The NUC6 has issues bringing up Bluetooth early in the boot sequence. We
see:
[ 4.091790] Bluetooth: hci0: Minimum firmware build 1 week 10 2014
[ 4.097326] Bluetooth: hci0: Found device firmware: intel/ibt-11-5.sfi
[ 4.145317] Bluetooth: hci0: Failed to send firmware data (-38)
Followed by this later on:
[ 11.509870] Bluetooth: hci0: Minimum firmware build 1 week 10 2014
[ 11.509988] Bluetooth: hci0: Found device firmware: intel/ibt-11-5.sfi
[ 13.090308] Bluetooth: hci0: Waiting for firmware download to complete
[ 13.090829] Bluetooth: hci0: Firmware loaded in 1549114 usecs
[ 13.090987] Bluetooth: hci0: Waiting for device to boot
[ 13.101958] Bluetooth: hci0: Device booted in 10818 usecs
Bluetooth does successfully come up and the firmware is loaded. This
behavior is consistent across all kernels I've tested.
[YOCTO #10628].
(From OE-Core rev: 045ee89342ea8ce16e78fea9f1c73d978d66a337)
Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were two missing cases to be migrated to the new framework: _qemutiny and
_ptest.
qemutiny was straightforward.
ptest on the other hand wasn't working even in previous releases; it has been
migrated from smart to dnf, and how ptest packages are gathered to be
installed, adapted to use unicode, and removed a lot of code that wasn't needed
anymore.
(From OE-Core rev: ee7c19546b686e852d01df25143504d9798d10d6)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently fetch,configure,build, and install cvs as our test for the
sdk. cvs unfortunately, requires a default editor in order to run. The
change in 94790a8254 that checks to see if you have something like vi
installed is fragile since you may have a different default editor. This
patch switches from using cvs as a test to using cpio. cpio also uses
autotools so the functionality tested is equivalent.
(From OE-Core rev: 233d36b0382a8b2e430c3377e50885d1a0c3ba21)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All documentation refers to dnf binary as 'dnf' yet make install
does not create one - it's done by Fedora's spec file when building
the rpm. Let's replicate this behavior.
(From OE-Core rev: 456c4a8ffc9a292d7a3e036d92baf4a8f14d1f45)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Printing a message which says "configure failed" without the log output
is effectively useless. If a command fails, print the output by default
and simplify the calling code which makes debugging any of these failures
much easier.
(From OE-Core rev: b6352ff001c29f0bff10c18879b92c5618ec645c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Debian based distros has a builtin syslog module so when
try to load tests using unittest it references the builtin
module instead of runtime/cases.
[YOCTO #10964]
(From OE-Core rev: 9923e3cdb58c2b3c54ec5fe99b2cec4cdc9fff92)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
smart test requires to build the image using rpm packages, this check was
included, but it checked for PACKAGE_CLASSES=='package_rpm', and this is
not true when building packages for rpm and deb/ipk. So this would check
IMAGE_PKGTYPE instead.
[YOCTO #10964]
(From OE-Core rev: 570f50b30072a10c5eadfbbfea2647339ee85176)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeRuntimeTest class is not used anymore as part of runtime
migration, this particular case was missed, so fix it.
[YOCTO #10964]
(From OE-Core rev: d928e5166d8812e7e20bc95c0a8b5cb5439ec497)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This migrates the smart test from the old framework to
the new one. This has its own commit because smart
test was using bb and oe libraries that are available
when exporting the test cases to run in a different host.
Because of the removal of bb and oe libraries index and
packages feeds creation will be managed in testimage bbclass.
[YOCTO #10234]
(From OE-Core rev: 8d64ac4208e8dcb8a6fde6ea2959c9b3edfe2172)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This migrates current runtime test suite to be used with the new framework.
[YOCTO #10234]
(From OE-Core rev: b39c61f2d442c79d03b73e8ffd104996fcb2177e)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
