Commit Graph

2050 Commits

Author SHA1 Message Date
Peter Kjellerstedt
c4681cc02e oe-selftest: devtool: Support meta being a symbolic link
oe-selftest's devtool tests have been broken since commit 2457cd57
(oe-selftest: devtool: avoid parallel races by using temporary copy of
core) if meta is a symbolic link.

(From OE-Core rev: daba6c5a991b370709d17e51305334f55a3858ec)

(From OE-Core rev: 3eb59559ecd2e93fb590a330b47de1db0750fc0b)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Paul Eggleton
e5eadb501b oe-selftest: devtool: avoid parallel races by using temporary copy of core
Some of the devtool tests make changes to files under meta/ -
legitimately since we want these tests to be working with real recipes
and associated files. Unfortunately with the new oe-selftest
parallelisation this can break other tests if files go missing at the
wrong time (among other scenarios). To avoid this issue, simply take a
copy of the core repository and use that for these tests. (We copy the
entire repository since changing the path of meta/ influences COREBASE
and thus we need to have things like scripts/ alongside as well).

(From OE-Core rev: 2457cd57b4195924ef127f497efa2f34f411e660)

(From OE-Core rev: 9cb8353a4f0137823d6ed3e467db9dd7ead7b3de)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
9947f6ad5b oeqa/utils/httpserver: Rework to avoid hangs and improve logging
testimage.bbclass installs a SIGTERM handler which conflicts with the
use of multiprocessing here. This is paritcularly problematic if the http
service is terminated before its started and hence before its had a chance
to reset the default signal handler (as the code was written).

Instead, temporarily remove testimage's handler whilst forking the http process
which means the correct handler is installed and won't deadlock.

Also take the opportunity to add in some log messages about the server start
and shutdown so that future debugging is easier and its clearer what the code
is doing.

(From OE-Core rev: cc0471439aa0085ca87deccf061c5b676ef12388)

(From OE-Core rev: 4eac9a5337d93b6cbd3916af97f62bb04881c9cd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Yeoh Ee Peng
4ca53b38ce oeqa/runtime/dnf: Make sure test_dnf_install does not skipped
During debugging dnf issue, we found that the test_dnf_install
PASSED the testing even though the environment does not allow
dnf install to run successfully. Further debugging had identified
that current test_dnf_install will execute dnf install even when
the package to be installed already exist, thus dnf install
will just skipped and this test will PASSED even though it was
not.

To solve this, added additional logic to check if the package
to be installed already exist, if yes then remove the package
before actually run dnf install. This will make sure dnf install
was tested as expected.

(From OE-Core rev: 4f662b253f7313c4e02bfafb527cdac076b6309a)

(From OE-Core rev: a80498e62898110e2ed6b01cbb9f5dd85995d13b)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Armin Kuster
f663c39ce2 perl: add testdepends for ssh
fixes:

 DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.4 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; perl -e '$_="Uryyb, jbeyq"; tr/a-zA-Z/n-za-mN-ZA-M/;print'
| DEBUG: time: 1548816904.4024463, endtime: 1548817204.397057
| DEBUG: Partial data from SSH call: ssh: connect to host 192.168.7.4 port 22: Connection refused

for master/thud/sumo

(From OE-Core rev: 76c66e061cdcdcbad73ed503668115120feb0ea3)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Ross Burton
5a8fcc4d0c oeqa: rationalise Perl tests
As with the Python test, this can be both better and faster.  No need to copy a
file, just run a one-liner.

(From OE-Core rev: c6eef46747fe58bb2310be4f06d2fa9b67901d72)

(From OE-Core rev: 9188ef8d1edbba8041a73d3bb8a9bfd194db0e92)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
4b21bf658f oeqa/utils/buildproject: Only clean files if we've done something
We should only be wiping out things on target if the tests have
actually run.

(From OE-Core rev: d38c3eac0a5a1a9b0eb98385832e92f48145655e)

(From OE-Core rev: dab22dc58eabaeb421afa3c7de1cc08c5ec34c61)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Robert Yang
0bda550627 oeqa/selftest/runqemu: Enable kvm when QEMU_USE_KVM is set
(From OE-Core rev: 564de3681353fe8e203425388e8be9703a89d2da)

(From OE-Core rev: 1207949c986cc9f6b3940a99e87bc4d8cfb86d5b)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Yeoh Ee Peng
3c7d4f0526 oeqa/qemu & runtime: qemu do not need ip input from external
Qemu do not use the ip input from external. It will
retrieve ip from QemuRunner instance and assign
ip value.

(From OE-Core rev: 14d99dc6c39c963ba3e0d9a30274846bd5369210)

(From OE-Core rev: e4990ae01c9f3f486b4b745a2602795e9d496109)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Richard Purdie
e1b97539c8 oeqa/runtime/cases: Improve dependencies of kernel/gcc/build tests
Mark up these tests as needing a compiler, make and kernel source code
as appropriate, the image feature requirements can then be retired.

(From OE-Core rev: 9f64e6c25abdf494fb511e9cd401f8dcaa08be2a)

(From OE-Core rev: 544c56b6950f0598d5eaf62d9ae9d9de5fcb7eaf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Richard Purdie
d853fa2680 oeqa/runtime/cases: Improve test dependency information
Add the OEHasPackage decorator to a variety of tests so they determine
automatically if they should run against a given image.

To ensure tests can do this we need to move target operations such
as scp commands into the tests and out of the class startup/teardown.

(From OE-Core rev: 60d6580b85714b8960a964e775d76a7f937f5e5a)

(From OE-Core rev: 03b7658369bb7c1c8fbbaac7d9e281617cc16135)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Ross Burton
e9055b90c0 oeqa/runtime/python: clean up Python test
Currently this is three test cases:

1) test_python_exists.  Fail if python3 isn't in PATH.
2) test_python_stdout.  Run a Python script and check the output is as expected
3) test_python_testfile.  Check that a file test_python_stdout wrote to exists.

(1) should be a setup and skip the test module if it isn't present.
(2) and (3) should be merged, there's no point copying over a two line Python
file, and the test doesn't verify that the file doesn't exist in the first
place.

Rewrite the test to check that Python is present in a class setup so the entire
test is skipped if it isn't and do some simple rot13 to verify that bytecode is
being executed correctly.

(From OE-Core rev: a35be5f32b4fe70b18ac1e2eccfd94558cecfbba)

(From OE-Core rev: babdd2f6addb06c23e4882107be07034ca49bcb8)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Alexander Kanavin
128b07d200 oeqa/runtime/cases/python: use python 3 rather than python 2
For example, core-image-sato skipped the test alltogether, as it
no longer pulls in Python 2.x at all.

(From OE-Core rev: 5ad0fe9ac6b6362011a17afaa7bee8e788093915)

(From OE-Core rev: bde50fc78ae75fd585f8914e458d65c328857fb2)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Chen Qi
0676240f4c oeqa/runtime/cases/rpm.py: change to use base-passwd-doc
The test cases assume that rpm-doc is built out, as it says it its
comment. This is not always true. And it sometimes results in
following error.

  |     cls.tc.target.copyTo(test_file, dst)
  | UnboundLocalError: local variable 'test_file' referenced before assignment

Change to use base-passwd-doc, as this package is more likely to be
built out than rpm-doc.

(From OE-Core rev: 2e23543b48921182307065c1fa9e8b9d7fbb3cdc)

(From OE-Core rev: 1ce378ce07d2c49f40054893a623456c8471e177)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Alexander Kanavin
c172c8fdc6 oeqa: replace Alex Kanavin's @intel email address with a personal one
As I will be leaving Intel, this address will no longer be valid,
so swap it for my personal one for now.

(From OE-Core rev: ea58ff101e28dfda3410de66d775df3d8a1e5a96)

(From OE-Core rev: 61bca5a7f6b3fc03cd88cbf8867708d28e0a3522)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Robert Yang
e8b9772ab8 oeqa: make it work for multiple users
There are failures when multiple users run oe-selftest on the same
host:

PermissionError: [Errno 13] Permission denied: '/tmp/oe-saved-
tests/201812250324_qemu'

This is because /tmp/oe-saved-tests was created by user A, while user B tries
to write data in it, then the error will happen. This patch can fix the
problem.

Move the dumped data to ${LOG_DIR}/runtime-hostdump/ rather than
/tmp/oe-saved-tests/ to fix the problem.

(From OE-Core rev: e219fe5329599cd6c3682f521eaee3852a2c8980)

(From OE-Core rev: 872c6e5101f4f27dcfc63d141d1b6568f46d2b5f)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Jose Perez Carranza
79d42d2b2f runtime/dnf: Add new dnf test cases
Add test cases to test “exclude” and “installroot“ options, also modify
the logic of filtering packages on the feed to have all the packages
needed by the tests.

[YOCTO #10744]

(From OE-Core rev: 1121806603c6f621d084b692216f3f616a0768dc)

(From OE-Core rev: e1b050f53ece2a31cd6866d2d737d7c67a44cea4)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Robert Yang
122d638e22 oeqa: Fix for QEMU_USE_KVM
Fixed:
MACHINE = "qemux86"
QEMU_USE_KVM = "qemux86"
IMAGE_CLASSES += "testimage"

$ oe-selftest -r runqemu.RunqemuTests.test_boot_rootfs

[snip]
  File "/buildarea1/lyang1/poky/meta/lib/oe/types.py", line 122, in boolean
    raise ValueError("Invalid boolean value '%s'" % value)
ValueError: Invalid boolean value 'qemux86'

Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more, kvm
will be enabled if target_arch == build_arch or both of them are x86 archs.

(From OE-Core rev: 7c1a8a624cad8d967635c6cb5f99cf655bde3d44)

(From OE-Core rev: de1b80f7f7b787f6b5b62c576ca6c62d2440031c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Richard Purdie
fa5ebe62e2 testimage/testsdk/selftest: Avoid platform.distro_identifier deprecation warnings
Use our own lsb function instead as used elsewhere by the codebase.

(From OE-Core rev: acac45a6fd604d28ef7c23d67482af3d7e8bcfe3)

(From OE-Core rev: 570256a64af5a3fa994a20a5cc4c74d59ffc361f)

(From OE-Core rev: d58fe9d352ae7de857e7f55b88f6e7d35b2cd706)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Robert Yang
7375570a24 oeqa/utils/qemurunner: set timeout to 60s for run_serial
The 5s timeout for non-kvm is too short, especially when the load is high,
which leads to unexpected errors, so set timeout to 60s by default.

(From OE-Core rev: 8197d0f638a760fc03062c7a9009117d083d7ead)

(From OE-Core rev: 6c930a8f5c11947c3b916efe23f33af462701bee)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Robert Yang
13ab7c2229 oeqa/utils/qemurunner: Print output when failed to login
This is useful for debugging.

(From OE-Core rev: da527d73f5eeae0f29b5f99aab757491d3f87ec7)

(From OE-Core rev: 7fb953573635afb1b6482dbd9f51eb0f05731eae)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:45 +00:00
Richard Purdie
6512ffb090 oeqa/selftest/esdk: Fix typo causing test failure
2018-12-06 23:19:24,564 - oe-selftest - INFO - Traceback (most recent call last):
  File "/media/build1/poky-sumo/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/media/build1/poky-sumo/meta/lib/oeqa/selftest/cases/eSDK.py", line 78, in setUpClass
    cls.tmpdir_eSDKQA = cls.tempdirobj.name
AttributeError: type object 'oeSDKExtSelfTest' has no attribute 'tempdirobj'

(From OE-Core rev: 2c60908a2039d333a9fe2651622750ff6ed4cce1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
766a95ecb2 oeqa/selftest/esdk: Ensure parent directory exists
INFO - ======================================================================
INFO - ERROR: setUpClass (eSDK.oeSDKExtSelfTest)
INFO - ----------------------------------------------------------------------
INFO - Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/eSDK.py", line 76, in setUpClass
    cls.tmpdirobj = tempfile.TemporaryDirectory(prefix="selftest-esdk-", dir=bb_vars["WORKDIR"])
  File "/usr/lib/python3.5/tempfile.py", line 929, in __init__
    self.name = mkdtemp(suffix, prefix, dir)
  File "/usr/lib/python3.5/tempfile.py", line 507, in mkdtemp
    _os.mkdir(file, 0o700)
FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-debian/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-q7ln84gc'

(From OE-Core rev: 170a601a99836d13b69e5287bee0d3e71983dd46)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
5df29d420a image-buildinfo,oeqa/selftest/containerimage: Ensure image-buildinfo doesn't break tests
Having image-buildinfo enabled causes containerimage.ContainerImageTests.test_expected_files
to fail due to the presence of an unexpected file:
  ['./',
   './etc/',
-  './etc/build',
   './etc/default/',
   './etc/default/postinst',

Tweak the class to allow it to be disabled and disable it from the test just in
case it was enabled.

(From OE-Core rev: af67bf422a4df5b7e07894512ff73a5f493682ab)

(From OE-Core rev: f49ab8b1610c045acaed7b964d12f07f969df856)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
1f1367a59d oeqa/utils/metadata: Allow to function without the git module
The python git module may or may not be enabled, allow this code to
function without it, falling back to the same method as metadata_scm.bbclass
uses. This will be cleaned up in the next round of feature development.

(From OE-Core rev: 6350586ba9f4a4107a2d457590824cd4d662d5b9)

(From OE-Core rev: 32c9169b76e13e53b6a9ab4a59932cea7863d992)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Yeoh Ee Peng
152a347f8d oeqa/selftest: Standardize json logging output directory
Currently sdk & sdkext will output json file to LOG_DIR, while
selftest will output json file to TOPDIR/log.

Standardize selftest json output file to LOG_DIR.

(From OE-Core rev: 9f2e39684cbbe9f87eeef6a81961e6db783439e3)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
f50977c3b5 oeqa/selftest: Improvements to the json logging
Tweak the preceeding commit to:

* Add STARTTIME to the identifier to make it unique
* Use LOG_DIR
* Store the layer config in a more natural json format
* Drop '_' function prefixes

(From OE-Core rev: 173f59acf9722e2ef27fdd49c20f7d3d664917eb)

(From OE-Core rev: 3b69099edc7db99c11bfb41eab2af50bd0e3d4f2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Yeoh Ee Peng
5a99880992 oeqa/selftest/context: write testresult to json files
As part of the solution to replace Testopia to store testresult,
OEQA selftest need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.

By default, json testresult file will be written to "oeqa"
directory under the oe-selftest log directory.

To configure multiple instances of bitbake to write json testresult
to a single testresult file at custom directory, user will define
the variable "OEQA_JSON_RESULT_DIR" with the custom directory for
json testresult.

(From OE-Core rev: a95218525a4c8228fff9908ffbda85c6b85e101c)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
3f2c5e0e24 oeqa/runner: Sort the test result output by result class
We want to see failures/errors listed last since this is the most easily
visible part of the log on consoles or autobuilder output and makes
human processing easier rather than having to scroll up and scan for
a single failure.

(From OE-Core rev: 2cc07ab253f1ba6a1f07a66051c9ba6d98cd2357)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
1123da4368 oeqa/runner: Always show a summary of success/fail/error/skip counts
Its useful to have the counts of success/failure/error/skipped at the end of the
results to allow for easier human reading of what happened.

(From OE-Core rev: 5942318a261ce7a885f351e214669068ff9d8931)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
ed5c12f11f oeqa/runtime/ptest: Inject results+logs into stored json results file
This allows the ptest results from ptest-runner, run in an image to be
transferred over to the resulting json results output.

Each test is given a pass/skip/fail so individual results can be monitored
and the raw log output from the ptest-runner is also dumped into the
results json file as this means after the fact debugging becomes much easier.

Currently the log output is not split up per test but that would make a good
future enhancement.

I attempted to implement this as python subTests however it failed as the
output was too confusing, subTests don't support any kind of log
output handling, subTest successes aren't logged and it was making things
far more complex than they needed to be.

We mark ptest-runner as "EXPECTEDFAILURE" since its unlikely every ptest
will pass currently and we don't want that to fail the whole image test run.
Its assumed there would be later analysis of the json output to determine
regressions. We do have to change the test runner code so that
'unexpectedsuccess' is not a failure.

Also, the test names are manipuated to remove spaces and brackets with
"_" used as a replacement and any duplicate occurrences truncated.

(From OE-Core rev: a13e088942e2a3c3521e98954a394e61a15234e8)

(From OE-Core rev: 526ceab9d0e43f73635bb92e8dd7763ef75ad33b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Yeoh Ee Peng
eee5b0d104 oeqa/core/runner: write testresult to json files
As part of the solution to replace Testopia to store testresult,
OEQA need to output testresult into single json file, where json
testresult file will be stored in git repository by the future
test-case-management tools.

The json testresult file will store more than one set of results,
where each set of results was uniquely identified by the result_id.
The result_id would be like "runtime-qemux86-core-image-sato", where
it was a runtime test with target machine equal to qemux86 and running
on core-image-sato image. The json testresult file will only store
the latest test content for a given result_id. The json testresult
file contains the configuration (eg. COMMIT, BRANCH, MACHINE, IMAGE),
result (eg. PASSED, FAILED, ERROR), test log, and result_id.

Based on the destination json testresult file directory provided,
it could have multiple instances of bitbake trying to write json
testresult to a single testresult file, using locking a lockfile
alongside the results file directory to prevent races.

Also the library class inside this patch will be reused by the future
test-case-management tools to write json testresult for manual test
case executed.

(From OE-Core rev: 00e03b5004f1eb6d59295544b3a8620504278f51)

(From OE-Core rev: 045511425577ccbe89d8eb91e2a87e385390cabf)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Yeoh Ee Peng
b3b337adb3 oeqa/core/runner: refactor for OEQA to write json testresult
Refactor the original _getDetailsNotPassed method to return
testresult details (test status and log), which will be reused
by future OEQA code to write json testresult.

Take the opportunity to consolidate and simplify the logic used
to gather test status and log within the TestResult instance.

(From OE-Core rev: 79ee7d1c371a86edeb61c99679985118da657e5d)

(From OE-Core rev: bc444181f9658423856621b2f2c60364642ae5b1)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
52fba04068 oeqa: Remove xmlrunner
This isn't present on modern distros by default and doesn't work with
testtools, needing multiple code paths in the code. Remove it in favour
of finding a better replacement for results collection/analysis.

(From OE-Core rev: 8001d933a8dc86004db014777f094d718086687d)

(From OE-Core rev: 02449e89d62a7714d30ab6e2e58c476dc9441f7a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
d8dc75de8b oeqa/runner: Simplify code
There doesn't appear to be any reason we need this _results indirection
any more so remove it.

(From OE-Core rev: b618261811c48ff3b98eab1b340a8cd09ef183c6)

(From OE-Core rev: ab271b49d9b55ea271d519c3a4da0b639a07f0bb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
58678f5aa7 oeqa/core/threaded: Remove in favour of using concurrenttests
We have several options for parallel processing in oeqa, parallel
execution of modules, threading and mulitple processes for the runners.

After much experimentation is appears the most scalable and least
invasive approach is multiple processes using concurrenttestsuite
from testtools. This means we can drop the current threading code
which is only used by the sdk test execution.

oeqa/decorator/depends: Remove threading code

Revert "oeqa/sdk: Enable usage of OEQA thread mode"
This reverts commit adc434c063.

Revert "oeqa/core/tests: Add tests of OEQA Threaded mode"
This reverts commit a4eef558c9.

Revert "oeqa/core/decorator/oetimeout: Add support for OEQA threaded mode"
This reverts commit d3d4ba902d.

(From OE-Core rev: a98ab5e560e73b6988512fbae5cefe9e42ceed53)

(From OE-Core rev: bb9a85e157e669d7a91c3bbefc8d5138e7b8b6ae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
783e1ae8fa oeqa/runner: Ensure we don't print misleading results output
The current code assumes if something isn't a failure of some
kind, it was a pass. When test case IDs weren't matching, this lead
to very confusing output where things would fail, then be listed as
passing.

This adds code to track successes, ensuring we don't end up in this
position again with unmatched entries being listed as UNKNOWN.

(From OE-Core rev: 4374c296d8963e4f6a1aa7bef7983ad0a1c2fcff)

(From OE-Core rev: bcb2948773d76befef2be787be6d25cf544e49a9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
521d02e6f5 oeqa/core/runner: Improve test case comparision
We can directly compare the test case IDs rather than representations,
then if we're using subunit to split the tests, the comparisions still
work as intended.

(From OE-Core rev: 72e5f46f75454ba4c445c65c1cbc616a9e72fc6e)

(From OE-Core rev: 4c53aac5315f9d0a0ed95cbeb48b7704e274e3a9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
fa12ad627b oeqa/selftest/context: Improve log file handling
The existing logfile is simply placed in the current directory. Since the test
changes cwd to BUILDDIR, the symlink to the log can be placed in an invalid
directory. We also see trackbacks if the symlink is invalid.

Improve things by:

* Placing logs in LOG_DIR (or BUILDDIR if unset).
* Using a full path to the log meaning the log and link are placed in the same directory.
* Using lexists instead of exists so invalid symlinks are handled correctly.

(From OE-Core rev: 750ece11bed0e62a11e0003d1d16a81f7c219761)

(From OE-Core rev: 9c4c3c876dd5d224133571fcad1095af1098ae1d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
c7a841625d oeqa/utils/qemurunner.py: Fix python regex warnings
Fix the warnings:

meta/lib/oeqa/utils/qemurunner.py:250: DeprecationWarning: invalid escape sequence \.
  ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1])
meta/lib/oeqa/utils/qemurunner.py:343: DeprecationWarning: invalid escape sequence \-
  if re.search("root@[a-zA-Z0-9\-]+:~#", output):
poky/meta/lib/oeqa/utils/qemurunner.py:350: DeprecationWarning: invalid escape sequence \-
  if re.search("root@[a-zA-Z0-9\-]+:~#", output):
meta/lib/oeqa/utils/qemurunner.py:448: DeprecationWarning: invalid escape sequence \-
  if re.search("[a-zA-Z0-9]+@[a-zA-Z0-9\-]+:~#", data):

by correctly marking the regexs.

(From OE-Core rev: 8e6987735002560fca714f77ea8ece9d4b28f7fa)

(From OE-Core rev: 7eb1f0be82d7ee12f893cdd40384da306fa597a7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
ea1539b872 oeqa/selftest/context: Replace deprecated imp module usage
Avoid the warning:

meta/lib/oeqa/selftest/context.py:8: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp

In this case importlib is a direct replacement.

(From OE-Core rev: db7a60c36a2d3eefc61ae6e1ede01680dc932035)

(From OE-Core rev: 351a7b75959593922909d7e1929a6429a2bf94a7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
76d7cf2415 oeqa/utils/commands: Avoid unclosed file warnings
Avoid warnings such as:

meta/lib/oeqa/utils/commands.py:213: ResourceWarning: unclosed file <_io.BufferedReader name=4>
  return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)

(From OE-Core rev: 6a68c42de08cffbadb59ebda63fa5e19f6e5acef)

(From OE-Core rev: 682d7b2810b235e86a28a8afe034e3853dbe8c45)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
eb0dd09cf5 oeqa/loader: Fix deprecation warning
Clean up the warning:
meta/lib/oeqa/core/loader.py:27: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  _failed_test_args = inspect.getargspec(unittest.loader._make_failed_test).args

(From OE-Core rev: d2deb66830be2d44532fea3d5db763b57778252a)

(From OE-Core rev: f9ab10bb08446052fd6af2a21f38d8454e466d51)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Ross Burton
5f74f3c639 oeqa/selftest/esdk: run selftest inside workdir not /tmp
We've seen issues with rootfs size calculations and we've seen systems
like opensuse which have btrfs mounted on /tmp causing selftest failures.

(From OE-Core rev: 61be3cd748d1b7321a1fc4cfe84efa9b26a6aee0)

(From OE-Core rev: d936faabfb29ea377d74e77332a2a91603747ac7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Ross Burton
899f4baef7 oeqa: don't litter /tmp with temporary directories
If we need to create a temporary directory in targetbuild or buildproject use
tempfile.TemporaryDirectory so that when the test case is finished, the
directory is deleted.

Also synchronise the logic and don't possibly store the temporary directory in
self.tmpdir as nothing uses that.

(From OE-Core rev: db0e658097130d146752785d0d45f46a3e0bad71)

(From OE-Core rev: d39252324a13580cc96f0694b88bc10515e030a0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
2fae64eae4 oeqa/utils/qemurunner: Avoid tracebacks on closed files
Reorder the shutdown/teardown to avoid:

  File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 224, in launch
    op = self.getOutput(output)
  File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 90, in getOutput
    fl = fcntl.fcntl(o, fcntl.F_GETFL)
ValueError: I/O operation on closed file

(From OE-Core rev: 8e7d756862d2a8d62f3c87497d6d65ddb3c1b962)

(From OE-Core rev: 48979ffbe25351f92179021a973207a71bbe7a4e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
9bd0b59f73 oeqa/selftest/runqemu: Improve testcase failure handling
assertTrue doesn't give good debug information when things fail. Update
several to use assertIn which gives information upon failure, for the
others print the log information upon failure.

(From OE-Core rev: c29cb75d5ce6b0873a934f4709b0c8824f7164d3)

(From OE-Core rev: 6f2bb2c12289422396deb793ae4b4e99cfa88c7b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Ross Burton
765f90b383 oeqa/oelib/path: don't leak temporary directories
setUp() is used to populate a directory of temporary files, and deleted in
__del__.  However setUp() is called once *per test* so __del__ would only be
able to remove the last directory created.

Fix the code by using the natural counterpart to setUp, tearDown(), to clean up.

(From OE-Core rev: 68b4723e6fb11d171869185bccf28f32f6284c18)

(From OE-Core rev: edd2ecbc86d8b02a0467491451306f67e81d9ead)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
202097fe5b oeqa/selftest/buildoptions: Ensure diskmon tests run consistently
Heartbeat events default to once a second and we need to ensure we have
enough time in the task to see them.

Add a nostamp delay task 5s long so we can have a consistently timed
task which doesn't need cleanup or have unneeded dependencies. This
ensures we should deterministically see the disk moinitor events
regardless of the state of the build. This is done in a way which
doesn't corrupt build state or need cleanup and is efficient.

(From OE-Core rev: ecc49ee8986929e2429d948000a0ca588fe63959)

(From OE-Core rev: d5ce38168238181423c9dcd1d258253e3515d0a5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00
Richard Purdie
26f65c2612 oeqa/selftest/buildoptions: Improve ccache test
This test occisionally fails as m4 doesn't recompile, meaning the logfile test
then doesn't find mention of ccache.

To ensure m4 does recompile, clean m4 before force compiling it.

(Reading the test is confusing due to the test cleanup also involving a clean)

(From OE-Core rev: 6e0b9214a0d57ed45a5df0ba5c9887a9045b89b1)

(From OE-Core rev: b729a46820cce38b3d500271df19bdaf31973140)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:27 +00:00