Commit Graph

1373 Commits

Author SHA1 Message Date
Richard Purdie
f89486cda1 oeqa/utils/logparser: Simplify ptest log parsing code
logparser is only used by ptest. Its slightly overcomplicated as it was
intended to be reusable but wasn't. Simplify it as a dedicated parser is
likely to me more readable and maintainable.

(From OE-Core rev: c7478345b2b4a85cb1fec40e762633871f0e94cb)

(From OE-Core rev: 1a6fcc97f3842deed3e78ededa8a21da274c0572)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:39 +00:00
Richard Purdie
31e10dd927 oeqa/runtime/ptest: Avoid traceback for tests with no section
Some tests end up without a section, avoid tracebacks trying to use
None as a string in that case.

(From OE-Core rev: 86fb5d898a29761f120c2eaa538a32cf2e078487)

(From OE-Core rev: fca18914013b2f05b36e6985a7ff4a7c8dddf8a5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:39 +00:00
Kai Kang
b023be7714 selftest/wic: update test case test_qemu
This test case boots the image in qemu and checks for mounted partitions.
But the outputs of mount are different between sysvinit and systemd:

sysvinit:
/dev/root /\r\n/dev/sda1 /boot\r\n/dev/sda3 /media\r\n/dev/sda4 /mnt
systemd:
/dev/sda1 /boot\r\n/dev/sda2 /\r\n/dev/sda3 /media\r\n/dev/sda4 /mnt

So check mounted partitions by egrep rather than check output of runqemu.

(From OE-Core rev: 8e3311d45ec68d74e934e966f67cf5102634981c)

(From OE-Core rev: 828db9026c61d0a6ad12b8e05929eef31abfd57d)

Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:39 +00:00
Armin Kuster
a9b4266f62 perl: add testdepends for ssh
fixes:

 DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.4 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; perl -e '$_="Uryyb, jbeyq"; tr/a-zA-Z/n-za-mN-ZA-M/;print'
| DEBUG: time: 1548816904.4024463, endtime: 1548817204.397057
| DEBUG: Partial data from SSH call: ssh: connect to host 192.168.7.4 port 22: Connection refused

for master/thud/sumo

(From OE-Core rev: a590e7805e3bec5dd995f7ea0b9e79a21f82b48b)

(From OE-Core rev: 7dbccf13493dad534c0cc604bc4c5ab16e5f0fc0)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:39 +00:00
Chen Qi
90dec2ffa5 eSDK.py: unset BBPATH and BUILDDIR to avoid eSDK failure
When executing eSDK test case, the following error appears.

  WARNING: attempting to use the extensible SDK in an environment
  	   set up to run bitbake - this may lead to unexpected
	   results. Please source this script in a new shell session
	   instead.

  FileExistsError: [Errno 17] File exists: '/.../tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-fcuyzsqu/tmp/sysroots/x86_64/bin/pigz' -> '/.../tmp/hosttools/pigz'

So unset these two vars to avoid messing things up.

(From OE-Core rev: bc07d825ce8bb3b337623c232fef61f2781c82af)

(From OE-Core rev: 748f946ee74f7480200a7eb0bb0b695467b08f0a)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 16:38:31 +00:00
Chen Qi
ecec5a8fdb eSDK.py: avoid error in tearDownClass due to race condistion
When removing the temporary directory, it's possible that bitbake.lock
file is removed by bitbake during the cleanup. And this leads to the
following error.

  FileNotFoundError: [Errno 2] No such file or directory: 'bitbake.lock'

So add a check to remove this file before cleaning up the temporary
directory.

(From OE-Core rev: 984f56b37bd0014e5bf9509fc8ed181973e61773)

(From OE-Core rev: 6a5cce862f1e950439dc8ee1d84a10397a7fac4f)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 16:38:31 +00:00
Richard Purdie
6731390a1a oeqa/utils/buildproject: Only clean files if we've done something
We should only be wiping out things on target if the tests have
actually run.

(From OE-Core rev: d38c3eac0a5a1a9b0eb98385832e92f48145655e)

(From OE-Core rev: 34efb674999781448411a04f876d6b3423ebe26c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 16:38:31 +00:00
Robert Yang
22243e44c1 oeqa/selftest/runqemu: Enable kvm when QEMU_USE_KVM is set
(From OE-Core rev: 564de3681353fe8e203425388e8be9703a89d2da)

(From OE-Core rev: ffea871575d0e80f73daf6b36c90798ce5e055e6)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 16:38:31 +00:00
Robert Yang
5af780686d oeqa/manual/bsp-qemu.json: Update for QEMU_USE_KVM
Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more.

(From OE-Core rev: 37e5035a2ba27595eeef22062580a5b04a73ba15)

(From OE-Core rev: 63f0ce0e084d2d24daeacf226d9c432e4a53fd21)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 16:38:31 +00:00
Yeoh Ee Peng
5e3b9adf28 oeqa/qemu & runtime: qemu do not need ip input from external
Qemu do not use the ip input from external. It will
retrieve ip from QemuRunner instance and assign
ip value.

(From OE-Core rev: 14d99dc6c39c963ba3e0d9a30274846bd5369210)

(From OE-Core rev: c3d51e92f412becd3e067c637ffbdc5ad13e8894)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 16:38:31 +00:00
Richard Purdie
22ebbe5c2a oeqa/runtime/cases: Improve dependencies of kernel/gcc/build tests
Mark up these tests as needing a compiler, make and kernel source code
as appropriate, the image feature requirements can then be retired.

(From OE-Core rev: 9f64e6c25abdf494fb511e9cd401f8dcaa08be2a)

(From OE-Core rev: e65d1680d406e50313f90e659527b9aa795eeb45)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 16:38:31 +00:00
Richard Purdie
aaab0dfa83 oeqa/runtime/cases: Improve test dependency information
Add the OEHasPackage decorator to a variety of tests so they determine
automatically if they should run against a given image.

To ensure tests can do this we need to move target operations such
as scp commands into the tests and out of the class startup/teardown.

(From OE-Core rev: 60d6580b85714b8960a964e775d76a7f937f5e5a)

(From OE-Core rev: 601f87016c6772a4ec8410bccb195989bcf05b55)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 16:38:31 +00:00
Joshua Watt
676046bc36 classes/testsdk: Split implementation into classes
Splits the SDK test implementation into configurable Python classes. The
classes used for the normal and extensible SDKs are
${TESTSDK_CLASS_NAME} and ${TESTSDKEXT_CLASS_NAME} respectively.

This allows SDK machines to override the classes used to implement the
tests. For the traditional SDK, a common "run()" function is provided by
the class (oeqa.sdk.testsdk.TestSDK), with several hook member functions
that can be overridden in child classes, making it easier to have
consistent behavior. The extensible SDK class
(oeqa.sdkext.testsdk.TestSDKEXT) also has a common "run()" function, but
no hooks have yet been added as there is not currently a known use case
for create derived classes.

These changes should be purely organizational; no functional changes
have been made to either the standard SDK or extensible SDK tests.

[YOCTO #13020]

(From OE-Core rev: a06d53928b22d5f88276023c4d57b206db2f27f9)

(From OE-Core rev: d0828522fc3a4bd3f86488edcedcce10833c18dc)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 16:38:30 +00:00
Robert Yang
f54aac732b oeqa: make it work for multiple users
There are failures when multiple users run oe-selftest on the same
host:

PermissionError: [Errno 13] Permission denied: '/tmp/oe-saved-
tests/201812250324_qemu'

This is because /tmp/oe-saved-tests was created by user A, while user B tries
to write data in it, then the error will happen. This patch can fix the
problem.

Move the dumped data to ${LOG_DIR}/runtime-hostdump/ rather than
/tmp/oe-saved-tests/ to fix the problem.

(From OE-Core rev: e219fe5329599cd6c3682f521eaee3852a2c8980)

(From OE-Core rev: 310a2b137f2cb249885844fa12830365f94866f0)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 16:38:30 +00:00
Robert Yang
6b6b442f3e oeqa: Fix for QEMU_USE_KVM
Fixed:
MACHINE = "qemux86"
QEMU_USE_KVM = "qemux86"
IMAGE_CLASSES += "testimage"

$ oe-selftest -r runqemu.RunqemuTests.test_boot_rootfs

[snip]
  File "/buildarea1/lyang1/poky/meta/lib/oe/types.py", line 122, in boolean
    raise ValueError("Invalid boolean value '%s'" % value)
ValueError: Invalid boolean value 'qemux86'

Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more, kvm
will be enabled if target_arch == build_arch or both of them are x86 archs.

(From OE-Core rev: 7c1a8a624cad8d967635c6cb5f99cf655bde3d44)

(From OE-Core rev: 2c12e1d9c262fb7e5fe02ae2d092789d6536413f)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 16:38:30 +00:00
Robert Yang
938ada24fb oeqa/utils/qemurunner: set timeout to 60s for run_serial
The 5s timeout for non-kvm is too short, especially when the load is high,
which leads to unexpected errors, so set timeout to 60s by default.

(From OE-Core rev: 8197d0f638a760fc03062c7a9009117d083d7ead)

(From OE-Core rev: 70e942e77493cb5851a78c5592b4e13de860d931)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 16:38:30 +00:00
Robert Yang
fc6d6411aa oeqa/utils/qemurunner: Print output when failed to login
This is useful for debugging.

(From OE-Core rev: da527d73f5eeae0f29b5f99aab757491d3f87ec7)

(From OE-Core rev: b09f261867fbdf206afc1059fc5a84a52d49a97a)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 16:38:30 +00:00
Richard Purdie
074cd4f22c oeqa/selftest/esdk: Fix typo causing test failure
2018-12-06 23:19:24,564 - oe-selftest - INFO - Traceback (most recent call last):
  File "/media/build1/poky-sumo/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/media/build1/poky-sumo/meta/lib/oeqa/selftest/cases/eSDK.py", line 78, in setUpClass
    cls.tmpdir_eSDKQA = cls.tempdirobj.name
AttributeError: type object 'oeSDKExtSelfTest' has no attribute 'tempdirobj'

(From OE-Core rev: 75cd4edaa8a42f76c0594ce26df05c7a51d620df)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
2cafee8620 oeqa/selftest/esdk: Ensure parent directory exists
INFO - ======================================================================
INFO - ERROR: setUpClass (eSDK.oeSDKExtSelfTest)
INFO - ----------------------------------------------------------------------
INFO - Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/eSDK.py", line 76, in setUpClass
    cls.tmpdirobj = tempfile.TemporaryDirectory(prefix="selftest-esdk-", dir=bb_vars["WORKDIR"])
  File "/usr/lib/python3.5/tempfile.py", line 929, in __init__
    self.name = mkdtemp(suffix, prefix, dir)
  File "/usr/lib/python3.5/tempfile.py", line 507, in mkdtemp
    _os.mkdir(file, 0o700)
FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-debian/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-q7ln84gc'

(From OE-Core rev: eca3c0a6aa4bce48f295ba25f613da8dcaefac20)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
237ff3e462 testimage/testsdk/selftest: Avoid platform.distro_identifier deprecation warnings
Use our own lsb function instead as used elsewhere by the codebase.

(From OE-Core rev: acac45a6fd604d28ef7c23d67482af3d7e8bcfe3)

(From OE-Core rev: 570256a64af5a3fa994a20a5cc4c74d59ffc361f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
ea1ffae258 oeqa/concurrencytest: Avoid unclosed file warnings
Avoid an unclosed file per thread warning when running selftests concurrently
by closing the result stream.

(From OE-Core rev: 33a4a076e8aa72a872807332501e7f5ae1cee0e2)

(From OE-Core rev: a7dceca55b169bcdb8d1528238cbdedfd131f37f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
19287294c1 oeqa/utils/qemurunner.py: Fix python regex warnings
Fix the warnings:

meta/lib/oeqa/utils/qemurunner.py:250: DeprecationWarning: invalid escape sequence \.
  ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1])
meta/lib/oeqa/utils/qemurunner.py:343: DeprecationWarning: invalid escape sequence \-
  if re.search("root@[a-zA-Z0-9\-]+:~#", output):
poky/meta/lib/oeqa/utils/qemurunner.py:350: DeprecationWarning: invalid escape sequence \-
  if re.search("root@[a-zA-Z0-9\-]+:~#", output):
meta/lib/oeqa/utils/qemurunner.py:448: DeprecationWarning: invalid escape sequence \-
  if re.search("[a-zA-Z0-9]+@[a-zA-Z0-9\-]+:~#", data):

by correctly marking the regexs.

(From OE-Core rev: 8e6987735002560fca714f77ea8ece9d4b28f7fa)

(From OE-Core rev: a980cb8a0940d4db4bb5d338650cf848cd292f5b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
0a5b7850e5 oeqa/selftest/context: Replace deprecated imp module usage
Avoid the warning:

meta/lib/oeqa/selftest/context.py:8: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp

In this case importlib is a direct replacement.

(From OE-Core rev: db7a60c36a2d3eefc61ae6e1ede01680dc932035)

(From OE-Core rev: 1e7bbdaf78fd2a75d03c6a0c48afed13fffd4397)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
01d00cd776 oeqa/utils/commands: Avoid unclosed file warnings
Avoid warnings such as:

meta/lib/oeqa/utils/commands.py:213: ResourceWarning: unclosed file <_io.BufferedReader name=4>
  return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)

(From OE-Core rev: 6a68c42de08cffbadb59ebda63fa5e19f6e5acef)

(From OE-Core rev: f65a5fbd4fd13a52b54c808a6f5d2afab426e050)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
f349ec0424 oeqa/core/runner: Correctly markup regexs
Avoid the warning "DeprecationWarning: invalid escape sequence \(" by marking
the regexs correctly.

(From OE-Core rev: cb49980fa4a158d5529902df731dec61a8c9b3d4)

(From OE-Core rev: bb2cddb08d7c4c6c56dfe6b2f0d26dce1cdf20e4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
e09a79e997 oeqa/loader: Fix deprecation warning
Clean up the warning:
meta/lib/oeqa/core/loader.py:27: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  _failed_test_args = inspect.getargspec(unittest.loader._make_failed_test).args

(From OE-Core rev: d2deb66830be2d44532fea3d5db763b57778252a)

(From OE-Core rev: e65e9492acad4861b22e6f29d5f470a82778f2b4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
86dbdfa00d oeqa/runner: Sort the test result output by result class
We want to see failures/errors listed last since this is the most easily
visible part of the log on consoles or autobuilder output and makes
human processing easier rather than having to scroll up and scan for
a single failure.

(From OE-Core rev: 7954b19020c28a4120bc1671aa81b9e1e2b05fa2)

(From OE-Core rev: 260738158b09aea0beeca85b778aa4ab08ba1c4c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
359ce89adf oeqa/runner: Always show a summary of success/fail/error/skip counts
Its useful to have the counts of success/failure/error/skipped at the end of the
results to allow for easier human reading of what happened.

(From OE-Core rev: 080d8900d470a8e7f929b0c5c2765ad461744fbb)

(From OE-Core rev: 5bc862d8f16df5611537cb51da95812e519c61f6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Ross Burton
510e6e2143 oeqa/selftest/esdk: run selftest inside workdir not /tmp
We've seen issues with rootfs size calculations and we've seen systems
like opensuse which have btrfs mounted on /tmp causing selftest failures.

(From OE-Core rev: 61be3cd748d1b7321a1fc4cfe84efa9b26a6aee0)

(From OE-Core rev: 63eefbe21612e58e88f1eb3fde3d314da53927fa)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Ross Burton
29361d7aeb oeqa: don't litter /tmp with temporary directories
If we need to create a temporary directory in targetbuild or buildproject use
tempfile.TemporaryDirectory so that when the test case is finished, the
directory is deleted.

Also synchronise the logic and don't possibly store the temporary directory in
self.tmpdir as nothing uses that.

(From OE-Core rev: db0e658097130d146752785d0d45f46a3e0bad71)

(From OE-Core rev: 6e2c6668791a80ee0ffe44c756cc0caceebea0e2)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
c7ff99ab44 oeqa/utils/qemurunner: Avoid tracebacks on closed files
Reorder the shutdown/teardown to avoid:

  File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 224, in launch
    op = self.getOutput(output)
  File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 90, in getOutput
    fl = fcntl.fcntl(o, fcntl.F_GETFL)
ValueError: I/O operation on closed file

(From OE-Core rev: 8e7d756862d2a8d62f3c87497d6d65ddb3c1b962)

(From OE-Core rev: 11d4bf460030eb6f072bd0e15550e26e055e632b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
d69080b9da oeqa/selftest/runqemu: Improve testcase failure handling
assertTrue doesn't give good debug information when things fail. Update
several to use assertIn which gives information upon failure, for the
others print the log information upon failure.

(From OE-Core rev: c29cb75d5ce6b0873a934f4709b0c8824f7164d3)

(From OE-Core rev: e9c6e974b49b8821cce8f2c7e3ba0dc16a5b46a0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
d6ba484506 lib/oe/utils: Improve multiprocess_lauch exception handling
We've seen a cryptic:

"ERROR: Fatal errors occurred in subprocesses, tracebacks printed above"

message from oe-selftest with no other traceback information. Improve the
traceback logging to try and give a better indication of any errors that is
ocurring.

(From OE-Core rev: 521dd3d00979a27b6932e58d5497de68abac26e1)

(From OE-Core rev: 2696e69af0b32e03692d8644cc01b28dcf221aa1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Ross Burton
72d7b5b5d4 oeqa/oelib/path: don't leak temporary directories
setUp() is used to populate a directory of temporary files, and deleted in
__del__.  However setUp() is called once *per test* so __del__ would only be
able to remove the last directory created.

Fix the code by using the natural counterpart to setUp, tearDown(), to clean up.

(From OE-Core rev: 68b4723e6fb11d171869185bccf28f32f6284c18)

(From OE-Core rev: eee30d799ff892443d21de4128e7d643215b0ac1)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
3c27f3b3a7 oeqa/selftest/context: Improve log file handling
The existing logfile is simply placed in the current directory. Since the test
changes cwd to BUILDDIR, the symlink to the log can be placed in an invalid
directory. We also see trackbacks if the symlink is invalid.

Improve things by:

* Placing logs in LOG_DIR (or BUILDDIR if unset).
* Using a full path to the log meaning the log and link are placed in the same directory.
* Using lexists instead of exists so invalid symlinks are handled correctly.

(From OE-Core rev: 750ece11bed0e62a11e0003d1d16a81f7c219761)

(From OE-Core rev: 1ec53b8d82491aeb9f49e7a78f531e98b5608f0f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
c414b06ea6 oeqa/selftest/buildoptions: Ensure diskmon tests run consistently
Heartbeat events default to once a second and we need to ensure we have
enough time in the task to see them.

Add a nostamp delay task 5s long so we can have a consistently timed
task which doesn't need cleanup or have unneeded dependencies. This
ensures we should deterministically see the disk moinitor events
regardless of the state of the build. This is done in a way which
doesn't corrupt build state or need cleanup and is efficient.

(From OE-Core rev: ecc49ee8986929e2429d948000a0ca588fe63959)

(From OE-Core rev: a883aa053ddeb4591109c7c1374525e63a59bd80)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:33 +00:00
Richard Purdie
d7594fd2ae oeqa/selftest/buildoptions: Improve ccache test
This test occisionally fails as m4 doesn't recompile, meaning the logfile test
then doesn't find mention of ccache.

To ensure m4 does recompile, clean m4 before force compiling it.

(Reading the test is confusing due to the test cleanup also involving a clean)

(From OE-Core rev: 6e0b9214a0d57ed45a5df0ba5c9887a9045b89b1)

(From OE-Core rev: bddb09a411ce69f49a37260e2188bbd9b02f5902)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:32 +00:00
Richard Purdie
5e12af7587 oeqa/qemurunner: Remove resource python warnings
If runqemu fails it would leak an unclosed socket and file. Ensure we
close these in all cases to remove the resource warning.

(From OE-Core rev: ed80e46ccbc8fe8e9148d80723152066fa00ba28)

(From OE-Core rev: 81ce67fb1274faafb9e7386c5982aeaf4c74294d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:32 +00:00
Richard Purdie
1cd8b3b789 oeqa/utils/commands: Avoid log message duplication
Each time a runqemu() fails, the log handler would be left behind meaning
messages from any subsequent run would be duplicated (or worse/more).

This ensures we remove the handler regardless and means we no longer
have the duplication.

(From OE-Core rev: 532984708436bdfa3a8cac2c684a425eb249bad0)

(From OE-Core rev: 018549bf94d1e5d33344691fde0fb27ca50f675f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:32 +00:00
Richard Purdie
ce291652c5 oeqa/utils/qemurunner: Fix python ResourceWarning for unclosed file
Fixes:

Stderr:
/media/build1/poky/meta/lib/oeqa/utils/qemurunner.py:381: ResourceWarning: unclosed file <_io.BufferedWriter name=16>
  self.runqemu = None

(From OE-Core rev: b9e0bf919e6fc1a58e02145a363ebe7066e5bf4f)

(From OE-Core rev: 43b7ee731d2cfc3fd4a4c26381c8d035b5fbcdeb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:32 +00:00
Richard Purdie
594679bb48 oeqa/utils/commands: Add extra qemu failure logging
Rather than just referring the user to the logs containing the failure, print
them on the console. This aids debugging with oe-selftest with parallelisation
as the logs may otherwise be lost.

(From OE-Core rev: 36a018e245a232f520ff946f152cc875927a6fb4)

(From OE-Core rev: 85b373f571cf2076d93e96db2aca295c53d3c16a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:32 +00:00
Richard Purdie
9cf4132158 oeqa/selftest/buildoptions: Improve ccache test failure output
The current failure mode doesn't show us what the logs actually looked like
and later cleans can lose them. Show the whole log in case of failure
to aid debugging intermittent problems on the autobuilder.

(From OE-Core rev: 7c3a0dc5978cea898b1ca51decf4d6e7cf9d519f)

(From OE-Core rev: 8a9fa1597245d13db89361c40db7867786f137ff)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:32 +00:00
Richard Purdie
835cf20001 oeqa/selftest/case: Use bb.utils.remove() instead of shutil.remove()
This avoids problems where shutil.remove will error with:

  File "/usr/lib/python3.5/shutil.py", line 436, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'S.gpg-agent.extra'

when there are races over file deletion (gpg agent may be slow to exit).

We already worked around speed and race issues in bb.utils.

(From OE-Core rev: 00a8fd5b93a5c19ce0b7498e2bc653ce8ad58aaf)

(From OE-Core rev: ee8255128e11d69d82e6093b58b761dfe773e71d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:32 +00:00
Richard Purdie
588480ff19 oeqa/selftest/signing: Use do_populate_lic target instead of do_package
This should speed the test up signficiantly without any loss of functionality
for the purposes of the test.

(From OE-Core rev: 3dde0b749643575878bfbca2f8d2d9ec30bad166)

(From OE-Core rev: 9fcfc43e1a51520a86761cf5534cff9e70167b77)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:32 +00:00
Richard Purdie
920a95e424 oeqa/selftest/signing: Allow tests not to need gpg on the host
We ideally don't want to use gpg from the host. This is straightforward for package
management but not for sstate.

For sstate, create a second build directory to run the test in using gnupg-native
from the original build directory.

(From OE-Core rev: 10afa94c3f0d7eb7524a26deda86949073d55fde)

(From OE-Core rev: 8c91e74ebcd4b1b851026c7ed5e984aa6a39cf90)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:32 +00:00
Richard Purdie
5778c709b5 oeqa/selftest/signing: Skip tests if gpg isn't found
Raising an assertionError in the class setup isn't a particuarly good way to
indicate gpg isn't installed. Instead skip the tests if the required binary
isn't present. For the signing tests we do require it to be present and can't
use a prebuilt one.

(From OE-Core rev: 2d486af97e51b9daa9c40482c31d637c9ab4ae79)

(From OE-Core rev: 9bb190d5bee70d2d3d5ec2eeee3439cd24548cb8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-16 14:31:32 +00:00
Richard Purdie
1c88c939f8 oeqa/utils/httpserver: Rework to avoid hangs and improve logging
testimage.bbclass installs a SIGTERM handler which conflicts with the
use of multiprocessing here. This is paritcularly problematic if the http
service is terminated before its started and hence before its had a chance
to reset the default signal handler (as the code was written).

Instead, temporarily remove testimage's handler whilst forking the http process
which means the correct handler is installed and won't deadlock.

Also take the opportunity to add in some log messages about the server start
and shutdown so that future debugging is easier and its clearer what the code
is doing.

(From OE-Core rev: cc0471439aa0085ca87deccf061c5b676ef12388)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-24 22:02:47 +00:00
Mohamad Noor Alim Hussin
4f59608a32 oeqa/selftest/recipetool: Fix problems from changing upstream source
The upstream source tarball checksums changed. Use the copy from our source
mirror to avoid failures.

[YOCTO #12979]

(From OE-Core rev: e97a31e6bbaec5cb56d4750bf5171dbba510ee33)

Signed-off-by: Mohamad Noor Alim Hussin <mohamad.noor.alim.hussin@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-09 15:27:34 +00:00
Richard Purdie
d3b55a3f67 image-buildinfo,oeqa/selftest/containerimage: Ensure image-buildinfo doesn't break tests
Having image-buildinfo enabled causes containerimage.ContainerImageTests.test_expected_files
to fail due to the presence of an unexpected file:
  ['./',
   './etc/',
-  './etc/build',
   './etc/default/',
   './etc/default/postinst',

Tweak the class to allow it to be disabled and disable it from the test just in
case it was enabled.

(From OE-Core rev: af67bf422a4df5b7e07894512ff73a5f493682ab)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-09 15:27:34 +00:00
Yeoh Ee Peng
b1c3b40146 oeqa/selftest: Standardize json logging output directory
Currently sdk & sdkext will output json file to LOG_DIR, while
selftest will output json file to TOPDIR/log.

Standardize selftest json output file to LOG_DIR.

(From OE-Core rev: 2012d6b076fc833864a0254d56d78536314a6799)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-09 15:27:34 +00:00