Commit Graph

171 Commits

Author SHA1 Message Date
Richard Purdie
4d0e1a7f5e resulttool: Use single space indentation in json output
Using 4 space indentation in resulted in hundreds of megabytes of extra file size
in general use. Reduce this to make filesizes more managable and reduce the processing
cost. Some level of indentation and spacing does make the files more readable and allows
use of git diff so we need to retain some of it.

(From OE-Core rev: cae6106f152c8c44e2d85179ad7e6831b974ffd5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a274cdcaf852cca9497f0358f44dda99c06aacbe)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-12-09 07:54:03 -08:00
Richard Purdie
464b034745 oeqa/ssh: Further improve process exit handling
It looks like there were further cases where orphaned processes may be left
behind since the .kill() calls may be unsuccessful if the process terminated
due to the terminate or through normal exit. In that situation .wait()
wouldn't have been called.

Further tweak the exit code paths to ensure .wait() is called to update the
returncode value before returning in all cases.

(From OE-Core rev: e1e038ab01a599fcdd4aa6211b6d15cd01a5e2e3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0a0a1731e38edfa72a141e8fd8f2de52be562e94)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2023-08-26 04:24:02 -10:00
Richard Purdie
1efc676afc target/ssh: Ensure exit code set for commands
As spotted by Joshua Watt, the returncode isn't set until .poll() or .wait()
is called so we need to call this after the .kill() call.

This fixes return code reporting so that timeouts for example now return an
exit code when they didn't before.

(From OE-Core rev: c70b05ea667e7bd280470b0b6ca10efb0f648e0f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3924e94214b5135369be2551d54fb92097d35e95)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2023-08-26 04:24:02 -10:00
Richard Purdie
0418f9112c oeqa/target/ssh: Ensure EAGAIN doesn't truncate output
We have a suspicion that the read() call may return EAGAIN on the non-blocking
fd and this may truncate test output leading to some of our intermittent failures.
Tweak the code to avoid this potential issue.

(From OE-Core rev: 4c02f7407d7afaefe1bc72aea25087b3f2271ac2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a8920c105725431e989cceb616bd04eaa52127ec)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2023-08-26 04:24:02 -10:00
Mikko Rapeli
4cd4e6d9ba oeqa ssh.py: fix hangs in run()
When qemu machine hangs, the ssh commands done by tests
are not timing out. do_testimage() task has last logs like this:

DEBUG: time: 1673531086.3155053, endtime: 1673531686.315502

The test process is stuck for hours, or for ever if the
executing command or test case did not set a timeout correctly.
The default 300 second timeout is not working when target hangs.
Note that timeout is really a "inactive timeout" since data returned
by the process will reset the timeout.

Make the process stdout non-blocking so read() will always return
right away using os.set_blocking() available in python 3.5 and later.

Then change from python codec reader to plain read() and make
the ssh subprocess stdout non-blocking. Even with select()
making sure the file had input to be read, the codec reader was
trying to find more stuff and blocking for ever when process hangs.

While at it, add a small timeout to read data in larger chunks if
possible. This avoids reading data one or few characters at a time
and makes the debug logs more readable.

close() the stdout file in all cases after read loop is complete.

Then make sure to wait or kill the ssh subprocess in all cases.
Just reading the output stream and receiving EOF there does not mean
that the process exited, and wait() needs a timeout if the process
is hanging. In the end kill the process and return the return value
and captured output utf-8 encoded, just like before these changes.

This fixes ssh run() related deadlocks when a qemu target hangs
completely.

(From OE-Core rev: 04f080802b4a28709a105e4f0ead56a7a2da42b4)

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit 9c63970fce3a3d6029745252a6ec2bf9b9da862d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-09 13:19:03 +00:00
Mikko Rapeli
758c32745e oeqa ssh.py: add connection keep alive options to ssh client
Configure ssh client to test that connection with server is up.
If the server does not respond within a minute then the connection,
target machine or sshd daemon are stuck and it's better to exit
the command execution with errors.

Some tests can execute a long time without returning stdout/stderror
data and it's difficult to adjust timers for those cases if
connection to target machine or the target machine itself hangs
and output is not expected in minutes or even hours.

(From OE-Core rev: 94021c8571242dd491ee3c7c40144c5a12ca8e42)

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit ba68ff04c5786eca7cd8dd44056705867dea8ac4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-24 16:41:46 +00:00
Mikko Rapeli
de82d38730 oeqa ssh.py: move output prints to new line
The output from is garbled otherwise and it's not
easy to remove debug output form real command output on target.

(From OE-Core rev: 36b572fbc5c88b9aaf4e146ecdb00c8d4ea6ff70)

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit 917a70cbc43ac1c70c477b220c4115735457ef04)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-24 16:41:46 +00:00
Richard Purdie
e2c6d1436a oeqa/concurrencytest: Add number of failures to summary output
When running oe-selftest and seeing the end of a running log, it is
extremely helpful to know if there have been failures or not to save
looking at the rest of the log. Add the number of failures to the summary
line so that people monitoring builds have an easier time before the end
totals are printed.

(From OE-Core rev: 5fa9d58f59bebfc9eb495b50ef9a767a5c765b77)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6b23996911d91f7f99774646c6db9f3490b4cb62)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-06 17:33:23 +00:00
Ross Burton
fcc9c0ab37 oeqa: rationalise skipifqemu decorators
(From OE-Core rev: 1a3a37cc2b16a8d5cd2258b0b35be43baa363f67)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-01 23:11:37 +01:00
Ross Burton
4e5f84902e oeqa/core/decorators/data: improve has_* logic
has_feature() should be splitting the feature string into substrings and
then looking for membership instead of looking for simple substrings.

has_machine() should be using equality instead of substrings.

(From OE-Core rev: a4c63819234e252c58e040af8bbdbfb96b6feccf)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-01 23:11:37 +01:00
Ross Burton
f4205dcf3d oeqa/core/utils/misc: remove redundant file
This file dates back to 2016. Half of the functions have never been used,
the rest are used in one place and have now been replaced.

(From OE-Core rev: 5a053b2a84e7a671925fb4a78005145786c57f6a)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-01 23:11:37 +01:00
Ross Burton
ccc03581f5 oeqa/core/decorator: remove redundant code
There's no need to wrap *tags in a potential list, as *tags will always
be a tuple.

(From OE-Core rev: 54210c518bcb76d80c8ec9564d1ddf344e9d8924)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-01 23:11:37 +01:00
Ross Burton
48587b5e63 oeqa/target/ssh: don't assume target_dumper is set
The target_dumper property is only set by the QemuTarget subclass, so
assign a default value and check it isn't None before calling it.

(From OE-Core rev: add36cf85380bf7dc5339a9d65f951923317440b)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-21 23:44:08 +01:00
Sakib Sajal
b44849c32c oeqa/core/target/qemu.py: display contents of dumped files
During do_testimage, if the target is not started within a certain
timeout, TEST_QEMUBOOT_TIMEOUT, host data is dumped to files for
each command in
${TMPDIR}/log/runtime-hostdump/<datetime>_qemu/host_<seq>_<command>.

Display the first 20 lines of top output and the last 20 lines of
bootlog to standard output for more context for the target not being
started up.

(From OE-Core rev: 441390b707bf681bc308c9ebd45ea2ae20c37d7c)

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-12 08:38:29 +01:00
Alexander Kanavin
377a73d5b7 oeqa/core/tests/test_data.py: use weston image instead of sato
(From OE-Core rev: c2ccd8c8144cdda52b858589f7d5d3a15ab28b90)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27 15:11:47 +01:00
Saul Wold
3acbec85b0 qemurunner: Add support for qmp commands
This adds support for the Qemu Machine Protocol [0] extending
the current dump process for Host and Target. The commands are
added in the testimage.bbclass.

Currently, we setup qemu to stall until qmp gets connected and
sends the initialization and continue commands, this works
correctly. If the UNIX Socket does not exist, we wait an timeout
to ensure to socket file is created.

With this version, the monitor_dumper is created in OEQemuTarget
but then set in OESSHTarget as that's where we get the SSH failure
happens. Python's @property is used to create a setter/getter type
of setup in OESSHTarget to get overridden by OEQemuTarget.

By default the data is currently dumped to files for each command in
TMPDIR/log/runtime-hostdump/<date>_qmp/unknown_<seq>_qemu_monitor as
this is the naming convenstion in the dump.py code.

We use the qmp.py from qemu, which needs to get installed in the
recipe-sysroot-native of the target image.

[0] https://github.com/qemu/qemu/blob/master/docs/interop/qmp-spec.txt

(From OE-Core rev: 42af4cd2df72fc8ed9deb3fde4312909842fcf91)

Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27 15:11:47 +01:00
Alexander Kanavin
c13be2e1dc meta/lib/oeqa/core/tests/cases/timeout.py: add a testcase for the previous fix
This is the sequence that didn't properly operate:

- a test case that skips and isn't executed
- a second test case that is skipped via a dependency decorator, and sets a timeout
- a third test case that takes longer than the timeout from the second
test case

Without the fix, the timeout is not cleared, and the third test case is
erroneously aborted. With the fix, the timeout is cleared and the third
test case is able to complete.

(From OE-Core rev: 54ef07a9aa1af8f41cfb9a4802929c918efc43c8)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-23 10:12:10 +01:00
Alexander Kanavin
b40f78a5c7 oeqa: tear down oeqa decorators if one of them raises an exception in setup
Some of the decorators need proper cleanup, such as OETimeout
which sets a signal handler that needs to be cleared via teardown.
If this is not done then the signal gets called later with unpredictable effects.

This can be seen if there's a test that is skipped via a decorator and sets a timeout
at the same time: the timeout isn't cleared, and is invoked later in a
completely unrelated context. The test case for this is added in the
next commit.

(From OE-Core rev: f42a08e1aabf1ca57e0c09d69fb69cc717c7f156)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-23 10:12:10 +01:00
Richard Purdie
232cb7b055 oeqa/runqemu: Support RUNQEMU_TMPFS_DIR as a location to copy snapshot images to
We have a working theory that IO queues on the autobuilder are impacting
runtime testing under qemu, particularly async writes which inice does not
influence. We already pass the snapshot option to qemu which copies the
image and runs out of the copy. Add in the ability to copy the image to
a specificed location which can be a tmpfs. This means that writes to the
image would no longer be blocked by other writes to disk in the system.

Preliminary tests show that this does improve the qemu errors at the expense
of sometimes showing qemu startup timeouts as on a loaded system with a large
test image, it can take longer than 120s to copy the image to tmpfs. Having
a most consistent failure mode for loaded tests is probably desireable though.

(From OE-Core rev: fd1c26ab426c3699ffd8082b83d65a84c8eb8bff)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-06 09:30:10 +01:00
Richard Purdie
855fbb6f99 oeqa/concurrencytest: Fix display of test stdout/stderr
If oe-selftest is run with -j, the output to stdout/stderr is being
lost at present. Capture this and display it upon test failure. We
have code that previously tried to enable this but it wasn't functioning
correctly. This should give more usable error reports on the autobuilder.

This code will mix stdout and stderr as the output is streamed from the test
server without markup. This is most in keeping with subunit/testools though
and the easiest way to handle the various challenges here as far as I can
see.

(From OE-Core rev: 6a954ce5834c8026adecff8478c3d827640bc647)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-06 09:30:10 +01:00
Richard Purdie
f0c5a904d9 oeqa/concurrencytest: Rename variables to improve the code
Each time I look at this code I get confused about what the different
variables represent. Rename a few of them to better indicate what they
represent.

(From OE-Core rev: e39d97c0b191add9281bac463ca059685288c81a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-06 09:30:10 +01:00
Konrad Weihmann
3c4147880e oeqa/core/context: initialize _run_end_time
with _run_start_time as value. For partial results of interrupted runs,
this info might be otherwise missing for at least one testcase

(From OE-Core rev: 1c5e8baf57fa2a33b9ef507b11d9ea9acaa77238)

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-11 10:08:12 +00:00
Konrad Weihmann
02be651f88 oeqa/core/context: expose results as variable
register an unittest handler for testresults and expose it as
variable result.
With this even partial results from an interrupted test suite run
can be made available

(From OE-Core rev: a97ae47525157871b6c098ffc352293e365a4335)

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-11 10:08:12 +00:00
Saul Wold
78b473e7f8 target/ssh.py: Add dump_target support
This adds the dump_target support when the ssh command fails with
a 'No route to host'. This is will provide additional data when a
Qemu target fails to respond during autobuilder testing. This does
not fix 14002 [0], but may help track down why qemu looses networking

[0] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14002

(From OE-Core rev: cef1a2b03b359c018911abc29db1895d0f46814c)

Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-30 15:01:51 +01:00
Saul Wold
406857b019 testimage: Add testimage_dump_target to kwargs
This passes the list of commands to run on the OEQemuTarget when
the TargetDumper needs to run in a test context due to a failure
on the target.

This is added here as a kwargs because the 'd' dictionary is not
available in the staticmethod getTarget in the
OERuntimeTestContextExecutor class. The OEQemuTarget is different
from the QemuTarget which already uses the list of commands from
testimage_dump_target from 'd'. The create_dir() is needed to
initialize the TargetDumper's dump_dir variable.

(From OE-Core rev: a63675fab4d9f638570912b15a07932f549cc4d1)

Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-30 15:01:51 +01:00
Alexander Kanavin
3ee166c0dd oeqa: write @OETestTag content into json test reports for each case
This allows using these tags for classification and filtering of test results
according to various organization-specific criteria, such as teams
responsible for the test, internal test ids, feature domains and so on.

Test name itself meanwhile can stay short and human-readable.

(From OE-Core rev: 3801b126eb52cd46efe417111afcd27f05d8f72b)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-08 09:17:49 +01:00
Matthew
b6b6a39ee8 ltp: make copyFrom scp command non-fatal
[YOCTO #13802]

Make the scp failure non-fatal so the ltp tests continue to run and
the rest of the logs will be available to see afterwards.

(From OE-Core rev: 0f7d093038274f4f21f6cca39a96aac4f6c32ee3)

Signed-off-by: Mingde (Matthew) Zeng <matthew.zeng@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-06 15:12:39 +01:00
Vacek, Patrick
6474ff808b oeqa/core/loader: fix regex to include numbers
The previous version only included the numbers 1 and 2 in the allowed characters
for the module name. In the past, this was (\w+) so all numbers were allowed.
Now it explicitly includes all numbers again.

(From OE-Core rev: f241fa493536ac953c1dac303917c6e75b459e28)

Signed-off-by: Patrick Vacek <patrickvacek@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-08 10:37:12 +01:00
Lee Chee Yang
ebfa049fc0 oeqa/core/loader: refine regex to find module
test case in format <module name>.<class name>.<test case name>
this is clear when test cases is only 3 item deep.

but confused when it is 4 item deep, eg,
oelib.types.TestList.test_list_nosep

in this case, oelib and oelib.types can both be treated as module
since module name contains only lower cases and class name should
contain atleast one upper case.
so, always treat leading item without upper case as module also allow
module name to contain dot.

[YOCTO #13941]

(From OE-Core rev: ad81ea90a815389e45ff302a85151724c71f71c3)

Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-03 11:38:24 +01:00
Richard Purdie
79856aaf93 oeqa/selftest: Clean up separate builddir in success case when non-threaded
If oe-selftest is run without -j, the separate build directory "build-st"
isn't cleaned up afterwards. Mirror the behaviour of the -j option to
handle this the same way, only preserve upon failure.

To do this, the remove function needs to be moved to the selftest
context module so that it can be accessed without requiring the
testtools and subunit modules the -j option requires.

A dummy wrapper class is used to wrap the tests and clean up afterwards.

[YOCTO #13953]

(From OE-Core rev: 1b376ade430d40d3cfe9c18f200c764d622710e5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-28 08:36:00 +01:00
Paul Barker
18a69b1cd2 oe-selftest: Support verbose log output
(From OE-Core rev: 6a607da9a6aa05f86b0e77cccd066b9f286cfada)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04 13:27:32 +01:00
Steve Sakoman
6b3b5ab4ba oeqa/concurrencytest: don't delete build directory for failed tests
(From OE-Core rev: 3d5aa170d2e88b852bd2a4452aab9311a24badef)

Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-24 23:56:03 +01:00
Alexander Kanavin
03bb031125 testresults.json: add duration of the tests as well
This is printed by testimage, but isn't actually saved.
It's a useful metric for tracking execution times.

(From OE-Core rev: 8fc19639f47b959a141dae231395bbababa644e1)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-21 12:35:39 +01:00
Konrad Weihmann
da0ab97f61 testimage: enable ovmf support
Add support for running wic images with EFI as testimage.
Introduces a variable called QEMU_USE_OVMF for configuration.

(From OE-Core rev: 3af8aaff68ed332d812ea7dc184d392700ad7882)

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-24 14:10:08 +01:00
Stefan Kral
80c1002c2e oeqa: default runtime json results for testexport
Set the json result output dir in the oeqe runtime context to create
testresults.json file by default for exported runtime test runs.

Use current datetime for the json result property name (not DATETIME
from build) to allow multiple result entries.

(From OE-Core rev: e6c73ed8d3c5d45f387cab619ca73c21e850582f)

Signed-off-by: Stefan Kral <sk@typedivision.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29 12:05:31 +01:00
Stefan Kral
b10131ea74 oeqa: enable testresults.json for testexport
Add the option --json-result-dir to oeqa core context to enable
testresults.json creation for test runs via testexport.

Eg. oe-test runtime --json-result-dir .

(From OE-Core rev: 9d8edf33d1f5d89b310923b0aa3cc967317c7c49)

Signed-off-by: Stefan Kral <sk@typedivision.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12 22:49:28 +00:00
Richard Purdie
c569157d9d oeqa/selftest: Standardise seperate builddir for concurrent and non-concurrent selftest
Currently oe-selftest reuses the current build directory and the concurrent
version run with -j does not.

Standardise and use a separate new build directory in both cases. This will lead
to simpler code and more reliable user run tests.

(From OE-Core rev: 50238b0717b04e0a1fa69d618e8c8aa8445a80b0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-21 09:39:00 +00:00
Trevor Gamblin
450688b010 concurrencytest.py: add outSideTestaddSkip for subunit
see: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13663

When running oe-selftest in concurrency mode (e.g. with oe-selftest
--run-tests oescripts.OEPybootchartguyTests -j 4), if a skip occurred
during setUpClass() rather than within individual tests, the entire
suite would show "UNKNOWN" as each test's result. This is because
subunit doesn't know how to handle skips outside of individual tests.
An example of where this occurs is when running the above call to
oe-selftest in concurrency mode on a host machine that does not have
python3-cairo installed.

Patch subunit inside concurrencytest.py to provide a method called
outSideTestaddSkip, which will allow subunit to correctly detect the
skip in setUpClass().

(From OE-Core rev: 9b8734b584d6e8d9c32ff2a721b29f3f3e61cca7)

Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-18 23:53:55 +00:00
Alejandro Hernandez Samaniego
a3416a5933 testimage: Extend runtime testing infrastructure to allow unconventional booting processes to be tested
The current runtime infrastructure contains hardcoded values which Ill refer to
as patterns, these patterns are either searched through or sent via the serial
terminal to communicate between HOST and TARGET.

These patterns are required since they allow us to check when a device has
finished booting, to log in, and to check whether a command sent from our tests
has returned, this way we are able to check both the status of the commands that
were sent along with its output.

The testing process goes somewhat as follows:
1. Launch QEMU and start booting.
2. Check when the device has booted by looking for the pattern login:.
3. Log in as the root user (default for our images).
4. Check that we were able to log in succesfully.
5. Start running the runtime test cases defined by TEST_SUITES.
6. One of such test cases could send a command to the QEMU target.
7. Check whether that command returned.
8. Check its output and status, return whether the test case passed or failed.

This patch allows this set of patterns to be defined instead of being hardcoded,
but it also automatically sets the defaults that we have been using in the past
if they have not been manually defined, for this reason, the patch is less
invasive and should not affect in any way how tests are currently being run.

Cases that can be enabled with this patch:
- A customized image that does not use the root user (or maybe we want to check
what happens if we dont use the root user).
- An image where the PS1 env variable has been modified, and the prompt pattern
wouldnt match the default.
- Baremetal applications, which do not follow the conventional way of booting
Linux and would probably not show a prompt for a user to log in, same applies
for testing bootloaders.
- poky-tiny: Using DISTRO=poky-tiny and an image such as the core-image-tiny
from meta-intel, which boots directly to RAM, and does not show a log in prompt
since it does not contain a conventional init process.

The code itself contains comments that should be self explanatory but here is an
example on how these patterns can be defined in a hypothetical case where we
want to run test cases as the webserver user instead:

TESTIMAGE_BOOT_PATTERNS = "send_login_user search_login_succeeded"
TESTIMAGE_BOOT_PATTERNS[send_login_user] = "webserver\n"
TESTIMAGE_BOOT_PATTERNS[search_login_succeeded] = "webserver@[a-zA-Z0-9\-]+:~#"

The variable TESTIMAGE_BOOT_PATTERNS defines which patterns to override when
used to communicate with the target when booting, anyone familiar with the
PACKAGECONFIG syntax should have no trouble setting these.

Other patterns would still be set up as default, e.g.
search_reached_prompt would still be login:

The accepted flags for TESTIMAGE_BOOT_PATTERNS are the following:
search_reached_prompt, send_login_user, search_login_succeeded,
search_cmd_finished.

They are prefixed with either search/send, to differentiate if the pattern is
meant to be sent or searched to/from the target terminal.

A working example of this code that falls under the baremetal case mentioned
above along with a test case is present on the meta-freertos layer, which tests
an RTOS image built with OpenEmbedded and automatically runs a test case on it
after booting such image:

As usual, INHERIT += "testimage" needs to be present on local.conf
$ bitbake freertos-demo -c testimage

RESULTS:
RESULTS - freertos_echo.FreeRTOSTest.test_freertos_echo: PASSED (2.00s)
SUMMARY:
freertos-demo () - Ran 1 test in 2.006s
freertos-demo - OK - All required tests passed (successes=1, skipped=0,
failures=0, errors=0)

(From OE-Core rev: 3ab2cbfeff371e8791b031a2852eeef80101a831)

Signed-off-by: Alejandro Hernandez Samaniego <aehs29@gmail.com>
Signed-off-by: Alejandro Hernandez Samaniego <alejandro@enedino.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-13 12:19:14 +00:00
Alejandro Hernandez Samaniego
e597809307 testimage: Allow testing on QEMU machines with a single serial port
commmit 6cde67d0a8 enables the use of qemurunner on machines that
only have a single serial port, but still sets the default value
as serial_ports=2 if not provided.

The testimage class does not call qemurunner with a serial_ports
argument, hence always defaulting to two.

Pass the serial_ports argument from the testimage class to allow
tests to run on QEMU machines with a single serial port.

(From OE-Core rev: a50d0163770f0b405a8de8a8a9cccd48c1de4112)

Signed-off-by: Alejandro Hernandez Samaniego <alejandro@enedino.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-13 12:19:14 +00:00
André Draszik
e4893d4830 oeqa: modernise subprocess: preexec_fn=os.setsid -> start_new_session=True
start_new_session was added to python3 subprocess in v3.2 and
is meant to take the place of the common use of preexec_fn to
call os.setsid() in the child - as done here.

Update to use the new equivalent.

(From OE-Core rev: c3bd502f31a621394a2f6ea3ae90443ff4ac98a4)

Signed-off-by: André Draszik <git@andred.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-04 15:56:29 +00:00
André Draszik
eeefadc738 oeqa/target/ssh oeqa/target/qemu: expose server listening port to tests
Allow tests to access the listening port as just introduced.
Note that when using qemu this infrastructure shouldn't be
needed, but we still need to set the port to 0 so that
a listening port is determined automatically (e.g. by the
python http server).

(From OE-Core rev: ca58559fc30842c123907fb5cc68b3356c530862)

Signed-off-by: André Draszik <git@andred.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:25:49 +00:00
Richard Purdie
055695994a oeqa/concurrencytest: Optimise for autobuilder/clobberdir
We're seeing huge delays on the autobuilder during oe-selftest builddir
deletion. For example there is a currently running selftest we could do
with the results from and its been going 13 hours, at least 8 of which
was in deletion of the builddirs.

There are a variety of ways we could solve this problem however the
autobuilder has a mechanism for deferred deletion, "clobberdir" which
it already uses for this kind of work.

Whilst in general hardcoding things like this is horrible, I believe
in this case the benefits (and resulting improvements on my sanity
if nothing else) mean this is a case where we should do it.

If/as/when someone can come up with a better solution that is fine
and this can be replaced.

(From OE-Core rev: 51a84937d32a85cbdb7d5b1d1ec69a290d0c303b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-06 14:41:28 +00:00
Armin Kuster
97c2021b8a oeqa/core: Add qemu checks
Some test should not be run in QEMU systems so
add some checks to make that easier

(From OE-Core rev: 1b1e53c4ad33a71d526887d1133598f255a476ec)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:21:00 +00:00
Armin Kuster
1351614b8d oeqa/core: Add a check for MACHINE
(From OE-Core rev: e69f963e548e2f6f211a56406694c029111d7203)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:21:00 +00:00
André Draszik
34cce61ece oeqa/core/decorator: add skipIfFeature
skipIfFeature will skip a test if a given DIST_FEATURE
or IMAGE_FEATURE is enabled.

(From OE-Core rev: ff2218f7cc3992725dd35499c14ec3396120dcc5)

Signed-off-by: André Draszik <git@andred.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-19 23:18:33 +01:00
Nathan Rossi
ec1104fd7f oeqa/core/case.py: Encode binary data of log
Do not decode the log content into a string only to re-encode it as
binary data again. Some logs might un-intentionally contain bytes that
do not decode as utf-8, as such preserve the log file content as it was
on disk.

Handle the decoding on the resulttool side, but also handle the failure
to decode the data.

(From OE-Core rev: 20531dc0b8f76a6e37cc856f36cd94077b6aba50)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 16:55:21 +01:00
Nathan Rossi
02a334c98f oeqa/core/utils/concurrencytest.py: Handle exceptions and details
Handle the streaming of exception content with details data. The
testtools package allows both 'err' and 'details' kwargs but can only
pass one of them to the parent.

To handle the passing of exception traceback and details data at the
same time, encode the traceback into the details object and remove the
'err' arg from the add* result call. This encodes the traceback similar
to how 'err' is handled without any details object. Decoding is already
done by testtools when the traceback is encoded in the details object.

(From OE-Core rev: 3613451825b251784b7673d89db465b9782c3a31)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 16:55:21 +01:00
Richard Purdie
e9f859106c oeqa/concurrenttest: Use ionice to delete build directories
Autobuilder type infrastructure can benefit from deletion of certain files as
background IO due to the way Linux filesystem priority works.

We have problems where build directories as part of oe-selftest being
delete starves the running tasks of IO to the point builds take much
longer to compelte.

Having this option of running the deletion at "idle" helps a lot with
that. Use the new option added to bb.utils.prunedir().

(From OE-Core rev: d41e7018be56902b7a1be4590e468cd15e02a3b5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-19 20:30:35 +01:00
Nathan Rossi
e08f657220 oeqa/core/case.py: Add OEPTestResultTestCase for ptestresult helpers
Add the OEPTestResultTestCase class as a mix-in class to provide helper
functions for interacting with ptestresults within the extraresults
object generated by the test case.

This class also provides default compression of log text and log files.

Also add support to resulttool for decoding/decompressing log files
embedded in the test results.

(From OE-Core rev: 06cba9883a5964320969301fd05eeb6bec3e786d)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:43 +01:00