We want to see failures/errors listed last since this is the most easily
visible part of the log on consoles or autobuilder output and makes
human processing easier rather than having to scroll up and scan for
a single failure.
(From OE-Core rev: 2cc07ab253f1ba6a1f07a66051c9ba6d98cd2357)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to have the counts of success/failure/error/skipped at the end of the
results to allow for easier human reading of what happened.
(From OE-Core rev: 5942318a261ce7a885f351e214669068ff9d8931)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows the ptest results from ptest-runner, run in an image to be
transferred over to the resulting json results output.
Each test is given a pass/skip/fail so individual results can be monitored
and the raw log output from the ptest-runner is also dumped into the
results json file as this means after the fact debugging becomes much easier.
Currently the log output is not split up per test but that would make a good
future enhancement.
I attempted to implement this as python subTests however it failed as the
output was too confusing, subTests don't support any kind of log
output handling, subTest successes aren't logged and it was making things
far more complex than they needed to be.
We mark ptest-runner as "EXPECTEDFAILURE" since its unlikely every ptest
will pass currently and we don't want that to fail the whole image test run.
Its assumed there would be later analysis of the json output to determine
regressions. We do have to change the test runner code so that
'unexpectedsuccess' is not a failure.
Also, the test names are manipuated to remove spaces and brackets with
"_" used as a replacement and any duplicate occurrences truncated.
(From OE-Core rev: a13e088942e2a3c3521e98954a394e61a15234e8)
(From OE-Core rev: 526ceab9d0e43f73635bb92e8dd7763ef75ad33b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As part of the solution to replace Testopia to store testresult,
OEQA need to output testresult into single json file, where json
testresult file will be stored in git repository by the future
test-case-management tools.
The json testresult file will store more than one set of results,
where each set of results was uniquely identified by the result_id.
The result_id would be like "runtime-qemux86-core-image-sato", where
it was a runtime test with target machine equal to qemux86 and running
on core-image-sato image. The json testresult file will only store
the latest test content for a given result_id. The json testresult
file contains the configuration (eg. COMMIT, BRANCH, MACHINE, IMAGE),
result (eg. PASSED, FAILED, ERROR), test log, and result_id.
Based on the destination json testresult file directory provided,
it could have multiple instances of bitbake trying to write json
testresult to a single testresult file, using locking a lockfile
alongside the results file directory to prevent races.
Also the library class inside this patch will be reused by the future
test-case-management tools to write json testresult for manual test
case executed.
(From OE-Core rev: 00e03b5004f1eb6d59295544b3a8620504278f51)
(From OE-Core rev: 045511425577ccbe89d8eb91e2a87e385390cabf)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Refactor the original _getDetailsNotPassed method to return
testresult details (test status and log), which will be reused
by future OEQA code to write json testresult.
Take the opportunity to consolidate and simplify the logic used
to gather test status and log within the TestResult instance.
(From OE-Core rev: 79ee7d1c371a86edeb61c99679985118da657e5d)
(From OE-Core rev: bc444181f9658423856621b2f2c60364642ae5b1)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This isn't present on modern distros by default and doesn't work with
testtools, needing multiple code paths in the code. Remove it in favour
of finding a better replacement for results collection/analysis.
(From OE-Core rev: 8001d933a8dc86004db014777f094d718086687d)
(From OE-Core rev: 02449e89d62a7714d30ab6e2e58c476dc9441f7a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There doesn't appear to be any reason we need this _results indirection
any more so remove it.
(From OE-Core rev: b618261811c48ff3b98eab1b340a8cd09ef183c6)
(From OE-Core rev: ab271b49d9b55ea271d519c3a4da0b639a07f0bb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current code assumes if something isn't a failure of some
kind, it was a pass. When test case IDs weren't matching, this lead
to very confusing output where things would fail, then be listed as
passing.
This adds code to track successes, ensuring we don't end up in this
position again with unmatched entries being listed as UNKNOWN.
(From OE-Core rev: 4374c296d8963e4f6a1aa7bef7983ad0a1c2fcff)
(From OE-Core rev: bcb2948773d76befef2be787be6d25cf544e49a9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can directly compare the test case IDs rather than representations,
then if we're using subunit to split the tests, the comparisions still
work as intended.
(From OE-Core rev: 72e5f46f75454ba4c445c65c1cbc616a9e72fc6e)
(From OE-Core rev: 4c53aac5315f9d0a0ed95cbeb48b7704e274e3a9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The value could be False in which case we should pass that through.
(From OE-Core rev: d0a3379bbcbcd8153bd59ccdb56d40fff7ad6c6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In general we don't need to see the output of runqemu however if it fails
we do. Use the buffer option that already exists in TestResult but allow
us to trigger it on a per test basis.
(From OE-Core rev: 86bef95bccbe9dad3dc3445ea365439861966179)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since OEQA framework uses Python logging functionality to report test
results there is a class that wraps PyUnit writes into logging commands
(OEStreamLogger), so don't buffer the actual test execution to have
insight of what is currently executing.
This fix will change a little the test output format adding an '\n'
previous the test result, for example:
From:
test_nonmatching_checksum (lic_checksum.LicenseTests) ... ok
To:
test_nonmatching_checksum (lic_checksum.LicenseTests)
... ok
This is because the new line added by the PyUnit StreamLogger because
currently we don't have a manner to identify when a test execution
starts at report level (write msg).
[YOCTO #11827]
(From OE-Core rev: 4c2276469f58a88f864eb374c00dbbaace702de4)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The details of a test failure is upper on the unittest output
so don't log twice the actual failure.
[YOCTO #11622]
(From OE-Core rev: 0f2e81c2a4458ad0ec6bab2710952ac2c2bbf1af)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently there was a restriction to only execute tests that's
inherits from OETestCase but in some circunstancies the features
from the OEQA framework isn't needed so we need to support
basic unittests.
[YOCTO #10828]
(From OE-Core rev: baac26f1b36e89e07637b738dd31ec7356f05a02)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A common operation is to list tests, currently only selftest
support it, this changes enables this functionality into the
core framework.
(From OE-Core rev: 7e803f1a855d3091a772b13efd3cc8e9c0c766e9)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This method is to assign results into the TestContext, create
an internal one to support change implementation in Thread version.
[YOCTO #11450]
(From OE-Core rev: 8a6a9cb816d78e4cf71b79c35b579918d31053f2)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Those methods are used to write in the log the results so
it makes sense to have defined at OETestResult because
is a format of the result itself.
[YOCTO #11450]
(From OE-Core rev: 33a783f59ed4e232f41f8b09dfa7955f2ddc2f80)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I override this method before for keep track of results and forget
to remove it, now isn't need.
(From OE-Core rev: 63606ffaaac1b84ddcad8a1c1006f8110050e20e)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OEStreamLogger class is used for redirect PyUnit output
to a certain logger so there is not need to expose at level
of OETestContext because only OETestRunner needs to know.
[YOCTO #11450]
(From OE-Core rev: 8787fba3df8acd9d2438669d20b1a5060caa9022)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was found a case (see the bugzilla entry) where two runners were running at the same
second, creating identical test result folders, so one of them (the second runner)
was not able to create the folder because the other has already created it,
raising the following exception (many text was removed from log)
NOTE: Executing RunQueue Tasks
NOTE: Running task 1 of 2 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/recipes-core/images/core-image-minimal.bb:do_testsdkext)
NOTE: Running task 2 of 2 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/recipes-sato/images/core-image-sato.bb:do_testsdkext)
NOTE: recipe core-image-sato-1.0-r0: task do_testsdkext: Started
NOTE: recipe core-image-minimal-1.0-r0: task do_testsdkext: Started
.
.
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_testsdkext(d)
0003:
File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/classes/testsdk.bbclass', lineno: 188, function: do_testsdkext
0184:
0185:testsdkext_main[vardepsexclude] =+ "BB_ORIGENV"
0186:
0187:python do_testsdkext() {
*** 0188: testsdkext_main(d)
0189:}
0190:addtask testsdkext
0191:do_testsdkext[nostamp] = "1"
0192:
File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/classes/testsdk.bbclass', lineno: 171, function: testsdkext_main
0167: except Exception as e:
0168: import traceback
0169: bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
0170:
*** 0171: result = tc.runTests()
.
.
File: '/usr/lib64/python3.5/os.py', lineno: 241, function: makedirs
0237: cdir = bytes(curdir, 'ASCII')
0238: if tail == cdir: # xxx/newdir/. exists if xxx/newdir exists
0239: return
0240: try:
*** 0241: mkdir(name, mode)
0242: except OSError:
0243: # Cannot rely on checking for EEXIST, since the operating system
0244: # could give priority to other errors like EACCES or EROFS
0245: if not exist_ok or not path.isdir(name):
Exception: FileExistsError: [Errno 17] File exists: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/build/TestResults_20170409130114'
[YOCTO #11318]
(From OE-Core rev: 10aa09d39c1b70ce2a88f59601f1f8dbc96ed817)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change conditional to avoid losing stack traces when log
into the logger. The logic only needs to don't log empty lines.
(From OE-Core rev: 545dd180cc27a69732f2332565038bd06d01c014)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
case: Defines OETestCase base class that provides custom
methods/attrs defined by the framework.
Every OETestCase instance contains a reference to the test
data (d), the test context (tc) and the logger.
Also implements _oe{SetUp,TearDown}Class for make special
handling of OEQA decorators and validations.
runner: Defines OETestRunner/OETestResult with support for RAW
and XML result logs.
exception: Custom exceptions related to the OEQA framework based
on class OEQAException.
[YOCTO #10230]
[YOCTO #10233]
(From OE-Core rev: c466086ccc4d4bb02d578a821cfb945945bfd529)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
