Python modules aren't supposed to have uppercase characters in their names
according to python conventions. We have regexs in the code which work
on that assumption too. Rather than showing errors under some filtering
situations, make it clear and error if a problematic name is seen.
(From OE-Core rev: 7964a6ee54e881a6128707a7f2a42eec2ed63881)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* skipped modules were triggering an ERROR before:
poky $ oe-selftest -v -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers imagefeatures.ImageFeatures.test_image_gen_debugfs -K -B /OE/build/poky/build-eSDK
2023-03-13 15:07:53,430 - oe-selftest - ERROR - Not found eSDK.oeSDKExtSelfTest.test_install_libraries_headers in loaded test cases
* but didn't show the reason why it wasn't loaded and more importantly -r
was ignored when all selected modules were silently skipped
* add a warning when skipping some module and abort if some modules were
selected, but all ended being skipped:
poky $ oe-selftest -v -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers -K -B /OE/build/poky/build-eSDK
2023-03-13 15:11:51,028 - oe-selftest - WARNING - module 'eSDK.oeSDKExtSelfTest.test_install_libraries_headers' was skipped from selected modules, because it doesn't match with module name assumptions: package and module names do not contain upper case characters, whereas class names do
2023-03-13 15:11:51,028 - oe-selftest - ERROR - All selected modules were skipped, this would trigger selftest with all tests and -r ignored.
* I was hit by this in oe-selftest -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers
which is skipped due to upper case characters in module name and selftest started to run
all tests (archiver.Archiver.test_archiver_allows_to_filter_on_recipe_name as first from 529)
poky $ oe-selftest -v -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers -K -B /OE/build/poky/build-eSDK
2023-03-13 14:00:52,955 - oe-selftest - DEBUG - Selected tests with -r: ['eSDK.oeSDKExtSelfTest.test_install_libraries_headers']
2023-03-13 14:00:55,531 - oe-selftest - INFO - Changing cwd to /OE/build/poky/build
..
2023-03-13 14:00:58,128 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver.test_archiver_allows_to_filter_on_recipe_name)
I'll rename eSDK to esdk in next commit to avoid this.
* also fix small typo in context I've noticed when debugging this
(From OE-Core rev: 15229c9abaf4cc398c31d97d171b9fac57141873)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous version only included the numbers 1 and 2 in the allowed characters
for the module name. In the past, this was (\w+) so all numbers were allowed.
Now it explicitly includes all numbers again.
(From OE-Core rev: f241fa493536ac953c1dac303917c6e75b459e28)
Signed-off-by: Patrick Vacek <patrickvacek@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
test case in format <module name>.<class name>.<test case name>
this is clear when test cases is only 3 item deep.
but confused when it is 4 item deep, eg,
oelib.types.TestList.test_list_nosep
in this case, oelib and oelib.types can both be treated as module
since module name contains only lower cases and class name should
contain atleast one upper case.
so, always treat leading item without upper case as module also allow
module name to contain dot.
[YOCTO #13941]
(From OE-Core rev: ad81ea90a815389e45ff302a85151724c71f71c3)
Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rework OETestTag so that it does not rely on the existing decorator code
base and instead inserts the tags into an attribute on the decorated
target (e.g. class/type or method). This allows the use of OETestTag on
classes and method.
In order to filter tagged tests rework the loaders filtering code,
removing the generic-ness (with validation and attributes/etc.) and
replace it with a "tags_filter" parameter which is a function that
filters a test based on the tags it has. This allows the loader user to
filter on tags in more specific ways (e.g. include all untagged tests
and any tests tagged with foo). Plumb all this through the context code
and testing code.
Update the associated tests to pass correctly with the changes.
(From OE-Core rev: b8a4a4c2de68110d74607cb9807c9e741ca9441c)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.
The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).
More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.
The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.
(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Clean up the warning:
meta/lib/oeqa/core/loader.py:27: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
_failed_test_args = inspect.getargspec(unittest.loader._make_failed_test).args
(From OE-Core rev: d2deb66830be2d44532fea3d5db763b57778252a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oe-selftest -r <invalid test name>
currently triggers a traceback. Ensure this doesn't happen and the user gets a
sensible error message.
(From OE-Core rev: f2042bf3638ed4edfb167e7f7d4be6da60997ead)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In previous OEQA, having 'auto' in TEST_SUITES results in executing
as many test cases as possible.
This behaviour is broken for now. From the codes in core/loader.py,
I can see that it tries to use another keyword 'all'. But in fact,
it does not work.
I've checked the current manual. The manual says using 'auto'.
Below is the current information in manual.
"""
Alternatively, you can provide the "auto" option to have all applicable
tests run against the image.
TEST_SUITES_append = " auto"
"""
So we should restore this behaviour. This patch does so.
Also, output warning message is some module is named as 'auto', as this
is a reserved keyword.
(From OE-Core rev: a65460a063a958cc887c756db5f7ab18e3f5a8c1)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For test modules in a package, e.g., oelib.license, running
`oe-selftest -r oelib.license` or `oe-selftest -r
oelib.license.TestSimpleCombinations` would fail with a message that
the specified test cases could not be found. This was due to the
parsing in _built_modules_dict(), which failed to distinguish between
<package>.<module>.<class> and <module>.<class>.<testcase> and treated
both cases as the latter.
(From OE-Core rev: 80db3d999ae26d298d9d5418a32b11a4f27af9d5)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If some test module/case is specified to run and isn't found the OEQA
framework didn't notice it, so complete the implementation using
modules_required and validate for the test case prescense.
Raise an exception when the test module/case required isn't found.
[YOCTO #11645]
(From OE-Core rev: e50b415aaaa1581473f85f0a8afa278b5f95129b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python versions has different features from branches 3.4.x, 3.5.x and
3.6.x, i expected in wrong mode that was incremental for example changes
in 3.4.4 be in 3.5.x but that's not true.
The _make_failed_test internal method differs and is only available in
certain versions >= 3.4.4 and in 3.5.x and 3.6.x branches but not
realeses have been made including it.
So to avoid futher problems inspect the _make_failed_test and generates
function definition according what parameters are needed, the unique
supossition is that exception argument is always passed.
Related to,
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=d8380d098a290510b442a7abd2dd5a50cabf5844
(From OE-Core rev: b4740d2b325a80bcecc5e56dff9add9081fcd31b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python unittest change the signature of the _make_failed_test
after python 3.4.4 don't pass the method name.
(From OE-Core rev: 767b68e6ca22512ff80e6fbc42154f3f0c2206c0)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The small module name was added to support run a whole suite that
has more that 3 levels in the test case name, but this broke the
behaviour for use a full test case name.
[YOCTO #11632]
(From OE-Core rev: 9ab20ceb5801bee8dd8b218b3928720da5e1d403)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently there was a restriction to only execute tests that's
inherits from OETestCase but in some circunstancies the features
from the OEQA framework isn't needed so we need to support
basic unittests.
[YOCTO #10828]
(From OE-Core rev: baac26f1b36e89e07637b738dd31ec7356f05a02)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Our filtering allows to specify which tests to run using,
<module_name>.[test_class].[test_name]
But the module name logic was restricted to only accept one level,
for example: runtime_test vs oelib.types, to support multiple
submodules use only the first part for filtering.
This allows to run the whole tests in a module with more than tree
levels.
Due to the ambiguity on the test filtering options with test cases
with more than tree levels the supported sytnax is,
<module>
or
<module>.[submoduleN].[test_class].[test_name]
[YOCTO #11632]
(From OE-Core rev: cf2ee12b007e5570959ccfbb643159b21d90426e)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After Python 3.5 the init method contains initialization of
_loading_packages set to avoid infinite loops in recursive
loading.
Fix,
Traceback (most recent call last):
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/oe-selftest", line 70, in <module>
ret = main()
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/oe-selftest", line 57, in main
results = args.func(logger, args)
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/context.py", line 214, in run
rc = self._internal_run(logger, args)
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/context.py", line 175, in _internal_run
self.tc.loadTests(self.module_paths, **self.tc_kwargs['load'])
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/context.py", line 51, in loadTests
self.suites = self.loader.discover()
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/loader.py", line 268, in discover
pattern='*.py', top_level_dir=path)
File "/usr/lib/python3.5/unittest/loader.py", line 341, in discover
tests = list(self._find_tests(start_dir, pattern))
File "/usr/lib/python3.5/unittest/loader.py", line 398, in _find_tests
full_path, pattern, namespace)
File "/usr/lib/python3.5/unittest/loader.py", line 473, in _find_test_path
self._loading_packages.add(name)
AttributeError: 'OETestLoader' object has no attribute '_loading_packages'
(From OE-Core rev: 17b832ba05d640570a773adbff2fa4e9ff83fce3)
Signed-off-by: Anibal Limon <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will enable only run certain module tests and filter by
class and test name.
(From OE-Core rev: 0f7531803aa6ffef2b7c5be9acee3efe7d2974ef)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Underscore modules are meant to be run only when manually added to the test
suite, so far another mechanisms are in place to make this happen with
runtime, sdk, and esdk (mostly in test* bbclasses).
This will add such functionality in the core framework so other specific
frameworks can take use this without adding something else.
[YOCTO #10980]
(From OE-Core rev: 2c6eac774768aa610a8b3784483b9e90fb629c2d)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If importing a test with the same name as a built-in module,
it will silently import the built-in and check for tests in
built-in module. This happened with syslog module in debian
based machines, so add a raise to avoid this behavior.
[YOCTO #10978]
(From OE-Core rev: d9548f981448307b042807373e469f0d0b110bfe)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With this we get the class that is actually having the problem,
not just a TypeError with an unknown class causing the error.
(From OE-Core rev: d6ff4891376417504018af27e8e729a412feeeea)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
loader: Implements OETestLoader handling OETestDecorator
and filtering support when load tests. The OETestLoader is
responsible to set custom methods, attrs of the OEQA
frameowork.
[YOCTO #10231]
[YOCTO #10317]
[YOCTO #10353]
decorator: Add base class OETestDecorator to provide a common
way to define decorators to be used over OETestCase's, every
decorator has a method to be called when loading tests and
before test execution starts. Special decorators could be
implemented for filter tests on loading phase.
context: Provides HIGH level API for loadTests and runTests
of certain test component (i.e. runtime, sdk, selftest).
[YOCTO #10230]
(From OE-Core rev: 275ef03b77ef5f23b75cb01c55206d1ab0261342)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
