Commit Graph

1104 Commits

Author SHA1 Message Date
Ed Bartosh
3a4a738bc5 oe-selftest: wic: change mkfs.btrfs options
test_mkfs_extraopts test case fails on ab with error caused
by using -K --mixed options:
 output: extent-tree.c:2696: btrfs_reserve_extent: BUG_ON `ret` triggered, value -28

For this test case it's not important to use particular options,
so changing options to anything less influential is OK.

Changed extra options for mkfs.btrfs to '--quiet' to fix the failure.

(From OE-Core rev: 6a1489f8deb2b26181ca340a0f6ef62aa18a5188)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-27 22:36:52 +01:00
Jose Perez Carranza
859d9a477a oeqa/buildperf: Add 'bitbake -m' on sync function to ensure bitbake is unloaded
Add 'bitbake -m' to the sync method and ensure all process related to
bitbake are correctly unloaded before doing the different measurements.
Also add a call to sync funtion on Test4 before final measurment of
eSDK deploy dir disk usage.

(From OE-Core rev: 9210c9ce051dfffaa7afa36bb4a926cea289ffd4)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-25 23:15:53 +01:00
Ed Bartosh
7916a5f013 oe-selftest: wic: add test_mkfs_extraopts test case
Test new wks option --mkfs-extraopts with all filesystems
supported by wic.

(From OE-Core rev: 79e853c450708e1e06684cf1216337cbad964770)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 22:51:38 +01:00
Aníbal Limón
ad734fd64a oeqa/selftest/{context,case}: Handle KeyboardInterrupt/SIGINT and SIGTERM
In order to avoid corrupt local.conf and bblayers.conf adds
signal handler for SIGTERM and use try/finally (KeyboardIntrrupt) block
to restore previously backuped configuration.

[YOCTO #11650]

(From OE-Core rev: 9419c81e69d2facc82e39c846466670c09e6b444)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Aníbal Limón
f4a978485f selftest/cases/package: Call parent setUpClass method
Since config paths are now passed in Test context the setUpClass
method is expected to be call.

(From OE-Core rev: 7281c995ff2b009c3fb23c7af1d91fe106ca8f87)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Aníbal Limón
bec45829e3 oeqa/core/loader: Generate function _make_failed_test dynamically
Python versions has different features from branches 3.4.x, 3.5.x and
3.6.x, i expected in wrong mode that was incremental for example changes
in 3.4.4 be in 3.5.x but that's not true.

The _make_failed_test internal method differs and is only available in
certain versions >= 3.4.4 and in 3.5.x and 3.6.x branches but not
realeses have been made including it.

So to avoid futher problems inspect the _make_failed_test and generates
function definition according what parameters are needed, the unique
supossition is that exception argument is always passed.

Related to,
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=d8380d098a290510b442a7abd2dd5a50cabf5844

(From OE-Core rev: b4740d2b325a80bcecc5e56dff9add9081fcd31b)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Richard Purdie
1ae1df3853 oeqa/eSDK: Ignore errors during directory cleanup
The cleanup can fail with:

ERROR [0.000s]: tearDownClass (eSDK.oeSDKExtSelfTest)
FileNotFoundError: [Errno 2] No such file or directory: 'bitbake.sock'

which is due to bitbake taking a small amount of time to shut down the server.
The easiest fix is just to ignore these kinds of errors, bitbake shouldn't create
any new files during shutdown.

(From OE-Core rev: 68009f4eb323e07cec8c4421d546bd3125dec033)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Ed Bartosh
eeef9c2202 imagefeatures: add test_image_fstypes test case
Tested if core-image-minimal can be built for existing fstypes by
building an image and checking if result file <image>.<fstype> exists
in the image deploy directory.

(From OE-Core rev: 9db3dbde66e3590aea71400891eaea0ae2caf371)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:37 +01:00
Ed Bartosh
5a514183e1 selftest: fixed 5 wic test cases
Added core-image-minimal -> syslinux dependency to ensure
syslinux artifacts are available from core-image-minimal
build. This should fix test_iso_image, test_bootloader_config and
test_default_output_dir test cases.

Used --native-sysroot in test_image_vars_dir_long and
test_image_vars_dir_short test cases to point out to wic-tools
native sysrtoot.

(From OE-Core rev: b7bcc4ba756aa3941c89999e9d6761738ea5bb05)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:37 +01:00
Richard Purdie
854c8c2e4c oeqa/tinfoil: Improve test_wait_event for race issues
The test could break in a variety of ways:

a) If BB_HEARTBEAT_EVENT was less than ~0.25 it would hang indefinitely
b) The mask is set after draining the event queue meaning a heartbeat event
   could have happened
c) The test exits once it sees the events it wants, it doesn't check for
   spurious events such as heartbeats which shouldn't have occured.
d) The hardcoded delay of 0.25 is nasty and shouldn't be needed.

I found a bitbake bug and fixed that meaning we don't need the delay any
more which fixes d). That means a) is no longer an issue either.

We now set the mask, then drain the queue meaning no spurious events should
be able to sneak in. The test is also tweaked to wait for 5s in total to
ensure spurious events don't occur such as heartbeat events we shouldn't see.

[YOCTO #11045]

(From OE-Core rev: 7dd5dfc4d56f1201110d947ce1ca3c6d64fbc7da)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08 13:34:46 +01:00
Jose Perez Carranza
aac343acc0 oeqa/sdk: Replace buildiptables for buildlzip tests
Buildiptables test cases are conflicting with images built with “musl”
as standard C library, in order to avoid those issues lzip package was
selected to be used on the tests as this does not have any "musl"
dependency.

[YOCTO #11713]

(From OE-Core rev: b798284f62b3cb171373716b1ee84403439314aa)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08 13:34:46 +01:00
Jose Perez Carranza
37e0cd0e91 oeqa/runtime: Replace buildiptables for buildlzip on runtime tests
Buildiptables test cases are conflicting with images built with “musl”
as standard C library, in order to avoid those issues lzip package was
selected to be used on the tests as this does not have any "musl"
dependency.

This patch is applicable for testimage tests

[YOCTO # 11713]

(From OE-Core rev: 41683e0ab316049e28b1f4ceaf39f0fe17722d92)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08 13:34:46 +01:00
Patrick Ohly
8b6fde28e9 runcmd.py: unit testing for runCmd()
This covers the traditional API as well as the new output_log feature.
While testing, it was noticed that killing hanging commands does not
work when a shell is used to run the command(s). This might be worth
fixing.

(From OE-Core rev: 62489e58ca9975f58b48fc2bd8cf27fd22e25564)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28 20:55:09 +01:00
Patrick Ohly
ea34f20a06 commands.py: live output logging + result.error encoding fix
Tests that use bitbake("my-test-image") can run for a long time
without any indication to the user of oe-selftest about what's going
on. The test author has to log the bitbake output explicitly,
otherwise it is lost in case of test failures.

Now it is possible to use bitbake("my-test-image",
output_log=self.logger) to get more output both on the console and in
the XML output (when xmlrunner is installed). Example output:

2017-06-23 12:23:14,144 - oe-selftest - INFO - Running tests...
2017-06-23 12:23:14,145 - oe-selftest - INFO - ----------------------------------------------------------------------
2017-06-23 12:23:14,151 - oe-selftest - INFO - Running: bitbake my-test-image
2017-06-23 12:23:16,363 - oe-selftest - INFO - Loading cache...done.
2017-06-23 12:23:17,575 - oe-selftest - INFO - Loaded 3529 entries from dependency cache.
2017-06-23 12:23:18,811 - oe-selftest - INFO - Parsing recipes...done.
2017-06-23 12:23:19,659 - oe-selftest - INFO - Parsing of 2617 .bb files complete (2612 cached, 5 parsed). 3533 targets, 460 skipped, 0 masked, 0 errors.
2017-06-23 12:23:19,659 - oe-selftest - INFO - NOTE: Resolving any missing task queue dependencies

Because the implementation was already using threading, the same is
done to decouple reading and writing the different pipes instead of
trying to multiplex IO in a single thread. Previously the helper
thread waited for command completion, now that is done in the main
thread.

The most common case (no input data, joined stdout/stderr) still uses
one extra thread and a single read(), so performance should be roughly
the same as before.

Probably unintentionally, result.error was left as byte string when
migrating to Python3. OE-core doesn't seem to use runCmd() with split
output at the moment, so changing result.error to be treated the same
as result.output (i.e. decoded to a normal strings) seems like a
relatively safe API change (or rather, implementation fix).

(From OE-Core rev: 00b8c7ff17cd8f1920728fdc2653068e63d71724)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>

merge: wait()
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28 20:55:08 +01:00
André Draszik
68e763d54f selftest/archiver: only execute deploy_archives task
There should be no reason to execute a full build, as we're
just interested in the deployment of the archives.

The newly added tests already do the same.

(From OE-Core rev: 78c2897dff7cd9fe2cab511549cb146d5231e573)

Signed-off-by: André Draszik <adraszik@tycoint.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28 20:55:08 +01:00
André Draszik
99e1b01cdc selftest/archiver: add tests for recipe type filtering
The archiver used to be able to filter based on COPYLEFT_RECIPE_TYPES.

Unfortunately, this got broken with the fix for
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=6929
in commit ae9102bda3 ("copyleft_filter.bbclass: Allow to filter on name")

Add two tests to prevent that from happening again.

(From OE-Core rev: 709f02c5cb25983090251c6237bac4fc0a295c4f)

Signed-off-by: André Draszik <adraszik@tycoint.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28 20:55:08 +01:00
Jose Perez Carranza
f44c923bb9 selftest/seltest.py: Add test to check imports from other layers
This tests adds a check of selftest itself to verify if can
add test from other layers.

[YOCTO #9770]

(From OE-Core rev: 4fe4c408246b9a4a563106d097876e6caefca694)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28 20:55:08 +01:00
Mikko Rapeli
b090ff6a27 meta: Fix return value checks from subprocess.call()'s
Python function subprocess.call() returns the return value of the
executed process. If return values are not checked, errors may
go unnoticed and bad things can happen.

Change all callers of subprocess.call() which do not check for
the return value to use subprocess.check_call() which raises
CalledProcessError if the subprocess returns with non-zero value.

https://docs.python.org/2/library/subprocess.html#using-the-subprocess-module

All users of the function were found with:

$ git grep "subprocess\.call" | \
  egrep -v 'if.*subprocess\.call|=\ +subprocess\.call|return.*subprocess\.call'

Tested similar patch on top of yocto jethro. Only compile tested
core-image-minimal on poky master branch.

(From OE-Core rev: 578c8205fd14c48c6d30ef2889d86f1b4aee060a)

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28 20:55:08 +01:00
Jussi Kukkonen
ad0d9ce3c6 selftest/devtool: Don't use removed remake as test case
remake was removed from oe-core: use another recipe in the devtool
extract test.

(From OE-Core rev: dbf680ce09ce54f97c781fc7a8e5e05ed4706073)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28 15:52:17 +01:00
Alexander Kanavin
51c8ac2d6a oe-selftest: add a test for upstream version check regressions
The test runs an upstream version check, and then compares the
list of recipes that failed the check (i.e. those where latest
upstream version could not be established) against the list of
known-broken upstreams. Mismatches either way (upstream check failed,
recipe not in the exception list, or upstream check worked,
recipe in exception list) fail the test.

[YOCTO #11031]

(From OE-Core rev: 544e04a11ab6c26a367ac0b67774bf7ffdf43c88)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28 15:52:17 +01:00
Aníbal Limón
7a19bbcf81 oeqa/selftest/context: Reload testlayer_path when meta-selftest isn't added
When add meta-selftest by the script the testlayer_path needs to be
reloaded to avoid None value.

(From OE-Core rev: d1120849e12c46aa46b02905a3f43c0853bd11a0)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23 11:44:13 +01:00
Aníbal Limón
d8380d098a oeqa/core/loader.py: Fix _make_failed_test for python >= 3.4.4
Python unittest change the signature of the _make_failed_test
after python 3.4.4 don't pass the method name.

(From OE-Core rev: 767b68e6ca22512ff80e6fbc42154f3f0c2206c0)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23 11:44:12 +01:00
Aníbal Limón
aa6e333de9 oeqa/core/threaded: Don't assume that results exists on logDetails
(From OE-Core rev: 7beed75c97a78e945e44a55b28f0f463cd6c8dcd)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23 11:44:12 +01:00
Jose Perez Carranza
ab0780dfea selftest: Add Testopia ID to test cases
Add decorator @OETestID() with proper Tesopia TC ID to the test cases
that did not have it set.

(From OE-Core rev: d7bc697534db911a3ce98537d772d87482a0f702)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23 11:44:12 +01:00
Yeoh Ee Peng
f822525c58 selftest/devtool: replace file assertTrue/False with assertExists/NotExists
Current osselftest print confusing assertion message when using
self.assertTrue(os.path.exists(filepath)) to test file path,
example of confusing assertion message:
	AssertionError: False is not true

Replce assertTrue/assertFalse with assertExists/assertNotExists to test
file path, this will improve assertion message and simplify coding,
self.assertExists(filepath) will print below
	AssertionError: <filepath> does not exist

[YOCTO #11356]

(From OE-Core rev: 08b0702492536d41d3cf2c9c05d1fab36d32a566)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23 11:44:12 +01:00
Richard Purdie
5db4d0dbbb selftest/sstatetests: Fix potential failure on uniprocessor machines
It was pointed out +1 is safer than -1 for systems with one processor.

(From OE-Core rev: 78041e20e43d9583448ff31f8b9b1c6157da8625)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14 11:56:09 +01:00
Ed Bartosh
1de1a34a64 selftest: add test_wic_rm test case
Added test case for "wic rm" functionality.
    - remove file from vfat partition
    - remove directory from vfat partition

(From OE-Core rev: fc42914d426ad2dad8c3026668726c003b3ab10b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14 10:18:30 +01:00
Ed Bartosh
bb94a25b85 selftest: add test_wic_cp test case
Added test case for "wic cp" functionality.
     - copy file to vfat partition
     - copy directory to vfat partition

(From OE-Core rev: cb416d1feea042bcdedc9f522d588fef2c4929bc)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14 10:18:29 +01:00
Ed Bartosh
4917798395 selftest: add new test case test_wic_ls
Tested 'wic ls' functionality:
 - list of image partitions
 - list of directory content of vfat partition

(From OE-Core rev: 3b271f6a40d7ef93fff9700dd8ac98bddd92d581)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14 10:18:29 +01:00
Richard Purdie
891b5ee2ee sstatetests: Use higher parallelism value
Since the processing code for signature generation is now threaded,
use higher thread values as examples in this code for better performance.

(From OE-Core rev: f68ec7191546474f0bd688e57d2381a8e92be617)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14 10:18:28 +01:00
Aníbal Limón
77a358e41e oeqa/core/loader: Use full and small module name on filtering
The small module name was added to support run a whole suite that
has more that 3 levels in the test case name, but this broke the
behaviour for use a full test case name.

[YOCTO #11632]

(From OE-Core rev: 9ab20ceb5801bee8dd8b218b3928720da5e1d403)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-13 10:46:34 +01:00
Aníbal Limón
3c5c8ccee1 oeqa/core/runner: Don't log details twice if test fails
The details of a test failure is upper on the unittest output
so don't log twice the actual failure.

[YOCTO #11622]

(From OE-Core rev: 0f2e81c2a4458ad0ec6bab2710952ac2c2bbf1af)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-13 10:46:34 +01:00
Aníbal Limón
8573de63aa oeqa/sdk/cases: Fix skip of buildgcalculator test
The tc.hasTargetPackage uses a re.search to see if gtk+3 is on
the manifest but + in regex means 1 or more causing the test
to be skipped.

(From OE-Core rev: 271cd99d00bde0b9f2aa27141acbe06812f34638)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 23:01:23 +01:00
Francisco Pedraza
05697aa941 oeqa/sdk/cases: Added validation for SDK compatibility tests with eSDK
The manifests for eSDK are generated using shared states so there is a
need to validate to different "packages names" into the test cases.

For example for perl:

SDK provides nativesdk-perl
eSDK provides perl-native

[YOCTO #9090]

(From OE-Core rev: 8db06dd1290dd53d626050879c9c306f95d76ac2)

Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 23:01:23 +01:00
Aníbal Limón
0b168b269b oeqa/core/loader: Allow unittest.TestCase's to be executed
Currently there was a restriction to only execute tests that's
inherits from OETestCase but in some circunstancies the features
from the OEQA framework isn't needed so we need to support
basic unittests.

[YOCTO #10828]

(From OE-Core rev: baac26f1b36e89e07637b738dd31ec7356f05a02)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:08:30 +01:00
Aníbal Limón
c3587b7d10 oeqa/cases/oelib: Change default case class to unittest.case.TestCase
Some tests doesn't need call bitbake so it is better to use the
basic unittest case class.

[YOCTO #10828]

(From OE-Core rev: 4d01610f36eaee8da3126bb5045856279371fd17)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:08:30 +01:00
Aníbal Limón
652df69b02 oeqa/core/loader: Fix filtering on test modules with submodules
Our filtering allows to specify which tests to run using,

<module_name>.[test_class].[test_name]

But the module name logic was restricted to only accept one level,
for example: runtime_test vs oelib.types, to support multiple
submodules use only the first part for filtering.

This allows to run the whole tests in a module with more than tree
levels.

Due to the ambiguity on the test filtering options with test cases
with more than tree levels the supported sytnax is,

<module>

or

<module>.[submoduleN].[test_class].[test_name]

[YOCTO #11632]

(From OE-Core rev: cf2ee12b007e5570959ccfbb643159b21d90426e)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:08:30 +01:00
Aníbal Limón
e9a6fd9e97 oeqa: Change the order to logDetails and logSummary
Is better to log the summary at end to see in an easy way
the actual result of the test run.

[YOCTO #11622]

(From OE-Core rev: 4e3ab36e8c90abc740cce1ba31faf6595116e1e2)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-09 17:12:15 +01:00
Jose Perez Carranza
38c06ab450 selftest/devtool: Modify test to use default config
Modify “test_devtool_virtual_kernel_modify” to be executed with default
configuration defined on oe-selftest test suite. A shorter string was
added to the the header file to avoid overlapped sections when building
kernel for qemux86-64.

[Yocto #11300]

(From OE-Core rev: a8bda790fded43eac72b2ad76d47c27363f02070)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-09 17:12:14 +01:00
Humberto Ibarra
f1812e090b runtime-test.py: Split different tests inside selftest.py
There were two completely different tests inside selftest.py,
and the only reason for them to be together was that both needed
the same mechanism to execute (run testimage from within selftest)

This fixes the design issue and error-prone situation by separating
these tests. In add a new module for dnf-runtime and also has an
extra time added to running the tests, but it is minimal since
the builds reuse data from previuous build.

[YOCTO #11436]

(From OE-Core rev: 2f6a506018973f0eaf1306555df3e74a3ab25680)

Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-09 17:12:13 +01:00
Leonardo Sandoval
4a7612c7a1 buildhistory: skip tests if GitPython module is missing
Catching a possible exception when importing the git module,
allows us to skip test if the latter is missing on the host.
Also, import oe.buildhistory_analysis inside fuctions
because this module also needs git to work correctly.

[YOCTO #11620]

(From OE-Core rev: 186882ca62bf683b93cd7a250963921b89ba071f)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-07 16:00:49 +01:00
Anibal Limon
cfa8245e0e oeqa/core/loader: Call parent init method in OETestLoader
After Python 3.5 the init method contains initialization of
_loading_packages set to avoid infinite loops in recursive
loading.

Fix,

Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/oe-selftest", line 70, in <module>
    ret = main()
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/oe-selftest", line 57, in main
    results = args.func(logger, args)
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/context.py", line 214, in run
    rc = self._internal_run(logger, args)
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/context.py", line 175, in _internal_run
    self.tc.loadTests(self.module_paths, **self.tc_kwargs['load'])
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/context.py", line 51, in loadTests
    self.suites = self.loader.discover()
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/loader.py", line 268, in discover
    pattern='*.py', top_level_dir=path)
  File "/usr/lib/python3.5/unittest/loader.py", line 341, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/usr/lib/python3.5/unittest/loader.py", line 398, in _find_tests
    full_path, pattern, namespace)
  File "/usr/lib/python3.5/unittest/loader.py", line 473, in _find_test_path
    self._loading_packages.add(name)
AttributeError: 'OETestLoader' object has no attribute '_loading_packages'

(From OE-Core rev: 17b832ba05d640570a773adbff2fa4e9ff83fce3)

Signed-off-by: Anibal Limon <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:02:44 +01:00
Aníbal Limón
d95700ff6e oeqa/runtime/context.py: Add support to specify port in target_ip
(From OE-Core rev: 7fe4c074611eb21abdf811d8f216d4df7ab6a3ea)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:02:44 +01:00
Leonardo Sandoval
157c3be2ca oeqa/selftest/cases: Migrate test cases into the new oe-qa framework
New framework has different classes/decorators so adapt current test cases to
support these. Changes include changes on base classes and decorators.

Also include paths in selftest/__init__.py isn't needed because the
loader is the standard unittest one.

(From OE-Core rev: ddbbefdd124604d10bd47dd0266b55a764fcc0ab)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:02:43 +01:00
Leonardo Sandoval
10c512b60d scripts/oe-selftest: Migrate to new framework into oeqa.selftest.context
The new OEQA framework aims to re-use code into the different Test
components.

The previous oe-selftest implements it-self loading, run, and list test
cases in a non-standard way (unittest base) and other functionalities
like logging that is now on oeqa core. This ends on a compact oe-selftest
script.

All needed command line options was migrated but there are some of them
pending of implementation and others deprecated.

Deprecated options:

list-tags: The tag functionality into the old oeqa framework isn't
    work, the selftest doesn't has tag decorators.
{run, list}-tests-by: Ambiguos options it accepts all the posibilites module,
    class, name, id or tag.

Remaining to implement:

coverage: It enables covrage reports over a test run, currently isn't on
    on use and some bugs [1], i filed a bug to add support to OEQA core module in
    this way other Test components could enable it.
repository: It push XML results into a git repository and isn't in use,
    i filed a bug to implement this into OEQA core module. [2]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11582#c0
[2] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11583#c0

(From OE-Core rev: 3b2a20eee4a39f40287bf67545839eaa09fc892d)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:02:43 +01:00
Aníbal Limón
d09938a608 oeqa/selftest/case: Don't figure out the testlayer inside
The test layer is added at init of selftest and is the same
across test runs so pass it as a attr in the context.

(From OE-Core rev: 8441da8fbc7ff2237d9ccd3a5c5117345f5c87b2)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:02:43 +01:00
Leonardo Sandoval
5633866ea5 oeqa/selftest/case.py: Remove machine selection logic
The machine selection is an operation that needs to be made
in every test run, the best place to it is on the context
module.

Use self.tc.custommachine variable instead of use environment.

SIgned-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
(From OE-Core rev: f295b70ace0ffc28256140a21af5c3a8903297cb)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:02:43 +01:00
Leonardo Sandoval
628ba5801b scripts/oe-selftest: Move {add,remove}_include files to case
The oe-selftest creates include files to store custom configuration to
make specific tests, every class executes a different test and may be
uses custom configuration.

So move to case class in order to simplify oe-selftest script and later
implement later a build folder per class.

(From OE-Core rev: 1130b40c3dfa65e7ece08a95b3941e4d1d20bcf0)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:02:43 +01:00
Leonardo Sandoval
a80aa4c025 oeqa/selftest/case: Migrate case class to the new OEQA framework
Summary of the changes:

- Use OETestCase as base class instead of unittest.TestCase
- Remove LogResults decorator the new framework provides logging into
  the core functionality.
- Logger is now self.logger instead of self.log
- Move comments into docstrings in several help methods
- Use get_test_layer() method instead of access monkey patched variable
  in old oeSelfTest case class.

(From OE-Core rev: c38cab77893f9d8fd505f050cc880a15677b73db)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:02:43 +01:00
Aníbal Limón
cb1b1eeb3f oeqa/selftest: Move base class to case module
To match the new structure of the OEQA framework.

(From OE-Core rev: 0d3d97414c8166d09065f6f7f45e3260ce405692)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:02:43 +01:00