Now that the hangs in httpservice are fixed we can let these tests
auto skip as appropriate.
(From OE-Core rev: 42a0d70291d551578e21f590fcb85ca72a78ccb5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use our own lsb function instead as used elsewhere by the codebase.
(From OE-Core rev: acac45a6fd604d28ef7c23d67482af3d7e8bcfe3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now the test markup of the development tools tests is complete, this
can be further tweaked to auto run the correct tests.
(From OE-Core rev: bd4f8d12fe1f9f2643ee9e68fa2bb981134294fb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that the tests have correct markup to automatically determine which images
they can run against, clean up the default test suites logic to be simpler
and not image specific.
Some cleanup of the compiler tests still needs to be completed but this
is a good first step.
The only downside to this is more noise during testing as we now see
many skipped messages for simple images like core-image-minimal.
The auto type is being removed since it currently breaks badly due to the
socat mandatory inclusion from the meta-selftest layer which is a problem
which needs to be addressed seperately.
(From OE-Core rev: 4966bc33845752eb0aeae54b72e8ba0146a7ed52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the preceeding commit to:
* Add STARTTIME to the identifier to make it unique
* Log DISTRO
* Use LOG_DIR
* Store the layer config in a more natural json format
* Drop '_' function prefixes
(From OE-Core rev: dda94cfa708cf3c6e5408d6dbcece360b4e8cd27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As part of the solution to replace Testopia to store testresult,
OEQA testimage need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.
By default, json testresult file will be written to "oeqa"
directory under the "WORKDIR" directory.
To configure multiple instances of bitbake to write json testresult
to a single testresult file at custom directory, user will define
the variable "OEQA_JSON_RESULT_DIR" with the custom directory for
json testresult.
(From OE-Core rev: 5d135d4769b6bb60d575eb6ed196367f9e077cc4)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ptest has dependencies on other tests so contain them in a
variable. This allows layers to not worry about maintaining the depends
themselves.
(From OE-Core rev: c5c5ad9030ec7ddaa6387e2ba52e9e37b9ea76a2)
Signed-off-by: Armin Kuster <akuster@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently only the dnf package manager tests are automatically added to the
list of tests to run. Improve the code to handle automatic addition of
the apt and opkg tests too.
(From OE-Core rev: c9169e4d1fa67289d5beb0e1f8fb5c54153c4242)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the opkg/apt tests are run without a package-index they will fail.
Trigger this here for now as a dependency until the code can be improved.
(From OE-Core rev: 7805eec0f522da2d2b6a08c8a07a9243e26c014a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current qemuall override overwrites other dependencies unintentionally.
Tweak the code to avoid this by appending to the variable.
(From OE-Core rev: fc8df45e6b27dd7a26b7bf88a06ffa831fcf075c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
QEMU_USE_KVM can either be a boolean or a whitespace separated list
of kvm supported machines.
For the 'intel-corei7-64' machine, defined in meta-intel, kvm could not be
used as the 'x86' substring is not part of its machine name.
By changing the order of this 'or' statement and setting
the 'QEMU_USE_KVM' variable to 'intel-corei7-64', it is possible to run the
'testimage' task with kvm support successfully.
(From OE-Core rev: a22789253aa653dc50fb159b40910248c2f98dd4)
Signed-off-by: Emmanuel Roullit <emmanuel.roullit@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The testimage changes break the autobuilder. The plan is to revert these hacks
but due to various changes happening with the autobuilder, we need these for now
to keep things working until we can move to the new autobuilder codebase.
(From OE-Core rev: 309a02931779f32d1139cc1169a039cbe4638706)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The TEST_IMAGE interface has never particularly worked and image testing currently
gets configured manually.
This reworks the interface to better serve the needs of its users, replacing it
with TESTIMAGE_AUTO.
This does away with the need for the separate class due to better bitbake APIs for
changing tasks.
TESTIMAGE_AUTO will automatically boot under qemu any image that is built. It
also adds in dependencies so that any SDK for which testing is requested will
automatically be built first.
The code in bitbake.conf was error prone (e.g. testsdk wasn't considered), this
improves it to standardise on IMAGE_CLASSES as the standard configuration mechanism.
(From OE-Core rev: b34d44f3dfea8254826a46701a4fe3769a900434)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
testimage-auto is expected to run testimage task's codes automatically.
But in fact, it's currently missing some codes, including testimage_sanity
and create_rpm_index.
This leads to the problem of unexpected runtime failure of test_dnf_makecache.
The error message is as below.
RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR
This error is caused by the fact that create_rpm_index is not executed
before running the tests.
There's no reason why such codes should not be in testimage_main, so
move them into it.
(From OE-Core rev: fa7ba486ded13907f63f9300f66350ba2835a3f7)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having 'auto' in TEST_SUITES will also run the 'dnf' test cases,
so also check it to determine whether to create rpm index or not.
This is to fix the following error when TEST_SUITES = "auto".
RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR
(From OE-Core rev: c1f4177848d25a9121f2a85da655ee414cd424b1)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current behaviour of TEST_SUITES is very confusing.
setting: TEST_SUITES = ""
result: Execute all test cases.
setting: TEST_SUITES = "some_case_not_exist"
result: Error out with 'Empty test suite' message.
The expected behaviour of TEST_SUITES should be:
1. when 'auto' is in it, execute as many test cases as possible
2. when no valid test case is specified, error out and give user message
The first one is implemented by a previous patch.
The second one is fixed in this patch.
Also add debug message to show test cases to be executed. This is
for easier debugging.
(From OE-Core rev: 909568821fbad8a6a7034b10a2dc349a210fdfc6)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add test cases to test “exclude” and “installroot“ options, also modify
the logic of filtering packages on the feed to have all the packages
needed by the tests.
[YOCTO #10744]
(From OE-Core rev: 1121806603c6f621d084b692216f3f616a0768dc)
Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If no ptest packages are installed in the image, the test does nothing;
if ptest packages are installed in the image, then they should be
run without user having to enable that manually.
(From OE-Core rev: f57feab2727dca916744deb64825f3beaf07961d)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, the fact an error message is shown means the rest of the
task logs are suppressed. In this case we don't want that as it hides
the real errors and useful information. Therefore override this behaviour.
(From OE-Core rev: 7058ac23c5ce32fbdc4be6c05f85550e1fd9fe77)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I have no idea why we didn't do this but it means the code has nowhere
to log to unless we do this. This means we can then use the logger
to log data to the task logs.
(From OE-Core rev: e241fb128840889acff79dbf3fae7bfaafcb5357)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's very important to add IMAGE_CLASSES += "testimage" in local.conf firstly,
otherwise the var like TEST_LOG_DIR (defined in testimage.bbclass) will not be
in testdata.json.
[YOCTO #11547]
(From OE-Core rev: 5663ed989f0af5b1c61c74288ec421cbca2261e7)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the new filters by <module>.[class].[testname] the modules
arg is expected to be a list so use split into TEST_SUITES variable.
(From OE-Core rev: b588c5c65e55a31d61c86553158eef409cd71086)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Those methods are used to write in the log the results so
it makes sense to have defined at OETestResult because
is a format of the result itself.
[YOCTO #11450]
(From OE-Core rev: 33a783f59ed4e232f41f8b09dfa7955f2ddc2f80)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
qemu-native-helper has an additional task that needs to be run in order
for testimage to work. This task is usually run by default in a full
build but there are use cases where it might be skipped. This commit
adds the dependency explicitly.
Also, this commit adds a try/catch error message to make it clearer what
you need to do if you try to run testimage before you have built or
downloaded the image artifacts.
[YOCTO #11375]
(From OE-Core rev: 6e019537b9eb3af482e474a8cb248fe7312f4b58)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, QEMU_USE_KVM=True enabled the use of kvm only when "x86"
was in the MACHINE name. That is too limiting, because for example
intel-corei7-64 can also use kvm but it wasn't possible to enable that
without changing OE-core.
That traditional usage is still supported. In addition, QEMU_USE_KVM
can be set to a list of space-separated MACHINE names for which kvm is
to be enabled.
(From OE-Core rev: d5421dd00b9cf785fa77e77c6c739e8bd8822fa3)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently fetch,configure,build, and install cvs as our test for the
sdk. cvs unfortunately, requires a default editor in order to run. The
change in 94790a8254 that checks to see if you have something like vi
installed is fragile since you may have a different default editor. This
patch switches from using cvs as a test to using cpio. cpio also uses
autotools so the functionality tested is equivalent.
(From OE-Core rev: 233d36b0382a8b2e430c3377e50885d1a0c3ba21)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This causes MBs of log messages which overload the standard 200kb syslog limit
used by busybox syslog by default. This means some tests which rely on messages
being detected in syslog, e.g.
rpm.RpmInstallRemoveTest.test_check_rpm_install_removal_log_file_size
fail (AssertionError: 54 not greater than or equal to 80 : Cound not find sufficient
amount of rpm entries in /var/log/messages, found 54 entries).
We enabled this to aid debugging of some systemd race issues, those are
now resolved so we can disable this. Leave the log level parameter as a
comment to save others having to look it up if they need debugging.
(From OE-Core rev: 84d4daffaebcb4816d597671b287329e8a472f59)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea on getTarget is to use kwargs to send custom variables
to different targets, instead of this, a new variable was added
(just used for custom targets) and this broke testexport. So
in order to fix it, just add the custom variable to kwargs.
This fixes the use of getTarget() in testexport class that was
introduced in 1dc8010afd71fe46fb28bb86fb7c07a5fbd3d7cf.
(From OE-Core rev: cf138029a1f18f991fc7a28c81d85884942e9d56)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If for some reason (most likely incorrect TEST_SUITES var)
you end with an empty test suite, testimage will start the
target (qemu by default) and will run an empty suite, so
better have a check before starting the target.
[YOCTO #10979]
(From OE-Core rev: 8345f834a080737f03bee3f0c615dd9654401d93)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch implements custom target loading for testimage, currently
missing due to major changes to the test framework.
Custom targets can be defined in various meta-layers, so we
need an extra path information in order to find them.
Any other additional info is retrieved as usual via the variables
TEST_TARGET and TEST_SUITES
(From OE-Core rev: 1dc8010afd71fe46fb28bb86fb7c07a5fbd3d7cf)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Debian based distros has a builtin syslog module so when
try to load tests using unittest it references the builtin
module instead of runtime/cases.
[YOCTO #10964]
(From OE-Core rev: 9923e3cdb58c2b3c54ec5fe99b2cec4cdc9fff92)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some missing suites on sato-sdk and lsb-sdk images so add it.
The createrepo-native needs to be built before run testimage in
order to create a repository from install packages.
The DL_DIR data also needs to be updated from the current bitbake
environment instead of use the value from testdata file.
[YOCTO #10964]
(From OE-Core rev: 42f3026b39519ba98ae3153329622ce670d57c09)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the change to the new framework data store dependecy was
removed, instead a new file is generated and used in testimage.
When testing builds from the autobuilders the test data values
are from the autobuilder, including the paths.
Some tests require paths to current environment in order to run,
this commit will update such paths and fix the error of running
images donwloaded from autobuilders.
[YOCTO #10964]
(From OE-Core rev: 26ad5105fc2ce03b7ee8ecc6911fd40a52bd573a)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds systemd test automatically when an image
is build with systemd DISTRO_FEATURE
[YOCTO #10935]
(From OE-Core rev: 2e6ba167bee7c7b42b1e0e96eb5a7be2eb0fd2c0)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows to use the package install feature with
the new OEQA framework.
[YOCTO #10234]
(From OE-Core rev: 077dc19445574457769eb4f231de97e8059cb75e)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
testimage support the installation of packages without a package
manager in the target. This adds support for package extraction
required to support the installation feature.
[YOCTO #10234]
(From OE-Core rev: 8c7335290cb00ed0683241249297ca573ebd353a)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This migrates the smart test from the old framework to
the new one. This has its own commit because smart
test was using bb and oe libraries that are available
when exporting the test cases to run in a different host.
Because of the removal of bb and oe libraries index and
packages feeds creation will be managed in testimage bbclass.
[YOCTO #10234]
(From OE-Core rev: 8d64ac4208e8dcb8a6fde6ea2959c9b3edfe2172)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This migrates testimage class to use the new framework. Most of
the code added here is to get rid off the data store dependency.
[YOCTO #10234]
(From OE-Core rev: 2aa5a4954d7610f31875ba7e655f25f8892517b6)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TargetBuildProject was refactored to avoid bitbake dependency so
the instance don't allow to pass data store anymore.
classes/testimage: Export proxies before run tests
The TargetBuildProject based tests download archives from network.
(From OE-Core rev: e275f29de500a338a02402ecc570405309963b35)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
