Commit Graph

2294 Commits

Author SHA1 Message Date
Richard Purdie
ceccf911b5 utils/multiprocess_launch: Improve failing subprocess output
Output before this patch:

ERROR: bash-4.4.18-r0 do_package_write_ipk: Fatal errors occurred in subprocesses:
Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1.: Traceback (most recent call last):
  File "/media/build1/poky/meta/lib/oe/utils.py", line 272, in run
    ret = self._target(*self._args, **self._kwargs)
  File "/media/build1/poky/meta/classes/package_ipk.bbclass", line 230, in ipk_write_pkg
    shell=True)
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1.

Note how stdout/stderr from the failing command isn't shown.

After this patch:

ERROR: bash-4.4.18-r0 do_package_write_ipk: Fatal errors occurred in subprocesses:
Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1.
Subprocess output:Foobar
*** Error: Package name Foobar contains illegal characters, (other than [a-z0-9.+-])

opkg-build: Please fix the above errors and try again.

We suddenly get a much more usable error message. The traceback is supressed
as its distracting from the real problem in this case.

Ideally python itself would handle this but it doesn't so we have to
wrap the exception. We already do this in bitbake itself for the same reason.

(From OE-Core rev: c27cc2355d331e063f170b45728671988a967deb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-12 09:13:39 +01:00
Mardegan, Alberto
6322fb5b2b oeqa/core/runner: dump stdout and stderr of each test case
Some CI pipelines might perform further processing of the test output
(for instance, to plot some metrics into a chart). However, Since `thud`
we switched away from the XML-based jUnit reporting, and at the same
time we lost the ability of collecting the stdout and stderr of the
various tests.

We now restore this functionality by adding `stdout` and `stderr` keys
to the JSON reports. This behavior is off by default; in order to enable
it, one must set the `TESTREPORT_FULLLOGS` variable in the bitbake
configuration.

(From OE-Core rev: fd57b34d7c8a120273d65cd361be208fbdaeff50)

Signed-off-by: Alberto Mardegan <amardegan@luxoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-12 09:13:39 +01:00
Alexander Kanavin
8da99b16bc buildhistory: call a dependency parser only on actual dependency lists
Previously it was also called on filelists and possibly other items which
broke the parser.

(From OE-Core rev: cf525cbbc4277ba08415963cfa7dca384ad501dd)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-12 09:13:38 +01:00
Richard Purdie
3c67567ece oeqa/utils/qemurunner: Fix typo in previous commit
(From OE-Core rev: e2b4b0580cd719cf9d48576aa7d0b88e4a286921)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 16:04:09 +01:00
Andreas Müller
b54bff841b patch/insane: Rework patch fuzz handling
Currently there are three issues which can be enhanced:

1. Fuzz warnings cannot be configured as errors for hardening. It happened
   often to me that these warnings were overseen and detected after commits
   were already out.
2. The output is too verbose - particularly when more than one file is
   affected. Meanwhile all users should know why patch fuzz check is performed.
   So move links with background information to insane.bbclass.
3. Reduce copy & paste effort slightly by printing PN (nit: <recipe> was not
   a correct suggestion e.g for native extended recipe - see example below)

To achieve patch.py drops patch-fuzz info encapsulated by a header- and footer-
string into log.do_patch. With this insane.bbclass can drop warnings/errors
depending on 'patch-fuzz' in ERROR_QA or WARN_QA. Default remains unchanged:
Spit out warnings only.

A message for two fuzzed patches and 'pact-fuzz' in ERROR_QA now looks like:

| ERROR: autoconf-native-2.69-r11 do_patch: Fuzz detected:
|
| Applying patch autoreconf-exclude.patch
| patching file bin/autoreconf.in
| Hunk #1 succeeded at 73 with fuzz 1 (offset -3 lines).
| Hunk #2 succeeded at 143 (offset 6 lines).
| Hunk #3 succeeded at 167 (offset 6 lines).
| Hunk #4 succeeded at 177 (offset 6 lines).
| Hunk #5 succeeded at 281 (offset 15 lines).
| Hunk #6 succeeded at 399 (offset 15 lines).
| Hunk #7 succeeded at 571 (offset 20 lines).
| Hunk #8 succeeded at 612 (offset 20 lines).
| Hunk #9 succeeded at 636 (offset 20 lines).
| Hunk #10 succeeded at 656 (offset 20 lines).
| Hunk #11 succeeded at 683 (offset 20 lines).
|
| Applying patch autoreconf-gnuconfigize.patch
| patching file bin/autoreconf.in
| Hunk #1 succeeded at 55 with fuzz 1 (offset -3 lines).
| Hunk #3 succeeded at 663 (offset 18 lines).
|
| The context lines in the patches can be updated with devtool:
|
|     devtool modify autoconf-native
|     devtool finish --force-patch-refresh autoconf-native <layer_path>
|
| Don't forget to review changes done by devtool!
|
| ERROR: autoconf-native-2.69-r11 do_patch: QA Issue: Patch log indicates that patches do not apply cleanly. [patch-fuzz]

(From OE-Core rev: c762c0be43a3854a43cb4b9db559b03126d50706)

Signed-off-by: Andreas Müller <schnitzeltony@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 13:44:39 +01:00
Andre Rosa
0dd6823875 lib/oe/utils: Make prune_suffix prune a suffix
... instead of replacing a substring that could happen more than once and not only when it ends with it. Do the same for the prefix.

See related https://github.com/openembedded/bitbake/pull/24 . There it stops replacing sufixes once first one is matched but not here.

(From OE-Core rev: 610ac84170f8a91cc3321edfc336a9e39f24ebe3)

Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 13:44:39 +01:00
Enrico Scholz via Openembedded-core
1891a852b2 utils.py: added sh_quote() function
This function is a wrapper around "shlex.quote()" and can be used in
"${@...}" context where shlex (or pipes, which provides similar
functionality) is unavailable.

(From OE-Core rev: 127141f5023a7e3fc3963dc7d76cfce9067a9e8a)

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-04 22:57:32 +01:00
Richard Purdie
7f98d0ffaf oeqa/qemurunner: Dont mix binary and non-binary strings
self.msg is a str(), bootlog is b'' so this code clearly doesn't work. Add
in a decode since its being used as a string.

(From OE-Core rev: e7546759d3102b576974bbab52594796f084721e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25 23:19:53 +00:00
Yeoh Ee Peng
a9b2f3561e oeqa/manual/toaster: updated test id naming
All test id (eg. @alias) inside manual testcase file shall follow the same
test id naming convention from oeqa automated tests (eg. selftest,
runtime, sdk, etc), where the test id consists of
<test_module>.<test_suite>.<test_function>. Furthermore, there shall be
only 1 unique test_module per each manual testcases file, where
test_module match the file name itself.

This file was using test_module name that does not match the file name
itself. Fixed test_module name as well as the test_suite name.

(From OE-Core rev: f24178993e7b0830510900e80e879c7f9f8e28f0)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 10:40:35 +00:00
Yeoh Ee Peng
aeb87d0139 oeqa/manual/toaster: transfer manual toaster test cases to oeqa
As part of the solution to replace Testopia, manual test cases
need to be migrated to OEQA. These manual test case json files
will serve two use cases.

Use case#1: as input to the future commandline-based manual test
runner script, where this script will display actions and expected
result information in user friendly text, then it will capture
user input for test result and log, finally it will write test
result and log into existing standardize test result json format
from OEQA framework for automated tests.

Use case#2: QA will open and read these json file manually for
planning manual test execution. Any reader interested in
understanding manual test cases will open and read these files.

(From OE-Core rev: fbeecb059d9c006f50a2a11bed5a42740722e3e8)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-18 22:59:58 +00:00
sangeeta jain
66c4001422 oeqa/manual/kernel-dev.json: test id updated
All test id (eg. @alias) inside manual testcase file shall follow the same test id
naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where
the test id consists of  <test_module>.<test_suite>.<test_function>. Furthermore,
there shall be only 1 unique test_module per each manual testcases file.

This file was using more than 1 unique test_module for testcases. Furthermore,
some of the testcases were defined using different test_suite where it was not needed.
This patch fix the manual testcases file to have only 1 unique test_module
as well as test_suite to simplify test id naming.

(From OE-Core rev: be7cbe86f046bf29082c3da574377f9828aff5f4)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-18 11:12:26 +00:00
sangeeta jain
8b6c8ec14b oeqa/manual/compliance-test.json: test id updated and missing test cases added
Two changes made in oeqa/manual/compliance-test.json:

1. All test id (eg. @alias) inside manual testcase file shall follow the same test id
naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where
the test id consists of  <test_module>.<test_suite>.<test_function>. Furthermore,
there shall be only 1 unique test_module per each manual testcases file.

This file was using more than 1 unique test_module for testcases. Furthermore, some of
the testcases were defined using different test_suite where it was not needed. This patch
fix the manual testcases file to have only 1 unique test_module as well as test_suite to
simplify test id naming.

2. Added 2 missing test cases for Beaglebone Stress Test.

(From OE-Core rev: 5c5402bc6e6505da0203ca9245a3599bd4940dad)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-18 11:12:26 +00:00
sangeeta jain
437dcb901d oeqa/manual/bsp-hw.json: test id updated and obsolete test cases removed
Two changes made in oeqa/manual/bsp-hw.json:

1. All test id (eg. @alias) inside manual testcase file shall follow the same test id naming
convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the
test id consists of  <test_module>.<test_suite>.<test_function>. Furthermore,
there shall be only 1 unique test_module per each manual testcases file.

This file was using more than 1 unique test_module for testcases.
Furthermore, some of the testcases were defined using different test_suite where it was not
needed. This patch fix the manual testcases file to have only 1 unique
test_module as well as test_suite to simplify test id naming.

2. As per review by Intel and Windriver team, 7 testcases were found obsolete. Removed 7 testcases.

(From OE-Core rev: e1136bae81672894a277512d7084d27f4e3416b9)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-18 11:12:26 +00:00
Chen Qi
c8a0da6391 metadata.py: return info after fallback methods
If python3 git module is not installed, we fall back to some git
command to get info. A previous commit (b8d22ed6) accidently deleted
the return statement, causing errors like below.

  Exception: UnboundLocalError: local variable 'InvalidGitRepositoryError' referenced before assignment

(From OE-Core rev: cd8aedc408afa4f8589f79e4d85befc7cac76ac6)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-07 13:00:40 +00:00
Tom Rini
648cd455dc imagefeatures: Add test to blacklist building busybox
Now that we have a packagegroup that can be used to replace the overall
functionality of busybox as base-utils, add a test that we can continue
to build a fairly complex image without busybox being available.  This
also doubles as documentation-by-test of how to do this.

(From OE-Core rev: 6319a59c1d30eeb8ad4871d43641e3469fb543ba)

Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-06 23:12:27 +00:00
Jaewon Lee
206ca252c6 devtool: Support kmeta directory usage with devtool modify/finish
When using Kmeta directories, devtool finish will add every single file
in the directory to the bbappend. This is because in the current
implementation, the get_recipe_local_files function treats the kmeta
directory like a file. Modifying the function to loop through the
provided directories and return all included files instead of just the
top level directory. This will enable correct file to file comparison
when determing which files are new/changed and need to be added to the
bbappend.

Adding an extra check in devtool-source.bbclass to not copy the cfg file
if its already included somewhere in the kmeta directory

Also during 'modify', when moving necessary files in the kmeta directory
from the workdir to oe-local-files, the dangling parent directories are
left behind.  This in itself is not an issue as the temporary devtool
workspace is automatically deleted, but this causes an incorrect include
directory to be added in kernel-yocto.bbclass.  Changing the order of
the if statements to catch the correct conditional. This is safe to do
as when not in the devtool context, there will be no oe-local-files
directory.

(From OE-Core rev: 4671011b5b02d0989aa0fdcb50e18cb385a0e95e)

Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-06 10:39:25 +00:00
André Draszik
9120e52ba1 masterimage: remove code duplication
Two identical classes exist for SystemdbootTarget,
introduced in commit f9a61d3400ad
("gummiboot: Remove/change gummiboot references with systemd-boot")
poky commit 2dce2648e3

Remove the duplicated copy.

(From OE-Core rev: a59562c7ddbda1a266f0ea22ab78fb86de4861a3)

Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-04 22:57:05 +00:00
Alexander Kanavin
d3c7d1a036 selftest: add tests for virgl GL acceleration
Note that the tests require that the host machine has a X display,
has mesa development files installed and is able to create OpenGL contexts.

(From OE-Core rev: 2868e8dfb9e62b49cd06f6c2d010405079d3a71c)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-04 14:27:06 +00:00
Richard Purdie
d9018a3d9c selftest: Add multiconfig test
Add a test for a multiconfig build which mixes tiny and musl builds
along with using the mcextend class to combine and package multiple images
into another image. This gives the multiconfig a decent test in a scenario
users may use.

(From OE-Core rev: 0c7fa15a7350808242754944243f01155bc6784c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-04 14:27:06 +00:00
Naveen Saini
7519b0fc32 parselogs.py: ignore regulatory.db load failure messages for x86*
[YOCTO #13098]

(From OE-Core rev: 9c20f8c6e4bb5157550e347ffd6ef790eb6a1200)

Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-03 15:38:13 +00:00
Richard Purdie
5353c1af71 oeqa/utils/metadata: Add commit_count to fallback logic
Currently if python3-git isn't installed we can get odd behaviours when the
commit_count is absent. Avoid this set of bugs by adding a fallback here.

(From OE-Core rev: b8d22ed681141ce360d742a96cec5f2925a20222)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-28 17:47:10 +00:00
Richard Purdie
4a67dfb089 oeqa/oetest: Drop unused import
(From OE-Core rev: ac4eba5415f39cd797a08071c0efae296ae61a70)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-28 17:47:10 +00:00
Alexander Kanavin
a3c4aaf9b3 lib/oe/package_manager.py: turn nativesdk postinst warnings into notes
The warnings deal with two specific cases of failure:
1) unable to execute nativesdk postinsts for mingw SDKs because
they need to be run under wine
2) unable to execute target postinsts when there is no qemu
usermode support for the target

Neither of these should be a big problem as mingw issue was never
found to problematic, and target postinsts deal with things that
are needed at runtime and not at build time which is the purpose
of SDKs.

The specific reason to do this is to reduce the amount of warnings
shown by the Yocto autobuilder, to zero eventually.

(From OE-Core rev: cbc32fcd9b52e750600cce9dd84b33e3ce612eae)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 10:43:07 +00:00
Richard Purdie
9fdd172fb0 oeqa/utils/gitarchive: Handle case where parent is only on origin
The parent code currently assumed that any parent branch is locally
checked out which may not be the case.

Use the local branch by default but fall back to the origin. This
also means removing the later saftey check as the branch may not exist
locally.

This fixes the autobuilder resulttool test pushing code.

(From OE-Core rev: 0ee636bc89d93282cb3d3cd747b3dd046f6e5a9c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 09:28:28 +00:00
Richard Purdie
47eb3d00e9 resulttool: Improvements to allow integration to the autobuilder
This is a combined patch of the various tweaks and improvements I
made to resulttool:

* Avoid subprocess.run() as its a python 3.6 feature and we
  have autobuilder workers with 3.5.

* Avoid python keywords as variable names

* Simplify dict accesses using .get()

* Rename resultsutils -> resultutils to match the resultstool ->
  resulttool rename

* Formalised the handling of "file_name" to "TESTSERIES" which the code
  will now add into the json configuration data if its not present, based
  on the directory name.

* When we don't have failed test cases, print something saying so
  instead of an empty table

* Tweak the table headers in the report to be more readable (reference
  "Test Series" instead if file_id and ID instead of results_id)

* Improve/simplify the max string length handling

* Merge the counts and percentage data into one table in the report
  since printing two reports of the same data confuses the user

* Removed the confusing header in the regression report

* Show matches, then regressions, then unmatched runs in the regression
  report, also remove chatting unneeded output

* Try harder to "pair" up matching configurations to reduce noise in
  the regressions report

* Abstracted the "mapping" table concept used to pairing in the
  regression code to general code in resultutils

* Created multiple mappings for results analysis, results storage and
  'flattening' results data in a merge

* Simplify the merge command to take a source and a destination,
  letting the destination be a directory or a file, removing the need for
  an output directory parameter

* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression
  mappings

* Have the store command place the testresults files in a layout from
  the mapping, making commits into the git repo for results storage more
  useful for simple comparison purposes

* Set the oe-git-archive tag format appropriately for oeqa results
  storage (and simplify the commit messages closer to their defaults)

* Fix oe-git-archive to use the commit/branch data from the results file

* Cleaned up the command option help to match other changes

* Follow the model of git branch/tag processing used by oe-build-perf-report
  and use that to read the data using git show to avoid branch change

* Add ptest summary to the report command

* Update the tests to match the above changes

(From OE-Core rev: ff2c029b568f70aa9960dde04ddd207829812ea0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 12:34:00 +00:00
Yeoh Ee Peng
1fd5ebdb06 resulttool: enable merge, store, report and regression analysis
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.

These scripts were developed as a test result tools to manage
these testresults.json file.

Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.

Using the "merge" operation, user can merge multiple testresults.json
files to a target file.

Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.

Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.

These resulttool operations expect the testresults.json file to use
the json format below.
{
    "<testresult_1>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
    ...
    "<testresult_n>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
}

To use these scripts, first source oe environment, then run the
entry point script to look for help.
    $ resulttool

To store test result from oeqa automated tests, execute the below
    $ resulttool store <source_dir> <git_branch>

To merge multiple testresults.json files, execute the below
    $ resulttool merge <base_result_file> <target_result_file>

To report test report, execute the below
    $ resulttool report <source_dir>

To perform regression file analysis, execute the below
    $ resulttool regression-file <base_result_file> <target_result_file>

To perform regression dir analysis, execute the below
    $ resulttool regression-dir <base_result_dir> <target_result_dir>

To perform regression git analysis, execute the below
    $ resulttool regression-git <source_dir> <base_branch> <target_branch>

[YOCTO# 13012]
[YOCTO# 12654]

(From OE-Core rev: 78a322d7be402a5b9b5abf26ad35670a8535408a)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 12:34:00 +00:00
Richard Purdie
95bd530b77 oe-build-perf-report/gitarchive: Move common useful functions to library
These functions can be reused by the resulttool code so move to the common
function library for this purpose.

(From OE-Core rev: c66f848938c04e133259c5b6903dc592866ab385)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 12:34:00 +00:00
Mingli Yu
6f0b28f009 logrotate.py: restore /etc/logrotate.d/wtmp
During the test logrotate.LogrotateTest.test_1_logrotate_setup,
there is below logic:
 # mkdir $HOME/logrotate_dir
 # sed -i "s#wtmp {#wtmp {\n    olddir $HOME/logrotate_dir#" /etc/logrotate.d/wtmp

After all logrotate.LogrotateTest finished, only cleanup
$HOME/logrotate_dir as below, but don't restore
the config file /etc/logrotate.d/wtmp.
[snip]
def tearDownClass(cls):
    cls.tc.target.run('rm -rf $HOME/logrotate_dir')
[snip]

That's to say, there is one additional line added
to /etc/logrotate.d/wtmp and will make the logrotate
service start failed when run systemd.SystemdBasicTests.test_systemd_failed

Take an example as below when run test as root:
 # cat /etc/logrotate.d/wtmp
 # no packages own wtmp -- we'll rotate it here
 /var/log/wtmp {
    olddir /root/logrotate_dir
    missingok
    monthly
    create 0664 root utmp
    minsize 1M
    rotate 1
 }

 # ls /root/logrotate_dir
 ls: cannot access '/root/logrotate_dir': No such file or directory

 # systemctl start logrotate
 Job for logrotate.service failed because the control process exited with error code.
 See "systemctl status logrotate.service" and "journalctl -xe" for details.

 # systemctl status logrotate
  logrotate.service - Rotate log files
   Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset>
   Active: failed (Result: exit-code) since Wed 2019-02-13 03:35:19 UTC; 7s ago
     Docs: man:logrotate(8)
           man:logrotate.conf(5)
   Process: 540 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
   Main PID: 540 (code=exited, status=1/FAILURE)

  Feb 13 03:35:18 qemumips systemd[1]: Starting Rotate log files...
  Feb 13 03:35:19 qemumips logrotate[540]: error: wtmp:9 error verifying olddir path /root/logrotate_dir: No such file or directory
  Feb 13 03:35:19 qemumips logrotate[540]: error: found error in file wtmp, skipping
  Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
  Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Failed with result 'exit-code'.
  Feb 13 03:35:19 qemumips systemd[1]: Failed to start Rotate log files.

Add the logic to restore /etc/logrotate.d/wtmp to
make the cleanup complete to fix the above issue.

(From OE-Core rev: a2db9320d97d12d87524ff16a329f9c38a8da33f)

Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-20 11:30:35 +00:00
Richard Purdie
18880a9f1c scripts/oe-git-archive: Separate out functionality to library function
This turns the core of the script into a library function. Ultimately this
will let us call that code with custom 'keywords' rather than relying
on the data parsed from bitbake metadata which can't be used when archiving
historical results.

(From OE-Core rev: 4820ca2b0850e29b04a4fd5659a6e9837d6714d0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-18 10:33:13 +00:00
Richard Purdie
f3e56f1b57 lib/oe/utils: Fix hang in multiprocess_launch()
If large results values are returned by the subprocesses, we can hit a deadlock
where the subprocess is trying to write data back to the parent, the pipe is full
and the parent is waiting for the child to exit.

Avoid this by calling the update() method which would trigger reading a result
from the child, avoiding the deadlock. The issue is described in
https://bugs.python.org/issue8426

(From OE-Core rev: 0035e8066ecbbff94d6a1994a9f72b1368d660d2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-17 10:19:59 +00:00
Mazliana
d76ab279e6 oeqa/manual/bsp-hw.json: add non-IA tests
QA team found that 10 manual bsp test cases from Testopia for
Beaglebone, EdgeRouter, and MPC need to be up streamed

[YOCTO #12650]

(From OE-Core rev: f47410899eb97cfd7db428f45b7ba354c609b4d5)

Signed-off-by: Mazliana <mazliana.mohamad@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-16 08:19:34 +00:00
Richard Purdie
e1e38c9bd1 lib/oe,oeqa/selftest: Fix DeprecationWarning: invalid escape sequence
Fix another load of regex escape sequence warnings for newer
python versions.

(From OE-Core rev: bd2c125bb9c362b6122e99dfdf4e1cfe12c26a90)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-16 08:19:34 +00:00
Ross Burton
cdd0683e5e utils: add -src to system package name blacklist
oe.utils.packages_filter_out_system() returns PACKAGES after removing "system"
packages but it doesn't handle ${PN}-src as generated by
PACKAGE_DEBUG_SPLIT_STYLE=debug-with-srcpkg.

(From OE-Core rev: 162632d3d1e40c83ed9c5d49a026edf3912860a0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-15 08:17:49 +00:00
Alejandro del Castillo
d8e12f2bfc OpkgPM: use --add-ignore-recommends to process BAD_RECOMMENDATIONS
Currently, BAD_RECOMMENDATIONS on the opkg backed relies on editing the
opkg status file (it sets BAD_RECOMMENDATIONS pkg want state to
deinstalled and pinned). This is brittle, and not consistent across the
different solver backends. Use new --add-ignore-recommends flag instead.

(From OE-Core rev: 0d11e813ba9b4e8de9e6e5099ff85f5d914243bc)

Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-08 10:57:19 +00:00
Andrej Valek
391d4ee036 lib/oe/rootfs: prelink only when image-prelink is inherited
Use prelink only when image-prelink class is inherited. Inheriting this
class adds prelink-native as dependency to do_rootfs, which makes prelink
binary and configuration available.

|Exception: FileNotFoundError: [Errno 2] No such file or directory:
|'/home/.../1.0-r0/recipe-sysroot-native/etc/prelink.conf'
|
| if not os.path.exists(prelink_cfg):
|     shutil.copy(self.d.expand('${STAGING_DIR_NATIVE}${sysconfdir_native}/prelink.conf'),prelink_cfg)

(From OE-Core rev: 9e30e72920b3c661279499f2e94d5dc0087a1a8e)

Signed-off-by: Andrej Valek <andrej.valek@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 08:29:06 +00:00
Angus Lees
dbb832b8cf package_manager.py: Use data.tar.xz for ipkg too
Commit b95b6ba1a2959e2294a8848fa35f20163388eb06 changed package_ipk.bbclass
to xz when building packages.  This updates OpkgDpkgPM.extract() accordingly.

(From OE-Core rev: c09a22c421a57701f6b943eb50b9bae1545e5b39)

Signed-off-by: Angus Lees <gus@inodes.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 08:29:06 +00:00
Jacob Kroon
404f7cec87 terminal: Fix working directory in tmux terminals
Recent version of tmux seems to use PWD from the environment instead of the
actual working directory. Help it use the correct path.

Fixes setting the correct working directory when running
devshell and menuconfig tasks.

(From OE-Core rev: 938a089b52e7bf5f67406f55234ab2d22236e9b6)

Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-05 13:53:47 +00:00
Alexander Kanavin
84e9159827 lib/oeqa/runtime/cases/gi.py: fix deprecation warning
With newest Python, the following is printed:

PyGIDeprecationWarning: GObject.markup_escape_text is deprecated; use GLib.markup_escape_text instead)

(From OE-Core rev: 7b4302d84e6c9833089bd575c1083fe826082387)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-02 11:06:00 +00:00
Richard Purdie
927320844f oeqa/runtime/ptest: Ensure OOM errors are logged
Currently processed being killed by the OOM killer may not be spotted by
ptest-runner. After we complete the tests, check the logs and report if there
were any. This ensures the user is aware of OOM conditions affecting the
ptest results.

(From OE-Core rev: 20a441d53817f80e0ce1597e77f6e794422ac49a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-02 11:05:59 +00:00
Armin Kuster
1a4d265ac0 perl: add testdepends for ssh
fixes:

 DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.4 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; perl -e '$_="Uryyb, jbeyq"; tr/a-zA-Z/n-za-mN-ZA-M/;print'
| DEBUG: time: 1548816904.4024463, endtime: 1548817204.397057
| DEBUG: Partial data from SSH call: ssh: connect to host 192.168.7.4 port 22: Connection refused

for master/thud/sumo

(From OE-Core rev: a590e7805e3bec5dd995f7ea0b9e79a21f82b48b)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 23:23:27 +00:00
Chen Qi
b68a42596d pkgdata.py: avoid target-sdk-dummy-provides to mess things up
Sometimes we meet the following failure for the test_lookup_recipe
test case.

  AssertionError: 'zlib\nbusybox is in the RPROVIDES of target-sdk-provid[32 chars]ummy' != 'zlib\nbusybox'
    zlib
  + busybox- busybox is in the RPROVIDES of target-sdk-provides-dummy:
  - target-sdk-provides-dummy

This is because target-sdk-provides-dummy rprovides busybox.

So clean things up to avoid failure.

(From OE-Core rev: b3001770df6640549270361bfaa449cb3e79a0b7)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 23:23:26 +00:00
Richard Purdie
7b17274c30 oeqa/logparser: Various misc cleanups
Get rid of further unneeded code complications:

* value mappings we could just direct use
* ftools when we can write files easily ourself
* test result status filtering we don't use
* variable overwriting module imports

(From OE-Core rev: d6065f136f6d353c3054cc3f440a4e259509f876)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 23:23:26 +00:00
Richard Purdie
54d4694f32 oeqa/logparser: Improve results handling
Merge the results handling into the ptest log parser as a seperate
method.

Drop the weird "pass.skip.fail." prefix to the results filename, its
just bizarre.

Drop the code turning a list into a regex then searching the regex for
an item, "x in y" is perfectly capable.

Use a dict, sort the keys as needed and drop the list sorting code.

(From OE-Core rev: f317800e950b4a37b4034133bc52e0c47f04dc29)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 23:23:26 +00:00
Richard Purdie
3731033435 oeqa/utils/logparser: Add in support for duration, exitcode and logs by section
Allow parsing of the ptest duration, exit code and timeout keywords
from the logs, returning data on each section.

Also include the logs broken out per section.

(From OE-Core rev: a9a67dccaa5be0f06eedcab46dcff7cbf9202850)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 23:23:26 +00:00
Richard Purdie
88f390bcb6 oeqa/logparser: Reform the ptest results parser
Now we have a dedicated ptest parser, merge in the remaining ptest
specific pieces to further clarify and simplify the code, moving to
a point where we can consider extending/enhancing it.

(From OE-Core rev: 05991bb5bc8018275d03fdeecee3d5a757840c7c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 23:23:26 +00:00
Richard Purdie
4ee85b1cd6 oeqa/logparser: Further simplification/clarification
Rename the paster to be ptest specific and apply some further cleanups
to the code to simplify and clarify what its doing.

(From OE-Core rev: 45a5886f1ec458d4c306b8d68fd31d568bc36b47)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 23:23:26 +00:00
Richard Purdie
95427c47ab oeqa/utils/logparser: Simplify ptest log parsing code
logparser is only used by ptest. Its slightly overcomplicated as it was
intended to be reusable but wasn't. Simplify it as a dedicated parser is
likely to me more readable and maintainable.

(From OE-Core rev: c7478345b2b4a85cb1fec40e762633871f0e94cb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 23:23:26 +00:00
Richard Purdie
fa2db6a72a oeqa/runtime/ptest: Avoid traceback for tests with no section
Some tests end up without a section, avoid tracebacks trying to use
None as a string in that case.

(From OE-Core rev: 86fb5d898a29761f120c2eaa538a32cf2e078487)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 23:23:26 +00:00
sangeeta jain
b38554767b oeqa/manual/sdk.json: Updated toolchain tarball to core-image-sato
Manual test step for SDK is updated. Previously toolchain was "poky-glibc-x86_64-core-image-sato-sdk<type-arch>-toolchain-<release-version>.sh"
But toochain for core-image-sato-sdk is not available in releases after 2.1, hence changed it to "poky-glibc-x86_64-core-image-sato-<type-arch>-toolchain-<release-version>.sh

Other less significant update is to add exepcted result in intermediate test steps.

(From OE-Core rev: f5083ba27e667789c32dcd00bc1b883f4b76bd4f)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>

"poky-glibc-x86_64-core-image-sato-<type-arch>-toolchain-<release-version>.sh"

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-28 23:14:10 +00:00
Sudhir Sharma
8705139c73 oeqa/manual: Add manual runtime 'compliance' testcases to json
Added json file for compliance test cases to the manual runtime tests

(From OE-Core rev: 946f37d57f7b23c7875089ed98be9148ccc05890)

Signed-off-by: Sudhir Sharma <sudhirx.sharma@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-28 23:14:10 +00:00