35043 Commits

Author SHA1 Message Date
Richard Purdie
5ddf7fff99 devtool: Split tests into multiple classes
This allows better parallelism between the different tests as currently
this block takes the longest time to execute. devtool tests are still
all grouped into the "devtool" module for ease of exection.

This also makes it easier to execute some subset of devtool tests for
testing devtool changes.

(From OE-Core rev: 84f19e78d9b1f3d634cf1d46ce48f24670199d0b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Richard Purdie
9a8d6d9fd2 oeqa: Default to buffer mode for tests
Currently some tests run in buffer mode and some don't. Those that don't can
corrupt stdout/stderr. Switch to using buffer mode everywhere so we're consistent.

If there is useful output on stdout/stderr, it will be displayed if the test
fails.

(From OE-Core rev: 85c1b6fb516aae58240330a0aca659bfafcd3883)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Richard Purdie
65681147b2 oeqa/selftest/devtool: Ensure dbus is built befoe running test
If dbus isn't build first the dbus dependency of dbus-wait can't be detected
through pkgconfig and the test fails:

AssertionError: {'DEPENDS': {'dbus'}} != {}
- {'DEPENDS': {'dbus'}}
+ {} : Some expected variables not found in recipe: {'DEPENDS': {'dbus'}}

Ensure dbus is built and present in the sysroot.

(From OE-Core rev: af7ba26a603a12de0aed35e786674c92049c2bee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Chen Qi
8c7ca97dd6 logging: use warning instead warn
The warn method is deprecated. We should use the documented warning instead.

Quoting from the python's official doc:
"""
Note: There is an obsolete method warn which is functionally identical to warning.
      As warn is deprecated, please do not use it - use warning instead.
"""

(From OE-Core rev: cc771aa4b74f222f1bea38b0b50196b2fbc97ab4)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Khem Raj
8dcc23b5cc lib/oe: Fix collections ABCs DeprecationWarning in Python 3.7+
- Prefer collections.abc (new in Python 3.3) over collections for abstract base classes

- In Python 3.8, the abstract base classes in collections.abc will no longer be exposed in
  the regular collections module. This will help create a clearer distinction between
  the concrete classes and the abstract base classes."

- https://docs.python.org/3.7/whatsnew/3.7.html#deprecated

- see c66f9f8d39

(From OE-Core rev: b254ab6ce34da3d3241a51958b5770664d317fcc)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Chen Qi
56ddd6b3f4 pkgdata.py: avoid target-sdk-dummy-provides to mess things up
Sometimes we meet the following failure for the test_lookup_recipe
test case.

  AssertionError: 'zlib\nbusybox is in the RPROVIDES of target-sdk-provid[32 chars]ummy' != 'zlib\nbusybox'
    zlib
  + busybox- busybox is in the RPROVIDES of target-sdk-provides-dummy:
  - target-sdk-provides-dummy

This is because target-sdk-provides-dummy rprovides busybox.

So clean things up to avoid failure.

(From OE-Core rev: 5ae49260180adc0070287db01c01a0820ce99ad3)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Yeoh Ee Peng
4498fea205 oeqa/manual/toaster: updated test id naming
All test id (eg. @alias) inside manual testcase file shall follow the same
test id naming convention from oeqa automated tests (eg. selftest,
runtime, sdk, etc), where the test id consists of
<test_module>.<test_suite>.<test_function>. Furthermore, there shall be
only 1 unique test_module per each manual testcases file, where
test_module match the file name itself.

This file was using test_module name that does not match the file name
itself. Fixed test_module name as well as the test_suite name.

(From OE-Core rev: 05e08a64771e40c147bdd6f02492b85edc01e946)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Yeoh Ee Peng
442b9f2dfa oeqa/manual/toaster: transfer manual toaster test cases to oeqa
As part of the solution to replace Testopia, manual test cases
need to be migrated to OEQA. These manual test case json files
will serve two use cases.

Use case#1: as input to the future commandline-based manual test
runner script, where this script will display actions and expected
result information in user friendly text, then it will capture
user input for test result and log, finally it will write test
result and log into existing standardize test result json format
from OEQA framework for automated tests.

Use case#2: QA will open and read these json file manually for
planning manual test execution. Any reader interested in
understanding manual test cases will open and read these files.

(From OE-Core rev: 73f5446c38a5d7766c81cda7a7bb22df9cfe0f23)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
sangeeta jain
ad2fe82652 oeqa/manual/kernel-dev.json: test id updated
All test id (eg. @alias) inside manual testcase file shall follow the same test id
naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where
the test id consists of  <test_module>.<test_suite>.<test_function>. Furthermore,
there shall be only 1 unique test_module per each manual testcases file.

This file was using more than 1 unique test_module for testcases. Furthermore,
some of the testcases were defined using different test_suite where it was not needed.
This patch fix the manual testcases file to have only 1 unique test_module
as well as test_suite to simplify test id naming.

(From OE-Core rev: 35233e411db1a5978bfb2f4265b8e3a8fbdc4246)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
sangeeta jain
fcde665168 oeqa/manual/compliance-test.json: test id updated and missing test cases added
Two changes made in oeqa/manual/compliance-test.json:

1. All test id (eg. @alias) inside manual testcase file shall follow the same test id
naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where
the test id consists of  <test_module>.<test_suite>.<test_function>. Furthermore,
there shall be only 1 unique test_module per each manual testcases file.

This file was using more than 1 unique test_module for testcases. Furthermore, some of
the testcases were defined using different test_suite where it was not needed. This patch
fix the manual testcases file to have only 1 unique test_module as well as test_suite to
simplify test id naming.

2. Added 2 missing test cases for Beaglebone Stress Test.

(From OE-Core rev: 7d95d4c6d30ae2977cd9feb5a45f96d05ff9b763)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
sangeeta jain
baceab8697 oeqa/manual/bsp-hw.json: test id updated and obsolete test cases removed
Two changes made in oeqa/manual/bsp-hw.json:

1. All test id (eg. @alias) inside manual testcase file shall follow the same test id naming
convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the
test id consists of  <test_module>.<test_suite>.<test_function>. Furthermore,
there shall be only 1 unique test_module per each manual testcases file.

This file was using more than 1 unique test_module for testcases.
Furthermore, some of the testcases were defined using different test_suite where it was not
needed. This patch fix the manual testcases file to have only 1 unique
test_module as well as test_suite to simplify test id naming.

2. As per review by Intel and Windriver team, 7 testcases were found obsolete. Removed 7 testcases.

(From OE-Core rev: 862c96a98d2087cf05fbfeeafb57dcef1225d978)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Mazliana
f075c32754 oeqa/manual/bsp-hw.json: add non-IA tests
QA team found that 10 manual bsp test cases from Testopia for
Beaglebone, EdgeRouter, and MPC need to be up streamed

[YOCTO #12650]

(From OE-Core rev: 05f785937be3d06f38cff4de238d7bd85d4a452a)

Signed-off-by: Mazliana <mazliana.mohamad@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
sangeeta jain
ad60684664 oeqa/manual/sdk.json: Updated toolchain tarball to core-image-sato
Manual test step for SDK is updated. Previously toolchain was "poky-glibc-x86_64-core-image-sato-sdk<type-arch>-toolchain-<release-version>.sh"
But toochain for core-image-sato-sdk is not available in releases after 2.1, hence changed it to "poky-glibc-x86_64-core-image-sato-<type-arch>-toolchain-<release-version>.sh

Other less significant update is to add exepcted result in intermediate test steps.

(From OE-Core rev: 4e26c3a6aa647a322a8e8a3027a4114caed38f7d)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>

"poky-glibc-x86_64-core-image-sato-<type-arch>-toolchain-<release-version>.sh"

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:21 +01:00
Sudhir Sharma
a9b4c75d2f oeqa/manual: Add manual runtime 'compliance' testcases to json
Added json file for compliance test cases to the manual runtime tests

(From OE-Core rev: 3f143119def5a84430762790f69b426fdc8ad281)

Signed-off-by: Sudhir Sharma <sudhirx.sharma@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:18 +01:00
Robert Yang
febbb0f2c9 oeqa/manual/bsp-qemu.json: Update for QEMU_USE_KVM
Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more.

(From OE-Core rev: cd3c467b9419238494e8c525e86be0cf85321c4e)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:18 +01:00
Richard Purdie
004b3a6193 oeqa/manual/build-appliance: Drop test which is already automated
The nightly-build-appliance target already tests this particular build.

(From OE-Core rev: 90d9542900b6a941a3c3355c15dbfc2e2ac78232)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:18 +01:00
Richard Purdie
848fe75d30 oeqa/manual: Remove bitbake-selftest execution
The autobuilder runs bitbake-selftest already so we don't need
to execute this manually.

(From OE-Core rev: 772b2b883249c8606b2df380f749320cb683a7b2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:18 +01:00
Yeoh Ee Peng
ed668ec695 oeqa/manual: transfer manual test cases from testopia to oeqa
As part of the solution to replace Testopia, manual test cases
need to be migrated to OEQA. These manual test case json files
will serve two use cases.

Use case#1: as input to the future commandline-based manual test
runner script, where this script will display actions and expected
result information in user friendly text, then it will capture
user input for test result and log, finally it will write test
result and log into existing standardize test result json format
from OEQA framework for automated tests.

Use case#2: QA will open and read these json file manually for
planning manual test execution. Any reader interested in
understanding manual test cases will open and read these files.

(From OE-Core rev: a1675c33638288363b243e7412fe4cea63756ae6)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 15:01:18 +01:00
Richard Purdie
84b78df15f build-appliance-image: Update to sumo head revision
(From OE-Core rev: 0a2db923fd17019d07d88204b355aa46590f0b97)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-27 23:22:02 +00:00
Richard Purdie
c10a028e27 oeqa/utils/gitarchive: Handle case where parent is only on origin
The parent code currently assumed that any parent branch is locally
checked out which may not be the case.

Use the local branch by default but fall back to the origin. This
also means removing the later saftey check as the branch may not exist
locally.

This fixes the autobuilder resulttool test pushing code.

(From OE-Core rev: 633e024f5d492ba38debf745b59813f6a2f4a3f0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-27 23:21:47 +00:00
Richard Purdie
a01fba7973 build-appliance-image: Update to sumo head revision
(From OE-Core rev: ece8242187558011940d7c6762b64c7116e38689)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:54:21 +00:00
Richard Purdie
6690c4962c yocto-uninative: Update to 2.4
This supports glibc 2.29 which is appearing in distros like Ubuntu 19.04

(From OE-Core rev: 661ac7ed164965690a4334d372bf22f04c3bfd3a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[Fixed up for Sumo context; no aarch64]
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:28 +00:00
Richard Purdie
ee12af6556 oe-build-perf-report/gitarchive: Move common useful functions to library
These functions can be reused by the resulttool code so move to the common
function library for this purpose.

(From OE-Core rev: c66f848938c04e133259c5b6903dc592866ab385)

(From OE-Core rev: 94a3ca85d30fc957f8f01a216a75342be49f9143)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:28 +00:00
Richard Purdie
622e5728da scripts/oe-git-archive: Separate out functionality to library function
This turns the core of the script into a library function. Ultimately this
will let us call that code with custom 'keywords' rather than relying
on the data parsed from bitbake metadata which can't be used when archiving
historical results.

(From OE-Core rev: 4820ca2b0850e29b04a4fd5659a6e9837d6714d0)

(From OE-Core rev: ab5d49ba8bb02410fb9f3024f98ff19a04acb3ca)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:28 +00:00
Richard Purdie
c7eb843d7c resulttool: Improvements to allow integration to the autobuilder
This is a combined patch of the various tweaks and improvements I
made to resulttool:

* Avoid subprocess.run() as its a python 3.6 feature and we
  have autobuilder workers with 3.5.

* Avoid python keywords as variable names

* Simplify dict accesses using .get()

* Rename resultsutils -> resultutils to match the resultstool ->
  resulttool rename

* Formalised the handling of "file_name" to "TESTSERIES" which the code
  will now add into the json configuration data if its not present, based
  on the directory name.

* When we don't have failed test cases, print something saying so
  instead of an empty table

* Tweak the table headers in the report to be more readable (reference
  "Test Series" instead if file_id and ID instead of results_id)

* Improve/simplify the max string length handling

* Merge the counts and percentage data into one table in the report
  since printing two reports of the same data confuses the user

* Removed the confusing header in the regression report

* Show matches, then regressions, then unmatched runs in the regression
  report, also remove chatting unneeded output

* Try harder to "pair" up matching configurations to reduce noise in
  the regressions report

* Abstracted the "mapping" table concept used to pairing in the
  regression code to general code in resultutils

* Created multiple mappings for results analysis, results storage and
  'flattening' results data in a merge

* Simplify the merge command to take a source and a destination,
  letting the destination be a directory or a file, removing the need for
  an output directory parameter

* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression
  mappings

* Have the store command place the testresults files in a layout from
  the mapping, making commits into the git repo for results storage more
  useful for simple comparison purposes

* Set the oe-git-archive tag format appropriately for oeqa results
  storage (and simplify the commit messages closer to their defaults)

* Fix oe-git-archive to use the commit/branch data from the results file

* Cleaned up the command option help to match other changes

* Follow the model of git branch/tag processing used by oe-build-perf-report
  and use that to read the data using git show to avoid branch change

* Add ptest summary to the report command

* Update the tests to match the above changes

(From OE-Core rev: e4195565d2a50046d4378c97f7a593c41bed51bd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Yeoh Ee Peng
d038d97d48 resulttool: enable merge, store, report and regression analysis
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.

These scripts were developed as a test result tools to manage
these testresults.json file.

Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.

Using the "merge" operation, user can merge multiple testresults.json
files to a target file.

Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.

Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.

These resulttool operations expect the testresults.json file to use
the json format below.
{
    "<testresult_1>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
    ...
    "<testresult_n>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
}

To use these scripts, first source oe environment, then run the
entry point script to look for help.
    $ resulttool

To store test result from oeqa automated tests, execute the below
    $ resulttool store <source_dir> <git_branch>

To merge multiple testresults.json files, execute the below
    $ resulttool merge <base_result_file> <target_result_file>

To report test report, execute the below
    $ resulttool report <source_dir>

To perform regression file analysis, execute the below
    $ resulttool regression-file <base_result_file> <target_result_file>

To perform regression dir analysis, execute the below
    $ resulttool regression-dir <base_result_dir> <target_result_dir>

To perform regression git analysis, execute the below
    $ resulttool regression-git <source_dir> <base_branch> <target_branch>

[YOCTO# 13012]
[YOCTO# 12654]

(From OE-Core rev: bb0bc6368bb51ac0be77d13fe931601d493951ea)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Jonathan Rajotte-Julien
cefa08aeef lttng-tools: update to 2.9.11
(From OE-Core rev: 4af5a794497746629a35726ce515556b53d40da9)

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25 23:12:03 +00:00
Jonathan Rajotte-Julien
3fd4794b16 lttng-modules: update to 2.10.9
Pertinent fix for OE-Core since 2.10.6:
    Fix: out of memory error handling
    Fix: access migrate_disable field directly
    Prevent allocation of buffers if exceeding available memory

2.10.9 also contains the necessary fix to support kernel up to 5.0.

(From OE-Core rev: 9666e9f6058bfd8294cb4e6b03752f28159a76bc)

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25 23:12:03 +00:00
Jonathan Rajotte-Julien
26745f6b22 lttng-ust: update to 2.10.3
(From OE-Core rev: 9d8b3988a53449fa18398e9eaae1627923eeef97)

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25 23:12:03 +00:00
Stefan Müller-Klieser
61f17352ec coreutils: 8.29: fix native build
This fixes the native build on glibc 2.28 hosts, by backporting two
gnulib patches from master.

(From OE-Core rev: fe7c33cd470d7466be48391b11ea703746812014)

Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25 23:12:03 +00:00
Martin Jansa
2333147fa6 busybox: backport fix for issues introduced by CVE-2011-5325.patch
(From OE-Core rev: d0555e89514f2641387ef061f9ffcd1c8ced008c)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25 23:12:03 +00:00
Mingli Yu
e0fdb98b0f logrotate.py: restore /etc/logrotate.d/wtmp
During the test logrotate.LogrotateTest.test_1_logrotate_setup,
there is below logic:
 # mkdir $HOME/logrotate_dir
 # sed -i "s#wtmp {#wtmp {\n    olddir $HOME/logrotate_dir#" /etc/logrotate.d/wtmp

After all logrotate.LogrotateTest finished, only cleanup
$HOME/logrotate_dir as below, but don't restore
the config file /etc/logrotate.d/wtmp.
[snip]
def tearDownClass(cls):
    cls.tc.target.run('rm -rf $HOME/logrotate_dir')
[snip]

That's to say, there is one additional line added
to /etc/logrotate.d/wtmp and will make the logrotate
service start failed when run systemd.SystemdBasicTests.test_systemd_failed

Take an example as below when run test as root:
 # cat /etc/logrotate.d/wtmp
 # no packages own wtmp -- we'll rotate it here
 /var/log/wtmp {
    olddir /root/logrotate_dir
    missingok
    monthly
    create 0664 root utmp
    minsize 1M
    rotate 1
 }

 # ls /root/logrotate_dir
 ls: cannot access '/root/logrotate_dir': No such file or directory

 # systemctl start logrotate
 Job for logrotate.service failed because the control process exited with error code.
 See "systemctl status logrotate.service" and "journalctl -xe" for details.

 # systemctl status logrotate
  logrotate.service - Rotate log files
   Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset>
   Active: failed (Result: exit-code) since Wed 2019-02-13 03:35:19 UTC; 7s ago
     Docs: man:logrotate(8)
           man:logrotate.conf(5)
   Process: 540 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
   Main PID: 540 (code=exited, status=1/FAILURE)

  Feb 13 03:35:18 qemumips systemd[1]: Starting Rotate log files...
  Feb 13 03:35:19 qemumips logrotate[540]: error: wtmp:9 error verifying olddir path /root/logrotate_dir: No such file or directory
  Feb 13 03:35:19 qemumips logrotate[540]: error: found error in file wtmp, skipping
  Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
  Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Failed with result 'exit-code'.
  Feb 13 03:35:19 qemumips systemd[1]: Failed to start Rotate log files.

Add the logic to restore /etc/logrotate.d/wtmp to
make the cleanup complete to fix the above issue.

(From OE-Core rev: a2db9320d97d12d87524ff16a329f9c38a8da33f)

(From OE-Core rev: a5dcf0c4bd9a0be2f2484b0b1b3e77f1f2128dfc)

Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
George McCollister
0278e515fd systemd: fix CVE-2019-6454
Apply patches from systemd_237-3ubuntu10.13 to fix CVE-2019-6454.
CVE-2019-6454 is an issue in which systemd (PID1) can be crashed
with a specially formed D-Bus message.

For information see:
https://usn.ubuntu.com/3891-1/
https://git.launchpad.net/ubuntu/+source/systemd/commit/?h=applied/ubuntu/bionic-updates&id=d7584b894afcaa8a4a1abb69db2a9c81a6276e80

(From OE-Core rev: 342157b135e7493e5965b706ede93bee190fbe32)

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
George McCollister
4c55db6d5c systemd: fix CVE-2018-6954
Apply patches to fix CVE-2018-6954

NVD description from https://nvd.nist.gov/vuln/detail/CVE-2018-6954

systemd-tmpfiles in systemd through 237 mishandles symlinks present in
non-terminal path components, which allows local users to obtain
ownership of arbitrary files via vectors involving creation of a
directory and a file under that directory, and later replacing that
directory with a symlink. This occurs even if the fs.protected_symlinks
sysctl is turned on.

Patches from systemd_237-3ubuntu10.13.debian.

These patches shouldn't be required on newer OE releases since they use
systemd v239 or higher.

(From OE-Core rev: 607350d98aa4c65b71fe1f10900e205fad81d1ec)

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
Marcus Cooper
d4e0f92528 systemd: Security fix CVE-2018-16866
Affects < v240

(From OE-Core rev: 10fa35a75617e82650b12d3e353a554f05f036dd)

Signed-off-by: Marcus Cooper <marcusc@axis.com>

>From v2 patch on openembedded-core@lists.openembedded.org
Incresed file name number from 0026 to 0027.

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
George McCollister
7be61780af systemd: Security fix CVE-2018-16865
Affects < v240

Based on thud commit d5d2b821fc85b8cf39f683061ac2a45bddd2139f
The second patch in the thud commit doesn't apply against 237. Use the
version of the second patch CVE-2018-16865_2.patch from
systemd_237-3ubuntu10.13.debian.

(From OE-Core rev: da41e48567eb21a47426a6fbe23ea07ce780cd3c)

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
George McCollister
c3890467ff systemd: Security fix CVE-2018-16864
Affects < v240

Based on thud commit 403e74b07b6f3c4a2444e68c74a8434fb17aee49
The patch in the thud commit doesn't compile against 237. Use the
version of this patch, CVE-2018-16864.patch from
systemd_237-3ubuntu10.13.debian.

(From OE-Core rev: ddbe969d0c9052a3ae17ef8f1cec8da847c722d3)

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
Chen Qi
b3ed759360 systemd: fix CVE-2018-15688
Backport patch to fix the following CVE.

CVE: CVE-2018-15688

(From OE-Core rev: d490839e881f3ff30a4bde8137cb04cb0fd37acd)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Cherry-picked from thud 13591d7224393dc0ae529a03cdf74aceb3540ce9

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
George McCollister
0e63d91b2b systemd: fix CVE-2018-15687
Backport patch to fix the following CVE.

CVE: CVE-2018-15687

Based on thud commit eeb621aa19f690971caf862290a172a115578ba1
The patch in the thud commit doesn't compile against 237. Use the
version of this patch,  CVE-2018-15687.patch from
systemd_237-3ubuntu10.13.debian.

(From OE-Core rev: 3e8ba9af58253ed9db0f0376a8e2966e45ee089e)

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
Chen Qi
d5400f11dd systemd: fix CVE-2018-15686
Backport patch to fix the following CVE.

CVE: CVE-2018-15686

(From OE-Core rev: 06bf145cee24b677ab076498fe8399126971bc43)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Cherry-picked from thud 0ef70603bc983315eb0e8a97958d995a31198c35

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
ROGEZ Matthieu
0b554def18 systemd: Fix typo in root home variable.
This regression has been introduced while upgrading to version 237
(commit 906230a73b3ccfa4afd2a19a6b0aa18cd1d5fa08)
and seems to only affect sumo version.

(From OE-Core rev: 665b41d326654235d305649be4be69a1be8dc00b)

Signed-off-by: Matthieu Rogez <matthieu.rogez@fivesgroup.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
Richard Purdie
0a8f50fd31 oeqa/runtime/dnf: Fix test error when static libs are enabled
The test works by excluding curl-dev which curl-staticdev depends upon.
When static libraries aren't disabled, this leads to an odd looking test
failure.

Simply exclude curl-staticdev as well in case its enabled to make sure
the test always works.

(From OE-Core rev: dfded083d9456c8d2bb168dbe8e08b2ffb3e1a26)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
Stefan Agner
b4daa5ba96 run-postinsts: for dpkg/opkg, do not rely on /etc/*-postinsts
Start opkg/dpkg as soon as the respective package managers status
file is present, no matter whether /etc/$pm-postinsts exists. This
decouples the implicit link between postinsts scripts in /etc and
the package manager: Currently the package manager is only started
if those scripts are present, although the package manager does not
use those scripts at all! Package managers install their own set of
postinst scripts.

The behavior when using rpm packages stays the same.

Note that using the package managers capability to execute postinst
scripts is preferred for good reasons: It makes sure that the
package managers database reflects that the packages have been
completely installed and configured.

This change allows to drop installation of the postinsts scripts
when package management is present. This will be done in a separate
change.

Note: Before commit 5aae19959a44 ("rootfs.py: Change logic to
unistall packages") rootfs.py did not install /etc/$pm-postinsts
when package management is installed! The change caused YOCTO #8235
which lead to the behavior change of run-postinsts in first place.

(From OE-Core rev: 85e498a4671426999610d90c87c354d41cfe8443)

Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
Richard Purdie
d3dcc4f92e package_rpm/archiver: Apply bandaid to src.rpm creation
| error: create archive failed on file /media/build1/poky/build/tmp/work/all-poky-linux/xcursor-transparent-theme/0.1.1+gitAUTOINC+23c8af5ba4-r0/deploy-sources/allarch-poky-linux/xcursor-transparent-theme-0.1.1+gitAUTOINC+23c8af5ba4-r0/xcursor-transparent-theme-0.1.1+git0+23c8af5ba4-r0.src.rpm: cpio: read failed - No such file or directory
| Building target platforms: noarch-poky-linux
| Building for target noarch-poky-linux

This is caused by:

$ cat log.task_order

do_cleansstate (24289): log.do_cleansstate.24289
do_deploy_archives_setscene (24395): log.do_deploy_archives_setscene.24395
do_fetch (24407): log.do_fetch.24407
[..]
do_package_write_rpm (25448): log.do_package_write_rpm.25448
do_package_qa (25451): log.do_package_qa.25451

So do_deploy_archives can run from sstate, created a .src.rpm in WORKDIR/deploy-sources,
then it was removed when rpm was running. This leads to a broken Source line in the
spec file as the original file was found by the os.listdir().

This fix is just a bandaid over much more fundamental problems sadly.

(From OE-Core rev: a10020ace4c3cd863c782760f7cbecea557ec6e7)

(From OE-Core rev: 6d56e912fbbaa22830b4da5ab230586a3d15b23e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
31a84b4316 archiver/package_rpm: Fix the worst src.rpm generation race
The package_rpm code is writing outside the task's sstate directory into
the sstate of do_deploy_archives. This is "out of spec" since if the
task is installed from sstate, the files are not restored. This means
the files may appear/disappear, things are not deterministic and there are
races.

Extend the do_package_write_rpm code to handle writing the src.rpm into
place to avoid these issues. There are other problems but this avoids races
around this file.

(From OE-Core rev: c6e151ba7fe0f14044537cf0ab2cac436f1496e3)

(From OE-Core rev: b119872fb794a36b6eb9ef5e9c42a9c6c991e835)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
6ab2cac421 libtool-cross: Handle ccache sstate 'infection' issues
On a system without ccache, f you:

INHERIT += "ccache"
bitbake libtool-cross
<remove INHERIT>
bitbake apmd

then it fails due to being unable to find ccache. The references to ccache are
coded into libtool-cross but the sstate checksum doesn't reflect this due to the
way the class is coded (output should be the same regardless).

The simplest solution is to remove references to ccache from the libtool script.
The output then works regardless of whether ccache is present or not. The
libtool-cross script is only used in a handful of cases (most of the time its
dynamically generated by autoconf) so any performance issue is minor.

(From OE-Core rev: ed550a49d2114c56e5bc033ecd0e83073d2d4067)

(From OE-Core rev: ee6a2e0ccb11e5f5267bc2e406203c78b0443415)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
ddd9042420 libtool: Fix patch status tag
(From OE-Core rev: 28fc470e5e10ee9cce893d037ed5e518bc5612f5)

(From OE-Core rev: c7c4920fc287bdb5f7a0bca7b2ec2ab7a43f58fd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
656897e761 libtool: Fix problem with libtoolize in multilib installations
Without this patch /usr/bin/libtoolize is different for each multilib
since their host-triplets are different, despite there being no difference in
the functionality of libtoolize itself.

This change just patches out the problematic line since its just a comment for the user
in help text. Ugly but solves the problem. This fixes issues where libtool and
libXX-libtool couldn't be installed into the same system.

(From OE-Core rev: f70040fd3ca3508d33ed24c749c0b8095b020dab)

(From OE-Core rev: 8480fff287e660f85a99fcc28119fe80d517e0b2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
6bbdd7b569 multilib_script: Add support for multilib scripts
Whilst the package managers handle multilib ELF binaries well, they don't
handle scripts in the *bindir directories well. This adds support for
marking these up so that they can be handled using update-alternatives.

Its done this way so that non-multilib systems don't see any changes and
there is standardisation amongst the multilibs on how the alternatives are
named and prioritiesd. The priotitisation code needs to be added but this
change means there is somewhere to add it.

Recipe needs to set MULTILIB_SCRIPTS in the form <pkgname>:<scriptname>, e.g.
MULTILIB_SCRIPTS = "${PN}-dev:${bindir}/file1 ${PN}:${base_bindir}/file2"
to indicate which script files to process from which packages.

libtool is used a as a reference to stop the libtool scripts conflicting
in a multilib case and allows the kernel-devsrc change to be merged.

(From OE-Core rev: 18e837433d07cfdce4019c13f682c6676425a2ad)

(From OE-Core rev: 97e2d65d1c406bc58fe693e500fcc939459bac1a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Ross Burton
5d249ac250 image_types: use cpio-native to build cpio images
As per the previous commit, upstream cpio has a bug which means it crashes on
append. If the image being built has already had testimage ran then cpio-native
will be in the sysroot.  It's also possible that some distributions are shipping
this broken CVE patch too.

Now that our cpio-native is fixed, until we can be sure that the host cpio isn't
broken depend on cpio-native if building a cpio image.

[ YOCTO #13042 ]

(From OE-Core rev: c3b9aedcbe538d7fa74bd814644b4899769dec46)

(From OE-Core rev: a75eba71145efa1c3d206c5e5c00608a50f013bc)

(From OE-Core rev: 9e1c69932add702b9c5bc1faa9ef5db975de0ee3)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00