Commit Graph

50721 Commits

Author SHA1 Message Date
Yeoh Ee Peng
86aa896b9e resulttool/report: Enable roll-up report for a commit
Enable roll-up all test results belong to a commit
and to provide a roll-up report.

(From OE-Core rev: 8f23018bcb76c1b431f69183f993f08adb660fae)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:28 +00:00
Yeoh Ee Peng
3e6a012462 scripts/resulttool: Enable manual result store and regression
To enable store for testresults.json file from manualexecution,
add layers metadata to configuration and add "manual" map to
resultutils.store_map.

To enable regression for manual, add "manual" map to
resultutils.regression_map. Also added compulsory configurations
('MACHINE', 'IMAGE_BASENAME') to manualexecution.

(From OE-Core rev: 514a9165790dda2afb412c519cb1444493dff1f9)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:28 +00:00
Yeoh Ee Peng
43823fbe11 resulttool/regression: Ensure regressoin results are sorted
Sorted regression results to provide friendly viewing of report.

(From OE-Core rev: de222c3c9e3aa34e42e1ad6ba6cbb9e1a18ccd80)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Richard Purdie
fa60137844 resulttool/store: Fix missing variable causing testresult corruption
(From OE-Core rev: 811d2be3cef340ee945335552892a2a49a46bf4b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Richard Purdie
e2a49f2679 resulttool/report: Ensure ptest results are sorted
(From OE-Core rev: f182217970029ba0260df3c3089101352da083ce)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Richard Purdie
c381ceb8eb resulttool/report: Ensure test suites with no results show up on the report
ptest suites with no results don't show up on the reports even though we have
a duration for them. Fix this so the fact they report no tests is visible.

(From OE-Core rev: a61ad19d5f997cb2377ca307c94ca2d53f8c0c49)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Richard Purdie
652b502456 resulttool/report: Handle missing metadata sections more cleanly
Currently some older results files cause the code to give tracebacks.
Handle these missing sections more cleanly.

(From OE-Core rev: 829754c380ca2b614afa23500e53a8c5b850932f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Richard Purdie
a1a31bb856 resulttool/store: Handle results files for multiple revisions
Currently we cant store results if the results files span multiple
different build revisons. Remove this limitation by iterating.

(From OE-Core rev: 22a5982dbcb624ffcdbacef92b2a235851214663)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Richard Purdie
da75c0b5b4 resulttool/resultutils: Avoids tracebacks for missing logs
(From OE-Core rev: d5d62b31b192b88c6a829099b74bfb30583b066e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Richard Purdie
c7eb843d7c resulttool: Improvements to allow integration to the autobuilder
This is a combined patch of the various tweaks and improvements I
made to resulttool:

* Avoid subprocess.run() as its a python 3.6 feature and we
  have autobuilder workers with 3.5.

* Avoid python keywords as variable names

* Simplify dict accesses using .get()

* Rename resultsutils -> resultutils to match the resultstool ->
  resulttool rename

* Formalised the handling of "file_name" to "TESTSERIES" which the code
  will now add into the json configuration data if its not present, based
  on the directory name.

* When we don't have failed test cases, print something saying so
  instead of an empty table

* Tweak the table headers in the report to be more readable (reference
  "Test Series" instead if file_id and ID instead of results_id)

* Improve/simplify the max string length handling

* Merge the counts and percentage data into one table in the report
  since printing two reports of the same data confuses the user

* Removed the confusing header in the regression report

* Show matches, then regressions, then unmatched runs in the regression
  report, also remove chatting unneeded output

* Try harder to "pair" up matching configurations to reduce noise in
  the regressions report

* Abstracted the "mapping" table concept used to pairing in the
  regression code to general code in resultutils

* Created multiple mappings for results analysis, results storage and
  'flattening' results data in a merge

* Simplify the merge command to take a source and a destination,
  letting the destination be a directory or a file, removing the need for
  an output directory parameter

* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression
  mappings

* Have the store command place the testresults files in a layout from
  the mapping, making commits into the git repo for results storage more
  useful for simple comparison purposes

* Set the oe-git-archive tag format appropriately for oeqa results
  storage (and simplify the commit messages closer to their defaults)

* Fix oe-git-archive to use the commit/branch data from the results file

* Cleaned up the command option help to match other changes

* Follow the model of git branch/tag processing used by oe-build-perf-report
  and use that to read the data using git show to avoid branch change

* Add ptest summary to the report command

* Update the tests to match the above changes

(From OE-Core rev: e4195565d2a50046d4378c97f7a593c41bed51bd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Mazliana
075cd5e7fe scripts/resulttool: enable manual execution and result creation
Integrated “manualexecution” operation to resulttool scripts.
Manual execution script is a helper script to execute all manual
test cases in baseline command, which consists of user guideline
steps and the expected results. The last step will ask user to
provide their input to execute result. The input options are
passed/failed/blocked/skipped status. The result given will be
written in testresults.json including log error from the user
input and configuration if there is any.The output test result
for json file is created by using OEQA library.

The configuration part is manually key-in by the user. The system
allow user to specify how many configuration they want to add and
they need to define the required configuration name and value pair.
In QA perspective, "configuration" means the test environments and
parameters used during QA setup before testing can be carry out.
Example of configurations: image used for boot up, host machine
distro used, poky configurations, etc.

The purpose of adding the configuration is to standardize the
output test result format between automation and manual execution.

To use these scripts, first source oe environment, then run the
entry point script to look for help.
        $ resulttool

To execute manual test cases, execute the below
        $ resulttool manualexecution <manualjsonfile>

By default testresults.json store in <build_dir>/tmp/log/manual/

[YOCTO #12651]

(From OE-Core rev: cd2ef260569a67907801299b1397db94aa871cb5)

Signed-off-by: Mazliana <mazliana.mohamad@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Yeoh Ee Peng
d038d97d48 resulttool: enable merge, store, report and regression analysis
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.

These scripts were developed as a test result tools to manage
these testresults.json file.

Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.

Using the "merge" operation, user can merge multiple testresults.json
files to a target file.

Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.

Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.

These resulttool operations expect the testresults.json file to use
the json format below.
{
    "<testresult_1>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
    ...
    "<testresult_n>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
}

To use these scripts, first source oe environment, then run the
entry point script to look for help.
    $ resulttool

To store test result from oeqa automated tests, execute the below
    $ resulttool store <source_dir> <git_branch>

To merge multiple testresults.json files, execute the below
    $ resulttool merge <base_result_file> <target_result_file>

To report test report, execute the below
    $ resulttool report <source_dir>

To perform regression file analysis, execute the below
    $ resulttool regression-file <base_result_file> <target_result_file>

To perform regression dir analysis, execute the below
    $ resulttool regression-dir <base_result_dir> <target_result_dir>

To perform regression git analysis, execute the below
    $ resulttool regression-git <source_dir> <base_branch> <target_branch>

[YOCTO# 13012]
[YOCTO# 12654]

(From OE-Core rev: bb0bc6368bb51ac0be77d13fe931601d493951ea)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 15:38:27 +00:00
Jonathan Rajotte-Julien
cefa08aeef lttng-tools: update to 2.9.11
(From OE-Core rev: 4af5a794497746629a35726ce515556b53d40da9)

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25 23:12:03 +00:00
Jonathan Rajotte-Julien
3fd4794b16 lttng-modules: update to 2.10.9
Pertinent fix for OE-Core since 2.10.6:
    Fix: out of memory error handling
    Fix: access migrate_disable field directly
    Prevent allocation of buffers if exceeding available memory

2.10.9 also contains the necessary fix to support kernel up to 5.0.

(From OE-Core rev: 9666e9f6058bfd8294cb4e6b03752f28159a76bc)

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25 23:12:03 +00:00
Jonathan Rajotte-Julien
26745f6b22 lttng-ust: update to 2.10.3
(From OE-Core rev: 9d8b3988a53449fa18398e9eaae1627923eeef97)

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25 23:12:03 +00:00
Stefan Müller-Klieser
61f17352ec coreutils: 8.29: fix native build
This fixes the native build on glibc 2.28 hosts, by backporting two
gnulib patches from master.

(From OE-Core rev: fe7c33cd470d7466be48391b11ea703746812014)

Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25 23:12:03 +00:00
Martin Jansa
2333147fa6 busybox: backport fix for issues introduced by CVE-2011-5325.patch
(From OE-Core rev: d0555e89514f2641387ef061f9ffcd1c8ced008c)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25 23:12:03 +00:00
Mingli Yu
e0fdb98b0f logrotate.py: restore /etc/logrotate.d/wtmp
During the test logrotate.LogrotateTest.test_1_logrotate_setup,
there is below logic:
 # mkdir $HOME/logrotate_dir
 # sed -i "s#wtmp {#wtmp {\n    olddir $HOME/logrotate_dir#" /etc/logrotate.d/wtmp

After all logrotate.LogrotateTest finished, only cleanup
$HOME/logrotate_dir as below, but don't restore
the config file /etc/logrotate.d/wtmp.
[snip]
def tearDownClass(cls):
    cls.tc.target.run('rm -rf $HOME/logrotate_dir')
[snip]

That's to say, there is one additional line added
to /etc/logrotate.d/wtmp and will make the logrotate
service start failed when run systemd.SystemdBasicTests.test_systemd_failed

Take an example as below when run test as root:
 # cat /etc/logrotate.d/wtmp
 # no packages own wtmp -- we'll rotate it here
 /var/log/wtmp {
    olddir /root/logrotate_dir
    missingok
    monthly
    create 0664 root utmp
    minsize 1M
    rotate 1
 }

 # ls /root/logrotate_dir
 ls: cannot access '/root/logrotate_dir': No such file or directory

 # systemctl start logrotate
 Job for logrotate.service failed because the control process exited with error code.
 See "systemctl status logrotate.service" and "journalctl -xe" for details.

 # systemctl status logrotate
  logrotate.service - Rotate log files
   Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset>
   Active: failed (Result: exit-code) since Wed 2019-02-13 03:35:19 UTC; 7s ago
     Docs: man:logrotate(8)
           man:logrotate.conf(5)
   Process: 540 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
   Main PID: 540 (code=exited, status=1/FAILURE)

  Feb 13 03:35:18 qemumips systemd[1]: Starting Rotate log files...
  Feb 13 03:35:19 qemumips logrotate[540]: error: wtmp:9 error verifying olddir path /root/logrotate_dir: No such file or directory
  Feb 13 03:35:19 qemumips logrotate[540]: error: found error in file wtmp, skipping
  Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
  Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Failed with result 'exit-code'.
  Feb 13 03:35:19 qemumips systemd[1]: Failed to start Rotate log files.

Add the logic to restore /etc/logrotate.d/wtmp to
make the cleanup complete to fix the above issue.

(From OE-Core rev: a2db9320d97d12d87524ff16a329f9c38a8da33f)

(From OE-Core rev: a5dcf0c4bd9a0be2f2484b0b1b3e77f1f2128dfc)

Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
George McCollister
0278e515fd systemd: fix CVE-2019-6454
Apply patches from systemd_237-3ubuntu10.13 to fix CVE-2019-6454.
CVE-2019-6454 is an issue in which systemd (PID1) can be crashed
with a specially formed D-Bus message.

For information see:
https://usn.ubuntu.com/3891-1/
https://git.launchpad.net/ubuntu/+source/systemd/commit/?h=applied/ubuntu/bionic-updates&id=d7584b894afcaa8a4a1abb69db2a9c81a6276e80

(From OE-Core rev: 342157b135e7493e5965b706ede93bee190fbe32)

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
George McCollister
4c55db6d5c systemd: fix CVE-2018-6954
Apply patches to fix CVE-2018-6954

NVD description from https://nvd.nist.gov/vuln/detail/CVE-2018-6954

systemd-tmpfiles in systemd through 237 mishandles symlinks present in
non-terminal path components, which allows local users to obtain
ownership of arbitrary files via vectors involving creation of a
directory and a file under that directory, and later replacing that
directory with a symlink. This occurs even if the fs.protected_symlinks
sysctl is turned on.

Patches from systemd_237-3ubuntu10.13.debian.

These patches shouldn't be required on newer OE releases since they use
systemd v239 or higher.

(From OE-Core rev: 607350d98aa4c65b71fe1f10900e205fad81d1ec)

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
Marcus Cooper
d4e0f92528 systemd: Security fix CVE-2018-16866
Affects < v240

(From OE-Core rev: 10fa35a75617e82650b12d3e353a554f05f036dd)

Signed-off-by: Marcus Cooper <marcusc@axis.com>

>From v2 patch on openembedded-core@lists.openembedded.org
Incresed file name number from 0026 to 0027.

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
George McCollister
7be61780af systemd: Security fix CVE-2018-16865
Affects < v240

Based on thud commit d5d2b821fc85b8cf39f683061ac2a45bddd2139f
The second patch in the thud commit doesn't apply against 237. Use the
version of the second patch CVE-2018-16865_2.patch from
systemd_237-3ubuntu10.13.debian.

(From OE-Core rev: da41e48567eb21a47426a6fbe23ea07ce780cd3c)

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
George McCollister
c3890467ff systemd: Security fix CVE-2018-16864
Affects < v240

Based on thud commit 403e74b07b6f3c4a2444e68c74a8434fb17aee49
The patch in the thud commit doesn't compile against 237. Use the
version of this patch, CVE-2018-16864.patch from
systemd_237-3ubuntu10.13.debian.

(From OE-Core rev: ddbe969d0c9052a3ae17ef8f1cec8da847c722d3)

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
Chen Qi
b3ed759360 systemd: fix CVE-2018-15688
Backport patch to fix the following CVE.

CVE: CVE-2018-15688

(From OE-Core rev: d490839e881f3ff30a4bde8137cb04cb0fd37acd)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Cherry-picked from thud 13591d7224393dc0ae529a03cdf74aceb3540ce9

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
George McCollister
0e63d91b2b systemd: fix CVE-2018-15687
Backport patch to fix the following CVE.

CVE: CVE-2018-15687

Based on thud commit eeb621aa19f690971caf862290a172a115578ba1
The patch in the thud commit doesn't compile against 237. Use the
version of this patch,  CVE-2018-15687.patch from
systemd_237-3ubuntu10.13.debian.

(From OE-Core rev: 3e8ba9af58253ed9db0f0376a8e2966e45ee089e)

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
Chen Qi
d5400f11dd systemd: fix CVE-2018-15686
Backport patch to fix the following CVE.

CVE: CVE-2018-15686

(From OE-Core rev: 06bf145cee24b677ab076498fe8399126971bc43)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Cherry-picked from thud 0ef70603bc983315eb0e8a97958d995a31198c35

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
ROGEZ Matthieu
0b554def18 systemd: Fix typo in root home variable.
This regression has been introduced while upgrading to version 237
(commit 906230a73b3ccfa4afd2a19a6b0aa18cd1d5fa08)
and seems to only affect sumo version.

(From OE-Core rev: 665b41d326654235d305649be4be69a1be8dc00b)

Signed-off-by: Matthieu Rogez <matthieu.rogez@fivesgroup.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
Richard Purdie
0a8f50fd31 oeqa/runtime/dnf: Fix test error when static libs are enabled
The test works by excluding curl-dev which curl-staticdev depends upon.
When static libraries aren't disabled, this leads to an odd looking test
failure.

Simply exclude curl-staticdev as well in case its enabled to make sure
the test always works.

(From OE-Core rev: dfded083d9456c8d2bb168dbe8e08b2ffb3e1a26)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
Stefan Agner
b4daa5ba96 run-postinsts: for dpkg/opkg, do not rely on /etc/*-postinsts
Start opkg/dpkg as soon as the respective package managers status
file is present, no matter whether /etc/$pm-postinsts exists. This
decouples the implicit link between postinsts scripts in /etc and
the package manager: Currently the package manager is only started
if those scripts are present, although the package manager does not
use those scripts at all! Package managers install their own set of
postinst scripts.

The behavior when using rpm packages stays the same.

Note that using the package managers capability to execute postinst
scripts is preferred for good reasons: It makes sure that the
package managers database reflects that the packages have been
completely installed and configured.

This change allows to drop installation of the postinsts scripts
when package management is present. This will be done in a separate
change.

Note: Before commit 5aae19959a44 ("rootfs.py: Change logic to
unistall packages") rootfs.py did not install /etc/$pm-postinsts
when package management is installed! The change caused YOCTO #8235
which lead to the behavior change of run-postinsts in first place.

(From OE-Core rev: 85e498a4671426999610d90c87c354d41cfe8443)

Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 16:49:54 +00:00
Filip Jareš
7aae52eae2 ref-manual: Fixed SRC_URI cut-and-paste error.
The update fixes a copy-paste-and-forget-to-update-everything error
in the documentation of the `SRC_URI` entry in the Reference Manual.
The `maxdate` paragraph just repeated what is there for the
neighboring `mindate` paragraph which was incorrect.

(From yocto-docs rev: 26c5549580e108c40aa7c95860d12c675596820c)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:08 +00:00
Richard Purdie
b28f5672ab bitbake: COW: Fix StopIteration warning
Fix the warning:

WARNING: lib/bb/data_smart.py:235: DeprecationWarning: generator 'COWDictMeta.iter' raised StopIteration
  for k, v in self.variables.iteritems():

by using return from the generator, not raising StopIteration.

(Bitbake rev: c0af6c81f8d5487ea2cef54a78fd1cb1d0dc6520)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 407d6e07b09123c12c382b4a92107f002c314b05)
Signed-off-by: Luca Boccassi <luca.boccassi@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-13 14:38:56 -07:00
Paul Eggleton
d3ad243822 bitbake: utils: add optional callback to edit_bblayers_conf()
Add a callback that lets you modify or remove items in addition to the
current scheme where you can only add or remove. This enables you to for
example replace a layer with a temporary copy (which is what we will use
this for first in OE's oe-selftest).

(Bitbake rev: 4f6ba26e8335f975038d90b9e1c1767160bd5272)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
9f8d417c32 bitbake: process: Rewrite multiple connection handling
If the bitbake server recieved multiple connections, it currently closes
ones it can't handle (while its dealing with another). This is rather
antisocial behaviour which causes clients to quickly run through their
retries and abort.

Instead, queue any other connections until the current one is closed. This
way the client can decide when it wants to stop waiting for the server. If the
client is gone by the time we handle it, we handle that gracefully.

This also fixes a number of bugs in the connection handling where connections
which did drop early were badly handled causing tracebacks in the logs.

Also, handle queue incomming connections in a loop to ensure that the main
client handling doesn't starve that piece of the system.

This code was stress tested by running 50 connection attempts in parallel at
once, ensuring the code correctly handled them.

(Bitbake rev: f675293d86504db66442532258d99b26333742bf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
d1baf887cd bitbake: process: Handle EWOULDBLOCK in socket connect
Now that we set a timeout for the socket, it can return EWOULDBLOCK
if a signal or other event happens to wake up even if we don't timeout.

If this happens, retry the connection, else we simply see it quickly
loop through the retries and abort the connection in a very short
interval.

(Bitbake rev: f770d6a332812031682dc6bef1a2a84da52a4c32)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
83dbc768d3 bitbake: process.py: Set socket timeout to 10 seconds
The current value of 2 seconds has shown to be short in
wider testing.

(Bitbake rev: c4a940991f261959eb08273d2250d3866b868938)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
d3b7f2ead5 bitbake: server/process: Ensure socket has a timeout set
We're seeing hangs in oe-selftest where server startup and shutdown are
racing. The assumption was a connect would timeout however no timeout is
set which can leave processes hanging. Set a short timeout for
the connection to avoid this.

(Bitbake rev: e53c1009356cc49c57d3b9af1e3dda6927acd78d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
3690c081c1 bitbake: cooker: Add some timing debug messages to the server startup
We're seeing slow startup in bitbake, add some timeing debug messages so
the logs are more useful for debugging when its slow.

(Bitbake rev: 39548791c84982c44c872a579e5b42d2720af98f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
245e1c251a bitbake: server/process: Increase server startup time delay
On loaded production systems we've seen bitbake server take over
40s to start up. Increase the timeout to 90s which tries to avoid
failures in selftests.

The delays come from setting up the inotify watches (31s) so can't
really be avoided.

After 5s delay we now warn the user we're waiting for 90s so the
interactive exeperience shouldn't be much changed and its very
unlikely the user would see that anyway.

(Bitbake rev: 64055fdb0d8485b40ba710a762e9a0af5c4540b4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Peter Kjellerstedt
caad85d90b bitbake: bitbake-diffsigs: Support recursive deps with signature files
Follow dependent hash changes recursively also when specifying two
signature files explicitly. Previously this was only done when using the
--task option.

(Bitbake rev: 65c2a64d6dd5dc85cb14b9e808964c699e890517)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Peter Kjellerstedt
6b6567cb6b bitbake: bitbake-diffsigs: Merge with bitbake-dumpsig
The functionalities of bitbake-diffsigs and bitbake-dumpsig are so
similar that they can be merged into one. Add an option --dump to make
bitbake-diffsigs dump the last signature data instead of comparing it.
Keep bitbake-dumpsig as a symbolic link to bitbake-diffsigs. When it is
called as bitbake-dumpsig, it behaves as if --dump was specified.

Also make -D the short option for --debug again (the way it used to be,
and still was for bitbake-dumpsig), so that -d can be used as the short
option for --dump.

(Bitbake rev: de298ff0b0cfeb29b569f946ab95329be102caa8)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
27d5f46f45 bitbake: server/process: Show last 60 lines of the log if the server didn't start
We're seeing issues where the server doesn't start with no logs as to why. Allow
the server to print the last 60 log lines just in case this shows us something useful
about what is failing.

(Bitbake rev: da54a3fef9cee308dfa87eea9b9638796d734abd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
ba9f093318 bitbake: server/process: Show the last 60 log lines, not the last 10
10 log lines may not capture any full traceback, increase the number of
lines to 60 which covers most tracebacks.

(Bitbake rev: 340a7d91b5030e4d62680f427286fc419509879b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Robert Yang
633d22650c bitbake: server/process: print a message when no logfile
[YOCTO #12898]

There might be no bitbake-cookerdaemon.log, print a message for debugging.

(Bitbake rev: cc32e703a8e68107353702edb230296503f5dad0)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
feee885613 bitbake: server/process: Make lockfile handling clearer
This simplifies the code and makes it easier to read but has the
same functionality.

(Bitbake rev: 21dcec291d58ab7ac5d2c07186c68e01d85e6f65)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Ross Burton
6ddeba54a4 bitbake: layerindex: don't use shell=True when cloning
(Bitbake rev: fd1504a5e92390b8e1c0e689b4c2cb0ac7096b1a)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Ross Burton
4e75f5cc6b bitbake: fetch: don't use shell=True when listing ar files
(Bitbake rev: d150e50355444b0a67cad4d6a100d9bf6fadb2ae)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
d3dcc4f92e package_rpm/archiver: Apply bandaid to src.rpm creation
| error: create archive failed on file /media/build1/poky/build/tmp/work/all-poky-linux/xcursor-transparent-theme/0.1.1+gitAUTOINC+23c8af5ba4-r0/deploy-sources/allarch-poky-linux/xcursor-transparent-theme-0.1.1+gitAUTOINC+23c8af5ba4-r0/xcursor-transparent-theme-0.1.1+git0+23c8af5ba4-r0.src.rpm: cpio: read failed - No such file or directory
| Building target platforms: noarch-poky-linux
| Building for target noarch-poky-linux

This is caused by:

$ cat log.task_order

do_cleansstate (24289): log.do_cleansstate.24289
do_deploy_archives_setscene (24395): log.do_deploy_archives_setscene.24395
do_fetch (24407): log.do_fetch.24407
[..]
do_package_write_rpm (25448): log.do_package_write_rpm.25448
do_package_qa (25451): log.do_package_qa.25451

So do_deploy_archives can run from sstate, created a .src.rpm in WORKDIR/deploy-sources,
then it was removed when rpm was running. This leads to a broken Source line in the
spec file as the original file was found by the os.listdir().

This fix is just a bandaid over much more fundamental problems sadly.

(From OE-Core rev: a10020ace4c3cd863c782760f7cbecea557ec6e7)

(From OE-Core rev: 6d56e912fbbaa22830b4da5ab230586a3d15b23e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
31a84b4316 archiver/package_rpm: Fix the worst src.rpm generation race
The package_rpm code is writing outside the task's sstate directory into
the sstate of do_deploy_archives. This is "out of spec" since if the
task is installed from sstate, the files are not restored. This means
the files may appear/disappear, things are not deterministic and there are
races.

Extend the do_package_write_rpm code to handle writing the src.rpm into
place to avoid these issues. There are other problems but this avoids races
around this file.

(From OE-Core rev: c6e151ba7fe0f14044537cf0ab2cac436f1496e3)

(From OE-Core rev: b119872fb794a36b6eb9ef5e9c42a9c6c991e835)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
6ab2cac421 libtool-cross: Handle ccache sstate 'infection' issues
On a system without ccache, f you:

INHERIT += "ccache"
bitbake libtool-cross
<remove INHERIT>
bitbake apmd

then it fails due to being unable to find ccache. The references to ccache are
coded into libtool-cross but the sstate checksum doesn't reflect this due to the
way the class is coded (output should be the same regardless).

The simplest solution is to remove references to ccache from the libtool script.
The output then works regardless of whether ccache is present or not. The
libtool-cross script is only used in a handful of cases (most of the time its
dynamically generated by autoconf) so any performance issue is minor.

(From OE-Core rev: ed550a49d2114c56e5bc033ecd0e83073d2d4067)

(From OE-Core rev: ee6a2e0ccb11e5f5267bc2e406203c78b0443415)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00
Richard Purdie
ddd9042420 libtool: Fix patch status tag
(From OE-Core rev: 28fc470e5e10ee9cce893d037ed5e518bc5612f5)

(From OE-Core rev: c7c4920fc287bdb5f7a0bca7b2ec2ab7a43f58fd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 22:27:46 +00:00