Commit Graph

1000 Commits

Author SHA1 Message Date
Jaewon Lee
88db9e41c1 populate_sdk_ext: Introduce mechanism to keep nativesdk* sstate in esdk
When doing a devtool build-sdk from within an esdk all nativesdk
components would be rebuilt. This patch introduces SDK_INCLUDE_NATIVESDK
flag to toggle the inclusion of nativesdk packages when creating the
esdk sstate

Currently locked-sigs.inc is generated during do_sdk_depends which
doesn't pull in nativesdk packages. Generating another locked-sigs.inc
in do_populate_sdk_ext and pruning it to only nativesdk* packages by
using a modified version of the already existing function
prune_locked_sigs and merging it with the current locked-sigs.inc
Also adding SDK_INCLUDE_NATIVESDK tasklistfn to the logic surrounding
setting tasklist file to not prune esdk sstate during creation

[YOCTO #13261]

(From OE-Core rev: d046afd12e1c209b29dca6ba402b9aa14680c5ce)

Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-19 15:27:02 +01:00
Joshua Watt
34923e4f77 Remove SSTATE_HASHEQUIV_SERVER
Removes all references to the SSTATE_HASHEQUIV_SERVER variable. This
variable is redundant now that BB_HASHSERVE is present.

(From OE-Core rev: 54b3adb6bc90a8e4b9e92952688772ee074d36e5)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-18 17:52:01 +01:00
Joshua Watt
f12e41c1bb sstatesig: Update server URI
The server no longer uses a "http://" URI, since it has been updated to
use a different protocol.

(From OE-Core rev: 519561172c48bc7f7a61a3d02edd418fc0895b7b)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-18 17:52:01 +01:00
Changqing Li
da61ee5541 classextend.py: don't extend file for file dependency
Fix error like:
lib32-e2fsprogs-1.45.3-r0 do_package_qa: QA Issue: /usr/sbin/e2scrub_all
contained in package lib32-e2fsprogs-e2scrub requires /bin/bash,
but no providers found in RDEPENDS_lib32-e2fsprogs-e2scrub

For some lib32 packages(eg: lib32-bash, lib32-sed) which probvides files,
extend is not needed

Eg: RPROVIDES of lib32-bash expects to have /bin/bash, with original
extend, it will become lib32-/bin/bash, then will cause above error

Fix by don't extend file dependency, and skip multilib check for file dependency
in do_package_qa to avoid error like:
WARNING: lib32-bash-5.0-r0 do_package: QA Issue: lib32-bash package lib32-bash
 - suspicious values '/bin/bash /bin/sh' in RPROVIDES [multilib]

(From OE-Core rev: a9163120ed52534e7dbf4db50dc2b03bbf69f06b)

Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:45 +01:00
Vijai Kumar K
550d3541ae lib/oe/gpg_sign.py: Remove unwanted import
Remove unwanted import of oe.utils.

(From OE-Core rev: 42e9de68384fb24d5c9a5f161810be90c45942d2)

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 14:58:09 +01:00
Richard Purdie
359d61067a sstatesig: Fix leftover splitting issue from siggen change
(From OE-Core rev: 8b76048da99ec3be7d763b7ccc81f767c7015bc7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-12 16:23:57 +01:00
Richard Purdie
a0542ed3ff sstatesig: Updates to match bitbake siggen changes
Update the metadata to correspond to the bitbake siggen task specification
format change. This standardises on "<fn>:<task>" everywhere rather than
the "." delimiter that was being used in some places.

This is an API breaking change but means we now have a consistent format
being used throughout the codebase without compatibility APIs.

(From OE-Core rev: 23db236a054ee7a989cdbbcb42ad5c6eefd4a6ae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
bcc9f1b5f1 sstate/sstatesig: Update to new form of BB_HASHCHECK_FUNCTION
Bitbake has updated to a cleaned up form of BB_HASHCHECK_FUNCTION,
adapt to this cleanup. This is an API breaking change.

(From OE-Core rev: a4d413d8d809132b0e0a5dd673a36e2bd0e0be4c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
8d7f663136 sstatesig: Update to handle BB_HASHSERVE
Update the code to handle BB_HASHSERV, an autostarted bitbake internal hash
equivalency server suited to local developer usage.

(From OE-Core rev: 9b3d7b0b0ff27281391c2b8f2511d312d6995ed6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
56b178ca2f sstatesig: Adpat to recent bitbake hash equiv runqueue changes
Upstream bitbake now hanes preserving the unihash data itself, drop
this usage of persist_data which was extremely problematic due to
concurrent task access issues, particulary on loaded systems.

(From OE-Core rev: 034d91c2c94b201797a7830b0af6141132f9bad1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
00f40114b4 sstatesig: Add debug for incorrect hash server settings
If the hash server settings are incorrect, show the user useful
error messages instead of tracebacks.

(From OE-Core rev: 2df5d95b9b63e30ddaa7c72a4173b9a05b3f15f9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
62c8b8c4d9 sstatesig: Move unihash siggen code to bitbake
This code is closely tied with the hash server in bitbake and also means
we can't relibably test the hashserv runqueue functionality without OE
metadata. Moving this to bitbake as a MixIn class makes most sense
and encourages code collaboration and reuse as well as enabling easier
and more accurate testing of the APIs.

(From OE-Core rev: a2a9c6092d4dde706ed071b08a972d1d87184295)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
546c32753b classes/lib: Remove bb.build.FuncFailed
Whilst seemingly a good idea, this exception doesn't really serve any purpose
that bb.fatal() doesn't cover. Wrapping exceptions within exceptions isn't
pythonic.

Its not used in many places, lets clean up those and remove usage of it
entirely. It may ultimately be dropped form bitbake entirely.

(From OE-Core rev: efe87ce4b2154c6f1c591ed9d8f770c229b044ad)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
41a5dbd16b package: Improve determinism
Its possible in cases with multiple shlib providers we were not being
deterministic. Add in a couple of sorted() calls to fix the shlibs and
pkgconfig cases with this potential issue.

(From OE-Core rev: 5b9a4214ee17e1a39dd5a1742f2ac5ed25a11310)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-26 08:41:39 +01:00
Alistair Francis
991cd48af5 package_manager: Ensure the base-feed directory exists
Ensure that the /etc/opkg directory exists before we try to create a
file there.

(From OE-Core rev: 30ff50223cd0b79fd3b8aa393ea1e621282773ac)

Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-18 12:16:19 +01:00
Andrej Valek
c41acbcf8a oe/copy_buildsystem: move layer into layers directory
Layers could be located outside from poky but inside the build directory.
This case should be covered in eSDK.
meta-abc
meta-def/meta-ghi
meta-def/poky
meta-def/meta-oe/meta-oe
...

It should take all enabled layers and put them into 'layers' dir during
build-time with respecting new relative path to poky.
layers/meta-abc
layers/meta-ghi
layers/poky
layers/meta-oe/meta-oe
...

(From OE-Core rev: 55ecf6988d3e3c0935cb6324a6ad2c75f1191a1d)

Signed-off-by: Andrej Valek <andrej.valek@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-18 12:16:19 +01:00
Ross Burton
1b7876b060 buildhistory_analysis: ignore ownership for sysroot diffs
The sysroot isn't populated under pseudo, so ownership differences should be ignored.

(From OE-Core rev: 01b816be4adff8f3992c1369810bdcf11a26fd6c)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-05 12:00:20 +01:00
Ross Burton
adaba4ee4f buildhistory: report sysroot changes
Now that the sysroot is written into the build history, write it out.

(From OE-Core rev: e9df98ba63e6d2baefee550170dbdd11ed2ad03a)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-28 13:28:37 +01:00
Ricardo Ribalda Delgado
931ed3e4f8 rootfs: Fix dependency for every dpkg run
Avoid getting a warning on do_rootfs.

Fixes:

NOTE: Installing the following packages: libgdk-pixbuf-2.0-locale-en-gb glibc-locale-en-gb libatspi-locale-en-gb gstreamer1.0-locale-en-gb gtk+3-locale-en libatk-1.0-locale-en-gb gtk+3-locale-en-gb gstreamer1.0-plugins-good-locale-en-gb gstreamer1.0-plugins-base-locale-en-gb gstreamer1.0-plugins-bad-locale-en-gb libglib-2.0-locale-en-gb avahi-locale-en-gb vte-locale-en-gb xkeyboard-config-locale-en-gb
WARNING: Unable to install packages. Command '/workdir/build/tmp/work/qt5122-poky-linux/bottlecam-image/1.0-r0/recipe-sysroot-native/usr/bin/apt-get  install --force-yes --allow-unauthenticated libgdk-pixbuf-2.0-locale-en-gb glibc-locale-en-gb libatspi-locale-en-gb gstreamer1.0-locale-en-gb gtk+3-locale-en libatk-1.0-locale-en-gb gtk+3-locale-en-gb gstreamer1.0-plugins-good-locale-en-gb gstreamer1.0-plugins-base-locale-en-gb gstreamer1.0-plugins-bad-locale-en-gb libglib-2.0-locale-en-gb avahi-locale-en-gb vte-locale-en-gb xkeyboard-config-locale-en-gb' returned 100:
Reading package lists...
Building dependency tree...
Reading state information...

(From OE-Core rev: 4e315dd23c52b75fd64dca89e7bffe25a0b0e6f9)

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-19 22:13:42 +01:00
Richard Purdie
32df42cff5 multiconfig: Adapt to bitbake switch 'multiconfig' -> 'mc'
(From OE-Core rev: 8a6f7c1e455156966f467008645fef14db679ccf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10 14:46:38 +01:00
Richard Purdie
1334b064d3 gpg_sign/selftest: Fix secmem parameter handling
We keep seeing "cannot allocate memory" errors from rpm when signing packages
on the autobuilder. The following were tried:

* checking locked memory use (isn't hitting limits)
* Restricting RPM_GPG_SIGN_CHUNK to 1
* Limiting to 10 parallel do_package_write_rpm tasks
* Allowing unlimied memory overcommit
* Disabling rpm parallel compression

and the test still failed. Further invetigation showed that the --auto-expand-secmem
wasn't being passed to gpg-agent which meant the secmem couldn't be expanded hence the
errors when there was pressure on the agent.

The reason this happens is that some of the early gpg commands can start the agent
without the option and it sticks around in memory so a version with the correct
option may or may not get started.

We therefore add the option to all the key gpg calls.

(From OE-Core rev: c7e131a76e522503df55e211dd261829feacfa28)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-07 09:11:49 +01:00
Hongxu Jia
c113a83dd5 lib/oe/reciputils.py: support character `+' in git pv
While pv contains char `+' (such as ncurses 6.1+20181013),
it was incorrectly converted to `6.1'

In this commit:
- Convert [^\+]* to .*?
  Support pv to contain char `+' and not greedy match
  (Such as ncurses 6.1+20181013)

- Add [^\+] to sfx match
  Support sfx contains extra chars between `+' and `git'
  (such as asciidoc "8.6.9+py3-gitAUTOINC+618f6e6f6b")

- Make sfx and rev greedy match

Run `devtool check-upgrade-status --all' in poky and compare results,
only one difference on ncurses version:
Without the commit:
INFO: ncurses                   6.1             6.1+20181013    Hongxu Jia <hongxu.jia@windriver.com> 7a97a7f937762ba342d5b2fd7cd090885a809835

With the commit:
INFO: ncurses                   6.1+20181013    MATCH           Hongxu Jia <hongxu.jia@windriver.com> 7a97a7f937762ba342d5b2fd7cd090885a809835

(From OE-Core rev: 8049bd34b89e710f7bb20883813ba3f929d9e997)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-04 09:09:42 +01:00
Ross Burton
92290d3b70 patch: add minver and maxver parameters
Add minver/maxver parameters to limit patch application by comparing PV.

(From OE-Core rev: 3fac9f884ac6bcc0280e1bb5d0f0e397bb53678f)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:56 +01:00
Richard Purdie
ffae400179 meta/lib+scripts: Convert to SPDX license headers
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.

The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).

More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.

The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.

(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:55 +01:00
Paul Barker
10a14af4ca oe.path: Add copyhardlink() helper function
This function creates hard links if possible, falling back to copying
the file if the destination is on a different volume to the source.

The docstring for copyhardlinktree() is also updated to make the
difference between the two functions a little clearer.

(From OE-Core rev: 5437efa16f9bec914e417c6c939a39c247084f52)

Signed-off-by: Paul Barker <paul@betafive.co.uk>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-07 10:12:53 +01:00
Richard Purdie
244cbcce0e utils/multiprocess_launch: Improve failing subprocess output
Output before this patch:

ERROR: bash-4.4.18-r0 do_package_write_ipk: Fatal errors occurred in subprocesses:
Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1.: Traceback (most recent call last):
  File "/media/build1/poky/meta/lib/oe/utils.py", line 272, in run
    ret = self._target(*self._args, **self._kwargs)
  File "/media/build1/poky/meta/classes/package_ipk.bbclass", line 230, in ipk_write_pkg
    shell=True)
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1.

Note how stdout/stderr from the failing command isn't shown.

After this patch:

ERROR: bash-4.4.18-r0 do_package_write_ipk: Fatal errors occurred in subprocesses:
Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1.
Subprocess output:Foobar
*** Error: Package name Foobar contains illegal characters, (other than [a-z0-9.+-])

opkg-build: Please fix the above errors and try again.

We suddenly get a much more usable error message. The traceback is supressed
as its distracting from the real problem in this case.

Ideally python itself would handle this but it doesn't so we have to
wrap the exception. We already do this in bitbake itself for the same reason.

(From OE-Core rev: 09276dc76a8bda237b0b0b6d117a1980ae9dbfcc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-26 10:09:08 +01:00
Alexander Kanavin
ac5b7cef59 buildhistory: call a dependency parser only on actual dependency lists
Previously it was also called on filelists and possibly other items which
broke the parser.

(From OE-Core rev: 90bbe1bbc1667bf836d93df1e1ecca0c43315d06)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 00:05:20 +01:00
Andreas Müller
b54bff841b patch/insane: Rework patch fuzz handling
Currently there are three issues which can be enhanced:

1. Fuzz warnings cannot be configured as errors for hardening. It happened
   often to me that these warnings were overseen and detected after commits
   were already out.
2. The output is too verbose - particularly when more than one file is
   affected. Meanwhile all users should know why patch fuzz check is performed.
   So move links with background information to insane.bbclass.
3. Reduce copy & paste effort slightly by printing PN (nit: <recipe> was not
   a correct suggestion e.g for native extended recipe - see example below)

To achieve patch.py drops patch-fuzz info encapsulated by a header- and footer-
string into log.do_patch. With this insane.bbclass can drop warnings/errors
depending on 'patch-fuzz' in ERROR_QA or WARN_QA. Default remains unchanged:
Spit out warnings only.

A message for two fuzzed patches and 'pact-fuzz' in ERROR_QA now looks like:

| ERROR: autoconf-native-2.69-r11 do_patch: Fuzz detected:
|
| Applying patch autoreconf-exclude.patch
| patching file bin/autoreconf.in
| Hunk #1 succeeded at 73 with fuzz 1 (offset -3 lines).
| Hunk #2 succeeded at 143 (offset 6 lines).
| Hunk #3 succeeded at 167 (offset 6 lines).
| Hunk #4 succeeded at 177 (offset 6 lines).
| Hunk #5 succeeded at 281 (offset 15 lines).
| Hunk #6 succeeded at 399 (offset 15 lines).
| Hunk #7 succeeded at 571 (offset 20 lines).
| Hunk #8 succeeded at 612 (offset 20 lines).
| Hunk #9 succeeded at 636 (offset 20 lines).
| Hunk #10 succeeded at 656 (offset 20 lines).
| Hunk #11 succeeded at 683 (offset 20 lines).
|
| Applying patch autoreconf-gnuconfigize.patch
| patching file bin/autoreconf.in
| Hunk #1 succeeded at 55 with fuzz 1 (offset -3 lines).
| Hunk #3 succeeded at 663 (offset 18 lines).
|
| The context lines in the patches can be updated with devtool:
|
|     devtool modify autoconf-native
|     devtool finish --force-patch-refresh autoconf-native <layer_path>
|
| Don't forget to review changes done by devtool!
|
| ERROR: autoconf-native-2.69-r11 do_patch: QA Issue: Patch log indicates that patches do not apply cleanly. [patch-fuzz]

(From OE-Core rev: c762c0be43a3854a43cb4b9db559b03126d50706)

Signed-off-by: Andreas Müller <schnitzeltony@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 13:44:39 +01:00
Andre Rosa
0dd6823875 lib/oe/utils: Make prune_suffix prune a suffix
... instead of replacing a substring that could happen more than once and not only when it ends with it. Do the same for the prefix.

See related https://github.com/openembedded/bitbake/pull/24 . There it stops replacing sufixes once first one is matched but not here.

(From OE-Core rev: 610ac84170f8a91cc3321edfc336a9e39f24ebe3)

Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 13:44:39 +01:00
Enrico Scholz via Openembedded-core
1891a852b2 utils.py: added sh_quote() function
This function is a wrapper around "shlex.quote()" and can be used in
"${@...}" context where shlex (or pipes, which provides similar
functionality) is unavailable.

(From OE-Core rev: 127141f5023a7e3fc3963dc7d76cfce9067a9e8a)

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-04 22:57:32 +01:00
Jaewon Lee
206ca252c6 devtool: Support kmeta directory usage with devtool modify/finish
When using Kmeta directories, devtool finish will add every single file
in the directory to the bbappend. This is because in the current
implementation, the get_recipe_local_files function treats the kmeta
directory like a file. Modifying the function to loop through the
provided directories and return all included files instead of just the
top level directory. This will enable correct file to file comparison
when determing which files are new/changed and need to be added to the
bbappend.

Adding an extra check in devtool-source.bbclass to not copy the cfg file
if its already included somewhere in the kmeta directory

Also during 'modify', when moving necessary files in the kmeta directory
from the workdir to oe-local-files, the dangling parent directories are
left behind.  This in itself is not an issue as the temporary devtool
workspace is automatically deleted, but this causes an incorrect include
directory to be added in kernel-yocto.bbclass.  Changing the order of
the if statements to catch the correct conditional. This is safe to do
as when not in the devtool context, there will be no oe-local-files
directory.

(From OE-Core rev: 4671011b5b02d0989aa0fdcb50e18cb385a0e95e)

Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-06 10:39:25 +00:00
Alexander Kanavin
a3c4aaf9b3 lib/oe/package_manager.py: turn nativesdk postinst warnings into notes
The warnings deal with two specific cases of failure:
1) unable to execute nativesdk postinsts for mingw SDKs because
they need to be run under wine
2) unable to execute target postinsts when there is no qemu
usermode support for the target

Neither of these should be a big problem as mingw issue was never
found to problematic, and target postinsts deal with things that
are needed at runtime and not at build time which is the purpose
of SDKs.

The specific reason to do this is to reduce the amount of warnings
shown by the Yocto autobuilder, to zero eventually.

(From OE-Core rev: cbc32fcd9b52e750600cce9dd84b33e3ce612eae)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 10:43:07 +00:00
Richard Purdie
f3e56f1b57 lib/oe/utils: Fix hang in multiprocess_launch()
If large results values are returned by the subprocesses, we can hit a deadlock
where the subprocess is trying to write data back to the parent, the pipe is full
and the parent is waiting for the child to exit.

Avoid this by calling the update() method which would trigger reading a result
from the child, avoiding the deadlock. The issue is described in
https://bugs.python.org/issue8426

(From OE-Core rev: 0035e8066ecbbff94d6a1994a9f72b1368d660d2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-17 10:19:59 +00:00
Richard Purdie
e1e38c9bd1 lib/oe,oeqa/selftest: Fix DeprecationWarning: invalid escape sequence
Fix another load of regex escape sequence warnings for newer
python versions.

(From OE-Core rev: bd2c125bb9c362b6122e99dfdf4e1cfe12c26a90)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-16 08:19:34 +00:00
Ross Burton
cdd0683e5e utils: add -src to system package name blacklist
oe.utils.packages_filter_out_system() returns PACKAGES after removing "system"
packages but it doesn't handle ${PN}-src as generated by
PACKAGE_DEBUG_SPLIT_STYLE=debug-with-srcpkg.

(From OE-Core rev: 162632d3d1e40c83ed9c5d49a026edf3912860a0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-15 08:17:49 +00:00
Alejandro del Castillo
d8e12f2bfc OpkgPM: use --add-ignore-recommends to process BAD_RECOMMENDATIONS
Currently, BAD_RECOMMENDATIONS on the opkg backed relies on editing the
opkg status file (it sets BAD_RECOMMENDATIONS pkg want state to
deinstalled and pinned). This is brittle, and not consistent across the
different solver backends. Use new --add-ignore-recommends flag instead.

(From OE-Core rev: 0d11e813ba9b4e8de9e6e5099ff85f5d914243bc)

Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-08 10:57:19 +00:00
Andrej Valek
391d4ee036 lib/oe/rootfs: prelink only when image-prelink is inherited
Use prelink only when image-prelink class is inherited. Inheriting this
class adds prelink-native as dependency to do_rootfs, which makes prelink
binary and configuration available.

|Exception: FileNotFoundError: [Errno 2] No such file or directory:
|'/home/.../1.0-r0/recipe-sysroot-native/etc/prelink.conf'
|
| if not os.path.exists(prelink_cfg):
|     shutil.copy(self.d.expand('${STAGING_DIR_NATIVE}${sysconfdir_native}/prelink.conf'),prelink_cfg)

(From OE-Core rev: 9e30e72920b3c661279499f2e94d5dc0087a1a8e)

Signed-off-by: Andrej Valek <andrej.valek@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 08:29:06 +00:00
Angus Lees
dbb832b8cf package_manager.py: Use data.tar.xz for ipkg too
Commit b95b6ba1a2959e2294a8848fa35f20163388eb06 changed package_ipk.bbclass
to xz when building packages.  This updates OpkgDpkgPM.extract() accordingly.

(From OE-Core rev: c09a22c421a57701f6b943eb50b9bae1545e5b39)

Signed-off-by: Angus Lees <gus@inodes.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 08:29:06 +00:00
Jacob Kroon
404f7cec87 terminal: Fix working directory in tmux terminals
Recent version of tmux seems to use PWD from the environment instead of the
actual working directory. Help it use the correct path.

Fixes setting the correct working directory when running
devshell and menuconfig tasks.

(From OE-Core rev: 938a089b52e7bf5f67406f55234ab2d22236e9b6)

Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-05 13:53:47 +00:00
Tomasz Dziendzielski
b73dc9f6f2 lib/oe/patch.py: Fix applying a directory as a patch
If a SRC_URI content ends with '.patch' bitbake is
trying to apply it as it's a patch file.

It causes that if we use git repository for 'patch' package
the bare clone is extracted to a directory
(i.e. build/downloads/git2/git.mirror.org.patch/) which is considered
to be a patch file, so patch.py tries to apply that directory as a patch
which ends up with a failure.

(From OE-Core rev: 1e38d74a2ca7638b1f54e2bb5617903c2683e484)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-27 13:05:17 +00:00
Tomasz Dziendzielski
ad38a64a26 lib/oe/patch.py: Fix the function comment
If it's not a patch the function returns nothing.

(From OE-Core rev: cc4a75c7ebf24fb3b7b35552a090e8844985e45e)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-27 13:05:17 +00:00
Alexander Kanavin
411e2b7047 lib/oe/reciputils.py: parallelize upstream version checks
Previously this was done via bitbake tasks, and when this
was rewritten to a for loop, performance sufered significantly:
from 90 seconds to about 12 minutes for oe-core. This change
restores the previous run time, and makes it possible
to perform such checks with command line utilities in an
interactive way.

Implementation note: we have to create a copy of the recipe
data, as Tinfoil API can't be used from multiple threads
and only allows one process to access the data at a time.

(From OE-Core rev: b1d01911fa2a0a4945da071d66fb50e9f14ded81)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-26 13:39:37 +00:00
Alexander Kanavin
32309ba1de lib/oe/package_manager: turn postinst_intercept warnings into failures for nativesdk
The few cases where they failed should be now all fixed. The only allowed
exception is when building mingw32 SDKs, as there is currently no support for running
postinst_intercepts through wine.

(From OE-Core rev: 3dd2fea51110950ec3d8c444f599ff855b4b936c)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-26 13:39:37 +00:00
Joshua Watt
1d86f65ff5 classes/sstate: Update output hash
Updates the output hash calculation for determining if tasks are
equivalent. The new algorithm does the following based on feedback:
 1) The output hash function was moved to the OE library.
 2) All files are printed in a single line tabular format
 3) Prints the file type and mode in a user-friendly ls-like format
 4) Includes the file owner and group (by name, not ID). These are only
    included if the task is run under pseudo since that is the only time
    they can be consistently determined.
 5) File size is included for regular files

(From OE-Core rev: 4bd297dfe92851f3b44f6b5560bac9d8f9ccf9f2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-22 14:35:58 +00:00
Richard Purdie
cd4b8a8553 meta: Fix Deprecated warnings from regexs
Fix handling of escape characters in regexs and hence fix python
Deprecation warnings which will be problematic in python 3.8.

Note that some show up as:

"""
meta/classes/package.bbclass:1293: DeprecationWarning: invalid escape sequence \.  

"""

where the problem isn't on 1293 in package.bbclass but in some _prepend to a
package.bbclass function in a different file like mesa.inc, often from
do_package_split() calls.

(From OE-Core rev: 4b1c0c7d5525fc4cea9e0f02ec54e92a6fbc6199)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:07 +00:00
Richard Purdie
f08808340e lib/sstatesig: Update to longer sha256 hash length
Update the code to match the recent switch to sha256 hashes

(From OE-Core rev: 910a8f3e8e350ff5d316faf5e07fb40c6c51f45e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-14 11:35:56 +00:00
Alexander Kanavin
e858593584 rpm: produce sane amount of logging when creating a rootfs
Previously there was lots of irrelevant noise in the logs,
because we also wanted to be able to debug postinst issues
easily. I have adjusted the logging levels so that
postinst info is still written to the logs, but other
things are not.

[YOCTO #13119]

(From OE-Core rev: ffb7b8f70937a7d95814c1a99527d5ea7cbf7cee)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-14 11:35:55 +00:00
Robert Yang
0c98ff2c31 ccache.bbclass: Refactor it to make it more reliable
The previous ccache.bbclass has the following problems:
- It uses host's ccache for native recipes, but this may not work on some
  hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always
  build failures (m4-native failed at do_configure, and others will also be
  failed if I disable CCACHE for m4-native)

- native/nativesdk/cross/crosssdk recipes use host's ccache, but target uses
  ccache-native, this may confuse user.

- The target recipes may use both host's ccache and ccache-native, this may
  cause unexpected problems and be hard to debug. This is because ccache-native is
  in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when
  rebuild target recipes, and then it would use hosttools/ccache, but the
  previous ccache files were generated by ccache-native.

- Target recipes can't use ccache when no ccache is installed on the host:
  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"

After refactored:
All types recipes (native, target and others) will use ccache-native except
ccache-native itself, host's cache won't be used any more. It is more
reliable now, which will work everywhere when ccache-native can be built.

And now we need use "CCACHE_DISABLE = '1'" to disable ccache for the recipe
rather than "CCACHE = ''" since we set CCACHE in anonymous function, and
d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we
can't check whether CCACHE is set or not in anonymous function since it is
always set. Use CCACHE_DISABLE to disable it would be more clear.

(From OE-Core rev: b25271b65262f70d849a4861da216c9be6c54d53)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-14 11:35:55 +00:00
Richard Weinberger
c5f7615505 sdk: Fix SDKIMAGE_LINGUAS handling
Currently SDKIMAGE_LINGUAS is broken for any inputs except "all".
In the non-"all" case, each enabled language package is installed via
pm.install("nativesdk-glibc-binary-localedata-%s.utf-8" % lang)
This will throw a python exception since pm.install() expects a list of
strings and not a string.

Fix the problem by constructing a list.
That way it is now also possible to call the package installer just
once.

Cc: "Burton, Ross" <ross.burton@intel.com>
Fixes: 67615e0175 ("rootfs_rpm.bbclass: migrate image creation to dnf")
(From OE-Core rev: 475a5d9ec21a329be973691734f9e8bcb332338c)

Signed-off-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-11 10:39:09 +00:00
Robert Yang
dda8084d7a oeqa: Fix for QEMU_USE_KVM
Fixed:
MACHINE = "qemux86"
QEMU_USE_KVM = "qemux86"
IMAGE_CLASSES += "testimage"

$ oe-selftest -r runqemu.RunqemuTests.test_boot_rootfs

[snip]
  File "/buildarea1/lyang1/poky/meta/lib/oe/types.py", line 122, in boolean
    raise ValueError("Invalid boolean value '%s'" % value)
ValueError: Invalid boolean value 'qemux86'

Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more, kvm
will be enabled if target_arch == build_arch or both of them are x86 archs.

(From OE-Core rev: 7c1a8a624cad8d967635c6cb5f99cf655bde3d44)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-08 11:16:44 +00:00