Commit Graph

712 Commits

Author SHA1 Message Date
Tobias Hagelborn
90c1e0da59 package.py: strip_execs: Support for .ko modules
* Support stripping of .ko modules verifying file extension and
  check of content "vermagic="

(From OE-Core rev: 61a20502a8433729e2c25b8c718e6f93e3bb6614)

Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-27 22:30:06 +01:00
Tobias Hagelborn
f46240c7ba package.py: Fix some lint errors
- rename type to exec_type not to shadow type
- rename isELF is_elf

(From OE-Core rev: c72bd726d3e8495aae3e57f524c43b3be6367796)

Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-27 22:30:06 +01:00
Jia Zhang
c0e0887001 gpg_sign: depress the owership warnings on homedir
A gpg command, e.g, called by get_gpg_version(), may trigger such a warning:
gpg: WARNING: unsafe ownership on homedir '/home/user/.gnupg'

This sort of warning is harmless so it can be depressed through specifying
--no-permission-warning.

(From OE-Core rev: 0a740aa6f02deada56e0b7deb2bc1f7401e58726)

Signed-off-by: Jia Zhang <lans.zhang2008@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-24 13:48:10 +01:00
Ioan-Adrian Ratiu
e36fbb3382 oe/path.py: copyhardlinktree: don't overwrite existing symlinks
Before GNU tar 1.24, only the archive creation command had the '-h'
argument to preserve and follow symlinks. After >= 1.24 via commit
14efeb9f956e38d7be (tar: --dereference consistency) the capability to
preserve symlinks was also added to the archive extraction command.

-h is default at archive creation but is not default at extraction,
meaning that it will replace symlinks with directories even if the
original filesystem directory tree and archive contains them.

Add -h to the copyhardlinktree extraction step so the build can
support symlinks in variables like ${DEPLOY_DIR_IPK/RPM/DEB}.

(From OE-Core rev: c1b73fe26540c38086e259be6eb313e0826dceed)

Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23 12:06:51 +01:00
Juan M Cruz Alcaraz
04dc76713b devtool/copy_buildsystem: adds meta-skeleton layer in the eSDK installation.
The eSDK installation requires the meta-skeleton layer.
The build system might use the meta-skeleton recipes as layout
to create custom recipes. An example is the recipetool script
that uses the meta-skeleton kernel recipe when creating a custom
kernel recipe.

[YOCTO #11102]

(From OE-Core rev: 5c9ef0734d23909b5694ed43cdbb205c2ba9ca95)

Signed-off-by: Juan M Cruz Alcaraz <juan.m.cruz.alcaraz@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-19 22:15:39 +01:00
Andrej Valek
1f033e4442 copy_buildsystem: include layer tree during build structure creation
When buildsystem with layer structure is going to be copied, only the last
meta-XXX layer is taken.
For example, during ext_sdk bblayers creating:
layers/oe/meta \
layers/oe/meta-oe \
layers/oe/meta-networking \
layers/oe/meta-webserver \
...
It restructured meta-oe, meta-networking,... contents into meta-oe.
Recipes from meta-oe will be on the same level like meta-networking,
meta-webserver, ... .

It should take the whole meta path instead of the last one.
layers/oe/meta \
layers/oe/meta-oe/meta-oe \
layers/oe/meta-oe/meta-networking \
layers/oe/meta-oe/meta-webserver \
...
Now the directory structure is the same like during build creation.

(From OE-Core rev: 5a59a6997f41e606d088e3e86812de56f72f543b)

Signed-off-by: Andrej Valek <andrej.valek@siemens.com>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-18 23:46:37 +01:00
Leonardo Sandoval
e461f32d68 sign_rpm: Allow pkg signing by chunks through RPM_GPG_SIGN_CHUNK
Commit d58b1d196 moved from chunk to serial signing, but neither of both approaches
allowed the user to select the chunks size. This patch allows the user to select
a chunk size through RPM_GPG_SIGN_CHUNK defaulting to BB_NUMBER_THREADS, considered
a good default. Indirectly, this change reduces the number of processes spawn
to number-of-packages/RPM_GPG_SIGN_CHUNK.

(From OE-Core rev: f7f78e73f1cd15f4233a231364b14438af758628)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-18 23:46:37 +01:00
Peter Kjellerstedt
1b5d1b36bb package.bbclass: Restore functionality to detect RPM dependencies
During the transition to dnf and rpm4, the functionality to
automatically make RPM determine dependencies was lost.

Before the transition, an OE specific tool called rpmdeps-oecore had
been added to the rpm suit. It was based on the rpmdeps tool that is
part of rpm. For each file specified on its command line, it would
output the provides and requires that RPM could determine.

During the transition to rpm4, rpmdeps-oecore was replaced with the
standard rpmdeps. However, what no one noticed was that unless rpmdeps
is given options, e.g., -P or -R, to tell it what it should output, it
will not output anything. Thus, it would do all the work to determine
the requirements, but would keep silent about it. And since no output
from rpmdeps is expected unless there are requirements, there were no
warnings indicating that everything was not working as expected.

Porting the old rpmdeps-oecore to work with rpm4 is not really
possible since it relied on being able to access internals of RPM that
are no longer available. However, it turned out that rpmdeps had a
debug option, --rpmfcdebug, that would output exactly the information
that we need, albeit in a different format and to stderr. To make this
usable, rpmdeps has now received a new option, --alldeps, which sends
the information we need to stdout.

(From OE-Core rev: 958501b3d9201aaabb81ec644c6049e0c9b737e7)

(From OE-Core rev: bf017930036f19b3d6df8e5b50d9979ee7045c5c)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-18 10:40:26 +01:00
Leonardo Sandoval
e25d7a9b9c gpg_sign: perform rpm signing serially
gpg signing in file batches (which was default to 100) is a memory expensive
computation, causing trouble in some host machines (even on production AB
as seen on the bugzilla ID). Also, in terms of performance, there is no real
gain when rpm signing is done in batches. Considering the latter issues, perform the
rpm signing serially.

Log showing errors observed recently at AB workers:

    | gpg: signing failed: Cannot allocate memory
    | gpg: signing failed: Cannot allocate memory
    | error: gpg exec failed (2)
    | /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/core2-64-poky-linux/base-passwd/3.5.29-r0/deploy-rpms/core2_64/base-passwd-dev-3.5.29-r0.core2_64.rpm:

[YOCTO #11914]

(From OE-Core rev: d58b1d196f87128892b7b624bfb725afe01581f1)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-17 00:21:14 +01:00
Markus Lehtonen
efcecc318b package_manager.py: enable dnf's repo_gpgcheck if feed signing is enabled
If package feed signing is enabled enable repo gpg signature check for
rpm repositories added via PACKAGE_FEED_URIS. This has the implication
that all repositories added via this mechanism must be signed with the
same key.

[YOCTO #11209]

(From OE-Core rev: f7716f1de0791dfe778bb70f1769a7e1e83c7a54)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16 00:03:15 +01:00
Markus Lehtonen
d27f45a692 lib/oe/package_manager: re-implement rpm feed signing
[YOCTO #11209]

(From OE-Core rev: 7a33426561ddc72296e33c5e7b8ca38763a7ab07)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16 00:03:15 +01:00
Richard Röjfors
0a3bb8b827 package_manager: Fix support for NO_RECOMMENDATONS
When support for dnf was introduced the check of the
no NO_RECOMMENDATIONS variable got broken.

This fixes the issue by compairing to the string "1"
rather than the number 1.

(From OE-Core rev: 1849ce3bd7c0af055f3e849a6508e746b6a0dca5)

Signed-off-by: Richard Röjfors <richard@puffinpack.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:16 +01:00
Leonardo Sandoval
6cbe18ff83 terminal: wait for terminal task to finish before procedding
This commit generalizes the work done in [1] and [2], both fixing
issues on several areas (the former with -c patch and gnome-terminal and
the latter with -c menuconfig and several terminals, including
gnome-terminal and tmux). The main idea is to get the PID
from the new spawned terminal and loop there until finished.

[1] 76e8ab47c9: terminal: Fix gnome-terminal to work with recent versions
[2] 7d02ea283b: cml1.bbclass: wait until menuconfig terminal finishes

(From OE-Core rev: 55707401d5bfb1f7686c273fc2d0db89df206395)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:15 +01:00
Aníbal Limón
c8330a053b oe/copy_buildsystem: check_sstate_task_list also pop BBPATH from env
The BBPATH environment could be set and can make a failure when try
to build an extensible sdk because it will look the bitbake.lock
file in the original build folder.

Example:

$ export BBPATH=`pwd`
$ bitbake core-image-minimal -c populate_sdk_ext

ERROR: bitbake failed:
ERROR: Only one copy of bitbake should be run against a build directory
ERROR: core-image-minimal-1.0-r0 do_populate_sdk_ext: Function failed:
copy_buildsystem

(From OE-Core rev: 33634b4c38d84e1c5d06056766933f1fe4f47e8d)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Lans Zhang
30ba8b6894 sign_rpm: support signing files in RPM payload
Currently, RPM4 supports to sign the files in RPM payload with plugin
mechanism. We introduce more definitions to make the file signing
available for the users:

- RPM_FILE_CHECKSUM_DIGEST
  Global switch to enable file signing.
- RPM_FSK_PATH
  The file signing key.
- RPM_FSK_PASSWORD
  The password of file signing key.
- RPM_FILE_CHECKSUM_DIGEST
  The file checksum digest.

(From OE-Core rev: 95b9ee33d5595078e90c633f6155ec9ba3d184f0)

Signed-off-by: Lans Zhang <jia.zhang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:37 +01:00
Lans Zhang
946a3dae15 gpg_sign.py: fix overwriting the RPM macro __gpg in a wrong way
The RPM macro used in --define option should not be prefixed by %%.

(From OE-Core rev: 2fda41cef26b600d599dfb7f2e61c5829e2b309c)

Signed-off-by: Lans Zhang <jia.zhang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:37 +01:00
Ross Burton
173760abc6 lib/oe/sstatesig: exclude ccache-native from signature hashes
Enabling ccache should not cause a complete rebuild, so filter out ccache-native
from the dependencies the same way we do for quilt-native (so the world doesn't
repatch if quilt changes).

This doesn't effect the actual dependencies, just the dependencies that impact
the hash.

[ YOCTO #11417 ]

(From OE-Core rev: ffeca7dcc1abe82cad8374a31bf72b36a472fa1b)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-17 14:01:37 +01:00
Amarnath Valluri
a4ada62a21 meta/lib/oe/sdk.py: support added for executing pre-target commands
Added a new POPULATE_SDK_PRE_TARGET_COMMAND variable, which can contain
functions need to be executed at pre traget sysroot creation phase.

classes/populate_sdk_base.bbclass: Added POPULATE_SDK_PRE_TARGET_COMMAND to sdk
command variables list.

(From OE-Core rev: b0c65c8a64cd0b77629c9f3c65fc827d4cdcf026)

Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-06 14:38:13 +01:00
Tobias Hagelborn
0e23081d70 package.py: Add function strip_execs
Strip all executables in a directory.
Utility function  placed in oe-package together with run_strip.
strip_execs is based on strip_sysroot from staging.bbclass
Moving out datastore references in favor of function parameters.

(From OE-Core rev: a350bfc41e8a19dfdc5b16e5fb8f2b198e7c55c1)

Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28 20:55:07 +01:00
Joshua Lock
25b410d851 lib/oe/distro_check: drop Mandriva from create_distro_packages_list()
Mandriva is no longer maintained, with its last release having been in
2011. It's no longer useful as a yard-stick distro, therefore drop it from
distro_check.create_distro_packages_list()

(From OE-Core rev: 4e1bef31e04044d26a948a9682a5122b46d15c0f)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28 15:52:16 +01:00
Joshua Lock
c36ca7c5c6 lib/oe/distro_check: update openSUSE to use Leap url formats
Since openSUSE switched to their dual development model of Leap and
Tumbleweed the urls we need to check for package lists has changed. Update
get_latest_released_opensuse_source_package_list() to use the new Leap
urls.

(From OE-Core rev: 263da717ba689eb7efe99234566e2b474e8ce95a)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28 15:52:16 +01:00
Alexander Kanavin
ee7c7150f0 package_manager.py: set dnf's releasever setting from DISTRO_CODENAME
So that:

1) dnf does not complain anymore about releasever not being set and then fail
for the same reason;

2) it's possible to refer to $releasever in dnf package feed configuration
(repo paths in particular) without hardconding the release name (pyro, morty, etc.)

(From OE-Core rev: 789e3fc225adbb61f10aaa3bbc3677856f5f0238)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23 11:44:14 +01:00
Ed Bartosh
d72227fd2b buildhistory-diff: exclude paths from the output
Implemented -e/--exclude-path command line option to
exclude paths from buildhistory-diff output.

[YOCTO #11459]

(From OE-Core rev: 86393230e0ce33bf7d6d69d3019113e704081d30)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23 11:44:12 +01:00
Martin Jansa
1460e8aa99 package_manager: flush installed_pkgs file before oe-pkgdata-util uses it
* since this commit:
  commit f5a1013ffa9815f22e13989e2bcb83f966e7ce2c
  Author: Ross Burton <ross.burton@intel.com>
  Date:   Tue Apr 18 16:19:12 2017 +0100

    package_manager: don't race on a file when installing complementary
    packages

  the file isn't closed before oe-pkgdata-util uses it and this
  temporary file might look empty to oe-pkgdata-util, because it
  wasn't flushed yet. Which resulted in almost empty debugfs tarballs
  and no locale packages in regular rootfs.
* without this change:
  124K May 30 07:41 core-image-full-cmdline-raspberrypi3-64-20170530054003-dbg.rootfs.tar.gz
* with this change:
  173M May 30 07:29 core-image-full-cmdline-raspberrypi3-64-20170530052715-dbg.rootfs.tar.gz

(From OE-Core rev: 877d38db08aa7060d16405443cf70539c559fe82)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14 14:53:58 +01:00
Richard Purdie
1847d2aa40 Revert "package.bbclass: Restore functionality to detect RPM dependencies"
This reverts commit aea90e9ee6f34e7c1c08307080b1e29646668df6.

RP hadn't meant to merge this yet and its causing problems so delay it until its
ready.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-13 06:18:49 +01:00
Peter Kjellerstedt
bd07f8c9ad package.bbclass: Restore functionality to detect RPM dependencies
During the transition to dnf and rpm4, the functionality to
automatically make RPM determine dependencies was lost.

Before the transition, an OE specific tool called rpmdeps-oecore had
been added to the rpm suit. It was based on the rpmdeps tool that is
part of rpm. For each file specified on its command line, it would
output the provides and requires that RPM could determine.

During the transition to rpm4, rpmdeps-oecore was replaced with the
standard rpmdeps. However, what no one noticed was that unless rpmdeps
is given options, e.g., -P or -R, to tell it what it should output, it
will not output anything. Thus, it would do all the work to determine
the requirements, but would keep silent about it. And since no output
from rpmdeps is expected unless there are requirements, there were no
warnings indicating that everything was not working as expected.

Porting the old rpmdeps-oecore to work with rpm4 is not really
possible since it relied on being able to access internals of RPM that
are no longer available. However, it turned out that rpmdeps had a
debug option, --rpmfcdebug, that would output exactly the information
that we need, albeit in a different format and to stderr. To make this
usable, rpmdeps has now received a new option, --alldeps, which sends
the information we need to stdout.

(From OE-Core rev: aea90e9ee6f34e7c1c08307080b1e29646668df6)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 23:01:23 +01:00
Francisco Pedraza
08a4705af9 lib/oe/sdk: Adds get_extra_sdk_info to reuse code in buildhistory
This function is going to be used for generating the target and host
manifest files packages for eSDK. Added some fixes for buildhistory.bblclass,
and docstring for get_extra_sdkinfo at oe.sdk

[YOCTO #9038]

(From OE-Core rev: f696b3bbe01969ce7ecb8174d63d3e1e172b473e)

Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 23:01:23 +01:00
Peter Kjellerstedt
3a9a5e85e8 package_manager.py: Generate correct RPM package names again
During the transition to RPM4, the package names returned by
RpmPM.list_installed() changed from the expected names of the packages
that were installed into the image to some fictitious source RPM
names.

This restores the original functionality so that the
installed-packages.txt files produced by inheriting buildhistory yet
again contains a list of the names of the installed packages.

(From OE-Core rev: 25a36bcea4c02002d4895ba17f49ced09ae821c2)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 23:01:22 +01:00
Patrick Ohly
baeafe6dcc utils.py: helper function for optional include files
The main intention is to provide easy-to-use and read helper functions
for including files only when certain distro features are
set. Functionally they are the same as bb.utils.contains and
bb.utils.contains_any.

Distro features are part of the base configuration and thus safe to
use for conditional inclusion in recipes and bbappends, in contrast to
recipe variables which might still change during parsing. Therefore
the check is limited to DISTRO_FEATURES. This is the reason for having
this in OE-core instead of bitbake.

Default values are set so that no redundant parameters have to be
passed for conditional includes. As a secondary usage, the functions
can also be used in boolean checks.

(From OE-Core rev: 13024ce5aae453769b546d5fbe533443aec3d6fd)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:08:32 +01:00
Patrick Ohly
23ecae6d67 archiver.bbclass: do not cause kernel rebuilds
Adding or removing archiver.bbclass from a build configuration causes
rebuilds of linux-yocto-based kernels because of the
do_kernel_configme->do_unpack_and_patch task dependency.

This particular dependency can be ignored for the do_kernel_configme
sstate signature calculcation. Idea for the fix from Richard Purdie.

Note that building the kernel and adding archiver.bbclass later to
archive sources leads to do_unpack_and_patch running after
do_kernel_configme (because that already ran in the first build),
which might be problematic. This is independent of the change here.

The use case in YOCTO #11441 is to removed archiver.bbclass between a
production build with archiving enabled and builds via oe-selftests
without archiving. That direction is fine.

Fixes: YOCTO #11441

(From OE-Core rev: fed0ed82928e6a7846fbad233ac657bd17bcefc7)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-18 14:01:46 +01:00
Markus Lehtonen
ef8c15852c oe.lsb: add get_os_release()
Move get_os_release() from oeqa.utils.metadata to oe.lsb, merging the
code with release_dict_osr() from oe.lsb. This removes some code
duplication and makes get_os_release() more robust.

(From OE-Core rev: 56b883f7765f6bd72e83dec26a5db8c7108c835d)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-16 14:08:28 +01:00
Richard Purdie
a11e87f179 useradd: Create lib/oe/useradd function library
The code in useradd-staticids is generally useful for user addition
functionality but is not reusable in its current form. Refactor the
code into a set of library functions.

(From OE-Core rev: a638ef304b1f1acb4c88f4f90b1ef22526cb8d2f)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-16 14:08:28 +01:00
brian avery
ecd485650c meta: add search, replace strings to export2json
We want to be able to save relative paths so that we can relocate the
deploy dir images and kernels, yet still have qemu and testimage work
correctly.  This extends export2json with 2 named arguments so a
search/replace operation can be done to remove the leading path.

[YOCTO #11375]

(From OE-Core rev: 4829f1ebd89dc91860cf72fbbdc7b6bb0d5822bc)

Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-20 07:55:26 +01:00
Jussi Kukkonen
adea8003ab package_manager.py: Reverse rpm arch order
The architecture list used by dnf/libsolv was in the wrong order.
As a result, the images were built with wrong and unpredictable
packages.

$ MACHINE=intel-corei7-64 bitbake core-image-sato
$ MACHINE=qemux86-64 bitbake core-image-sato
$ MACHINE=intel-corei7-64 bitbake -ccleansstate core-image-sato
$ MACHINE=intel-corei7-64 bitbake core-image-sato

The first image had 0 core2_64 packages in it, but the last one had
583 core2_64 packages (which were built for the qemu image in
between).

Reverse the arch order in etc/dnf/vars/arch.

Fixes [YOCTO #11384].

(From OE-Core rev: 4a82433de42943f8219beca3286f40b67157172f)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-20 07:55:25 +01:00
Ross Burton
ae5d643114 package_manager: don't race on a file when installing complementary packages
PackageManager.install_complementary() uses WORKDIR/installed_pkgs.txt as a
temporary file but if two tasks are executing for the same recipe which uses
this file (e.g. bitbake my-image my-image:do_populate_sdk) then it's possible
for the file to be overwritten or deleted.

Instead of using a static filename, use tempfile to generate a unique name and
ensure it is cleaned up when finished.

Also move the glob generation/expansion earlier in the function as if there are
no globs to install, we don't need to generate a package list.

(From OE-Core rev: f5a1013ffa9815f22e13989e2bcb83f966e7ce2c)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-19 10:18:43 +01:00
Ian.Arkver
2bc2e67dd0 package_manager.py: Generate separate repo entries per arch
dnf requires a serparate repo for each architecture. This patch
writes one config file per PACKAGE_FEED_URIS entry with an entry
for each architecture, if any.

It also uses a space separated version of the repo id as the repo
name instead of just the id again.

(From OE-Core rev: 3eed822b5c5661aa9f43af6582c1481bacf0d39a)

Signed-off-by: Ian.Arkver <ian.arkver.dev@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-14 09:47:08 +01:00
Ian.Arkver
f2e41a197f package_manager.py: Add a name for the generated dnf repo.
This adds repo_name as the name, which is the same as the
repo ID and hence a bit pointless, but it stops dnf from
complaining that the repo doesn't have a configured name.

(From OE-Core rev: c92cff23893e87cfb7ef9f77b16761310104543a)

Signed-off-by: Ian.Arkver <ian.arkver.dev@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:09:58 +01:00
Paul Eggleton
1f19d9dfe5 buildhistory-diff: add option to compare actual signature differences
Use the code underpinning bitbake-diffsigs to add an option to
buildhistory-diff to determine and display the differences between the
actual signature inputs, with a twist - we collapse identical
changes across different tasks, showing only the most recent task to
have that difference, meaning that there's less noise to wade through
when you just want to know what changed in order to cause some
rebuilding you're seeing.

(From OE-Core rev: 86cb4b01f2020553902554e512c02147eb4e0f51)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:10:17 +01:00
Paul Eggleton
9049c09793 buildhistory-diff: add option to compare task signature list
Having added writing out of the task signature list to buildhistory
(when BUILDHISTORY_FEATURES includes "task"), we now need a way to
compare the list. This just shows which tasks have been added / changed
signature / removed.

(From OE-Core rev: 63bd7e9f780a98dda458d612877495756bcc5463)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:10:17 +01:00
Paul Eggleton
6774995322 classes/buildhistory: write out task signatures on every build
If we want to determine what changed since the last build, one angle
from which to look at it is to check the signatures. However, if we
don't actually have the signatures from the last build we don't have
anywhere to start. Save the signatures on each build in order to give us
the starting point.

NOTE: you need to set your BUILDHISTORY_FEATURES value to include
"task" to enable collection of these signatures as it is is disabled by
default.

(From OE-Core rev: 11f68f65c46c5bc26ddeeade3021e83b3a7f895a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:10:17 +01:00
Ian.Arkver
ae65b95354 package_manager.py: Split feed_archs for RPM repo URIs
By default the feed_archs variable is split into single characters
resulting in very many broken short repo_uris. Add a split() to
split the string into words first.

(From OE-Core rev: 7bd6a4d56c08bf7f9ced7c07327e6163216fee08)

Signed-off-by: Ian.Arkver <ian.arkver.dev@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:10:17 +01:00
Juro Bystricky
34db499e6b classutils.py: deterministic sorting
The method "prioritized" returns a list sorted by the value
of the "priority" field, in descending order. However, if several
list items have the same priority, the ordering of those items
within the priority-sorted list becomes random. As a consequence,
we may end up with a non-deterministic oe-terminal spawning, as several
terminals have the same priority. So running commands such as

$ bitbake xxx -cdevshell
$ bitbake yyy -cmenuconfig

may spawn a different terminal each time, for example sometimes Gnome
and sometimes Konsole as hey have the same priority.
Rather than modifying the priorities so they all differ, we sub-sort
the list based on (terminal) names.
This way we achieve a deterministic outcome.

[YOCTO#10938]

(From OE-Core rev: 82ae62f28dd2d5fb2fb261478ac6161315ac6c38)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:10:17 +01:00
Paul Eggleton
23095a6b01 lib/oe/sstatesig: avoid reporting duplicate siginfo files from sstate
In find_siginfo(), which is used by bitbake-diffsigs among other things,
avoid adding a siginfo file from the sstate-cache where we've already
collected a sigdata file from the stamps directory with the same hash.
This avoids the possibility that the top two files (as picked by default
using the bitbake-diffsigs -t option) are for the same signature and
thus the tool would report no differences. In order to do that, just use
the hashfiles dict that we already have - we just need to change the
code to populate that even if we're collecting matching files without
looking for a fixed set of hashes (i.e. taskhashlist isn't set).

This replaces previous code in bitbake-diffsigs that attempted to filter
these out with limited success.

(From OE-Core rev: 5b69eef40868180c59400624096d7ebbbbea446b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:44 +01:00
Paul Eggleton
27b7de94f0 lib/oe/sstatesig: fix finding native siginfo files in sstate-cache
When comparing signatures with bitbake-diffsigs -t or bitbake -S
printdiff, we use this find_siginfo() function implemented in this
module to find the siginfo/sigdata files corresponding to the tasks
we're looking for. However, native sstate files go into a
NATIVELSBSTRING subdirectory and there was no handling for this when
asking about native recipes.

I'm not even sure why we were walking SSTATE_DIR in order to find
this - we don't need to, we just need to run glob.glob() on the filespec
we calculate, which should be a little bit more efficient.

(From OE-Core rev: 8cb472e4ed25e56ec0d9cf6d8d101d1ab6687a5b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:44 +01:00
Martin Jansa
fcbad38193 package_manager.py: respect OPKGLIBDIR
* respect it for incremental rootfs generation
* add lists_dir option to opkg.conf
* also fix setting info_dir and status_file when they use default value, the
  problem is that self.opkg_dir is already prefixed with rootfs directory,
  comparing it with /var/lib/opkg always returned false and the options were
  appended to config file unnecessary
* with opkg 0.3.4 we can use VARDIR prefix added in:
  commit d2a8e23dc669adc398f4bb8bcfcabfcf925708f7
  Author: Florin Gherendi <floring2502@gmail.com>
  Date:   Mon Dec 19 12:25:38 2016 +0200
  libopkg: make the /var and /etc directories configurable at compile time.

(From OE-Core rev: b14c11d062872c3dcf95e03b61017005dea5b754)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-24 23:43:32 +00:00
Martin Jansa
da8369b670 rootfs.py: Respect OPKGLIBDIR variable
* when OPKGLIBDIR doesn't have the default /var/lib value it will
  silently fail to copy package database from normal rootfs to debugfs
  rootfs and then when trying to install *-dbg complimentary packages
  it won't install anything, because installed_pkgs.txt file generated
  from debugfs is empty

(From OE-Core rev: bebdb056c8bd0efc056f07b275a154e5d5a3aa2b)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-24 23:43:32 +00:00
Alexander Kanavin
4581e2c3e6 sign_rpm.bbclass: do not set/use RPM_GPG_PUBKEY
This is entirely unnecessary (we can ask the signer backend to export the
key to a file when needed), and was causing confusing selftest failures
due to the variable being set from two different places.

[YOCTO #11191]

(From OE-Core rev: 74ea979044368dc28c24325e7e77471b70aa8fe8)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-23 13:19:49 +00:00
Joshua Lock
cc2744267f meta: replace uses of bb.data.expand(VARNAME, d) with d.expand(VARNAME)
bb.data.expand(x, d) is deprecated API.

[YOCTO #10678]

(From OE-Core rev: a361babe443da635aed83be46679067457fd6a58)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:19 +00:00
Robert Yang
08dfb89301 oe/path.py: fix for "Argument list too long"
Issue: LIN9-1648

Fixed when len(TMPDIR) = 410:
$ bitbake core-image-sato-sdk
[snip]
Subprocess output:
/bin/sh: /bin/cp: Argument list too long

ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs
[snip]

This is because "copyhardlinktree(src, dst)" does "cp -afl src/* dst",
while src/* is expanded to "src/file1 src/file2, src/file3..." which
causes the "Argument list too long", use ./* as src and change cwd in
subprocess.check_output() to fix the problem.

(From OE-Core rev: a3dc93eb25fba32109edd1db6e8766074fb52e4b)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-17 16:53:05 +00:00
Paul Gortmaker
52d2faad9f package_manager: fix "exlcude" vs. "exclude" typo
Which results in:

 --------------------------------
      0557:        package_exclude = self.d.getVar('PACKAGE_EXCLUDE')
  *** 0558:        exclude_pkgs = (bad_recommendations.split() if bad_recommendations else []) + (package_exlcude.split() if package_exclude else [])
      0559:
      0560:        output = self._invoke_dnf((["--skip-broken"] if attempt_only else []) +
      0561:                         (["-x", ",".join(exclude_pkgs)] if len(exclude_pkgs) > 0 else []) +
      0562:                         (["--setopt=install_weak_deps=False"] if self.d.getVar('NO_RECOMMENDATIONS') == 1 else []) +
 Exception: NameError: name 'package_exlcude' is not defined
 ERROR: cube-builder-initramfs-1.0-r0 do_rootfs: Function failed: do_rootfs
 ---------------------------------

Cc: Alexander Kanavin <alexander.kanavin@linux.intel.com>
(From OE-Core rev: cee1bdc09f4bbfedcd7cac06b48ba9d195c29e62)

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-17 16:53:04 +00:00