Commit Graph

64 Commits

Author SHA1 Message Date
Khem Raj
ea00dbbcf3 bitbake.conf: Switch prefix mapping to use -ffile-prefix-map
-ffile-prefix map is more comprehensive when it comes to reproducible
builds and its superset of all prefix-mapping options in compilers

This makes is cleaner and workable across gcc and clang, clang does not
support -fcanon-prefix-map and it has to be explicitly omitted when using
clang.

There are lambdas generated in templates by clang which still get the
absolute paths despite -fdebug-prefix-map, this helps with that as well.

nasm is an outlier and we have fixed it by adding -fdebug-prefix-map option
luckily we do not pass DEBUG_PREFIX_MAP to nasm, in all recipes which use
nasm either pass -fdebug-prefix-map explicitly to nasm or they rewrite it
to use nasm flags syntax.

We have discussed this in past [1]

[1] https://patchwork.yoctoproject.org/project/oe-core/patch/20230428032030.2047920-1-raj.khem@gmail.com/#10281

(From OE-Core rev: ff73fa7ef7666a6dbe34f15515bc3ab6e574c5b0)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Cc: Jacob Kroon <jacob.kroon@gmail.com>
Cc: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-23 09:47:42 +01:00
Richard Purdie
430ed4a884 lib/package/utils: Improve multiprocess_launch argument passing
The current code for multiple argument passing is horrible. Tweak the
multiprocess_launch function to only convert to a tuple if it isn't already
one, which means we can then use function arguments in a standard way.

(From OE-Core rev: 7c99f90079e722764ebdc30e8d0e781454b3a51a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-28 07:49:18 +00:00
Ross Burton
b67dce7ed0 lib/oe/package: remove PACKAGE_SNAP_LIB_SYMLINKS
This was added to OpenEmbedded in 2011[1] for the micro distro[2] and
subsequently ported to oe-core[3].  This feature aims to improve runtime
link speed by removing intermediate symlinks, i.e. libstdc++.so.6 is not
a symlink but the actual library.

However, there are several issues here:
- The meta-micro distribution has been unmaintained since 2012[4]
- The original creator of this function has confirmed that it is not in
  use anymore because the renaming doesn't affect the -dev packages, so
  on-target development or use of SDKs isn't possible.

Whilst the latter is possible to fix, the performance impact of removing
a single symlink resolution is negliable at best and the packaging code
is already very complex.

So, remove this functionality as it is known to be broken, isn't tested,
and removing it reduces complexity.

[1] https://git.openembedded.org/openembedded/commit/?id=cf7114179ead8ddff8f66e84d630811920ac9add
[2] https://git.openembedded.org/openembedded/tree/conf/distro/micro.conf
[3] https://git.openembedded.org/openembedded-core/commit/?id=600dbb7cb384c2290af38b993a9bea3a4dfc4494
[4] https://git.openembedded.org/meta-micro/

(From OE-Core rev: 9bfdf4c9eff16804588942a13ac7484059197f99)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-01 12:13:18 +00:00
Joshua Watt
f94bd6fe97 lib/package: Copy locale license
When creating split locales, copy the license from LICENSE:${PN}-locale
if set, otherwise leave it unspecified (which will result in falling
back to LICENSE)

(From OE-Core rev: 34b844d3a920574d81affbdd2096ffd7dea14067)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-25 15:37:10 +01:00
Joshua Watt
3cabb0e170 lib/package: Check incompatible licenses at packaging time
Instead of checking for incompatible licenses in the anonymous python
and setting '_exclude_incompatible-', (re)check all the packages in
populate_packages(). This ensures that all packages are processed, even
dynamically generated ones.

The use of the '_exclude-incompatible-' variable set in base.bbclass has
been the mechanism used for per-packages licenses since it was added as
a feature (although with different names for the variable throughout
history). However, since this misses dynamic packages, calling
oe.license.skip_incompatible_package_licenses() a second time on the
actual final package set is a better solution.

(From OE-Core rev: 1816c5a5e724a7f907c9afe4a47b6699ea081ace)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-25 15:37:10 +01:00
Mark Hatle
fa894486a9 package.py: Fix static library processing
When PACKAGE_STRIP_STATIC is enabled the system did not pay attention to
hardlinks.  This could trigger a race condition during stripping of static
libraries where multiple strips (through hardlinks) could run at the same
time triggering a truncated or modified file error.

The hardlink breaking code is based on the existing code for elf files, but
due to the nature of the symlinks needed to be done in a separate block of
code.

Add support for static-library debugfs hardlinking through the existing
inode processing code.

Print a note to the logs if the link target can't be found.  This isn't
strictly an error, but may be useful for debugging an issue where a file
isn't present.

(From OE-Core rev: ff371d69f60a1529ed456acb7d8e9305242e74bd)

Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-22 16:53:06 +01:00
Mark Hatle
def276f0c4 package.py: Fix static debuginfo split
Fix:
  NameError: name 'shutil' is not defined

(From OE-Core rev: 13bdd750ae54d57a5f459e4b7d8636c864978241)

Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-21 11:09:29 +01:00
Sreejith Ravi
07eb1bea80 package.py: Add Requires.private field in process_pkgconfig
Currently, the process_pkgconfig() only handles the "Requires" key
field from the .pc file and ignores the "Requires.private" field while
generating the dev dependency chain. This results in a broken dependency
list and requires the installation of recommended packages to resolve
the build dependencies when using the dev IPKs. This increases the
hard disk space usage and download time and installs many unnecessary
packages as part of the recommendations.

This patch ensures that the "Requires.private" field is also considered
when creating the dependency list for the dev IPKs. With this, the dev
IPK will have the proper dependencies listed, eliminating the need to
install recommended packages to resolve the build time dependencies.

Example: usr/lib/pkgconfig/libical.pc
----------
Libs: -L${libdir} -lical -licalss -licalvcal
Libs.private: -lpthread
Requires.private: icu-i18n
----------
Depends field generated for libical-dev
Depends: glib-2.0-dev, libical (= 3.0.7-r0)
------------

When trying to resolve the build time dependency with libical
package using “-dev” ipk generated, it will throw the below error.
-----------
Package icu-i18n was not found in the pkg-config search path.
Perhaps you should add the directory containing `icu-i18n.pc'
to the PKG_CONFIG_PATH environment variable
Package 'icu-i18n', required by 'libical', not found
-----------

This patch will fix the broken dependency list.

-------
libical-dev depends field generated with this patch
Depends: glib-2.0-dev, icu-dev, libical (= 3.0.7-r0)
-------

Other examples of packages generated with broken dev dependency.

libflac-dev : https://packages.debian.org/sid/libflac-dev
Without patch:
Depends: flac (= 1.3.3-r0), libflac, libflac++
with patch:
Depends: flac (= 1.3.3-r0), libflac, libflac++, libogg-dev

libglib2.0-dev : https://packages.debian.org/buster/libglib2.0-dev
without patch:
Depends: libffi-dev, libglib-2.0-0 (= 1:2.62.6-r0), libpcre-dev
with patch:
Depends: libffi-dev, libglib-2.0-0 (= 1:2.62.6-r0), libpcre-dev,
         util-linux-dev, zlib-dev

(From OE-Core rev: 4b5c8b7006aae2162614ba810ecf4418ca3f36b4)

Signed-off-by: Sreejith Ravi <sreejith.ravi087@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-06-27 13:03:35 +01:00
Joshua Watt
dd148a0a4e lib: package: Replace ":" in file names
The purpose of this translation API is to make sure that the file name
can be used in a bitbake variable without adding special meaning,
therefore the ":" should also be replaced since it is the override
separator

(From OE-Core rev: b87eb329f20e40c1658460af7ff649aa099f0f36)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-06-03 07:47:04 +01:00
Joshua Watt
242be0888c lib: package: Add file_reverse_translate
Adds API to reverse the removal of special characters from file names so
it can be correctly done in multiple places without open-coding it.

Replace the translation done in the package_rpm.bbclass with the new API

(From OE-Core rev: 4cb7e93c624987d146aaf626ce8e99568e938a70)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-06-03 07:47:04 +01:00
lixiaoyong
66f2f710e6 oe/package: enhance objdump command call with llvm
Replace `${HOST_PREFIX}objdump` with `${OBJDUMP}`.

When utilizing llvm for compiling packages, the invocation of GNU objdump
will consistently occur during package precess. This behavior is unfriendly
to llvm. So prefer `${OBJDUMP}` over `${HOST_PREFIX}objdump`.

(From OE-Core rev: fac4d33041d8feb041e617b9b16689c4d3a522cb)

Signed-off-by: lixiaoyong <lixiaoyong19@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-30 22:25:43 +00:00
Jonathan GUILLOT
ca25926d07 lib/oe/package: fix LOCALE_PATHS scan to create locale packages
split_locales() must only check subdirectories in paths added to
LOCALE_PATHS to avoid creating weird packages based on filenames also
present in paths.

Without such a filter, cups recipe adding ${datadir}/cups/templates to
LOCALE_PATHS creates the following incorrect packages:
- cups-locale-add-class.tmpl
- cups-locale-add-printer.tmpl
- cups-locale-admin.tmpl

(From OE-Core rev: ba3aee0d516bd066829d6edaa8d7bacdd75dd6ef)

Signed-off-by: Jonathan GUILLOT <jonathan@joggee.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-24 16:10:22 +00:00
Jonathan GUILLOT
c2a7c008e1 lib/oe/package: add LOCALE_PATHS to add define all locations for locales
Some packages may contain localized files not located in default path
${datadir}/locale. Add the new variable LOCALE_PATHS to allow a recipe
to define extra paths or even fully override the scanned directories.
LOCALE_PATHS is set at ${datadir}/locale by default to keep the exact
same behavior for the recipes which did not need modification.

(From OE-Core rev: 0ffc7cf01225743789ac30dd325fca05b9203be1)

Signed-off-by: Jonathan GUILLOT <jonathan@joggee.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-14 13:53:36 +00:00
Jonathan GUILLOT
c381ca7243 lib/oe/package: replace in place PN-locale-* packages in PACKAGES
split_locales() removes PN-locale from PACKAGES and adds PN-locale-* to the end.
As the PN-locale package typically appears before PN base package, it may result
in paths not installed in PN-locale-* packages if already catched by PN. Now
insert PN-locale-* exactly where PN-locale was existing in list to avoid such
an issue.

(From OE-Core rev: 108bc167ed0d43505af3140947a0ab79c89f0a7b)

Signed-off-by: Jonathan GUILLOT <jonathan@joggee.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-14 13:53:36 +00:00
Etienne Cordonnier
5031cf42ff package.py: fix Darwin support
- 'subprocess.Popen([d.expand("${HOST_PREFIX}otool)' requires text-mode (a more
  readable alias for the universal_newlines parameter), since otool produces
  text and the code 'out.split("\n")' expects a string, not a bytes object.
  otool is used on MacOS only, so this error isn't triggered on Linux.

- use 'startswith("darwin")' in order to support all darwin versions and not
  just specific versions (meta-darwin supports darwin21 at the moment).

(From OE-Core rev: 248ca79a6400e063c4965f9542c614bf837ff758)

Signed-off-by: Dominik Schnitzer <dominik@snap.com>
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-15 21:42:15 +00:00
Etienne Cordonnier
959b1f7de4 gdb/systemd: enable minidebuginfo support conditionally
Enabling minidebuginfo is not useful if gdb and systemd-coredump
are unable to parse it.

In order to parse it, gdb needs xz support. Systemd needs coredump enabled, as
well as elfutil enabled as well (systemd-coredump loads libdw which is part of elfutils using dlopen).

(From OE-Core rev: 0d2df803bebfd7e832ab7da54c4dacaaeeb424a9)

Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:50 +00:00
Richard Purdie
0769e8ef6b bitbake.conf: Drop oldincludedir
Autoconf defines this as:

"The directory for installing C header files for non-GCC compilers."

Whilst this is something autoconf does allow changing, I find it hard
to believe it has much use in the wild now and that headers don't get
split like this in reality, it would probably only be useful on really
old unixes.. The values are the same in our configuration anyway.

Drop the value and just use includedir everywhere.

(From OE-Core rev: 506c91cbc6a604a84e37e53ccff430436369802e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-20 15:30:52 +00:00
Adrian Freihofer
b31f99a6cb lib/oe/package: Refactor to make strip_execs callable without d
This allows to call strip_execs function from devtool without going
via tinfoil and a bitbake server process.

(From OE-Core rev: 3bde26d64a0c8c3ef8ffbcb398f2a268759321af)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-12 12:44:47 +01:00
Jean-Marie Lemetayer
fbcce87a3a package: always sort the conffiles
To improve package reproducibility, the conffiles order should not be
directly linked to the file system.

Sorting the conffiles solves this issue.

(From OE-Core rev: c7800fa825fa610327e3d2ee8c5707973b726a83)

Signed-off-by: Jean-Marie Lemetayer <j.lemetayer@kerlink.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-08-09 12:44:52 +01:00
Randolph Sapp
2f68c2d727 package: enable recursion on file globs
Enable recursion of file globs. This just allows the use of '**' in file
globs to match 0 or more subdirectories, it should not make all current
globs recursive [1].

[1] https://docs.python.org/3.6/library/glob.html#glob.glob

(From OE-Core rev: 4cd62f72dbaa0361e97d7c118752d0a2b02adfbe)

Signed-off-by: Randolph Sapp <rs@ti.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-05-25 10:29:09 +01:00
Ola x Nilsson
1d70f5b466 package: Use shlex instead of deprecated pipe
The pipe library is deprecated in Python 3.11 and will be removed in
Python 3.13.  pipe.quote is just an import of shlex.quote anyway.

(From OE-Core rev: 47731e0f7d9e92db39a3d551fe50df69c1cc6d6a)

Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-14 16:44:24 +01:00
Xiangyu Chen
810a71236d package: moving field data process before variable process in process_pkgconfig
Currently, the latest version abseil-cpp contains a new library named "absl_log_internal_format", it's
basic package config(.pc file) as below:

prefix=/usr
exec_prefix=${prefix}

......

Requires: absl_config = 20230125, absl_core_headers = 20230125, absl_log_internal_append_truncated = 20230125,
absl_log_internal_config = 20230125, absl_log_internal_globals = 20230125, absl_log_severity = 20230125,
absl_strings = 20230125, absl_str_format = 20230125, absl_time = 20230125, absl_span = 20230125
......

Normally, the process_pkgconfig() would process variable data before field data in a .pc file, but in the
absl_log_internal_format, the field data in "Requires" section contains "xxxx = xxxx" format, the
process_pkgconfig() treats them as normal variable and using the setVar() in bitbake's data_smart.py
try to process. The absl_log_internal_format field data contains "_append_", this hit the setVar() checking
and finally bitbake stop building and reporting an error as below:

"Variable xxx contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake."

This patch move the field data process before variable process to avoid the process_pkgconfig() treat the field
data as variable.

(From OE-Core rev: a73e269d3e591a10bb397b94b82e3fb960112d33)

Signed-off-by: Xiangyu Chen <xiangyu.chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-26 18:50:17 +01:00
Richard Purdie
e4f17fd72d package: Move package functions to function library
Move the bulk of the remaining package "processing" functions over
to the package function library for parsing efficiency.

(From OE-Core rev: f8785117074908330faca0b99afa7f60ed6ad952)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-05 11:52:50 +00:00
Richard Purdie
93be2cdf49 package: Move get_conffiles/files_from_filevars functions to lib
To avoid reparsing the bbclass code all the time, move the functions
to the python function library code which is more efficient.

(From OE-Core rev: 424e65627c018b3119050f515b0c7cfb43be5573)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-05 11:52:50 +00:00
Richard Purdie
ed07d52b47 package: Move fixup_perms function to bb function library
To avoid reparsing the bbclass code all the time, move the function
to the python function library code which is more efficient.

(From OE-Core rev: 7c8c4cfd9355a9cee1144e0444e1b54402f1951c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-05 11:52:50 +00:00
Richard Purdie
ce08cf4825 lib: Add copyright statements to files without one
Where there isn't a copyright statement, add one to make it explicit.
Also add license identifiers as MIT if there isn't one.

(From OE-Core rev: bb731d1f3d2a1d50ec0aed864dbca54cf795b040)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-12 12:00:43 +01:00
Saul Wold
d756b346f2 package: Add support for kernel stripping
Extend runstrip() to accept additional argument to enable
sharing it with the kernel do_strip() so that
KERNEL_IMAGE_STRIP_EXTRA_SECTIONS can be passed.

Since is_elf() understands kernel modules there is no need to keep a
seperate list for kernmodules or hardcode the values to runstrip.

(From OE-Core rev: e09a8fa931fe617afc05bd5e00dca5dd3fe386e8)

Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-13 13:57:26 +00:00
Jean-Marie LEMETAYER
3b0640c993 lib/oe/package: remove unneeded npm_split_package_dirs function
The npm_split_package_dirs function was used by the recipetool when
creating npm recipes. This is not the case anymore.

(From OE-Core rev: 6cd834737eaa42592e83300099c152e2cfef568c)

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27 16:48:09 +00:00
Richard Purdie
41a5dbd16b package: Improve determinism
Its possible in cases with multiple shlib providers we were not being
deterministic. Add in a couple of sorted() calls to fix the shlibs and
pkgconfig cases with this potential issue.

(From OE-Core rev: 5b9a4214ee17e1a39dd5a1742f2ac5ed25a11310)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-26 08:41:39 +01:00
Richard Purdie
ffae400179 meta/lib+scripts: Convert to SPDX license headers
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.

The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).

More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.

The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.

(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:55 +01:00
Richard Purdie
cd4b8a8553 meta: Fix Deprecated warnings from regexs
Fix handling of escape characters in regexs and hence fix python
Deprecation warnings which will be problematic in python 3.8.

Note that some show up as:

"""
meta/classes/package.bbclass:1293: DeprecationWarning: invalid escape sequence \.  

"""

where the problem isn't on 1293 in package.bbclass but in some _prepend to a
package.bbclass function in a different file like mesa.inc, often from
do_package_split() calls.

(From OE-Core rev: 4b1c0c7d5525fc4cea9e0f02ec54e92a6fbc6199)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:07 +00:00
Rasmus Villemoes
7b23f6124b lib/oe/package.py: use bb.utils.break_hardlinks helper
This does the same thing, but is more efficient in case st_nlinks
is (already) 1.

Depends on bitbake commit 7ae93cf40ab91965147055100432961436bce46c .

(From OE-Core rev: a09f8e32044c8daec2d2fb3ff0e830c21402df6e)

Signed-off-by: Rasmus Villemoes <rv@rasmusvillemoes.dk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23 09:04:29 +01:00
foocampo
8fc56faa5d package: skip strip on signed kernel modules
Executing strip action on kernel modules removes the signature.
Is not possible to strip and keep the signature, therefore avoid
strip signed kernel modules.

(From OE-Core rev: 4c47e5f171fa2603355e2f9183065ce8137a18c7)

Signed-off-by: Omar Ocampo <omar.ocampo.coronado@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-09 23:47:56 +01:00
Richard Purdie
adea90740d package/package_manager: multiprocess_exec -> multiprocess_launch
After this replacement, the parent exception handling works so we don't
need subprocess wrapping with bb.error in the underlying functions.

The underlying contexts also have better module handling so the imports
can be cleaned up.

(From OE-Core rev: aa8260adf53139d776a2affe6118d28b295c1fab)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24 11:52:27 +01:00
Richard Purdie
e1ba46109e package: Call file to determine elf status in parallel
This allows the calls to is_elf (which calls file) to happen in parallel
allowing a speedup of do_package and do_populate_sysroot for native
recipes.

(From OE-Core rev: bbe0d3e26484f3f347262d40a8a9d415ce21fb43)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24 11:52:27 +01:00
Richard Purdie
069a1c4a15 package: Refactor to remove isElf/is_elf function duplication
There are probably further cleanups needed here but this at least removes
the major code duplication between these two similar funcitons, keeping the
kernel module ".ko" extension check for efficiency to avoid opening and
reading file contents in the general case.

(From OE-Core rev: 7ad0c0d6ab12bebeac097fc0f5210c876dcfe9be)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24 11:52:27 +01:00
Richard Purdie
0e6a662ba2 package: Don't use subshell to execute file
We don't need any functionality from the shell here, its just extra fork
overhead. Therefore remove it and use subprocess directly.

(From OE-Core rev: bcc03ea19e103f6aa93bada2f49fcc5cc7bc0790)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24 11:52:27 +01:00
Andre McCurdy
fd5488ba75 package.py: use single quotes for path passed to file in is_elf()
Align package.py is_elf() with recent changes in package.bbclass
isELF():

  http://git.openembedded.org/openembedded-core/commit/?id=7877761534b0c2492da6289e9f2269d41b6ed464

(From OE-Core rev: ab056c7f6065f310be4dd256ceb45f85ff981f69)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-20 08:12:17 +01:00
Mark Hatle
49bfeb0a9c package.bbclass: Add '-b' option to file call in isELF
The isELF function works by running:

   result = file <pathname>
   if 'ELF' in result

By default 'file' will prepend the result with the path name of the file
that is being checked.  This usually works fine, such as:

$ file /home/foo/openembedded-core/meta/classes/package.bbclass
/home/foo/openembedded-core/meta/classes/package.bbclass: Python script, ASCII text executable, with very long lines

However, if the path includes 'ELF', ELF will end up in the result, and then
the check will return positive.

$ file /home/ELF/openembedded-core/meta/classes/package.bbclass
/home/ELF/openembedded-core/meta/classes/package.bbclass: Python script, ASCII text executable, with very long lines

This will then result in the isELF coming back true, and possibly causing the
checks that use isELF, such as the 'is it already stripped' check, to do the
incorrect thing.

Adding the '-b' option to file will result in the path being omitted in the
result:

$ file /home/ELF/openembedded-core/meta/classes/package.bbclass
Python script, ASCII text executable, with very long lines

(From OE-Core rev: 5a324e9b2cf6378f8eaa4e394f9cb36d4e2680ac)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-18 18:57:06 +01:00
Richard Purdie
55f0493e26 lib/oe/package: Improve filedeprunner subprocess handling
Currently the exit code of the spawned program isn't checked so it can
fail and the do_package task will continue merrily upon its way.

Use subprocess.check_output() to ensure we check the exit code and
redirect stderr to stdout so if it fails, we see the error output.

We can then drop the existing exception handling as the subprocess
exception gives a much better error.

(From OE-Core rev: ce11cb449222bc47fea4f6d66ff1cc7cdc529ab9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-02 00:52:10 +01:00
Tobias Hagelborn
90c1e0da59 package.py: strip_execs: Support for .ko modules
* Support stripping of .ko modules verifying file extension and
  check of content "vermagic="

(From OE-Core rev: 61a20502a8433729e2c25b8c718e6f93e3bb6614)

Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-27 22:30:06 +01:00
Tobias Hagelborn
f46240c7ba package.py: Fix some lint errors
- rename type to exec_type not to shadow type
- rename isELF is_elf

(From OE-Core rev: c72bd726d3e8495aae3e57f524c43b3be6367796)

Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-27 22:30:06 +01:00
Peter Kjellerstedt
1b5d1b36bb package.bbclass: Restore functionality to detect RPM dependencies
During the transition to dnf and rpm4, the functionality to
automatically make RPM determine dependencies was lost.

Before the transition, an OE specific tool called rpmdeps-oecore had
been added to the rpm suit. It was based on the rpmdeps tool that is
part of rpm. For each file specified on its command line, it would
output the provides and requires that RPM could determine.

During the transition to rpm4, rpmdeps-oecore was replaced with the
standard rpmdeps. However, what no one noticed was that unless rpmdeps
is given options, e.g., -P or -R, to tell it what it should output, it
will not output anything. Thus, it would do all the work to determine
the requirements, but would keep silent about it. And since no output
from rpmdeps is expected unless there are requirements, there were no
warnings indicating that everything was not working as expected.

Porting the old rpmdeps-oecore to work with rpm4 is not really
possible since it relied on being able to access internals of RPM that
are no longer available. However, it turned out that rpmdeps had a
debug option, --rpmfcdebug, that would output exactly the information
that we need, albeit in a different format and to stderr. To make this
usable, rpmdeps has now received a new option, --alldeps, which sends
the information we need to stdout.

(From OE-Core rev: 958501b3d9201aaabb81ec644c6049e0c9b737e7)

(From OE-Core rev: bf017930036f19b3d6df8e5b50d9979ee7045c5c)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-18 10:40:26 +01:00
Tobias Hagelborn
0e23081d70 package.py: Add function strip_execs
Strip all executables in a directory.
Utility function  placed in oe-package together with run_strip.
strip_execs is based on strip_sysroot from staging.bbclass
Moving out datastore references in favor of function parameters.

(From OE-Core rev: a350bfc41e8a19dfdc5b16e5fb8f2b198e7c55c1)

Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28 20:55:07 +01:00
Richard Purdie
1847d2aa40 Revert "package.bbclass: Restore functionality to detect RPM dependencies"
This reverts commit aea90e9ee6f34e7c1c08307080b1e29646668df6.

RP hadn't meant to merge this yet and its causing problems so delay it until its
ready.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-13 06:18:49 +01:00
Peter Kjellerstedt
bd07f8c9ad package.bbclass: Restore functionality to detect RPM dependencies
During the transition to dnf and rpm4, the functionality to
automatically make RPM determine dependencies was lost.

Before the transition, an OE specific tool called rpmdeps-oecore had
been added to the rpm suit. It was based on the rpmdeps tool that is
part of rpm. For each file specified on its command line, it would
output the provides and requires that RPM could determine.

During the transition to rpm4, rpmdeps-oecore was replaced with the
standard rpmdeps. However, what no one noticed was that unless rpmdeps
is given options, e.g., -P or -R, to tell it what it should output, it
will not output anything. Thus, it would do all the work to determine
the requirements, but would keep silent about it. And since no output
from rpmdeps is expected unless there are requirements, there were no
warnings indicating that everything was not working as expected.

Porting the old rpmdeps-oecore to work with rpm4 is not really
possible since it relied on being able to access internals of RPM that
are no longer available. However, it turned out that rpmdeps had a
debug option, --rpmfcdebug, that would output exactly the information
that we need, albeit in a different format and to stderr. To make this
usable, rpmdeps has now received a new option, --alldeps, which sends
the information we need to stdout.

(From OE-Core rev: aea90e9ee6f34e7c1c08307080b1e29646668df6)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 23:01:23 +01:00
Richard Purdie
c90246caeb package: Fix various rpmdeps relocation issues
There are several issues with rpmdeps after the rpm v5 -> v4 transition:
 * _rpmfc_magic_path  is an invalid option for rpm4
 * --rpmpopt is an invalid option for rpm4
 * we need to use the path to rpmrc since otherwise it poitns at the
   original build path
 * we need to set MAGIC in the environment so libmagic can find its
   files.

This patch addresses those and ensures rpmdeps works in relocated builds
from sstate (or with rm_work).

(From OE-Core rev: 806e37264d7102ae982867350ad8363ed3e5f475)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-14 14:42:18 +00:00
Anders Darander
2db1227fa2 lib/oe/package.py: remove @ from package name
@ isn't allowed in package names. Angular2 packages often have
@ in their names.

(From OE-Core rev: 3c4291bc58bcc5c66ef539eed29b7c37ac968a06)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-04 23:18:18 +00:00
Joshua Lock
c4e2c59088 meta: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Stephano Cetola
d8fbaebc48 package_manager: remove strings and migrate to direct arrays
When using subprocess call and check_output, it is better to use arrays
rather than strings when possible to avoid whitespace and quoting
problems.

[ YOCTO #9342 ]

(From OE-Core rev: b12cec9a5ef14ecb02be7feec65508cf5d65c795)

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08 10:31:30 +00:00