sstate filename generation was changed a while ago and taskname has to be
passed into core functions for the correct filename to be generated. Update
find_siginfo to match those changes and pass in taskname via SSTATE_CURRTASK.
Thanks to Gregory Lumen <gregorylumen@microsoft.com> for spotting.
[YOCTO #14774]
(From OE-Core rev: 3230134c0e5654afccf4f8831ab1101615502162)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 51d7310b6cf8ef9033d461868c07f686656070ba)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In bitbake commit 1ecc1d94 (process: Do not mix stderr with stdout),
bb.process.Popen() was changed to no longer combine stdout and stderr by
default. However, the Terminal class was not updated to reflect this and
subsequently only output stdout in case of failures.
(From OE-Core rev: f8f8e2e159a5ac03f619e6d0882011445e6a2545)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 116d0bb07ba044cf8847bf3d5c3996ad7e58b7ae)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a remote patch is compressed we need to have run the unpack task for
the file to exist locally. Currently cve_check only depends on fetch so
instead of erroring out, emit a warning that this file won't be scanned
for CVE references.
Typically, remote compressed patches won't contain our custom tags, so
this is unlikely to be an issue.
(From OE-Core rev: b52a508bd7693c40e6416db9c9076a8789499501)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit cefc8741438c91f74264da6b59dece2e31f9e5a5)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implement debian package repository signature.
For each Release file created in repository subdirectory, a signature
Release.gpg is created.
Signature is performed using gpg backend when the following variables
are set in local.conf:
PACKAGE_CLASSES += "sign_package_feed"
PACKAGE_FEED_GPG_NAME = "<Id of GPG key>"
PACKAGE_FEED_GPG_PASSPHRASE_FILE="<path to password file>"
(From OE-Core rev: fcc3cee276999efe6402959eb295e7a0e1e96f96)
Signed-off-by: Xavier Berger <xavier.berger@bio-logic.net>
Signed-off-by: Ferry Toth <ftoth@exalondelft.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
output_suffix: If defined, add output_suffix as file name extension.
use_sha256: If True, use sha256 for gpg as digest algorithm
(From OE-Core rev: cfcaa54dc73925df448099fb60f75b18350b2a3b)
Signed-off-by: Xavier Berger <xavier.berger@bio-logic.net>
Signed-off-by: Ferry Toth <ftoth@exalondelft.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add an option to output the CVE check in a JSON-based format.
This format is easier to parse in software than the original
text-based one and allows post-processing by other tools.
Output formats are now handed by CVE_CHECK_FORMAT_TEXT and
CVE_CHECK_FORMAT_JSON. Both of them are enabled by default.
The JSON output format gets generated in a similar way to the
text format with the exception of the manifest: appending to
JSON arrays requires parsing the file. Because of that we
first write JSON fragments and then assemble them in one pass
at the end.
(From OE-Core rev: df567de36ae5964bee433ebb97e8bf702034994a)
Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add ${IMAGE_ROOTFS} and ${IMGDEPLOYDIR} to do_rootfs[dirs] and
do_rootfs[cleandirs], this ensures do_rootfs run from a clean
workspace, with this change, we can now remove two bb.utils.mkdirhier
lines from meta/lib/oe/rootfs.py.
(From OE-Core rev: cb8b6f7eee4e059bb311330c57068e11bc477366)
Signed-off-by: Ming Liu <liu.ming50@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Layers could be located anywhere. The eSDK should work with them even if
they are not located in TOPDIR or in the same parent directory as
COREBASE.
For layers located in the same parent directory as COREBASE this preserves
the intent from the previous
copy_buildsystem: include layer tree during build structure creation
commit.
Related OE-Core rev: 5a59a6997f41e606d088e3e86812de56f72f543b
(From OE-Core rev: 16d330d42e03085769eddb1b60ba1df7228baf36)
Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On modern Power systems `uname -m` yields 'ppc64le' while the toolchain
knows the architecture as 'powerpc64le'. Provide a mapping from one to
the other to integrate with the existing architecture configuration
flags.
arch_to_rust_arch() only exists to map the OE *_ARCH variables before
any further processing, unlike arch_to_rust_target_arch() which is
specific to the internal triple handling of rust.
On Linux ppc64le systems the changes give the following config:
```
$ cat ./tmp/work/ppc64le-linux/rust-native/1.58.0-r0/targets/ppc64le-linux.json
{
"llvm-target": "powerpc64le-unknown-linux-gnu",
"data-layout": "e-m:e-i64:64-n32:64-v256:256:256-v512:512:512",
"max-atomic-width": 64,
"target-pointer-width": "64",
"target-c-int-width": "64",
"target-endian": "little",
"arch": "powerpc64",
"os": "linux",
"env": "gnu",
"vendor": "unknown",
"target-family": "unix",
"linker": "gcc",
"cpu": "generic",
"dynamic-linking": true,
"executables": true,
"linker-is-gnu": true,
"linker-flavor": "gcc",
"has-rpath": true,
"has-elf-tls": true,
"position-independent-executables": true,
"panic-strategy": "unwind"
}
```
Change-Id: Ief0c01189185d7d4da31d307270bec4e1de674ca
(From OE-Core rev: 9ab61e3cfef0157393cb870d606c2f362e190889)
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want to use SPDX identifiers in LICENSE variables. There is now a
conversion script to make most of the translations. Add a list of
strings which have been replaced so we can show warnings to users
if they're still used anywhere.
Add checks to the package as insane check. This is currently a warning
by default but can be turned off or made an error as per the other standard
checks.
(From OE-Core rev: 9379f80f484f94686a4d494e9e237fadfb72a938)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This re-writes the INCOMPATIBLE_LICENSE checking code to replace
the WHITELIST_<lic> with
INCOMPATIBLE_LICENSE_EXCEPTIONS = '<pkg>:<lic> <pkg>:<lic> ...'
This initial change leaves most of the code structure in place,
but the code in base.bbclass needs to be re-written to make
the check more consistent around packages (PKGS) and not recipe
names (PN). This also is taking into account the changes for SPDX
licenses.
The aim is to provide a mode consistent variable where the variable
name is known and can easily be queried.
(From OE-Core rev: 0d19c45ba6cf43518f380ca5afe9753a2eda0691)
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is redundant (target arch is already in PN), and breaks
compiling a cross-canadian toolchain, as that needs populating the
sysroot with two different native-hosted toolchains built from
cross recipes. Inserting TARGET_ARCH allows only one or the other.
(From OE-Core rev: 33fc1792cd782feb8dbb4285e3006bb588f7978f)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous commit introduced a small logic error. Fix the renaming issue.
(From OE-Core rev: b01e10b27d23ea1b4ac58376c2423505a70832d6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were few bugs in the _isInitialized() function which might trigger
git repo to be reinitialized and patches failing to apply.
(From OE-Core rev: 80500ecda4c1bc8812e6e078b6b0db5ec46624de)
Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, SPDX SBOMs are only created for images. Add support for
SDKs.
(From OE-Core rev: c3acbb936a339636153903daf127eec9f36de79b)
Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updates within bitbake now require this call to be triggered somewhere with
floating git source revs. Add the missing call. Issue can be reproduced with:
devtool check-upgrade-status dbus-wait
(From OE-Core rev: d7088b2b5012f9e386914a6837685f5869379cc5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was prompted by ltp's unreasonably long package_qa times; it has
a massive amount of executables and insane runs objdump for all of
them, serially.
This reduces the time from 4 minutes to 1m20s on my machine.
(From OE-Core rev: fac984b99fdb46949879516cb87153860f402c75)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds the set helper for list properties to allow list members to be
replaced with a new list. This is necessary as it is (currently) the
only way to replace the default value for a list member.
(From OE-Core rev: 8280b857c510646cd10167dd2ea9a0c2d5f1e800)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prelink is being dropped by glibc in 2.36. It already causes issues with
binary corruption, has a number of open bugs and is of questionable benefit
without disabling load address randomization and PIE executables.
We disabled it by default a while back but left people able to use it.
We would be unable to maintain it alone without glibc support so remove
the remaining pieces.
(From OE-Core rev: 23c0be78106f1d1e2bb9c724174a1bb8c56c2469)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Native task outputs are directly run on the build system during the build
after being built. Even if the output of a native recipe doesn't change, a
change in one of its dependencies may cause a change in the output it
generates (e.g. rpm output depends on the output of its dependent zstd
library).
This can cause poor interactions with hash equivalence, since this
recipe's output-changing dependency is "hidden" and downstream tasks only
see that this recipe has the same outhash and therefore is equivalent.
This can result in different output in different cases and issues with
reproducible builds in parcular (e.g. rpm compression changes for the same
content).
To resolve this, unhide the output-changing dependency by adding it's
unihash to this tasks outhash calculation. Unfortunately, we don't know
specifically know which dependencies are output-changing, so we have to
add all of them.
[YOCTO #14685]
(From OE-Core rev: d6c7b9f4f0e61fa6546d3644e27abe3e96f597e2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Extend runstrip() to accept additional argument to enable
sharing it with the kernel do_strip() so that
KERNEL_IMAGE_STRIP_EXTRA_SECTIONS can be passed.
Since is_elf() understands kernel modules there is no need to keep a
seperate list for kernmodules or hardcode the values to runstrip.
(From OE-Core rev: e09a8fa931fe617afc05bd5e00dca5dd3fe386e8)
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since honister host manifest stopped to generate, i.e.
manifest file is empty but all ipks/files into sdk is
ok.
(From OE-Core rev: 79a2392f5d2a4cb6509a83afb40bca01bac59914)
Signed-off-by: Oleksiy Obitotskyy <oobitots@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If PATCHTOOL="git" has been specified but workdir is not git repo
bitbake fails to apply the patches with error message:
Command Error: 'git rev-parse --show-toplevel' exited with 0 Output:
fatal: not a git repository (or any of the parent directories): .git
Fix this by initializing the repo before patching.
This allows binary git patches to be applied.
(From OE-Core rev: 6184b56a7a0fc6f5d19fdfb81e7453667f7da940)
Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
distutils is going away and we have functionality in bitbake which can
handle these comparisions so switch to the bb.utils function.
(From OE-Core rev: fe624b520e6c75e16a8f394785ab0216341402f9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Larger systems may have large numbers of cores, but beyond a certain
point they can't all be used for compiling: whilst purely
compute-intensive jobs can be parallelised to hundreds of cores,
operations such as compressing (needs lots of RAM) or compiling (lots of
I/O) don't scale linearly.
For example, the Marvel ThunderX2 has 32 cores, each capable of
executing four threads, and can be configured with two sockets, making
256 CPUs according to Linux. Zstd using 256 threads has been seen to
fail to allocate memory during even small recipes such as iso-codes.
Add a default cap of 64 CPUs to the cpu_count() method so that extreme
parallisation is limited. 64 is high enough that meaningful gains
beyond it are unlikely, but high enough that most systems won't be
effected.
(From OE-Core rev: 765d0f25ce48636b1838a5968e2dc15de2127428)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When looking at logs involving thread pools it is useful if the threads
can be named.
(From OE-Core rev: 18342945b021608794d83ecf567afd43f4379b24)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Application can depend on several overlayfs mount points. Provide a
systemd unit application can depend on to make sure all overlays are
mounted before it is started to avoid any race conditions
(From OE-Core rev: b38e194db0c6825f28c56123cf88af94d3f52beb)
Signed-off-by: Bruno Knittel <Bruno.Knittel@bruker.com>
Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having annotations on relationship can provide additional information
about the relationship such as how it was derived.
(From OE-Core rev: d98585aa89e1d3819f8139a07fb7376ef89b37f8)
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the function a ThreadedWorker is executing raises an exception, don't
use print() as that mostly disappears. Instead, output it to the logger.
This is done using bb.mainlogger.debug directly instead of bb.debug() as
this allows us to pass the exception instance directly, which is then
incorporated into the log stream.
(From OE-Core rev: 2f1ea25c222b344dd8b784b2bc73a6540ab30274)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Extract package_qa_write_error, package_qa_handle_error and
package_qa_add_message functions from insane.bbclass to lib/oe/qa.py and
drop the package_qa_ prefixes.
Update various bbclasses to use the new functions. No import is required
since base.bbclass puts oe.qa in OE_IMPORTS.
Stop requiring callers to manually track whether a fatal error has been
encountered via a "sane" flag. Instead replace the QA_SANE variable with
QA_ERRORS_FOUND and call oe.qa.exit_if_errors or
oe.qa.exit_with_message_if_errors at the end of each task.
Inspired by discussion resulting from
https://lists.openembedded.org/g/openembedded-core/message/156793 and
https://lists.openembedded.org/g/openembedded-core/message/156900
(From OE-Core rev: f0ad152ef4cc15c042bc9eeefb6af096d054b220)
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want things to be reproduicble and the variable doesn't really change
much any more. Drop the remaining uses and make those code paths always
active.
(From OE-Core rev: d15fb02c7ee7da50e322d74bc6a545234e20c7f3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reproducibility is here to stay and needs to be part of our default workflow.
Move the remaining code to base.bbclass so it is always a first class citizen
and it is clear people need to be mindful of it.
(From OE-Core rev: abb0671d2cebfd7e8df94796404bbe9c7f961058)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To try and avoid parse/memory overhead of functions within bitbake,
move the bulk of the reproducibility functions to the function library.
(From OE-Core rev: f2fd1c9d75e774c8a5271cdc1ec6f65c4492f941)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow a user to validate sstate objects against a list of keys, instead
of just any known key in the user's keychain.
(From OE-Core rev: 52ba0c5e6e2e3d5d01dc3f01404f0ab1bb29b3b5)
Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Python 3.8 visit_Num(), visit_Str() and so on are all deprecated
and replaced with visit_Constant. We can't yet remove the deprecated
functions until we require 3.8, but we can implement visit_Constant to
silence the deprecation warnings.
(From OE-Core rev: abc93390a3f19bc4cc159c5690a478b9e2270906)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, installed packages are listed for images in image-info.txt, but
not for SDKs in sdk-info.txt. Add TOOLCHAIN_HOST_TASK and
TOOLCHAIN_TARGET_TASK to the output variables in sdk-info.txt.
Moreover, package output files for the SDK host are empty because
PKGDATA_DIR defaults to the target directory. Fix this bug and create a new
variable called PKGDATA_DIR_SDK which stores the correct path for the SDK
host package data.
(From OE-Core rev: 82e6172c1df378dff4e503aa878501c08937b5bb)
Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We hardlink some files into the build, such as licence files in
do_populate_lic tasks. Depending on the umask that the source tree
was checked out with, the group permissions would vary. This
results in inconsistent task outhashes.
Avoid this by ignoring the group/other bits unless we're under
pseudo context.
Bump the ABI numbers to ensure we don't see cache corruption from
earlier builds.
(From OE-Core rev: 2bd9c806de7e5197168360e3bd1d223a04a92291)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some files in the populate_sysroot tasks have hardcoded paths in them,
particularly if they are postinst-useradd- files or crossscripts.
Add some filtering logic to remove these paths.
This means that the hashequiv "outhash" matches correcting in more
cases allowing for better build artefact reuse.
To make this work a new variable is added SSTATE_HASHEQUIV_FILEMAP
which maps file globbing to replacement patterns (paths or regex)
on a per sstate task basis. It is hoped this shouldn't be needed
in many cases. We are in the process to developing QA tests which
will better detect issues in this area to allow optimal sstate
reuse.
(From OE-Core rev: d9852ffbbe728dac33dc081538a08af98f52fd4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 2a76082363d189880613765ad339718e3614049d.
We have an issue where x86 host builds are not matching hashes with
aarch64 host builds. We'd expect that for a given target, the target
artefacts should work regardless of the host architecture, compiler
version etc. but this isn't happening and the hashes are differing.
This is due to issues from hash equivalence.
I believe the commit being reverted was added as a test and there were
other fixes at the time which resolved these issues.
As illustration of that, different gcc versions are not cauing issues
with hash equivalence. That should be similar to the aarch64 case vs.
x86-64 and hence if we're not seeing gcc verison issues, we also don't
need this special case. As such, revert it as we don't need it and it
is in fact breaking sstate reuse cross platform.
[YOCTO #14578]
(From OE-Core rev: be67dc551ca15a6f19855e8e99848aab2a33800d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds several comments to this file to explain the intention and how it
should be used
(From OE-Core rev: 0be0ba70e68b57a3617d3c1ea4b6233b37d5d648)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is added to allow the create-spdx code to create annotations
that store values properly according to the SPDX Specification.
Initialy they will be used to track if a recipe is a native type.
(From OE-Core rev: 8db551bba619f3b3612c59270043148214a462cc)
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
collections.abc is available since 3.3 and doesn't need special handling.
(From OE-Core rev: 01152c9410ba00274c8415a5d914dc33dfd0cf46)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
