Commit Graph

1105 Commits

Author SHA1 Message Date
Alejandro Hernandez Samaniego
26dbc1f641 rootfs.py: dont try to list installed packages for baremetal images
Theres not a rootfs for baremetal images, hence we should avoid
trying to list rootfs packages for them.

This fixes an issue where some classes (e.g. license_image) rely on
rootfs functionality when included for baremetal images even if its
nonexistent

(From OE-Core rev: e713e118b46aa0c8a5015b915a93718b7aa20a74)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandro@enedino.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5a7e13f46083ce3b08aa762238c1e93b7626dda4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-16 17:53:23 +01:00
Enrico Scholz
bf1987dbe5 lib:npm_registry: initial checkin
Helper module to:

- generate meta information from package.json content.  This data has
  a format as provided by https://registry.npmjs.org

- put this meta information and the corresponding tarball in the
  nodejs cache.  This uses an external, nodejs version specific helper
  script (oe-npm-cache) shipped in oe-meta

To avoid further nodejs version dependencies, future versions of this
module might omit the caching completely and serve meta information
and tarball by an http server.

(From OE-Core rev: 17132402031f4659db5cc1f84263278b82b27ffa)

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6cd5886ad05fee704e8a5892bd370c360c8c3b54)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-12 08:41:48 +01:00
Ernst Sjöstrand
6413ed11f8 cve-check: Don't use f-strings
Since we're keeping cve-check aligned between the active branches,
and dunfell is supported on Python 3.5, we can't use f-strings.

(From OE-Core rev: 057a6f826bd45ebf240df47f0baf7c8027f5711c)

Signed-off-by: Ernst Sjöstrand <ernstp@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1821cf7464cbba521b55a9c128fe8812c0cc5eca)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-28 07:51:29 +01:00
Mihai Lindner
d8c792f12c create-spdx: Fix supplier field
The correct field name is "supplier" according to SPDX schema.
The "supplier" field translates to "PackageSupplier", but that's for
tag-value format.

(From OE-Core rev: 504b50aec662f177fea452e05e29af8b36ca69fc)

Signed-off-by: Mihai Lindner <mihai.lindner@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ca8db0e0a2860ac1e3f537471fa71b43c3be0a58)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-23 15:22:52 +01:00
Shruthi Ravichandran
f6bc8dfadb package_manager/ipk: do not pipe stderr to stdout
Some opkg commands print an error during cleanup when the tmp_dir
does not exist and an attempt is made to delete it. The error messages
are harmless and the opkg commands eventually succeed.
When these commands are run and stderr is piped to stdout, the error
messages may clobber the stdout and cause unexpected results while
parsing the output of the command. Therefore, when parsing the output
of a command, do not pipe stderr to stdout. Instead, capture stderr
and stdout separately, and upon success, send stderr to bb.note().

(From OE-Core rev: fd5689696731fefa0d035fde86f27a0135dc31f1)

Signed-off-by: Shruthi Ravichandran <shruthi.ravichandran@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f2167ae80258253eb47a5b148546b265320284cc)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-08 16:23:37 +01:00
Richard Purdie
a78211a182 base/reproducible: Change Source Date Epoch generation methods
In bc, we found that there are files newer than Changelog (e.g. scan.l)
which means after install runs, the timestamp is clamped to SDE which means
"bitbake bc -C compile" would then try and regenerate generated files and
complain flex was missing.

Rather than assuming Changelog/NEWS are magic for dates, drop that scan
method and scan all unpacked files. This shouldn't be that slow as they'd
be in the disk cache already after an unpack.

For bc, this changes SDE by about two minutes and avoids the problems
rebuilding.

Add a version comment to the task definition as changes in python library
code won't trigger a rebuild and we need one here.

(From OE-Core rev: 400ed3bc77232b9d67594fa012b1278a6c8be3c6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 32dda0ad91a9e7946351c897578b4c97ae142341)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-04 16:29:15 +01:00
Joshua Watt
ca3a1f17cc sstatesig: Include all dependencies in SPDX task signatures
SDPX generation involves looking through BB_TASKDEPDATA for
dependencies, then linking to the generated documents for those
dependencies. These document links use a checksum to validate the
document, which means that if a upstream document changes, all
downstream documents must be regenerated to get the new checksum,
otherwise the compendium of documents produced by the build will have
broken links; therefore all dependent task should be included in the
signature (even from "ABI safe" recipes).

(From OE-Core rev: 7549429fc93218dee33b216010b2c36a9f814091)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit 5fe543b9ceec971cf0297ff0ae3b0ccc4703cece)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-01 16:27:30 +01:00
Paul Eggleton
ee60960da3 devtool: finish: handle patching when S points to subdir of a git repo
If devtool finish needs to create a patch and have it applied to the
sources for a recipe where S points to a subdirectory of the sources,
then the patch needs to be applied at the root of the repo i.e. we need
to add a patchdir= parameter to the SRC_URI entry.

(From OE-Core rev: 7cf53810b1dc3d14c4838a610b3d53170f552c19)

Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ad3736d9ca14cac14a7da22c1cfdeda219665e6f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-07-25 15:11:46 +01:00
Paul Eggleton
9e7a56bef7 patch: handle if S points to a subdirectory of a git repo
If PATCHTOOL = "git", SRC_URI fetches from a git repo and S points to
a subdirectory of the checked out sources, then we were erroneously
initialising the subdirectory as its own git repo. Check if the returned
top-level repo directory is a subdirectory of WORKDIR and do not
run initialise the source directory if that is the case.

(This was a regression introduced with OE-Core revision
6184b56a7a0fc6f5d19fdfb81e7453667f7da940, however we didn't have a test
that verified the behaviour.)

(From OE-Core rev: 577a69137eac6a44869d384b9027fbfdfea5740e)

Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9cca53a2bcbf6809615ce5626c86c6ee481a7a76)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-07-25 15:11:46 +01:00
Martin Jansa
4aeda14352 rootfs.py: close kernel_abi_ver_file
* fixes:
  oe-core/meta/lib/oe/rootfs.py:331: ResourceWarning: unclosed file <_io.TextIOWrapper name='pkgdata/mach/kernel-depmod/kernel-abiversion' mode='r' encoding='UTF-8'>
  kernel_ver = open(kernel_abi_ver_file).read().strip(' \n')

(From OE-Core rev: 0f7a8359ba370c7f5d5153453ed699e9566f5b1d)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f9dd8ee063c1132265248457fcd628e1e93727be)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-18 16:34:41 +01:00
Sean Anderson
6c51379206 rootfs.py: find .ko.zst kernel modules
With CONFIG_MODULE_COMPRESS_ZSTD enabled, kernel modules will have a
.ko.zst extension. This fixes depmod not being run.

Fixes: 1b696a45ddb ("rootfs.py: Add check for kernel modules before running depmod")
(From OE-Core rev: 6ade2bfdd9297896b9f251dd62f55e461fef1a3e)

Signed-off-by: Sean Anderson <sean.anderson@seco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 425efac7110f0f42d70643e0a448e834d0f01a7a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-11 10:06:13 +01:00
Martin Jansa
411fbde5ac patch.py: make sure that patches/series file exists before quilt pop
* Since quilt upgrade to 0.67 some recipes sometimes fail in do_patch with
  errors like:

  ERROR: Applying patch 'GPLv2.patch' on target directory '/OE/build/oe-core/tmp-glibc/work/qemux86_64-oe-linux/keymaps/1.0-r31'
  CmdError('quilt --quiltrc /OE/build/oe-core/tmp-glibc/work/qemux86_64-oe-linux/keymaps/1.0-r31/recipe-sysroot-native/etc/quiltrc push', 0, 'stdout:
  stderr: File series fully applied, ends at patch GPLv2.patch
  ')

* It affects only recipes with S = "${WORKDIR}", which wipe only
  ${S}/patches, because in other cases whole ${S} is wiped when
  do_unpack is re-executed.

* It was originally added in:
  https://git.openembedded.org/openembedded-core/commit/?id=5fe5e6a15f26f23f0c5b863fafad7a0d382a55e2

  since then it was extended to wipe whole ${S} when ${S} != ${WORKDIR} in:
  https://git.openembedded.org/openembedded-core/commit/?id=5fe5e6a15f26f23f0c5b863fafad7a0d382a55e2
  https://git.openembedded.org/openembedded-core/commit/?id=eccae514b71394ffaed8fc45dea7942152a334a1

  this is now causing issues to quilt-0.67 because it checks that
  ${S}/patches/series exists during 'quilt pop -a -f' which we call
  from QuiltTree.Clean to undo patches possibly already applied
  in ${S} in previous do_patch execution.

* There are couple recipes affected by this e.g. keymaps (.patch already
  removed in oe-core), makedevs (.patch removal sent to ML yesterday
  https://lists.openembedded.org/g/openembedded-core/message/166172),
  devmem2
  (https://lists.openembedded.org/g/openembedded-devel/message/97270), but
  there are other recipes with S = "${WORKDIR}" where you can trigger this
  e.g. by having a .patch file in DISTRO layer .bbappend (e.g. tzdata with
  webOS
  06e5298d9f/meta-webos/recipes-extended/tzdata/tzdata.bbappend (L10))

  This do_patch issue is caused by:
  https://git.savannah.nongnu.org/cgit/quilt.git/commit/?id=8b39a960afcf45cd4f5804ae62b6b0656bdb191d
  introduced in kirkstone with:
  https://git.openembedded.org/openembedded-core/commit/?h=kirkstone&id=fa71afcee9ab42198c619333b77a15bd2ae02b20

  The shortest sequence to reproduce this is just
  bitbake keymaps -c patch
  bitbake keymaps -c unpack -f
  bitbake keymaps -c patch
  with
  https://git.openembedded.org/openembedded-core/commit/?id=17d981005a0c0c97702ad88602b7181b69bcc9eb
  reverted.

  And the change in quilt behavior is causing QuiltTree.Clean (quilt pop -a -f) in:
  https://git.openembedded.org/openembedded-core/tree/meta/lib/oe/patch.py?id=17d981005a0c0c97702ad88602b7181b69bcc9eb#n601

  to silently fail with "No series file found" before undoing the
  patches in ${S} and then quilt push failing, because all the
  patches are _still_ applied in ${S}.

  Removing ".pc" doesn't help, because we really
  need quilt's help to undo the patches (in this case to delete COPYING
  file from WORKDIR before applying the .patch which tries to add it
  again), because do_unpack cannot just wipe S and start over (because S
  == WORKDIR) - nor selectively removing the files listed in SRC_URI,
  because COPYING file isn't listed there.

  Using skip_series_check in 'quilt pop' (partially reverting the change
  from upstream) does fix this as well and it's simple one line patch
  (just adding skip_series_check=1 in pop.in), but might be difficult
  to upstream, because it's this strange OE specific behavior that we
  remove 'patches' directory and then still need quilt pop to work.

(From OE-Core rev: 5323f0951923f201b0ad8f78aa3f42b8c3c97a17)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c9d36882044b1c633d8611a77df54cd68c9bee25)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-11 10:06:13 +01:00
Marta Rybczynska
7f22fb48eb cve-check: move update_symlinks to a library
Move the function to a library, it could be useful in other places.

(From OE-Core rev: 66bf08f2df478a909a12756c650dcc2ae09a0a47)

Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit debd37abcdde8788761ebdb4a05bc61f7394cbb8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-11 10:06:13 +01:00
Richard Purdie
983c257517 lib/sstatesig: Fix find_siginfo to match sstate filename generation
sstate filename generation was changed a while ago and taskname has to be
passed into core functions for the correct filename to be generated. Update
find_siginfo to match those changes and pass in taskname via SSTATE_CURRTASK.

Thanks to Gregory Lumen <gregorylumen@microsoft.com> for spotting.

[YOCTO #14774]

(From OE-Core rev: 3230134c0e5654afccf4f8831ab1101615502162)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 51d7310b6cf8ef9033d461868c07f686656070ba)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-12 16:44:05 +01:00
Peter Kjellerstedt
f8a450a39d terminal.py: Restore error output from Terminal
In bitbake commit 1ecc1d94 (process: Do not mix stderr with stdout),
bb.process.Popen() was changed to no longer combine stdout and stderr by
default. However, the Terminal class was not updated to reflect this and
subsequently only output stdout in case of failures.

(From OE-Core rev: f8f8e2e159a5ac03f619e6d0882011445e6a2545)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 116d0bb07ba044cf8847bf3d5c3996ad7e58b7ae)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-04 13:07:33 +01:00
Ross Burton
498bbee789 cve_check: skip remote patches that haven't been fetched when searching for CVE tags
If a remote patch is compressed we need to have run the unpack task for
the file to exist locally.  Currently cve_check only depends on fetch so
instead of erroring out, emit a warning that this file won't be scanned
for CVE references.

Typically, remote compressed patches won't contain our custom tags, so
this is unlikely to be an issue.

(From OE-Core rev: b52a508bd7693c40e6416db9c9076a8789499501)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit cefc8741438c91f74264da6b59dece2e31f9e5a5)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-04 13:07:31 +01:00
Ferry Toth
0b4231b597 package_manager: sign DEB package feeds
Implement debian package repository signature.
For each Release file created in repository subdirectory, a signature
Release.gpg is created.

Signature is performed using gpg backend when the following variables
are set in local.conf:
PACKAGE_CLASSES += "sign_package_feed"
PACKAGE_FEED_GPG_NAME = "<Id of GPG key>"
PACKAGE_FEED_GPG_PASSPHRASE_FILE="<path to password file>"

(From OE-Core rev: fcc3cee276999efe6402959eb295e7a0e1e96f96)

Signed-off-by: Xavier Berger <xavier.berger@bio-logic.net>
Signed-off-by: Ferry Toth <ftoth@exalondelft.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-05 22:25:13 +01:00
Xavier Berger
bd8f1f7787 gpg-sign: Add parameters to gpg signature function
output_suffix: If defined, add output_suffix as file name extension.
use_sha256: If True, use sha256 for gpg as digest algorithm

(From OE-Core rev: cfcaa54dc73925df448099fb60f75b18350b2a3b)

Signed-off-by: Xavier Berger <xavier.berger@bio-logic.net>
Signed-off-by: Ferry Toth <ftoth@exalondelft.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-05 22:25:13 +01:00
Marta Rybczynska
777f1d42b6 cve-check: add json format
Add an option to output the CVE check in a JSON-based format.
This format is easier to parse in software than the original
text-based one and allows post-processing by other tools.

Output formats are now handed by CVE_CHECK_FORMAT_TEXT and
CVE_CHECK_FORMAT_JSON. Both of them are enabled by default.

The JSON output format gets generated in a similar way to the
text format with the exception of the manifest: appending to
JSON arrays requires parsing the file. Because of that we
first write JSON fragments and then assemble them in one pass
at the end.

(From OE-Core rev: df567de36ae5964bee433ebb97e8bf702034994a)

Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-30 13:07:41 +01:00
Ming Liu
bbc60671ad image.bbclass: make sure do_rootfs run from a clean workspace
Add ${IMAGE_ROOTFS} and ${IMGDEPLOYDIR} to do_rootfs[dirs] and
do_rootfs[cleandirs], this ensures do_rootfs run from a clean
workspace, with this change, we can now remove two bb.utils.mkdirhier
lines from meta/lib/oe/rootfs.py.

(From OE-Core rev: cb8b6f7eee4e059bb311330c57068e11bc477366)

Signed-off-by: Ming Liu <liu.ming50@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-13 12:23:48 +00:00
Saul Wold
d9e500f83d meta/scripts: Improve internal variable naming
Update internal variable names to improve the terms used.

(From OE-Core rev: f408068e5d7998ae165f3002e51bc54b380b8099)

Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-10 08:00:28 +00:00
Andrew Jeffery
ef518a327d ipk: Decode byte data to string in manifest handling
```
File: '/home/andrew/src/openbmc/openbmc/meta/lib/oe/package_manager/ipk/manifest.py', lineno: 69, function: create_full
     0065:        output = pm.dummy_install(pkgs_to_install)
     0066:
     0067:        with open(self.full_manifest, 'w+') as manifest:
     0068:            pkg_re = re.compile('^Installing ([^ ]+) [^ ].*')
 *** 0069:            for line in set(output.split('\n')):
     0070:                m = pkg_re.match(line)
     0071:                if m:
     0072:                    manifest.write(m.group(1) + '\n')
     0073:
Exception: TypeError: a bytes-like object is required, not 'str'
```

Change-Id: Ifefb13bfa22c766d20ab9f73f7abe5163b3df86f
(From OE-Core rev: cf9df9e8d89fee9cea4785c94a1e3004a5f3469d)

Signed-off-by: Andrew Jeffery <andrew@aj.id.au>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-04 17:14:15 +00:00
Andrew Jeffery
1366335749 ipk: Import re in manifest module
```
File: '/home/andrew/src/openbmc/openbmc/meta/lib/oe/package_manager/ipk/manifest.py', lineno: 67, function: create_full
     0063:
     0064:        output = pm.dummy_install(pkgs_to_install)
     0065:
     0066:        with open(self.full_manifest, 'w+') as manifest:
 *** 0067:            pkg_re = re.compile('^Installing ([^ ]+) [^ ].*')
     0068:            for line in set(output.split('\n')):
     0069:                m = pkg_re.match(line)
     0070:                if m:
     0071:                    manifest.write(m.group(1) + '\n')
Exception: NameError: name 're' is not defined
```

Change-Id: I769a2ab5e57c7b60598ea0390b576d707356db9d
(From OE-Core rev: e240a8ede690e56bcf53a97b3be5592e6e4a5a15)

Signed-off-by: Andrew Jeffery <andrew@aj.id.au>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-04 17:14:15 +00:00
Daniel Wagenknecht
4c298459ab copy_buildsystem: allow more layer paths
Layers could be located anywhere. The eSDK should work with them even if
they are not located in TOPDIR or in the same parent directory as
COREBASE.

For layers located in the same parent directory as COREBASE this preserves
the intent from the previous
  copy_buildsystem: include layer tree during build structure creation
commit.

Related OE-Core rev: 5a59a6997f41e606d088e3e86812de56f72f543b

(From OE-Core rev: 16d330d42e03085769eddb1b60ba1df7228baf36)

Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-04 17:14:14 +00:00
Andrew Jeffery
102e4c029e rust: Introduce arch_to_rust_arch()
On modern Power systems `uname -m` yields 'ppc64le' while the toolchain
knows the architecture as 'powerpc64le'. Provide a mapping from one to
the other to integrate with the existing architecture configuration
flags.

arch_to_rust_arch() only exists to map the OE *_ARCH variables before
any further processing, unlike arch_to_rust_target_arch() which is
specific to the internal triple handling of rust.

On Linux ppc64le systems the changes give the following config:

```
$ cat ./tmp/work/ppc64le-linux/rust-native/1.58.0-r0/targets/ppc64le-linux.json
{
    "llvm-target": "powerpc64le-unknown-linux-gnu",
    "data-layout": "e-m:e-i64:64-n32:64-v256:256:256-v512:512:512",
    "max-atomic-width": 64,
    "target-pointer-width": "64",
    "target-c-int-width": "64",
    "target-endian": "little",
    "arch": "powerpc64",
    "os": "linux",
    "env": "gnu",
    "vendor": "unknown",
    "target-family": "unix",
    "linker": "gcc",
    "cpu": "generic",
    "dynamic-linking": true,
    "executables": true,
    "linker-is-gnu": true,
    "linker-flavor": "gcc",
    "has-rpath": true,
    "has-elf-tls": true,
    "position-independent-executables": true,
    "panic-strategy": "unwind"
}
```

Change-Id: Ief0c01189185d7d4da31d307270bec4e1de674ca
(From OE-Core rev: 9ab61e3cfef0157393cb870d606c2f362e190889)

Signed-off-by: Andrew Jeffery <andrew@aj.id.au>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-02 18:44:17 +00:00
Richard Purdie
321cf8962e license/insane: Show warning for obsolete license usage
We want to use SPDX identifiers in LICENSE variables. There is now a
conversion script to make most of the translations. Add a list of
strings which have been replaced so we can show warnings to users
if they're still used anywhere.

Add checks to the package as insane check. This is currently a warning
by default but can be turned off or made an error as per the other standard
checks.

(From OE-Core rev: 9379f80f484f94686a4d494e9e237fadfb72a938)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-02 18:43:25 +00:00
Saul Wold
d6449581c9 base/license: Rework INCOMPATIBLE_LICENSE variable handling
This re-writes the INCOMPATIBLE_LICENSE checking code to replace
the WHITELIST_<lic> with
INCOMPATIBLE_LICENSE_EXCEPTIONS = '<pkg>:<lic> <pkg>:<lic> ...'

This initial change leaves most of the code structure in place,
but the code in base.bbclass needs to be re-written to make
the check more consistent around packages (PKGS) and not recipe
names (PN). This also is taking into account the changes for SPDX
licenses.

The aim is to provide a mode consistent variable where the variable
name is known and can easily be queried.

(From OE-Core rev: 0d19c45ba6cf43518f380ca5afe9753a2eda0691)

Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-02 18:43:25 +00:00
Alexander Kanavin
c486c6609a sstate: do not add TARGET_ARCH to pkgarch for cross recipes.
This is redundant (target arch is already in PN), and breaks
compiling a cross-canadian toolchain, as that needs populating the
sysroot with two different native-hosted toolchains built from
cross recipes. Inserting TARGET_ARCH allows only one or the other.

(From OE-Core rev: 33fc1792cd782feb8dbb4285e3006bb588f7978f)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-02 18:43:24 +00:00
Peter Kjellerstedt
9da87f1119 license.py: Correct a comment
(From OE-Core rev: 1f7a34c6d246c6f42ab823ffd0bd0306705ad88d)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-27 12:34:40 +00:00
Richard Purdie
2a06ab3eb8 licenses: Fix logic error introduced in rename
The previous commit introduced a small logic error. Fix the renaming issue.

(From OE-Core rev: b01e10b27d23ea1b4ac58376c2423505a70832d6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-25 12:45:41 +00:00
Saul Wold
e629653463 license.py: rename variables
Update the comment to reflect new variable names

(From OE-Core rev: 7fbab1aefc127f0e1834f51a8a793b0d7e7b4f07)

Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-25 12:41:24 +00:00
Pavel Zhukov
da344db43c patch.py: Prevent git repo reinitialization
There were few bugs in the _isInitialized() function which might trigger
git repo to be reinitialized and patches failing to apply.

(From OE-Core rev: 80500ecda4c1bc8812e6e078b6b0db5ec46624de)

Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-25 12:41:23 +00:00
Andres Beltran
e43a9d15ea create-spdx: add support for SDKs
Currently, SPDX SBOMs are only created for images. Add support for
SDKs.

(From OE-Core rev: c3acbb936a339636153903daf127eec9f36de79b)

Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-25 12:41:23 +00:00
Richard Purdie
eb72e3c381 recipeutils: Add missing get_srcrev() call
Updates within bitbake now require this call to be triggered somewhere with
floating git source revs. Add the missing call. Issue can be reproduced with:

devtool check-upgrade-status dbus-wait

(From OE-Core rev: d7088b2b5012f9e386914a6837685f5869379cc5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-18 11:37:34 +00:00
Daiane Angolini
76b3d141c0 classes/lib/useradd: The option -P is deprecated
This is also covered in documentation since:
https://git.yoctoproject.org/yocto-docs/commit/?id=3f3e5574ac9801ad92940168b61b532e0bd53a80

[YOCTO 14605]

(From OE-Core rev: 3f9153986e4e6f667b4bbe97613ec0b279665a97)

Signed-off-by: Daiane Angolini <daiane.angolini@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-12 17:05:35 +00:00
Alexander Kanavin
18f165c9e8 insane.bbclass: use multiprocessing for collecting 'objdump -p' output
This was prompted by ltp's unreasonably long package_qa times; it has
a massive amount of executables and insane runs objdump for all of
them, serially.

This reduces the time from 4 minutes to 1m20s on my machine.

(From OE-Core rev: fac984b99fdb46949879516cb87153860f402c75)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-08 14:20:18 +00:00
Joshua Watt
d366a1a223 spdx: Add set helper for list properties
This adds the set helper for list properties to allow list members to be
replaced with a new list. This is necessary as it is (currently) the
only way to replace the default value for a list member.

(From OE-Core rev: 8280b857c510646cd10167dd2ea9a0c2d5f1e800)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-07 10:08:59 +00:00
Richard Purdie
a242274d98 prelink: Drop support for it
Prelink is being dropped by glibc in 2.36. It already causes issues with
binary corruption, has a number of open bugs and is of questionable benefit
without disabling load address randomization and PIE executables.

We disabled it by default a while back but left people able to use it.
We would be unable to maintain it alone without glibc support so remove
the remaining pieces.

(From OE-Core rev: 23c0be78106f1d1e2bb9c724174a1bb8c56c2469)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05 12:26:36 +00:00
Joshua Watt
85b520587c classes/native: Propagate dependencies to outhash
Native task outputs are directly run on the build system during the build
after being built. Even if the output of a native recipe doesn't change, a
change in one of its dependencies may cause a change in the output it
generates (e.g. rpm output depends on the output of its dependent zstd
library).

This can cause poor interactions with hash equivalence, since this
recipe's output-changing dependency is "hidden" and downstream tasks only
see that this recipe has the same outhash and therefore is equivalent.
This can result in different output in different cases and issues with
reproducible builds in parcular (e.g. rpm compression changes for the same
content).

To resolve this, unhide the output-changing dependency by adding it's
unihash to this tasks outhash calculation. Unfortunately, we don't know
specifically know which dependencies are output-changing, so we have to
add all of them.

[YOCTO #14685]

(From OE-Core rev: d6c7b9f4f0e61fa6546d3644e27abe3e96f597e2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-17 11:49:11 +00:00
Saul Wold
d756b346f2 package: Add support for kernel stripping
Extend runstrip() to accept additional argument to enable
sharing it with the kernel do_strip() so that
KERNEL_IMAGE_STRIP_EXTRA_SECTIONS can be passed.

Since is_elf() understands kernel modules there is no need to keep a
seperate list for kernmodules or hardcode the values to runstrip.

(From OE-Core rev: e09a8fa931fe617afc05bd5e00dca5dd3fe386e8)

Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-13 13:57:26 +00:00
Alexander Kanavin
51a8e28fe0 lib/oe/reproducible: correctly set .git location when recursively looking for git repos
(From OE-Core rev: ffdaa1a0527691d66dd28e86bd015bfad7a020f6)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-05 17:18:15 +00:00
Oleksiy Obitotskyy
2a9bcdb2ea package_manager: ipk: Fix host manifest generation
Since honister host manifest stopped to generate, i.e.
manifest file is empty but all ipks/files into sdk is
ok.

(From OE-Core rev: 79a2392f5d2a4cb6509a83afb40bca01bac59914)

Signed-off-by: Oleksiy Obitotskyy <oobitots@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-04 23:14:05 +00:00
Pavel Zhukov
ae57c83f03 patch.py: Initialize git repo before patching
If PATCHTOOL="git" has been specified but workdir is not git repo
bitbake fails to apply the patches with error message:
Command Error: 'git rev-parse --show-toplevel' exited with 0  Output:
fatal: not a git repository (or any of the parent directories): .git

Fix this by initializing the repo before patching.
This allows binary git patches to be applied.

(From OE-Core rev: 6184b56a7a0fc6f5d19fdfb81e7453667f7da940)

Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-03 23:37:16 +00:00
Richard Purdie
f062749071 sanity/lib: Replace usage of LooseVersion() with bb.utils.vercmp_string_op()
distutils is going away and we have functionality in bitbake which can
handle these comparisions so switch to the bb.utils function.

(From OE-Core rev: fe624b520e6c75e16a8f394785ab0216341402f9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-29 23:07:13 +00:00
Ross Burton
c6f23f1f0f oe/utils: by default cap cpu_count() to 64 cores
Larger systems may have large numbers of cores, but beyond a certain
point they can't all be used for compiling: whilst purely
compute-intensive jobs can be parallelised to hundreds of cores,
operations such as compressing (needs lots of RAM) or compiling (lots of
I/O) don't scale linearly.

For example, the Marvel ThunderX2 has 32 cores, each capable of
executing four threads, and can be configured with two sockets, making
256 CPUs according to Linux. Zstd using 256 threads has been seen to
fail to allocate memory during even small recipes such as iso-codes.

Add a default cap of 64 CPUs to the cpu_count() method so that extreme
parallisation is limited.  64 is high enough that meaningful gains
beyond it are unlikely, but high enough that most systems won't be
effected.

(From OE-Core rev: 765d0f25ce48636b1838a5968e2dc15de2127428)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-25 21:55:10 +00:00
Ross Burton
1fb99950a2 oe/utils: allow naming threads in ThreadedPool
When looking at logs involving thread pools it is useful if the threads
can be named.

(From OE-Core rev: 18342945b021608794d83ecf567afd43f4379b24)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-25 21:55:10 +00:00
Vyacheslav Yurkov
9c8ea9dcf1 overlayfs: all overlays unit
Application can depend on several overlayfs mount points. Provide a
systemd unit application can depend on to make sure all overlays are
mounted before it is started to avoid any race conditions

(From OE-Core rev: b38e194db0c6825f28c56123cf88af94d3f52beb)

Signed-off-by: Bruno Knittel <Bruno.Knittel@bruker.com>
Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-30 22:31:52 +01:00
Saul Wold
b2f859ee61 spdx.py: Add annotation to relationship
Having annotations on relationship can provide additional information
about the relationship such as how it was derived.

(From OE-Core rev: d98585aa89e1d3819f8139a07fb7376ef89b37f8)

Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-28 14:16:31 +01:00
Ross Burton
f59811aeeb oe/utils: log exceptions in ThreadedWorker functions
If the function a ThreadedWorker is executing raises an exception, don't
use print() as that mostly disappears.  Instead, output it to the logger.

This is done using bb.mainlogger.debug directly instead of bb.debug() as
this allows us to pass the exception instance directly, which is then
incorporated into the log stream.

(From OE-Core rev: 2f1ea25c222b344dd8b784b2bc73a6540ab30274)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-18 13:48:16 +01:00
Mike Crowe
51b1611e20 lib/oe/qa,insane: Move extra error handling functions to library
Extract package_qa_write_error, package_qa_handle_error and
package_qa_add_message functions from insane.bbclass to lib/oe/qa.py and
drop the package_qa_ prefixes.

Update various bbclasses to use the new functions. No import is required
since base.bbclass puts oe.qa in OE_IMPORTS.

Stop requiring callers to manually track whether a fatal error has been
encountered via a "sane" flag. Instead replace the QA_SANE variable with
QA_ERRORS_FOUND and call oe.qa.exit_if_errors or
oe.qa.exit_with_message_if_errors at the end of each task.

Inspired by discussion resulting from
https://lists.openembedded.org/g/openembedded-core/message/156793 and
https://lists.openembedded.org/g/openembedded-core/message/156900

(From OE-Core rev: f0ad152ef4cc15c042bc9eeefb6af096d054b220)

Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-17 11:56:32 +01:00