Commit Graph

1336 Commits

Author SHA1 Message Date
Ross Burton
79b6409ef7 oe/sdk: fix empty SDK manifests
The SDK manifests are generated by listing the sstate was that used, but
it hardcodes that the sstate data filenames end in .tgz.

This has not been the case since sstate switched to Zstd[1] in 2021,
which meant that all of the tests which checked for packages existing
were being skipped as the manifests were empty.  For example, see a
representative core-image-sato eSDK test run[2]:

RESULTS - cmake.CMakeTest.test_assimp: SKIPPED (0.00s)
RESULTS - gtk3.GTK3Test.test_galculator: SKIPPED (0.00s)
RESULTS - kmod.KernelModuleTest.test_cryptodev: SKIPPED (0.00s)
RESULTS - maturin.MaturinDevelopTest.test_maturin_develop: SKIPPED (0.00s)
RESULTS - maturin.MaturinTest.test_maturin_list_python: SKIPPED (0.00s)
RESULTS - meson.MesonTest.test_epoxy: SKIPPED (0.00s)
RESULTS - perl.PerlTest.test_perl: SKIPPED (0.00s)
RESULTS - python.Python3Test.test_python3: SKIPPED (0.00s)

All of those tests should have been ran.

Solve this by generalising the filename check so that it doesn't care
what specfic compression algorithm is used.

[1] oe-core 0710e98f40e ("sstate: Switch to ZStandard compressor support")
[2] https://autobuilder.yoctoproject.org/valkyrie/#/builders/16/builds/1517/steps/15/logs/stdio

(From OE-Core rev: b293c44f87b6a52e4239ce14066514e87d9b08d0)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-12 22:01:55 +01:00
Hongxu Jia
2207150bc7 spdx30: Provide software_packageUrl field in SPDX 3.0 SBOM
Define var-SPDX_PACKAGE_URL to provide software_packageUrl field [1][2]
in SPDX 3.0 SBOM, support to override with package name
SPDX_PACKAGE_URL:<pkgname>

Currently, the format of purl is not defined in Yocto, set empty for now
until we have a comprehensive plan for what Yocto purls look like.
But users could customize their own purl by setting var-SPDX_PACKAGE_URL

[1] https://spdx.github.io/spdx-spec/v3.0.1/model/Software/Properties/packageUrl/
[2] https://spdx.github.io/spdx-spec/v3.0.1/annexes/pkg-url-specification/

(From OE-Core rev: c8e6953a0b6f59ffca994c440069db39e60b12d2)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-08 10:28:18 +01:00
Chen Qi
0d5afd1778 lib/classes/conf: refactor qemu.bbclass functions into library functions
Move the functions in qemu.bbclass to meta/lib/oe/qemu.py as they are
generally useful.

The qemu.bbclass is still kept, and recipes can continue to use functions
from it, though they have become wrapper functions on qemu.py functions.

Note that the QEMU_OPTIONS settings are still kept in qemu.bbclass.
This sets a clear barrier for people to use qemu user mode.

(From OE-Core rev: 7b3563b3b3901c96c3e498799a83ab8cabcf84b4)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-01 14:22:54 +01:00
Khem Raj
d2a2c35c7f sstatesig: Handle special case of llvm-project-source shared-workdir
bitbake-dumpsig or bitbake-diffsig tools do not work on any of tasks
exposed by llvm-project-source recipe. This is due to it being a
shared-workdir recipe.

Fixes

bitbake-diffsigs -t llvm-project-source-20.1.2 do_preconfigure
NOTE: Starting bitbake server...
ERROR: No sigdata files found matching llvm-project-source-20.1.2 do_preconfigure

(From OE-Core rev: a6d46935939a94b8ea2b83c024aa86f05efbd7ce)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-01 14:22:53 +01:00
Peter Marko
9fd08fcd94 cve-check: extract extending CVE_STATUS to library function
The same code for extending CVE_STATUS by CVE_CHECK_IGNORE and
CVE_STATUS_GROUPS is used on multiple places.
Create a library funtion to have the code on single place and ready for
reuse by additional classes.

(From OE-Core rev: 45e18f4270d084d81c21b1e5a4a601ce975d8a77)

Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-24 11:27:06 +01:00
Olga Denisova
a0c83d9069 buildstats.py: Add tracking of network I/O per interface
This patch extends SystemStats to collect and store data from /proc/net/dev.
It extracts per-interface received and transmitted bytes, calculates deltas
between samples, and stores them for further analysis.

Useful for identifying network bottlenecks during long-running builds.

(From OE-Core rev: 09cbe17e43783fc6b8e3a341d564956452a04c0a)

Signed-off-by: denisova-ok <denisova.olga.k@yandex.ru>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-24 11:27:06 +01:00
denisova-ok
5b74c76964 buildstats.py: extend diskstats support for NVMe and flexible token count
Added support for NVMe devices in the diskstats regex pattern to ensure stats are properly collected from devices like nvme0n1.

Relaxed the check for the number of fields in /proc/diskstats from an exact match (14) to a minimum check (at least 14), to handle kernel variations and additional fields gracefully.

(From OE-Core rev: 87a31bc4ca3661aae94cf43f3f579b02f4fb4923)

Signed-off-by: denisova-ok <denisova.olga.k@yandex.ru>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-24 11:27:06 +01:00
Martin Jansa
f4d8e84fd9 buildcfg: get_metadata_git_*: catch also bb.process.NotFoundError
* bb.process.NotFoundError is triggered when e.g. oe.buildcfg.get_metadata_git_branch
  is called on non-existent directory

(From OE-Core rev: 34c1f66c4c689b26a4c3129eb62f4ff9b6ec14be)

Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-24 11:27:06 +01:00
Khem Raj
ea00dbbcf3 bitbake.conf: Switch prefix mapping to use -ffile-prefix-map
-ffile-prefix map is more comprehensive when it comes to reproducible
builds and its superset of all prefix-mapping options in compilers

This makes is cleaner and workable across gcc and clang, clang does not
support -fcanon-prefix-map and it has to be explicitly omitted when using
clang.

There are lambdas generated in templates by clang which still get the
absolute paths despite -fdebug-prefix-map, this helps with that as well.

nasm is an outlier and we have fixed it by adding -fdebug-prefix-map option
luckily we do not pass DEBUG_PREFIX_MAP to nasm, in all recipes which use
nasm either pass -fdebug-prefix-map explicitly to nasm or they rewrite it
to use nasm flags syntax.

We have discussed this in past [1]

[1] https://patchwork.yoctoproject.org/project/oe-core/patch/20230428032030.2047920-1-raj.khem@gmail.com/#10281

(From OE-Core rev: ff73fa7ef7666a6dbe34f15515bc3ab6e574c5b0)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Cc: Jacob Kroon <jacob.kroon@gmail.com>
Cc: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-23 09:47:42 +01:00
Joshua Watt
0ed51cf434 lib/oe/cve_check: Mark variable flag dependencies
Marks CVE check functions which depend on non-constant variable flags as
depending on the variables. This allows changes in the flags to
correctly trigger a rebuild

(From OE-Core rev: 2cc43c72ff28aa39a417dd8d57cd7c8741c0e541)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-16 17:41:30 +01:00
Joshua Watt
920efbf519 lib: oe: Add cve_check to BBIMPORTS
Adds cve_check.py to BBIMPORTS so the functions it exposes will be
correctly scanned for dependencies in the dependency scanner

(From OE-Core rev: 52ead33c6b6e2532c57b7b28b862ba38b575f9e3)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-10 11:05:34 +01:00
Peter Marko
057049c1b6 spdx30: handle Unknown CVE_STATUS
CVE_STATUS can be also "Unknown" since oe-core commit
d25f1817752bc8a84c40dcbef75f7559801ce15e

When this status type is used, build fails with e.g.
ERROR: openssl-3.4.1-r0 do_create_spdx: Unknown CVE-2025-0001 status 'Unknown'

Since this is now a valid status, it needs to be handled.
It cannot be mapped to any VEX status (see below), so just skip it.
Possible VEX statuses are: NOT AFFECTED, AFFECTED, FIXED, and UNDER INVESTIGATION.

(From OE-Core rev: 2d3081ef63c8a54df62a2a08bd36008c20eed65a)

Signed-off-by: Peter Marko <peter.marko@siemens.com>
cc: Marta Rybczynska <rybczynska@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-08 15:49:17 +01:00
Ross Burton
cb7e73b358 lib/oe: remove redundant __name__ == "__main__" checks
There's no point in checking if __name__ == "__main__" (i.e., is this
module being invoked) and then doing nothing.

(From OE-Core rev: 020b6b1411c9fd3adb208808c0d56623190873f8)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-03 11:06:20 +01:00
Changqing Li
4f9a3e751c patch.py: set commituser and commitemail for addNote
When PATCHTOOL is set to 'git', and user don't setup
user.name and user.email for git, do_patch fail with
the following error, fix by passing -c options.
CmdError("git notes --ref refs/notes/devtool append -m 'original patch: 0001-PATCH-increase-to-cpp17-version.patch' HEAD", 0, 'stdout:
stderr: Author identity unknown
*** Please tell me who you are.
Run
  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

(From OE-Core rev: a3c6706d31ae1345b571ca10b290a4e1f5a9384b)

Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-03 11:06:20 +01:00
Hongxu Jia
0834a9cdf6 spdx3: support to override the version of a package in SBOM 3
By default, still use ${PV} as the the version of a package in SBOM 3
$ bitbake acl
$ jq . tmp/deploy/spdx/3.0.1/core2-64/packages/package-acl.spdx.json
...
    {
      "type": "software_Package",
       ...
      "name": "acl",
      "software_packageVersion": "2.3.2"
    },
...

Support to override it by setting SPDX_PACKAGE_VERSION, such as
set SPDX_PACKAGE_VERSION = "${EXTENDPKGV}" in local.conf to append
PR to software_packageVersion in SBOM 3
$ echo 'SPDX_PACKAGE_VERSION = "${EXTENDPKGV}"' >> conf/local.conf
$ bitbake acl
$ jq . tmp/deploy/spdx/3.0.1/core2-64/packages/package-acl.spdx.json
...
    {
      "type": "software_Package",
       ...
      "name": "acl",
      "software_packageVersion": "2.3.2-r0"
    },
...

(From OE-Core rev: e6ff5f4d870624795bd36572f5c2bfeec90d83ce)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-24 17:29:16 +00:00
Peter Marko
769a4479e1 spdx30: handle links to inaccessible locations
This is the same as e105befbe4ee0d85e94c2048a744f0373e2dbcdf on
additional place in the code.

When a link is pointing to location inaccessible to build user (e.g. "/root/something"),
filepath.is_file() throws "PermissionError: [Errno 13] Permission denied".
Fix this by first checking if it is a link.

(From OE-Core rev: 26f35f866cf7888431963cf4fc5d2019cd28de74)

Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-20 17:49:16 +00:00
Richard Purdie
110b2c124b spdx: Update for bitbake changes
Bitbake is dropping the need for fetcher name iteration and multiple revisions
per url. Update the code to match (removal of the for loop).

(From OE-Core rev: 4859cdf97fd9a260036e148e25f0b78eb393df1e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-20 13:50:40 +00:00
hongxu
d029e4e033 spdx30: test the existence of directory before walking
Due to commit [spdx30: Improve os.walk() handling][1] applied,
it reported an error if walk directory failed

While SPDX_INCLUDE_SOURCES = "1", if recipe does not provide sysroots,
the walk in function add_package_files is broken

$ echo 'SPDX_INCLUDE_SOURCES = "1"' >> conf/local.conf
$ bitbake packagegroup-core-boot
|DEBUG: Adding sysroot files to SPDX
|ERROR: packagegroup-core-boot-1.0-r0 do_create_spdx: ERROR walking tmp/sysroots-components/intel_x86_64/packagegroup-core-boot: [Errno 2]
| No such file or directory: 'tmp/sysroots-components/intel_x86_64/packagegroup-core-boot'

Test the existence of directory before walking

[1] https://git.openembedded.org/openembedded-core/commit/?id=86b581e80637cd8136ce7a7e95db94d9553d2f60

(From OE-Core rev: cb1792e4950d5075be9bbe4c5337a5215db9669e)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 09:48:14 +00:00
Alexander Kanavin
05d23378c2 meta/lib/oe/recipeutils.py: handle fetcher errors when checking for new commits
Recent freedesktop instabilities are causing 'devtool check-upgrade-status' to
fail with:

bb.fetch2.FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all -c clone.defaultRemoteName=origin ls-remote https://gitlab.>
fatal: unable to access 'https://gitlab.freedesktop.org/mesa/kmscube/': The requested URL returned error: 502

and not print any results for this one or any unrelated recipes included in the check.

This change handles the error, so that if some upstream server isn't working
properly, latest upstream revision for that is marked as unknown, a
warning is printed and upstream version check for other recipes
isn't thwarted:

WARNING: Unable to obtain latest revision: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all -c clone.defaultRemoteName=origin ls-remote https://gitlab.freedesktop.org/mesa/piglit.git  failed with exit code 128, output:
remote: GitLab is not responding
fatal: unable to access 'https://gitlab.freedesktop.org/mesa/piglit.git/': The requested URL returned error: 502

piglit                    1.0             UNKNOWN_BROKEN  Ross Burton <ross.burton@arm.com>

(From OE-Core rev: c1056293f7cb32ee2bdf31441cc0b59d9ccfe556)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 09:48:14 +00:00
Hongxu Jia
99e1e8c0b9 lib: spdx30_tasks: remove duplicated patched CVEs
Due to commit [lib: spdx30_tasks: Handle patched CVEs][1] applied,
duplicated CVE identifier for each CVE which increased +25% build
time (image task: do_create_image_sbom_spdx)

$ bitbake binutils-cross-x86_64
$ jq . tmp/deploy/spdx/3.0.1/x86_64/recipes/recipe-binutils-cross-x86_64.spdx.json | grep CVE-2023-25584
      "spdxId": "http://spdx.org/spdxdocs/binutils-cross-x86_64-5de92009-80e6-55c5-8b1f-cc37f04fbe09/962efd5da447b81b017db54d3077be796d2e5b6e770a6b050467b24339c0995f/vulnerability/CVE-2023-25584",
          "https://rdf.openembedded.org/spdx/3.0/alias": "http://spdxdocs.org/openembedded-alias/by-doc-hash/594f521fb7a3a4e9a2d3905303ffb04b016c3ce7693a775cca08be5af4d06658/binutils-cross-x86_64/UNIHASH/vulnerability/CVE-2023-25584"
          "identifier": "CVE-2023-25584",
            "https://cveawg.mitre.org/api/cve/CVE-2023-25584",
            "https://www.cve.org/CVERecord?id=CVE-2023-25584"
      "spdxId": "http://spdx.org/spdxdocs/binutils-cross-x86_64-5de92009-80e6-55c5-8b1f-cc37f04fbe09/962efd5da447b81b017db54d3077be796d2e5b6e770a6b050467b24339c0995f/vulnerability/CVE-2023-25584",
          "https://rdf.openembedded.org/spdx/3.0/alias": "http://spdxdocs.org/openembedded-alias/by-doc-hash/594f521fb7a3a4e9a2d3905303ffb04b016c3ce7693a775cca08be5af4d06658/binutils-cross-x86_64/UNIHASH/vulnerability/CVE-2023-25584"
          "identifier": "CVE-2023-25584",
            "https://cveawg.mitre.org/api/cve/CVE-2023-25584",
            "https://www.cve.org/CVERecord?id=CVE-2023-25584"

Since the commit [cve-check: annotate CVEs during analysis][2] improved
function get_patched_cves to:
- Check each patch file;
- Search for additional patched CVEs from CVE_STATUS;

And return dictionary patched_cve for each cve:
{
  "abbrev-status": "xxx",
  "status": "xxx",
  "justification": "xxx",
  "resource": "xxx",
  "affected-vendor": "xxx",
  "affected-product": "xxx",
}

But while adding CVE in meta/lib/oe/spdx30_tasks.py, the cve_by_status
requires decoded_status
{
  "mapping": "xxx",
  "detail": "xxx",
  "description": "xxx",
}

This commit converts patched_cve to decoded_status

  patched_cve["abbrev-status"] --> decoded_status["mapping"]
  patched_cve["status"] --> decoded_status["detail"]
  patched_cve["justification"] --> decoded_status["description"]

And remove duplicated search for additional patched CVEs from CVE_STATUS
(calling oe.cve_check.decode_cve_status)

After applying this commit
$ bitbake binutils-cross-x86_64
$ jq . tmp/deploy/spdx/3.0.1/x86_64/recipes/recipe-binutils-cross-x86_64.spdx.json | grep CVE-2023-25584
      "spdxId": "http://spdx.org/spdxdocs/binutils-cross-x86_64-5de92009-80e6-55c5-8b1f-cc37f04fbe09/381bf593d99c005ecd2c2e0815b86bca2b9ff4cc2db59587aaddd3db95c67470/vulnerability/CVE-2023-25584",
          "https://rdf.openembedded.org/spdx/3.0/alias": "http://spdxdocs.org/openembedded-alias/by-doc-hash/594f521fb7a3a4e9a2d3905303ffb04b016c3ce7693a775cca08be5af4d06658/binutils-cross-x86_64/UNIHASH/vulnerability/CVE-2023-25584"
          "identifier": "CVE-2023-25584",
            "https://cveawg.mitre.org/api/cve/CVE-2023-25584",
            "https://www.cve.org/CVERecord?id=CVE-2023-25584"

[1] https://git.openembedded.org/openembedded-core/commit/?id=1ff496546279d8a97df5ec475007cfb095c2a0bc
[2] https://git.openembedded.org/openembedded-core/commit/?id=452e605b55ad61c08f4af7089a5a9c576ca28f7d

(From OE-Core rev: 08595b39b46ef2bf3a928d4528292ee31a990c98)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-12 15:58:14 +00:00
Joshua Watt
143103a1c4 lib: Fix dependencies on SPDX code
The SPDX library code was being ignored from taskhash calculations due
to accidentally being omitted from BBIMPORTS. This meant that changes in
the code or dependent variables would not cause the task to rebuild
correctly.

In order to add spdx_common, convert the `Dep` object from a named tuple
to a frozen dataclass. These function more or less equivalently, but the
bitbake code parser cannot handle named tuples.

Finally, the vardepsexclude that used to be present on the recipe tasks
needs to be moved to the python code in order for the variables to be
correctly ignored. Several unused exclusions were removed

(From OE-Core rev: eb597bf61cbcb0a4d43149404c93eec0894fb4c7)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-11 21:29:50 +00:00
Joshua Watt
e852d99018 lib: spdx30_tasks: Handle patched CVEs
The code to iterate over patched CVEs (e.g. those patched by a .patch
file in SRC_URI) was accidentally omitted when writing the SPDX 3
handling. Add it in now

[YOCTO #15789]

(From OE-Core rev: 1ff496546279d8a97df5ec475007cfb095c2a0bc)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-08 18:08:04 +00:00
Xiaotian Wu
5323603048 lib/oe/elf.py: Add loongarch64 architecture definition for musl
Add the ELF definition for the loongarch64 architecture when building
with musl as libc.

(From OE-Core rev: c6498e4ca43dc2f8bc326bc6b6dbc8fd7f0bef79)

Signed-off-by: Xiaotian Wu <wuxiaotian@loongson.cn>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-07 11:13:24 +00:00
Joshua Watt
5d7d2981bd lib: sbom30: Add action statement for affected VEX statements
VEX Affected relationships have a mandatory action statement that
indicates the mitigation for a vulnerability. Since we don't track this
add a statement indicating that no mitigation is known.

(From OE-Core rev: 39545c955474a43d11a45d74a88a5999b02cb8b3)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-06 16:33:20 +00:00
Richard Purdie
430ed4a884 lib/package/utils: Improve multiprocess_launch argument passing
The current code for multiple argument passing is horrible. Tweak the
multiprocess_launch function to only convert to a tuple if it isn't already
one, which means we can then use function arguments in a standard way.

(From OE-Core rev: 7c99f90079e722764ebdc30e8d0e781454b3a51a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-28 07:49:18 +00:00
Joshua Watt
54e4a89a75 spdx30: Improve os.walk() handling
There have been errors seen when assembling root file system SPDX
documents where they will references files that don't exist in the
package SPDX.

The speculation is that this is caused by os.walk() ignoring errors when
walking, causing files to be omitted. Improve the code by adding an
error handler to os.walk() to report errors when they occur.

In addition, sort the files and directories while walking to ensure
consistent ordering of the file SPDX IDs.

(From OE-Core rev: 86b581e80637cd8136ce7a7e95db94d9553d2f60)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-18 11:56:03 +00:00
Joshua Watt
9600cd875b spdx30: Include files in rootfs
Adds a "contains" relationship that relates the root file system package
to the files contained in it. If a package provides a file with a
matching hash and path, it will be linked, otherwise a new File element
will be created

(From OE-Core rev: e6fe754aef93e834e5226c8b13fdf75e03080ba2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-05 12:49:55 +00:00
Joshua Watt
c8dda4c735 lib/oe/sbom30: Fix SHA256 hash dictionary
Fixes a bug in the code that created the hash dictionary that was
accidentally excluding items with a SHA256 hash instead of including
them

(From OE-Core rev: 90765467e42241d8b572d035389d7062a0316a9f)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-05 12:49:55 +00:00
Igor Opaniuk
f186e405c5 lib/spdx30_tasks: support directories deployed by image recipes
create_image_spdx() implementation assumes that image is indeed a file.
If image recipe deploys a directory (for example, which contains an
hierarchy of flash artifacts, that is used by SoC vendor-specific
flashing tool) which follows ${IMAGE_NAME}.${IMAGE_TYPE} naming scheme,
create_image_spdx() function will fail after trying to hash a directory:

*** 0002:do_create_image_spdx(d)
     0003:
File: '.../meta/classes-recipe/create-spdx-image-3.0.bbclass', lineno: 48, function: do_create_image_spdx
     0044:addtask do_create_rootfs_spdx_setscene
     0045:
     0046:python do_create_image_spdx() {
     0047:    import oe.spdx30_tasks
 *** 0048:    oe.spdx30_tasks.create_image_spdx(d)
     0049:}
     0050:addtask do_create_image_spdx after do_image_complete do_create_rootfs_spdx before do_build
     0051:SSTATETASKS += "do_create_image_spdx"
...
File: '.../bitbake/lib/bb/utils.py', lineno: 536, function: _hasher
     0532:
     0533:def _hasher(method, filename):
     0534:    import mmap
     0535:
 *** 0536:    with open(filename, "rb") as f:
     0537:        try:
     0538:            with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
     0539:                for chunk in iter(lambda: mm.read(8192), b''):
     0540:                    method.update(chunk)
Exception: IsADirectoryError: [Errno 21] Is a directory: '...'

(From OE-Core rev: a0d63082a4db375a55586c7864e280cd8f45ff7b)

Signed-off-by: Igor Opaniuk <igor.opaniuk@foundries.io>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-29 15:52:58 +00:00
Hongxu Jia
2a8ea9790c classes/spdx-common: Fix patch error for recipes that inherit dos2unix.
While SPDX_INCLUDE_SOURCES = "1", do_create_spdx error happens for
these recipes inherit dos2unix

Refer [1] to fix the issue

[1] https://git.openembedded.org/openembedded-core/commit/?id=2ceda7c90c0087f52693c54d5ccab143b27f4d21

(From OE-Core rev: 7051bfb8ff67a6f0c394ce0f559eb53a05cde472)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-29 10:40:41 +00:00
Richard Purdie
90e0a0f7f4 classes/recipes: Switch virtual/XXX-gcc to virtual/cross-cc (and c++/binutils)
The idea of the base class dependency is to say "yes, I need a C cross compiler"
and this was never meant to be gcc specific. Looking at the codebase, whilst we
code triplets into this, it does overcomplicate things as there are only ever
limited, "target", "sdk" and the class extended versions like mutlilib.

After much thought, we can simplify this to virtual/cross-cc and virtual/nativesdk-cross-cc.

This lets us remove the "gcc" specific element as well as removing the over
complicated triplet usage.

At the same time, change the much less widely used "g++" variant to "c++" for
similar reasons and remove the triplet from virtual/XXX-binutils too.

Backwards compatibility mappings could be left but are just going to confuse
things in future so we'll just require users to update.

This simplification, whilst disruptive for any toolchain focused layers, will
make improved toolchain selection in the future much easier.

Since we no longer have overlapping variables, some code for that can just
be removed. The class extension code does need to start remapping some variables
but not the crosssdk target recipe names.

This patch is in two pieces, this one handles the renaming with the functional
changes separate in a second for easier review even if this breaks bisection.

(From OE-Core rev: 4ccc3bc8266c327bcc18c9a3faf7536210dfb9f0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-21 23:09:33 +00:00
Joshua Watt
d1947c5364 lib: spdx: Upgrade to final 3.0.1 release
The 3.0.1 release of SPDX has been officially released with a few minor
modifications. Regenerate the bindings to use this version.

(From OE-Core rev: 54233a7d6fe414d22449fb02fac26b66a820b17a)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-20 13:38:59 +00:00
hongxu
2abb0028ba meta/lib/oe/cve_check.py: fix patched_cves not updated
Due to commit [cve-check: Rework patch parsing] applied, it missed to
update patched_cves dictionary if cve_id not in patched_cves

(From OE-Core rev: 08796a8153666d93bb622c6a7497a85cef4def42)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-12 13:15:29 +00:00
Colin McAllister
35f4253a10 cve-check: Rework patch parsing
The cve_check functionality to parse CVE IDs from the patch filename and
patch contents have been reworked to improve parsing and also utilize
tests. This ensures that the parsing works as intended.

Additionally, the new patched_cves dict has a few issues I tried to fix
as well. If multiple patch files exist for a single CVE ID, only the
last one will show up with the "resource" key. The value for the
"resource" key has been updated to hold a list and return all patch
files associated with a given CVE ID. Also, at the end of
get_patch_cves, CVE_STATUS can overwrite an existing entry in the dict.
This could cause an issue, for example, if a CVE has been addressed via
a patch, but a CVE_STATUS line also exists that ignores the given CVE
ID. A warning has been added if this ever happens.

(From OE-Core rev: 87c6da681609b4f8e048eca2a27ae8e068c724e1)

Signed-off-by: Colin McAllister <colinmca242@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 13:25:11 +00:00
Colin McAllister
db2146dbe6 cve-check: Fix errors in log lines
Two warning lines in cve_check.py reference a variable that doesn't
exist. These would cause a runtime error if the conditions they are
hidden in were to be entered. The log lines have been updated to no
longer reference an undefined variable.

(From OE-Core rev: c9d059e8a362b3c9d604f7ebe8fd1dd994f0af6b)

Signed-off-by: Colin McAllister <colinmca242@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 13:25:11 +00:00
Hongxu Jia
ac546d2196 meta/lib/oe/sbom30.py: set alias for scan declared licenses
when using SPDX_INCLUDE_SOURCES, it calls scan_declared_licenses
to scan licenses from source file, set alias for the newly added
license and and license alias to hasDeclaredLicense relationship

$ echo 'MACHINE = "qemux86-64"' >> conf/local.conf
$ echo 'SPDX_INCLUDE_SOURCES = "1"' >> conf/local.conf
$ bitbake shadow
$ vim tmp/deploy/spdx/3.0.1/corei7-64/recipes/recipe-shadow.spdx.json
Before this commit
...
    {
      "type": "Relationship",
      ...
      "from": "http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/d0cdb0d02e54d55e52fccf8631f8290b161ad43fe31fffe09e8e25041d2280cf/sourcefile/11048",
      "relationshipType": "hasDeclaredLicense",
      "to": [
        "http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/d0cdb0d02e54d55e52fccf8631f8290b161ad43fe31fffe09e8e25041d2280cf/license/3_24_0/BSD-3-Clause",
      ]
    },
...

After this commit
...
    {
      "type": "Relationship",
      ...
      "from": "http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/d0cdb0d02e54d55e52fccf8631f8290b161ad43fe31fffe09e8e25041d2280cf/sourcefile/11048",
      "relationshipType": "hasDeclaredLicense",
      "to": [
        "http://spdxdocs.org/openembedded-alias/by-doc-hash/cc72db638e3f8e283e722af0ecc77d19f93cc6736700ee76477e3773b6b07b05/shadow/UNIHASH/license/3_24_0/BSD-3-Clause"
      ]
    },
...

(From OE-Core rev: 1f109e22dd22e6071685a8e55bdfe3069bc8fadb)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 12:50:14 +00:00
Hongxu Jia
4d8103bfed meta/lib/oe/spdx30_tasks.py: set license alias to hasConcludedLicense relationship
Set license alias other than spdxId to hasConcludedLicense relationship

$ echo 'MACHINE = "qemux86-64"' >> conf/local.conf
$ bitbake shadow
$ In tmp/deploy/spdx/3.0.1/corei7-64/recipes/recipe-shadow.spdx.json
Before this commit
...
    {
      "type": "Relationship",
      ...
      "from": "http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/c34a0920c2f689375e786a80ab3176b844997d543b73dd5f99e6ffa8fcc72fee/source/8",
      "relationshipType": "hasConcludedLicense",
      "to": [
        "http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/c34a0920c2f689375e786a80ab3176b844997d543b73dd5f99e6ffa8fcc72fee/license/3_24_0/BSD-3-Clause"
      ]
    },
...

After this commit
...
    {
      "type": "Relationship",
      ...
      "from": "http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/c34a0920c2f689375e786a80ab3176b844997d543b73dd5f99e6ffa8fcc72fee/source/8",
      "relationshipType": "hasConcludedLicense",
      "to": [
        "http://spdxdocs.org/openembedded-alias/by-doc-hash/cc72db638e3f8e283e722af0ecc77d19f93cc6736700ee76477e3773b6b07b05/shadow/UNIHASH/license/3_24_0/BSD-3-Clause"
      ]
    },
...

(From OE-Core rev: 0530b45a5f27adff55251ce909ad4f077c1076d6)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 12:50:14 +00:00
hongxu
0fee98b8e8 meta/lib/oe/sbom30.py: fix alias in simplelicensing_customIdToUri not extracted
After commit [spdx 3.0: Rework how SPDX aliases are linked] applied, it set
license_text_map with SPDX alias other than actual ID

The property of simplelicensing_customIdToUri is ListProp(ObjectProp(DictionaryEntry))),
and class DictionaryEntry has key and value, the property of value is StringProp other
than ObjectProp in which could not support to decode/extract SPDX alias with actual ID
in image jsonld file
-----------
      "simplelicensing_customIdToUri": [
        {
          "type": "DictionaryEntry",
          "key": "LicenseRef-PD",
          "value": "http://spdxdocs.org/openembedded-alias/by-doc-hash/d53e90e23b12c4ad640809a74a810e86f31c76cdbdf36487712d22a33d53362a/sqlite3-native/UNIHASH/license-text/PD"
        }
      ],
-----------

Add special code in the linking to manually go through all of the
simplelicensing_customIdToUri DictionaryEntry items and resolve
any aliases to actual objects
-----------
      "simplelicensing_customIdToUri": [
        {
          "type": "DictionaryEntry",
          "key": "LicenseRef-PD",
          "value": "http://spdx.org/spdxdocs/sqlite3-native-e5cc0672-d8dd-57e8-a2df-fe4615831fda/162c62b5b011cd3f82f413b3dae4d6d1542201552c964d5ce69fe170e0720b85/license-text/PD"
        }
      ]
-----------

Suggested-by: Joshua Watt <jpewhacker@gmail.com>

(From OE-Core rev: f16c2eae5a7e1a483e1d537d8711d83fede75691)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 12:49:00 +00:00
hongxu
446e74ccee meta/lib/oe/sbom30.py: correct alias
After commit [spdx 3.0: Rework how SPDX aliases are linked] applied,
it added extra "/" to namespace, which causing the replacement of
UNIHASH missing a "/"

  http://spdxdocs.org/openembedded-alias/by-doc-hash/0b308e4b9ad979f642d8787c61f76c31bdcad04837eeaaf8bc383f33f99bbeb8/flex-nativeUNIHASH/build/recipe

After applying this commit to remove "/" from namespace.

  http://spdxdocs.org/openembedded-alias/by-doc-hash/0b308e4b9ad979f642d8787c61f76c31bdcad04837eeaaf8bc383f33f99bbeb8/flex-native/UNIHASH/build/recipe

(From OE-Core rev: 708cbf00d46169ca05660b1f9f3e0653b499e9ef)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 12:49:00 +00:00
Alexander Kanavin
2d22ef8195 rpm: replace use of rpm2cpio with rpm2archive
rpm2cpio has been deprecated upstream, so this prepares for its
eventual removal.

rpm2archive produces a tar archive which can be uncompressed
with tar executable from the host.

(From OE-Core rev: ed824d3fb23f0c89d8dfdacb2c4ef0b7c21a5144)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-12 13:22:08 +00:00
Joshua Watt
102743c4df spdx 3.0: Rework how SPDX aliases are linked
The SPDX code needs to be able to look up an Element by its SPDX ID,
locating the file that (should) contain the SPDX ID and opening it for
parsing. Previously, the code would do this be hashing each Element
SPDX ID and Alias, and the creating a symbolic link to the file that
contains the element with a name of the hash.

This worked well as it was possible to look up any arbitrary SPDX ID or
alias by simply hashing it and following the symbolic link to get the
file. However, the down side of this approach is that it creates a lot
of symbolic links, since it will make one or two per Element in the
document. This can be a problem when using SPDX_INCLUDE_SOURCES, for
example.

This change reworks this strategy so that the only Element that gets a
symbolic link based on the hash is the singular SpdxDocument that is
create for each file. All other Elements are assigned an alias with a
special prefix that encodes the hash of SpdxDocument alias. Thus, when
attempting to look up an arbitrary alias, the code sees the special
prefix, extract the hash, opens the file based on the symlink with that
hash name, then finds the matching Element in the file. This drastically
reduces the number of symbolic links by making only one per file.

This also means that the custom link extension can be removed since it
is now superfluous.

(From OE-Core rev: 838d64c09657ac53175737fc4e7fd6f01f3dcf47)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-12 12:55:56 +00:00
Joshua Watt
30638546af lib: oeqa: spdx: Add tests for extra options
Adds a test for several of the extra options provided by the SPDX
classes. In particular, these are the options that can produce
non-reproducible results, so are not enabled by default in OE core. This
test takes care to configure the build so that the tests do run in a
reproducible manner so that pre-built test objects can be pulled from
sstate

(From OE-Core rev: 14f534f15f7fe6362723d7f064d39783c5bd758f)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-05 17:07:10 +00:00
Joshua Watt
eda4a8bc21 lib: spdx: Fix SPDX_BUILD_HOST
The call to make a new import for the SPDX_BUILD_HOST was incorrect,
since it was asking for the key named "SPDX_BUILD_HOST", instead of the
key name contained in that variable

(From OE-Core rev: 250c2bcc557040b4cd67cecf25f8351638cff45d)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-05 17:07:10 +00:00
Joshua Watt
ddbbc9e53c lib: sbom30: Fix agent reference
When a agent reference was being used, the code was not using the
correct base variable to look up the agent

(From OE-Core rev: f3f13f48e214b25cf302b8ce397b630f5aa283fa)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-05 17:07:10 +00:00
Joshua Watt
813d6b296c lib: sbom30: Fix import handling
Fixes a bug in the way that imports where being handled, due to a
misspelled variable

(From OE-Core rev: 10e2a1fbcf4828f3a6c7fd327976fefa3ffb252e)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-05 17:07:10 +00:00
Chen Qi
f4c4aa37fc meta/lib: move buildstats.py and rootfspostcommands.py into oe
These two files are the only ones that are left in meta/lib.
They logically belong to meta/lib/oe, so move them there.

(From OE-Core rev: c65dd0e3e463d6072b9364ac74e1fef0d998068f)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-05 17:07:10 +00:00
Joshua Watt
8bc93605d5 lib: spdx30_tasks: Fix supplied By
Fixes the supplied by property in SPDX to use the correct name

(From OE-Core rev: a33fb425262492f5e8f6cc5e7ec2adf072042817)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-05 17:07:10 +00:00
Joshua Watt
eae43f42b7 lib: oe: license: Add missing import
The code in this file uses oe.qa, but it was not imported resulting an
an exception when a license error was detected

(From OE-Core rev: e44cb7c5b7281d614ed51fdec06dad0a7211528a)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-28 00:07:30 +00:00
Joshua Watt
88489115ae lib/oe/sbom30: Prefix aliases with "http://spdx.org/spdxdocs/"
While the SPDX ID aliases are primarily intended for internal use by the
SPDX code, they are used in places where a valid SPDX ID is expected. In
order to make sure that they still conform to this, prefix them with
"http://spdx.org/spdxdocs/openembedded-alias"

(From OE-Core rev: 5e0ff36e025f5e842fa90b8219b53257d65ea66a)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-23 14:44:54 +00:00
Joshua Watt
43a7402255 lib: sbom30: Fix build parameters
Fixes the build parameters code, since the build_parameter property was
changed to be singular in SPDX 3.0.1

(From OE-Core rev: e6e1e4585198b819319bab61ac2676e82f177020)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-23 14:44:54 +00:00