Commit Graph

4068 Commits

Author SHA1 Message Date
Alexander Kanavin
70a650c26a bbconfigbuild/configfragments.py: set BBPATH in standalone datastore
Otherwise fragments with 'require' statements will not parse,
as the parser will not be able to find files referred to by
those statements.

Add such a statement to the test fragment so that the scenario
is tested.

[YOCTO #15707]

(From OE-Core rev: d88e63d1a7867b3a4ebfc57d472900136b5361d8)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-09 16:59:23 +00:00
Colin McAllister
35f4253a10 cve-check: Rework patch parsing
The cve_check functionality to parse CVE IDs from the patch filename and
patch contents have been reworked to improve parsing and also utilize
tests. This ensures that the parsing works as intended.

Additionally, the new patched_cves dict has a few issues I tried to fix
as well. If multiple patch files exist for a single CVE ID, only the
last one will show up with the "resource" key. The value for the
"resource" key has been updated to hold a list and return all patch
files associated with a given CVE ID. Also, at the end of
get_patch_cves, CVE_STATUS can overwrite an existing entry in the dict.
This could cause an issue, for example, if a CVE has been addressed via
a patch, but a CVE_STATUS line also exists that ignores the given CVE
ID. A warning has been added if this ever happens.

(From OE-Core rev: 87c6da681609b4f8e048eca2a27ae8e068c724e1)

Signed-off-by: Colin McAllister <colinmca242@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 13:25:11 +00:00
Colin McAllister
db2146dbe6 cve-check: Fix errors in log lines
Two warning lines in cve_check.py reference a variable that doesn't
exist. These would cause a runtime error if the conditions they are
hidden in were to be entered. The log lines have been updated to no
longer reference an undefined variable.

(From OE-Core rev: c9d059e8a362b3c9d604f7ebe8fd1dd994f0af6b)

Signed-off-by: Colin McAllister <colinmca242@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 13:25:11 +00:00
Hongxu Jia
ac546d2196 meta/lib/oe/sbom30.py: set alias for scan declared licenses
when using SPDX_INCLUDE_SOURCES, it calls scan_declared_licenses
to scan licenses from source file, set alias for the newly added
license and and license alias to hasDeclaredLicense relationship

$ echo 'MACHINE = "qemux86-64"' >> conf/local.conf
$ echo 'SPDX_INCLUDE_SOURCES = "1"' >> conf/local.conf
$ bitbake shadow
$ vim tmp/deploy/spdx/3.0.1/corei7-64/recipes/recipe-shadow.spdx.json
Before this commit
...
    {
      "type": "Relationship",
      ...
      "from": "http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/d0cdb0d02e54d55e52fccf8631f8290b161ad43fe31fffe09e8e25041d2280cf/sourcefile/11048",
      "relationshipType": "hasDeclaredLicense",
      "to": [
        "http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/d0cdb0d02e54d55e52fccf8631f8290b161ad43fe31fffe09e8e25041d2280cf/license/3_24_0/BSD-3-Clause",
      ]
    },
...

After this commit
...
    {
      "type": "Relationship",
      ...
      "from": "http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/d0cdb0d02e54d55e52fccf8631f8290b161ad43fe31fffe09e8e25041d2280cf/sourcefile/11048",
      "relationshipType": "hasDeclaredLicense",
      "to": [
        "http://spdxdocs.org/openembedded-alias/by-doc-hash/cc72db638e3f8e283e722af0ecc77d19f93cc6736700ee76477e3773b6b07b05/shadow/UNIHASH/license/3_24_0/BSD-3-Clause"
      ]
    },
...

(From OE-Core rev: 1f109e22dd22e6071685a8e55bdfe3069bc8fadb)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 12:50:14 +00:00
Hongxu Jia
4d8103bfed meta/lib/oe/spdx30_tasks.py: set license alias to hasConcludedLicense relationship
Set license alias other than spdxId to hasConcludedLicense relationship

$ echo 'MACHINE = "qemux86-64"' >> conf/local.conf
$ bitbake shadow
$ In tmp/deploy/spdx/3.0.1/corei7-64/recipes/recipe-shadow.spdx.json
Before this commit
...
    {
      "type": "Relationship",
      ...
      "from": "http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/c34a0920c2f689375e786a80ab3176b844997d543b73dd5f99e6ffa8fcc72fee/source/8",
      "relationshipType": "hasConcludedLicense",
      "to": [
        "http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/c34a0920c2f689375e786a80ab3176b844997d543b73dd5f99e6ffa8fcc72fee/license/3_24_0/BSD-3-Clause"
      ]
    },
...

After this commit
...
    {
      "type": "Relationship",
      ...
      "from": "http://spdx.org/spdxdocs/shadow-10e66933-65cf-5a2d-9a1d-99b12a405441/c34a0920c2f689375e786a80ab3176b844997d543b73dd5f99e6ffa8fcc72fee/source/8",
      "relationshipType": "hasConcludedLicense",
      "to": [
        "http://spdxdocs.org/openembedded-alias/by-doc-hash/cc72db638e3f8e283e722af0ecc77d19f93cc6736700ee76477e3773b6b07b05/shadow/UNIHASH/license/3_24_0/BSD-3-Clause"
      ]
    },
...

(From OE-Core rev: 0530b45a5f27adff55251ce909ad4f077c1076d6)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 12:50:14 +00:00
hongxu
0fee98b8e8 meta/lib/oe/sbom30.py: fix alias in simplelicensing_customIdToUri not extracted
After commit [spdx 3.0: Rework how SPDX aliases are linked] applied, it set
license_text_map with SPDX alias other than actual ID

The property of simplelicensing_customIdToUri is ListProp(ObjectProp(DictionaryEntry))),
and class DictionaryEntry has key and value, the property of value is StringProp other
than ObjectProp in which could not support to decode/extract SPDX alias with actual ID
in image jsonld file
-----------
      "simplelicensing_customIdToUri": [
        {
          "type": "DictionaryEntry",
          "key": "LicenseRef-PD",
          "value": "http://spdxdocs.org/openembedded-alias/by-doc-hash/d53e90e23b12c4ad640809a74a810e86f31c76cdbdf36487712d22a33d53362a/sqlite3-native/UNIHASH/license-text/PD"
        }
      ],
-----------

Add special code in the linking to manually go through all of the
simplelicensing_customIdToUri DictionaryEntry items and resolve
any aliases to actual objects
-----------
      "simplelicensing_customIdToUri": [
        {
          "type": "DictionaryEntry",
          "key": "LicenseRef-PD",
          "value": "http://spdx.org/spdxdocs/sqlite3-native-e5cc0672-d8dd-57e8-a2df-fe4615831fda/162c62b5b011cd3f82f413b3dae4d6d1542201552c964d5ce69fe170e0720b85/license-text/PD"
        }
      ]
-----------

Suggested-by: Joshua Watt <jpewhacker@gmail.com>

(From OE-Core rev: f16c2eae5a7e1a483e1d537d8711d83fede75691)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 12:49:00 +00:00
hongxu
446e74ccee meta/lib/oe/sbom30.py: correct alias
After commit [spdx 3.0: Rework how SPDX aliases are linked] applied,
it added extra "/" to namespace, which causing the replacement of
UNIHASH missing a "/"

  http://spdxdocs.org/openembedded-alias/by-doc-hash/0b308e4b9ad979f642d8787c61f76c31bdcad04837eeaaf8bc383f33f99bbeb8/flex-nativeUNIHASH/build/recipe

After applying this commit to remove "/" from namespace.

  http://spdxdocs.org/openembedded-alias/by-doc-hash/0b308e4b9ad979f642d8787c61f76c31bdcad04837eeaaf8bc383f33f99bbeb8/flex-native/UNIHASH/build/recipe

(From OE-Core rev: 708cbf00d46169ca05660b1f9f3e0653b499e9ef)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 12:49:00 +00:00
Joshua Watt
5cef6cbe2d lib/configfragements: enable/disable multiple fragements at once
Extends the 'enable-fragment' and 'disable-fragment' commands so that
they accept multiple fragments at once as a convenience for the user

(From OE-Core rev: 50c3cdb3a3b9c7daa55ff26d302d95e5f350e4d2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 12:37:20 +00:00
Joshua Watt
9054345fb8 lib/configfragments: Restrict fragment file checking
The current implementation of the config fragments is too aggressive in
checking files; any file in the fragment directory is checked, including
hidden files or files with weird extensions. In particular, if an editor
is creating temporary backup files when editing, these will be checked
and will almost assuredly fail, which prevents the tool from running.

Add a filter so that only non-hidden files that end with .conf are
checked.

(From OE-Core rev: 93edab0e2bccacb808421f0766d587c576c1a12b)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 12:37:20 +00:00
Adrian Freihofer
a3c5179414 oe-selftest: devtool ide-sdk use modify debug-build
(From OE-Core rev: 9ab1e35a0969af144caedcc25cd095a17d758bf9)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-17 11:41:53 +00:00
Alexander Kanavin
22f046d67c bitbake-config-build: add a plugin for config fragments
This allows fine-tuning local configurations with pre-frabricated
configuration snippets in a structured, controlled way. It's also
an important building block for bitbake-setup.

The tool requires that each fragment contains a one-line summary, and one or more
lines of description, as BB_CONF_FRAGMENT_SUMMARY style metadata.

There are three (and a half) operations (list/enable/disable/disable all), and here's the 'list' output:

alex@Zen2:/srv/storage/alex/yocto/build-64$ bitbake-config-build list-fragments
NOTE: Starting bitbake server...
Available fragments in selftest layer located in /srv/work/alex/poky/meta-selftest:

Enabled fragments:
selftest/test-fragment	This is a configuration fragment intended for testing in oe-selftest context

Unused fragments:
selftest/more-fragments-here/test-another-fragment	This is a second configuration fragment intended for testing in oe-selftest context

(From OE-Core rev: fdb611e13bd7aa00360d3a68e4818ef5f05c8944)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-13 11:11:18 +00:00
Alexander Kanavin
2d22ef8195 rpm: replace use of rpm2cpio with rpm2archive
rpm2cpio has been deprecated upstream, so this prepares for its
eventual removal.

rpm2archive produces a tar archive which can be uncompressed
with tar executable from the host.

(From OE-Core rev: ed824d3fb23f0c89d8dfdacb2c4ef0b7c21a5144)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-12 13:22:08 +00:00
Alexander Kanavin
ea9c2cfb69 rpm: update 4.19.1.1 -> 4.20.0
With this release, as expected, deprecated openpgp support
has been fully removed and replaced with sequoia:
https://github.com/rpm-software-management/rpm-sequoia/

Alas, it's written in rust, and has no recipe, so for now
signing rpm packages has to be disabled.

Remove package name parameter to %prep in source archiving, as it's
been dropped in the new rpm, and wasn't needed to begin with.

Drop 0001-perl-disable-auto-reqs.patch
(files removed upstream).

(From OE-Core rev: 8c15b4577d5e554cc2dd5adfb88b816894b05a9a)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-12 13:22:08 +00:00
Oleksandr Hnatiuk
7e3d21d0b2 oeqa/selftest: Fix failure when configuration contains BBLAYERS:append
When used with `--newbuilddir` option, it replaces relative paths in
BBLAYERS variable with absolute paths by evaluating the final value of
the variable, converting paths and saving result by assignment
(`BBLAYERS =`) at the end of bblayers.conf.

This breaks tests when bblayers.conf contains BBLAYERS:append because
:append statements are evaluated after all assignments and we end up
with the appended layer added twice - first by evaluating final value
of the variable, then again by bitbake evaluating BBLAYERS:append.

The error is as follows:
AssertionError: Command 'bitbake  -e' returned non-zero exit status 1:
ERROR: Found duplicated BBFILE_COLLECTIONS 'layer-name', check bblayers.conf or layer.conf to fix it.

(From OE-Core rev: 516c8e8f722ed0eb85a715a23b5e68d05abe9275)

Signed-off-by: Oleksandr Hnatiuk <ohnatiuk@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-12 12:59:35 +00:00
Joshua Watt
102743c4df spdx 3.0: Rework how SPDX aliases are linked
The SPDX code needs to be able to look up an Element by its SPDX ID,
locating the file that (should) contain the SPDX ID and opening it for
parsing. Previously, the code would do this be hashing each Element
SPDX ID and Alias, and the creating a symbolic link to the file that
contains the element with a name of the hash.

This worked well as it was possible to look up any arbitrary SPDX ID or
alias by simply hashing it and following the symbolic link to get the
file. However, the down side of this approach is that it creates a lot
of symbolic links, since it will make one or two per Element in the
document. This can be a problem when using SPDX_INCLUDE_SOURCES, for
example.

This change reworks this strategy so that the only Element that gets a
symbolic link based on the hash is the singular SpdxDocument that is
create for each file. All other Elements are assigned an alias with a
special prefix that encodes the hash of SpdxDocument alias. Thus, when
attempting to look up an arbitrary alias, the code sees the special
prefix, extract the hash, opens the file based on the symlink with that
hash name, then finds the matching Element in the file. This drastically
reduces the number of symbolic links by making only one per file.

This also means that the custom link extension can be removed since it
is now superfluous.

(From OE-Core rev: 838d64c09657ac53175737fc4e7fd6f01f3dcf47)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-12 12:55:56 +00:00
Mathieu Dubois-Briand
0c8b34b55d qemurunner: Fix stack trace generation in exception handler
Qemurunner exception handling code currently formats the stack trace
using traceback.format_exception(), with parameters introduced in python
3.10. This will fail on platforms with an older python version.

Change this to the old parameter order, still supported in current
python versions.

https://docs.python.org/3/library/traceback.html#traceback.format_exception

Fixes [YOCTO #15675]

(From OE-Core rev: 5f9ecf5f210e967594069f172728fd5b4d5b4f1d)

Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-09 15:23:28 +00:00
Joshua Watt
30638546af lib: oeqa: spdx: Add tests for extra options
Adds a test for several of the extra options provided by the SPDX
classes. In particular, these are the options that can produce
non-reproducible results, so are not enabled by default in OE core. This
test takes care to configure the build so that the tests do run in a
reproducible manner so that pre-built test objects can be pulled from
sstate

(From OE-Core rev: 14f534f15f7fe6362723d7f064d39783c5bd758f)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-05 17:07:10 +00:00
Joshua Watt
eda4a8bc21 lib: spdx: Fix SPDX_BUILD_HOST
The call to make a new import for the SPDX_BUILD_HOST was incorrect,
since it was asking for the key named "SPDX_BUILD_HOST", instead of the
key name contained in that variable

(From OE-Core rev: 250c2bcc557040b4cd67cecf25f8351638cff45d)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-05 17:07:10 +00:00
Joshua Watt
ddbbc9e53c lib: sbom30: Fix agent reference
When a agent reference was being used, the code was not using the
correct base variable to look up the agent

(From OE-Core rev: f3f13f48e214b25cf302b8ce397b630f5aa283fa)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-05 17:07:10 +00:00
Joshua Watt
813d6b296c lib: sbom30: Fix import handling
Fixes a bug in the way that imports where being handled, due to a
misspelled variable

(From OE-Core rev: 10e2a1fbcf4828f3a6c7fd327976fefa3ffb252e)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-05 17:07:10 +00:00
Richard Purdie
08e91d0d20 oeqa/gitarchive: Fix syntax warning
The backslash characters cause syntax warnings, mark the strings are raw
to avoid this.

(From OE-Core rev: f717f61a37ed83618d054fc4017b5f5386fb2e3c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-05 17:07:10 +00:00
Chen Qi
f4c4aa37fc meta/lib: move buildstats.py and rootfspostcommands.py into oe
These two files are the only ones that are left in meta/lib.
They logically belong to meta/lib/oe, so move them there.

(From OE-Core rev: c65dd0e3e463d6072b9364ac74e1fef0d998068f)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-05 17:07:10 +00:00
Joshua Watt
8bc93605d5 lib: spdx30_tasks: Fix supplied By
Fixes the supplied by property in SPDX to use the correct name

(From OE-Core rev: a33fb425262492f5e8f6cc5e7ec2adf072042817)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-05 17:07:10 +00:00
Joshua Watt
eae43f42b7 lib: oe: license: Add missing import
The code in this file uses oe.qa, but it was not imported resulting an
an exception when a license error was detected

(From OE-Core rev: e44cb7c5b7281d614ed51fdec06dad0a7211528a)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-28 00:07:30 +00:00
Joshua Watt
88489115ae lib/oe/sbom30: Prefix aliases with "http://spdx.org/spdxdocs/"
While the SPDX ID aliases are primarily intended for internal use by the
SPDX code, they are used in places where a valid SPDX ID is expected. In
order to make sure that they still conform to this, prefix them with
"http://spdx.org/spdxdocs/openembedded-alias"

(From OE-Core rev: 5e0ff36e025f5e842fa90b8219b53257d65ea66a)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-23 14:44:54 +00:00
Joshua Watt
43a7402255 lib: sbom30: Fix build parameters
Fixes the build parameters code, since the build_parameter property was
changed to be singular in SPDX 3.0.1

(From OE-Core rev: e6e1e4585198b819319bab61ac2676e82f177020)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-23 14:44:54 +00:00
Richard Purdie
369d4f05d2 oeqa/utils/gitarchive: Return tag name and improve exclude handling
Tweak the gitarchive exclude handling not to error if excluded files
don't match.

Also return the tagname created so that other code can then use it.

(From OE-Core rev: 1adba3430faffdf6217b6a00533a3b48a9388abc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-23 14:44:54 +00:00
Richard Purdie
9f57d44abe resulttool: Use single space indentation in json output
Using 4 space indentation in resulted in hundreds of megabytes of extra file size
in general use. Reduce this to make filesizes more managable and reduce the processing
cost. Some level of indentation and spacing does make the files more readable and allows
use of git diff so we need to retain some of it.

(From OE-Core rev: a274cdcaf852cca9497f0358f44dda99c06aacbe)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-23 14:44:54 +00:00
Richard Purdie
a5c9bff2b7 selftest/reproducible: Clean up pathnames
There are several problems with these paths. Firstly they contain full
system paths which depend upon where the test was run. These are pretty
pointless and just take up a lot of space making the results files large.

Secondly, they contain the same path twice. The reference and target path
will always be the same thing in two different locations.

Strip off the prefix and remove the duplication. This does change the output
data but that can't really be avoided. It does shrink the results data and makes
it more readable.

(From OE-Core rev: 81a44de36e864b08687451fd85aeba7c529fd7f7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-22 16:53:37 +00:00
Richard Purdie
68d40bd122 selftest/reproducible: Drop rawlogs
The "rawlogs" data consists of a long string of results data which is
already in a structured data format. I can't see this is adding much
value in duplciating the data but it does create a huge string with a
lot of long problematic pathnames and inflates the results data size.

I suggest we drop this data as obsolete and not necessary.

(From OE-Core rev: 5b2c70fab2ffa409b861d83f048b65d458d03a90)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-22 16:53:37 +00:00
Richard Purdie
0b674b09c1 qemu/testimage: Ensure GNU tar is in test images
We've seen cases where a ptest (strace) has a 43GB sparse file in the test
directory. busybox tar doesn't work well with this. The resulting 1.4GB archive
takes hours to extract too.

Ensure tar is added to our full images and use the sparse option to collect
files for debugging. This stops crazy build hangs.

Since tar is GPLv3, we have to exclude it from that test code. We don't boot
any of those images so the debug collection code is safe there, at least for now.

(From OE-Core rev: fefeb919696b6ac76f0997acfb0f612203ef7f1b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-21 12:16:28 +00:00
Richard Purdie
5f2d077c3e oeqa/ssh: Improve performance and log sizes
The current code is not fit for purpose when handling large files via ssh. In the strace
ptest case, we can end up with a 1.4GB archive being transferred for which every
byte is printed into the task logfile twice over. This is then sent over bitbake IPC
which compounds the problems.

Make the following improvements:
  * when the output is large (over 64kb), don't print it
  * use a bytearray for better concat performance since strings are slow for this
  * when there is no ssh output, say that
  * print periodic size status output rather than the data itself since this could be binary and/or large
  * fix the killed process message logic which appeared broken

(From OE-Core rev: e7dd009a17dc902852983a82bce41bf78bb1e242)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-21 12:16:28 +00:00
Alexander Kanavin
4ed3c83240 patchtest: use HEAD commit as base for the selftest and not master
Patchtest applies patches on top of poky master branch by default;
this means selftest does the same, and any commits from the branch-under-test
are then discarded.

This can cause issues for example, if bitbake-server process started by selftest
from the master branch tries to parse bitbake.conf from the branch under test:
https://valkyrie.yoctoproject.org/#/builders/71/builds/460

(From OE-Core rev: 03c6b2e0277c00faf55c12c4d0b4b5e3a4898f8c)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-19 12:54:32 +00:00
Ralph Siemsen
21eab81f95 rootfs-postcommands: Actually do the re-ordering
Previous commit added logic to move certain tasks to the end, but these
had no effect, because the result of the make_last() function was not
used to update the post_process_cmds variable.

Also, once this is fixed, it becomes evident that the commands need to
be joined using whitespace, otherwise they all run together, and cannot
be executed as individual commands anymore.

Fixes: 0ffff2c1f8 ("rootfs-postcommands: Try and improve ordering constraints")
(From OE-Core rev: 2a4e8f06bac1bff0a167f775f7babab94b32732a)

Signed-off-by: Ralph Siemsen <ralph.siemsen@linaro.org>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-19 11:38:16 +00:00
Yoann Congal
131872001a oeqa/selftest: add a test for bitbake "-e" and "-getvar" difference
This is a non-regression test for [YOCTO #15638]

(From OE-Core rev: 22b508da24e0f7e5ad8ce4e090832bd0829963f0)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-18 22:09:03 +00:00
Ross Burton
3975cc6f4f oeqa/runtime/ping: don't bother trying to ping localhost
If SLIRP is being used instead of TAP for networking to the guest then
the target IP will be localhost.  There's no point in pinging localhost
to see if the target is up but whilst you'd think it is harmless, in
some containers ping doesn't actually have enough rights to work:

  ping: socktype: SOCK_RAW
  ping: socket: Operation not permitted
  ping: => missing cap_net_raw+p capability or setuid?

Look at the target address and if it's localhost or 127.0.0.* return
immediately.

(From OE-Core rev: a06ef43d2a50e16c32bd6edbdc7b32c3528687d5)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-18 22:09:03 +00:00
Deepesh Varatharajan
d8407ca2d0 rust: Remove passing test case from the exclude list in the Rust OE self-test.
We had previously excluded this test case due to failure in earlier versions.
However, with the latest version of Rust, this test case is passing.
As a result, we have removed it from the exclude list and added back into
the test suite for execution.

The rust-1.79 has been successfully tested with the current test inclusion.

(From OE-Core rev: a33edee3efe67f81229496e03a098454dc7df051)

Signed-off-by: Deepesh Varatharajan <Deepesh.Varatharajan@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-12 23:54:35 +00:00
Mikko Rapeli
30cc8b17d0 oeqa runtime uki.py: add tests
If Unified Kernel Image was enabled via IMAGE_CLASSES, then
target should also boot the same uki at runtime.

(From OE-Core rev: 2ae651c869a19fd24b5a915451c1f99e1d7b32f0)

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-12 11:31:48 +00:00
Ross Burton
109fe2eade selftest/container: fix IMAGE_FEATURES assignment
Assigning IMAGE_FEATURES="" in local.conf doesn't actually do anything
useful, as bitbake.conf has IMAGE_FEATURES += EXTRA_IMAGE_FEATURES after
the include of local.conf.

In this test case, this results in IMAGE_FEATURES still using
EXTRA_IMAGE_FEATURES, so the image contains the files installed by the
post-install-logging feature.

As the intention here is that container-test-image is as minimal as
possible, move the IMAGE_FEATURES assignment into the image itself so
it actually works, and update the expected file list to remove the
post-install-logging files.

(From OE-Core rev: c65349e1238ddc6634dfa4759c57e6168a355200)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-06 12:18:05 +00:00
Yoann Congal
b94b85bef0 oeqa/selftest/reproducibility: restart bitbake for each build
Since [0], the reproducibility test tries to save logs of failed builds.
Using the memory persistent bitbake prevent this by concatenating logs in the
single persistent instance (See [YOCTO #15641]).
So, force a BitBake server restart for each build to ensure a new log
file is created.

This fixes this error (seen on AB [1]):
  2024-11-02 08:16:00,816 - oe-selftest - ERROR - reproducibleA build failed. Trying to compute built packages differences but the test will fail.
  2024-11-02 08:16:00,819 - oe-selftest - INFO -  ... ERROR
  2024-11-02 08:16:00,819 - oe-selftest - INFO - Traceback (most recent call last):
    File "/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/meta/lib/oeqa/selftest/cases/reproducible.py", line 321, in test_reproducible_builds
      self.copy_file(variables["BB_CONSOLELOG"], os.path.join(save_dir, "bitbake-%s.log" % name))
    File "/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/meta/lib/oeqa/selftest/cases/reproducible.py", line 216, in copy_file
      shutil.copyfile(source, dest)
    File "/usr/lib/python3.11/shutil.py", line 256, in copyfile
      with open(src, 'rb') as fsrc:
           ^^^^^^^^^^^^^^^
  FileNotFoundError: [Errno 2] No such file or directory: '/srv/pokybuild/yocto-worker/reproducible-meta-oe/build/build/build-st-meta-python/build-st/tmp/log/cooker/qemux86-64/20241102071141.log'

[0]: https://git.openembedded.org/openembedded-core/commit/?id=c78cc753843d4199443052e8deb0c9c3b7e4b580
[1]: https://valkyrie.yoctoproject.org/#/builders/87/builds/17/steps/40/logs/stdio

(From OE-Core rev: e89bbc00ba16574d719b199c01ffbf37646f4f54)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-05 22:45:17 +00:00
Alexander Kanavin
24c1a31481 oeqa/runCmd: print stderr when that is a separate stream
runCmd by default merges stderr into stdout, and only
needs to print stdout when errors occur. When stderr
is requested as a separate stream, and an error occurs,
stderr is discarded, obscuring useful error messages.
This changes the output to include both streams.

(From OE-Core rev: 72c747b37ccdd486ddae06e3d0a99fb2b93643ba)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-01 23:03:32 +00:00
Vince Chang
806c6b974b selftest: wic: wic_sector_size add wic-tools path
Add wic-tools to the PATH to avoid failures when running the
wic_sector_size test case on a build host that doesn’t have parted.

(From OE-Core rev: 013dcdf75669421bc38d699263cb1e8d5b95d398)

Signed-off-by: Vince Chang <vince_chang@aspeedtech.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-01 23:03:32 +00:00
Vince Chang
e07cda3174 selftest: wic: remove 2>/dev/null to help fail diagnostic
The wic test case previously used '2>/dev/null' to suppress error
messages. This commit updates the code to use 'stderr=subprocess.PIPE'
when calling runCmd().

Refer:
https://lists.openembedded.org/g/openembedded-core/topic/109308684

(From OE-Core rev: bd26d999a0ba1107ee5629a8e238f4fe945e9be5)

Signed-off-by: Vince Chang <vince_chang@aspeedtech.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-01 23:03:32 +00:00
Katariina Lounento
f023756ef8 patchtest: add "Inactive-Upstream"
The list of valid statuses (`upstream_status_literal_valid_status`) was
missing "Inactive-Upstream", which caused patchtest to fail the test
test_patch.TestPatch.test_upstream_status_presence_format for patches
containing lines like:

    +Upstream-Status: Inactive-Upstream [lastrelease: 2013 lastcommit: 2013]

with the error:

    FAIL: test Upstream-Status presence: Upstream-Status is in incorrect format (test_patch.TestPatch.test_upstream_status_presence_format)

"Inactive-Upstream" is documented in the Yocto Project and OpenEmbedded
Contributor Guide [1]:

    Inactive-Upstream [lastcommit: when (and/or) lastrelease: when]

        The upstream is no longer available. This typically means a
        defunct project where no activity has happened for a long time —
        measured in years. To make that judgement, it is recommended to
        look at not only when the last release happened, but also when
        the last commit happened, and whether newly made bug reports and
        merge requests since that time receive no reaction. It is also
        recommended to add to the patch description any relevant links
        where the inactivity can be clearly seen.

`upstream_status_nonliteral_valid_status` only seems to be used in
logging and the value was copied verbatim from the aforementioned
documentation.

After this change all upstream status options documented in [1] are
covered.

[1] https://docs.yoctoproject.org/5.0.3/contributor-guide/recipe-style-guide.html#patch-upstream-status

(From OE-Core rev: a39e6e6854b60fd10f21c6c3f1e7d676e12ba9ee)

Signed-off-by: Katariina Lounento <katariina.lounento@vaisala.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-01 12:58:19 +00:00
Peter Marko
b2a04c8fd8 cve-check: fix malformed cve status description with : characters
When CPE is not provided and character ":" is in cve status description,
current code takes only last part of split function.
This works only if there is no ":" in description, otherwise it drops
the other split parts.

Do a new split of the original string to take the whole description unchanged.
This fixes following entries from world build of poky+meta-oe+meta-python:

tiff-4.6.0-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2015-7313
CVE_STATUS:  fixed-version: Tested with check from https://security-tracker.debian.org/tracker/CVE-2015-7313 and already 4.3.0 doesn't have the issue
description: //security-tracker.debian.org/tracker/CVE-2015-7313 and already 4.3.0 doesn't have the issue
corrected:   Tested with check from https://security-tracker.debian.org/tracker/CVE-2015-7313 and already 4.3.0 doesn't have the issue

gnupg-2.5.0-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2022-3219
CVE_STATUS:  upstream-wontfix: Upstream doesn't seem to be keen on merging the proposed commit - https://dev.gnupg.org/T5993
description: //dev.gnupg.org/T5993
corrected:   Upstream doesn't seem to be keen on merging the proposed commit - https://dev.gnupg.org/T5993

libyaml-0.2.5-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2024-35325
CVE_STATUS:  upstream-wontfix: Upstream thinks this is a misuse (or wrong use) of the libyaml API - https://github.com/yaml/libyaml/issues/303
description: //github.com/yaml/libyaml/issues/303
corrected:   Upstream thinks this is a misuse (or wrong use) of the libyaml API - https://github.com/yaml/libyaml/issues/303

libyaml-0.2.5-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2024-35326
CVE_STATUS:  upstream-wontfix: Upstream thinks there is no working code that is exploitable - https://github.com/yaml/libyaml/issues/302
description: //github.com/yaml/libyaml/issues/302
corrected:   Upstream thinks there is no working code that is exploitable - https://github.com/yaml/libyaml/issues/302

libyaml-0.2.5-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2024-35328
CVE_STATUS:  upstream-wontfix: Upstream thinks there is no working code that is exploitable - https://github.com/yaml/libyaml/issues/302
description: //github.com/yaml/libyaml/issues/302
corrected:   Upstream thinks there is no working code that is exploitable - https://github.com/yaml/libyaml/issues/302

cpio-2.15-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2023-7216
CVE_STATUS:  disputed: intended behaviour, see https://lists.gnu.org/archive/html/bug-cpio/2024-03/msg00000.html
description: //lists.gnu.org/archive/html/bug-cpio/2024-03/msg00000.html
corrected:   intended behaviour, see https://lists.gnu.org/archive/html/bug-cpio/2024-03/msg00000.html

openssh-9.9p1-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2023-51767
CVE_STATUS:  upstream-wontfix: It was demonstrated on modified sshd and does not exist in upstream openssh https://bugzilla.mindrot.org/show_bug.cgi?id=3656#c1.
description: //bugzilla.mindrot.org/show_bug.cgi?id=3656#c1.
corrected:   It was demonstrated on modified sshd and does not exist in upstream openssh https://bugzilla.mindrot.org/show_bug.cgi?id=3656#c1.

cups-2.4.10-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2021-25317
CVE_STATUS:  not-applicable-config: This concerns /var/log/cups having lp ownership, our /var/log/cups is root:root, so this doesn't apply.
description: root, so this doesn't apply.
corrected:   This concerns /var/log/cups having lp ownership, our /var/log/cups is root:root, so this doesn't apply.

unzip-1_6.0-r0 do_cve_check: CVE_STATUS with 3 parts for CVE-2008-0888
CVE_STATUS:  fixed-version: Patch from https://bugzilla.redhat.com/attachment.cgi?id=293893&action=diff applied to 6.0 source
description: //bugzilla.redhat.com/attachment.cgi?id=293893&action=diff applied to 6.0 source
corrected:   Patch from https://bugzilla.redhat.com/attachment.cgi?id=293893&action=diff applied to 6.0 source

syslog-ng-4.7.0-r0 do_cve_check: CVE_STATUS with 6 parts for CVE-2022-38725
CVE_STATUS:  cpe-incorrect: cve-check wrongly matches cpe:2.3🅰️oneidentity:syslog-ng:*:*:*:*:premium:*:*:* < 7.0.32
description: syslog-ng:*:*:*:*:premium:*:*:* < 7.0.32
corrected:   cve-check wrongly matches cpe:2.3🅰️oneidentity:syslog-ng:*:*:*:*:premium:*:*:* < 7.0.32

(From OE-Core rev: cc33dd9176726cb4b2d2f142ed1bc655da8e0a9f)

Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-01 12:58:19 +00:00
Peter Marko
b97a5f9b0e cve-check: do not skip cve status description after :
Correct maxsplit parameter from 5 to 4 to not drop text if
description contains ":".

Example:
>>> "detail: cpe:vendor:product:description:cont".split(':', 5)
['detail', ' cpe', 'vendor', 'product', 'description', 'xxx']
>>> "detail: cpe:vendor:product:description:cont".split(':', 4)
['detail', ' cpe', 'vendor', 'product', 'description:xxx']

(From OE-Core rev: 3c4d8ca41ac0b429af92bf0ea84f1dfd0cda9e1f)

Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-01 12:58:19 +00:00
Ross Burton
b67dce7ed0 lib/oe/package: remove PACKAGE_SNAP_LIB_SYMLINKS
This was added to OpenEmbedded in 2011[1] for the micro distro[2] and
subsequently ported to oe-core[3].  This feature aims to improve runtime
link speed by removing intermediate symlinks, i.e. libstdc++.so.6 is not
a symlink but the actual library.

However, there are several issues here:
- The meta-micro distribution has been unmaintained since 2012[4]
- The original creator of this function has confirmed that it is not in
  use anymore because the renaming doesn't affect the -dev packages, so
  on-target development or use of SDKs isn't possible.

Whilst the latter is possible to fix, the performance impact of removing
a single symlink resolution is negliable at best and the packaging code
is already very complex.

So, remove this functionality as it is known to be broken, isn't tested,
and removing it reduces complexity.

[1] https://git.openembedded.org/openembedded/commit/?id=cf7114179ead8ddff8f66e84d630811920ac9add
[2] https://git.openembedded.org/openembedded/tree/conf/distro/micro.conf
[3] https://git.openembedded.org/openembedded-core/commit/?id=600dbb7cb384c2290af38b993a9bea3a4dfc4494
[4] https://git.openembedded.org/meta-micro/

(From OE-Core rev: 9bfdf4c9eff16804588942a13ac7484059197f99)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-01 12:13:18 +00:00
Alexander Kanavin
0d3901b768 selftest/sstatetests: run CDN mirror check only once
The first no-fail check was an attempt to work around the old
CDN's instability (and it didn't really help); it should not be necessary
with the new CDN, and only delays a-full completion.

(From OE-Core rev: 7f75c42b7fcf60a9ca58d3ded9047df675d76dc2)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-01 12:13:17 +00:00
Hongxu Jia
ac8be80638 oeqa/selftest: Add SPDX 3.0 include source case for work-share
Build gcc and check gcc-14.2.0/README in objset is available

$ oe-selftest -r spdx.SPDX30Check.test_gcc_include_source
...
2024-10-26 01:24:57,063 - oe-selftest - INFO - test_gcc_include_source (spdx.SPDX30Check.test_gcc_include_source)
2024-10-26 01:28:24,204 - oe-selftest - INFO - The spdxId of gcc-14.2.0/README in gcc.spdx.json is http://spdx.org/spdxdocs/gcc-f2eaeb0d-b54b-53ba-899a-8c36c21139bf/88d5068ffd41e5ea6b4e0dd390b23bf499bb2b6674a41e09eaf2a887eced16c8/sourcefile/42
2024-10-26 01:28:26,369 - oe-selftest - INFO -  ... ok
2024-10-26 01:28:33,315 - oe-selftest - INFO - ----------------------------------------------------------------------
2024-10-26 01:28:33,316 - oe-selftest - INFO - Ran 1 test in 216.457s
2024-10-26 01:28:33,316 - oe-selftest - INFO - OK
2024-10-26 01:28:45,254 - oe-selftest - INFO - RESULTS:
2024-10-26 01:28:45,254 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_gcc_include_source: PASSED (209.31s)
2024-10-26 01:28:45,260 - oe-selftest - INFO - SUMMARY:
2024-10-26 01:28:45,260 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 216.457s
2024-10-26 01:28:45,260 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)

(From OE-Core rev: ccd6dde301dc8c45c8f901ebd4676b488d638b08)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Reviewed-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-01 12:07:12 +00:00
Hongxu Jia
ce192383d5 create-spdx-{2.2,3.0}: support SPDX include source for work-share directory
Originally, while SPDX_INCLUDE_SOURCES = "1" [1], there is bug in scan
for gcc, libgcc in which the sources locates in work-share directory.
Copy source from ${WORKDIR} to ${SPDXWORK} did not satisfy the situation
while ${S} was not included in ${WORKDIR}

This commit aim to support SPDX include source for work-share directory

1. If is_work_shared_spdx, Copy source from ${S} to ${SPDXWORK},
normally the dest dir in ${SPDXWORK} has the same basename dir of ${S};
but for kernel source, rename basename dir 'kernel-source' to ${BP} (${BPN}-${PV})

2. For SPDX source copy, do hard link copy to save copy time

3. Move do_patch to no work shared situation along with do_unpack

4. Tweak task do_create_spdx dependencies to assure the patched source
in work share is ready for SPDX source copy

5. Remove bb.data.inherits_class('kernel', d) from is_work_shared_spdx,
the kernel source locates in 'work-shared', test kernel.bbclass is not
necessary

[1] https://docs.yoctoproject.org/dev/ref-manual/variables.html#term-SPDX_INCLUDE_SOURCES

(From OE-Core rev: 64454b1956a9b50d6c89a3f3d7c594c1272cb289)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Reviewed-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-01 12:07:12 +00:00