Commit Graph

409 Commits

Author SHA1 Message Date
Richard Purdie
2aeb91cdb3 bitbake: bitbake-setup: Rename bb-layers-relative to bb-layers-file-relative
The difference between bb-layers and bb-layers-relative is unclear as both
are relative paths. Rename one to "file-relative" which makes it clear it
is relative to the current file, without becomming a long name.

https://lists.openembedded.org/g/bitbake-devel/message/18296

Based on a patch from Alexander Kanavin <alex@linutronix.de> but
with different naming.

(Bitbake rev: dcb17758b99767ab6da4172cf60eabc9269082dd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-11-07 10:20:32 +00:00
Alexander Kanavin
516039e609 bitbake: bitbake-setup: replace {THISDIR} token with a keyword: bb-layers-relative
{THISDIR} is a special value token that can be used in the list of enabled
layers to specify the layer location relative to the confguration file:
https://git.openembedded.org/bitbake/commit/?id=b3153be29de8b8570b0c184369bd41f4c646cf92

This replaces the token with an explicit separate keyword for such layers:
so that special processing to determine the final value can be avoided, and
the feature can be formalized in the json schema:

instead of
   "bb-layers": [
        "{THISDIR}/meta-my-project"
    ]

this allows
   "bb-layers-relative": [
        "meta-my-project"

Going forward I think we should strive to avoid any further special value tokens.

(Bitbake rev: 90da82bd2bfcfd5590c9ae06015737b616074b56)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-11-06 18:04:13 +00:00
Johannes Schneider
9780ec24d0 bitbake: bitbake-setup: capture revisions while checking out layers
When initializing a build setup from a conf.json that only sets 'rev'
to a tag or branch, the actual revision would not be captured or
logged.

To capture the current layer state after an 'init' or 'update', the
checkout_layers function is extended to store the revision the
bb.fetch.Fetch pulled, and write that information into a
sources-fixed-revisions.json file. This file can then be fed back into
bitbake-setup init as: --sources-overrides

This new 'sources-fixed-revisions.json' is written during 'update_build' and
stored alongside the 'config-upstream.json' in the config dir. And put
with the later under version control by calling 'commit_config" after
'update_build'.

The use of 'deepcopy' is necessary to not modify the original input
data - which python passes around as reference.

(Bitbake rev: 95866ff03f78e987ae7e47daad053bc0f353eea4)

Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-11-06 18:03:46 +00:00
Johannes Schneider
13b7a749ec bitbake: bitbake-setup: tests: move duplicate code into 'check_setupdir_files'
All places that called into 'check_setupdir_files' did the same
preparation step to load the upstream-config.json and then pass it
into the function.

Since the 'setuppath' is already passed into the function, and the
name and relative location of the upstream-config.json is fixed,
constructing the file path and loading the json could be done in the
function.

De-duplicate code by loading the json inside the function instead.

(Bitbake rev: 16d77c83ae3ce92ddab84d714a93fd3bb7def5e2)

Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-11-06 18:03:46 +00:00
Alexander Kanavin
5d7abe740e bitbake: bitbake-setup: rename 'build' -> 'setup'
After a terminology review by Antonin we brainstormed
something less confusing than 'build' and 'build directory'
for the place where bitbake-setup clones layers and
creates a *bitbake* build directory in. People are bound
to get these two confused and mix them up, and 'setup' is
much more distinct and aligns nicely with 'bitbake-setup'.
It's also not claimed by anything else in OE/Yocto.

So before:

top-dir -> build-dir (can be several) -> bitbake build dir, layer dir, config dir

Now:

top-dir -> setup-dir (can be several) -> bitbake build dir, layer dir, config dir

This also updates the respective command line options, I understand it's
a breaking change, but as before the tweaks are simple and we need to get the
terminology right for the users, and now is the time to do it.

(Bitbake rev: eeb81a35bf0304451f7612950d5156ea7ff18bad)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-11-04 13:16:04 +00:00
Johannes Schneider
7320e59aec bitbake: bitbake-setup: commandline: use subsubparser for settings {list,set,unset}
Previously the sub-command 'settings' would take any number of
arguments and then silently do nothing if the number wasn't three.

The help text was also not clear about this, marking the positionals
separately as optional:

usage: bitbake-setup settings [-h] [--global] [--unset UNSET UNSET] [-l] [section] [key] [value]

The '--unset SECTION SETTING' also did not  integrate too well, as it
had its own positional arguments for section+setting.

For a bit more consistency and a explorable help, a sub-subparser is
added, that provides the commands:
  bitbake-setup settings list
  bitbake-setup settings set foo bar baz
  bitbake-setup settings unset foo bar
with a '--global' that is added from a stand-alone parent parser, so
that it shows up in all sub-command help texts.

The new help text now reads:
usage: bitbake-setup settings [-h] [--global] {list,set,unset} ...

and the respective sub commands:
usage: bitbake-setup settings list [-h] [--global]
usage: bitbake-setup settings set [-h] [--global] <section> <setting> <value>
usage: bitbake-setup settings unset [-h] [--global] <section> <setting>

(Bitbake rev: 8b582ef8dd0cef0192d4c0104bcd9b5d642d132c)

Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-10-14 11:24:57 +01:00
Alexander Kanavin
cd4fe0aada bitbake: bitbake-setup: further rework the settings handling
After some further feedback, additional changes are made:

1. 'setting' command is renamed to 'settings' to better reflect
that it is an interface to various ways of managing settings.

2. This command now has a -l/--list option to list all settings
with their values (same as 'git config -l').

3. A new level of settings (built-in defaults) is added,
and used as a last resort after command line options, top dir
settings file and global settings file.

4. This means bitbake-setup does not have to write and use a
global settings file, and it no longer does so when initializing
a build, avoiding default 'pollution' of ~/.config/bitbake-setup/
which can be problematic or unwelcome.

A global settings file is still created if a setting is explicitly
requested to be placed into it.

5. 'install-global-settins' is removed as the use case for it
(tweak default settings before using them to initialize a build)
can be achieved by setting the settings individually.

5. Similarly, a top dir settings file is no longer created by default
and only appears if a setting needs to be written into it.

6. Default dl-dir is again created inside a top directory and not
in ~/.cache/ to make default builds fully contained in the top
directory (which was also asked about).

(Bitbake rev: 664f8ec48d42d2ddc5f234c4f7d590fa597f489a)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-10-14 11:24:57 +01:00
Alexander Kanavin
401e7b6a10 bitbake: bitbake-setup: rework the settings handling
This is the outcome of various discussions, suggestions and pull
requests on github.

What has specifically changed?

1. The sources for the settings are no longer separated, but are stacked and given priorities,
from highest to lowest:

    a. '--setting section key value' on the command line

    b. a settings file in the top directory

    c. a global settings file in ~/.config/bitbake-setup/ (or in a file pointed to by --global-settings)

Any setting can be in any of these three locations (other than top dir name and prefix which do not
make sense in the settings file in the top directory).

2. A global settings file must contain all of the needed settings, while a settings file
in the top directory can be empty (and this is how they are written out if they do not exist).

Specifically, both dl-dir and registry settings have been relocated to the global file,
and dl-dir defaults to ~/.cache/bitbake-setup/downloads, rather than somewhere in top dir.

3. The file name for both global and top dir settings is now 'settings.conf'.

4. --top-dir-prefix and --top-dir-name options have been removed and superseded by
a generic, universal --setting option.

5. 'install-settings' command has been removed, as it is no longer does anything useful,
and is superseded by the 'setting' command (see below).

'install-global-settings' has been retained, to be able to have a set of global defaults
that can be changed without initializing a build.

6. 'change-setting', 'change-global-setting' and 'install-settings' have all been replaced
by a single 'setting' command that mimics 'git config' in its parameters:

    a. Changing a setting: bitbake-setup setting [--global] default dl-dir /path/to/downloads

    b. Removing a setting: bitbake-setup setting [--global] --unset default dl-dir

(Bitbake rev: 713e7f213c6d4a620be9ce34d5f4396af48e1d69)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-10-14 11:24:57 +01:00
Johannes Schneider
e9345af4f8 bitbake: bitbake-setup: tests: add environment-passthrough
Add a test configuration to cover the 'bb-env-passthrough-additions'
conf.json key, and add it to the test routine.

(Bitbake rev: 24f12b68692f9ebb5d3813bc3b1771e43298f640)

Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-10-14 11:24:57 +01:00
Johannes Schneider
151a0f6361 bitbake: bitbake-setup: tests: add a template-only test configuration
bitbake-setup conf.json can be purely template driven, to setup older
yocto LTS (e.g. scarthgap) based layer-collections - where fragment
support was not yet present in oe-core+bitbake.

Add a configuration for that scenario and add it to the test routine.

(Bitbake rev: 23e121befa0779fbb1f342984b583c04ccc637a0)

Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-10-14 11:24:57 +01:00
Yoann Congal
20f58af1b3 bitbake: bitbake-setup: allow using {THISDIR}/my-layer
This implement the ability to use "{THISDIR}/my-layer" in the
"bb-layers" list. "{THISDIR}" is remplaced by the directory containing
the configuration file.

In small projects, we try to keep the setup a simple as possible: a
single git repo containing both the build confguration (e.g.
a bitbake-setup configuration file) and the meta layer with project
recipes/machine/distro.

This change allows this kind of setup:
├── meta-my-project/ # the project layer
└── my-project.conf.json   # the bb-setup configuration file

by writing, in my-project.conf.json:
  "bitbake-setup": {
    "configurations": [{
      "bb-layers": [
        "{THISDIR}/meta-my-project"

Note: in this case meta-my-project is not present as a "source", so, not
handled by bb-setup update/status. It is expected of the user to handle
this on their own (is our case, a simple git workflow).

(Bitbake rev: b3153be29de8b8570b0c184369bd41f4c646cf92)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-10-14 11:24:57 +01:00
Alexander Kanavin
9f820ce0f4 bitbake: lib/bb/tests/setup.py: unset BBPATH to ensure isolation from the existing bitbake environment
bitbake-setup deduces top directory from BBPATH, which, if set, interferes with
the tests' own setup.

(Bitbake rev: e974d42eb5229f755e14b46a00ad06b23b53e143)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-10-14 11:24:57 +01:00
Alexander Kanavin
bebbf40426 bitbake: lib/bb/tests/setup.py: define test parameters in a single dictionary
This makes maintaining and extending them easier.

(Bitbake rev: 16dc8e3dad7dde7e7651cce13549e61574cafba1)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-10-14 11:24:57 +01:00
Alexander Kanavin
1a55c45617 bitbake: bitbake-setup: add support for specifying branches in repo checkouts
Previously bitbake-setup was checking out 'detached commits' using
fetcher's nobranch feature, as that is the only option when only a revision is in the config.

Branches are optional, but beneficial, as

- checkout directory will be on a branch, making it easier for users
to understand where they are if they need to make changes (also
bitbake will print branch information instead of saying 'HEAD:sha').

- supply chain security! Enforcing a branch means any specified revision
has to be on it, and no one can sneak in (accidentally or deliberately!)
some dangling commit, or something from their private branch in the same repo.

(Bitbake rev: 45ed9b9faebdaa8cb7cc8dd2a6d51ec8eea06e73)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-10-14 11:24:57 +01:00
Alexander Kanavin
266379f0be bitbake: bitbake-setup: add 'install-buildtools' command
This basically calls install-buildtools from oe-core/poky, but
it ensures via command line parameters that the installation
location is stable and the downloads are preserved for reproducibility:

$ bin/bitbake-setup install-buildtools
Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf

======
Buildtools archive is downloaded into /home/alex/bitbake-builds/yocto-master-testing/buildtools-downloads/20250319141333 and its content installed into /home/alex/bitbake-builds/yocto-master-testing/buildtools

... (output from install-buildtools script)
======

It also detects when buildtools are already installed, and will direct
users what to do:
======
alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup install-buildtools
Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf

Buildtools are already installed in /home/alex/bitbake-builds/yocto-master-testing/buildtools.
If you wish to use them, you need to source the the environment setup script e.g.
$ . /home/alex/bitbake-builds/yocto-master-testing/buildtools/environment-setup-x86_64-pokysdk-linux
You can also re-run bitbake-setup install-buildtools with --force option to force a reinstallation
======

This commits includes fixes by Gyorgy Sarvari <skandigraun@gmail.com>
https://github.com/kanavin/bitbake/pull/2

(Bitbake rev: 3fe3096847046110c72b23fce37fb4a459b1d748)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-10-14 11:24:57 +01:00
Alexander Kanavin
a66c929a6a bitbake: bitbake-setup: add tests to bitbake-selftest
Run like this:

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-selftest -v bb.tests.setup
test_setup (bb.tests.setup.BitbakeSetupTest.test_setup) ... ok

----------------------------------------------------------------------
Ran 1 test in 9.223s

OK

The test does a basic run-through of init, then status/update
on an unchanged configuration, then status/update on a
configuration changed via new commits to the test layer,
then status/update on configuration changed via the top
level json config file.

Note that nothing whatsoever is fetched from the network;
the test relies entirely on synthetic data contained inside
itself, including minimal stubs for oe-setup-build and
bitbake-config-build. This data is used to create temporary
git repositories then clone them via local filesystem URIs.

Later on this can be supplemented by an oe-selftest that
tests bitbake-setup against real config files in the
official configuration repository and real layers,
templates and fragments.

(Bitbake rev: e3aa3eb46bd3196fa5415fa36e3737636fd6a1c0)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-10-14 11:24:57 +01:00
Peter Kjellerstedt
44029f9fb5 bitbake: tests/parse: Add tests for include, require and include_all
Simple tests for various ways to include and require configuration
files.

(Bitbake rev: bdcb67824172ed5cd76fa0274dc3d27aeb52e77c)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-09-24 00:01:44 +01:00
Peter Kjellerstedt
6cfa08f301 bitbake: tests/parse: Remove unnecessary calls to file.flush()
There is no reason to call file.flush() as the last call within a file
context manager since ending the context will close the file and thus
flush it.

(Bitbake rev: 90d5ce926d434d60d6df28b7d77b3cbc9b62ce14)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-09-24 00:01:44 +01:00
Peter Kjellerstedt
c948feaae4 bitbake: tests/parse: Use with statement to avoid ResourceWarning
(Bitbake rev: 46a5e31713fc526a1807b1617fba8a01c1564641)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-09-24 00:01:44 +01:00
Mathieu Dubois-Briand
e90bc48bda bitbake: tests/fetch: Update tests after bitbake tag removal
Tags for bitbake 2.8.6 and 2.8.7 have been removed from the git: use
some other ones.

(Bitbake rev: dcf12947954f3184d99fbf9813b3f3f667dacc5f)

Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-09-23 09:37:35 +01:00
Gyorgy Sarvari
3d451f3452 bitbake: fetch2/git: verify if local clone contains tag
In case a recipe specifies a git SRC_URI along with revision and tag, but only the
revision is present in the local clone without the tag (because it was tagged after
it was cloned), then unpacking fails with the following error:

... rev-list -n 1 1.0 failed with exit code 128, output:\nfatal: ambiguous argument \'1.0\': unknown revision or path not in the working tree

This happens because the during the download step only the revision's presence is
verified to decide if the repository needs to be updated.

To avoid this, check also if the tag is present in the local repository, when the "tag"
tag is specified.

(Bitbake rev: 546b347b4d3d82c01ecc99f45296f66e44638adc)

Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-08-19 11:35:52 +01:00
Joshua Watt
6fcda5cecd bitbake: lib/bb: Add filter support
Add the python API for applying filters to a string and being able to
register functions as filters.

Filter functions are pure functions where an input is translated into
an output and there are no external data accesses. This means translations
can be cached as they won't change.

(Bitbake rev: 7d25d7511ca14213eea78ee739d260295cfa4045)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-08-12 10:05:11 +01:00
Joshua Watt
d363bc475a bitbake: Use a "fork" multiprocessing context
Python 3.14 changes the default multiprocessing context from "fork" to
"forkserver"; however bitbake heavily relies on "fork" to efficiently
pass data to the child processes. As such, make "fork" context in the bb
namespace and use it in place of the normal multiprocessing module.

Note that multiprocessing contexts were added in Python 3.4, so this
should be safe to use even before Python 3.14

[YOCTO #15858]

(Bitbake rev: 62be9113d98fccb347c6aa0a10d5c4ee2857f8b6)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-07-23 11:34:31 +01:00
Richard Purdie
b0f8af26a7 bitbake: test/fetch: Switch u-boot based test to use our own mirror
The upstream servers are having issues so switch to our own shadow copy
of the repo.

(Bitbake rev: e910c7cd24fd366d6756641cd599c4efeb492e2a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-07-22 11:14:21 +01:00
Gyorgy Sarvari
1f88be64bf bitbake: bitbake/tests: fix typo in test
The test behavior did not change visibly though.

"bitbake-selftest bb.tests.runqueue" passes completely, just like before.

(Bitbake rev: 1751aed08f8472f20fcfbadbb09d35f951904952)

Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-07-14 13:29:30 +01:00
Richard Purdie
58b0a65ada bitbake: utils: Refactor filemode variable conversion to a function
We have other places in the code where we need to take filemode/mask
information from a bitbake variable and turn it into a real python
number. Turn this internal code into public API in bb.utils and
add some tests for it.

(Bitbake rev: d89e30fb2fb15b09f2cb95c4e5aa9f749ca257ea)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-07-01 08:49:37 +01:00
Koch, Stefan
d8b0adfaac bitbake: tests/fetch: Add test case to check shallow cloning using PREMIRRORS
(Bitbake rev: 6e1434d93d489aa4bab07777a7a9dc58ba0ca5a7)

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-28 12:37:02 +01:00
Chen Qi
89d2b3876d bitbake: lib/bb/tests/fetch: add a test case to ensure git shallow fetch works for tag containing slash
Add a test case to ensure git shallow fetch succeeds for SRC_URI
with tag containing slash.

For example, we want to succeed for SRC_URI like below:
SRC_URI = "git://salsa.debian.org/debian/debianutils.git;protocol=https;branch=master;tag=debian/${PV}"

See the following link for more information:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15862

(Bitbake rev: 919d4cf6e688e67229c46d30c84d523b21936377)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-15 10:05:49 +01:00
Philip Lorenz
a7331c3992 bitbake: tests/fetch: Test gitsm with LFS
Add a test case to verify that the gitsm fetcher properly handles
repositories storing objects with LFS.

The test case verifies that LFS objects are fetched on the initial clone
but also ensures that consecutive updates extend the original clone with
any newly referenced LFS objects.

(Bitbake rev: 2a8722ddd155596862029f6ea34e1e92c77e0b7f)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-08 11:37:32 +01:00
Philip Lorenz
7642477dc8 bitbake: fetch2: Check for git-lfs existence before using it
So far, existence of `git-lfs` was only checked during unpacking. As the
binary is also used in earlier steps also check for its existence there.

Additionally, factor out the LFS existence check into a dedicated
function and call it wherever git-lfs is used for the first time.

(Bitbake rev: 5818367db9b261b7e07c347d38044e6cba8f9727)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-05-08 11:37:32 +01:00
Philip Lorenz
62c4daafce bitbake: tests/fetch: Move commonly used imports to top
Avoid multiple import statements for anything that is used more than
once. Additionally, drop no longer used imports.

(Bitbake rev: 7c74310440f4d6ec47cf5bacf597e18308b3bb20)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-29 13:11:24 +01:00
Julian Haller
1def15acc2 bitbake: bitbake: tests/fetch: Fix git PREMIRRORONLY test
Using a shallow clone to simulate an outdated git mirror tarball does
not work in the intended way. A shallow clone already contains the
latest commit which can hide certain fetcher behavior. Simulate an
outdated mirror tarball, as the test titles indicate, by removing the
newer commits from the mirror.

(Bitbake rev: a51ee01f0a586fefd5a4061f4a1ca6cbf81b7046)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-10 11:07:38 +01:00
Richard Purdie
49ec431d95 bitbake: fetch2/git: Handle srcrevs for annotated tags in tag check
If SRCREV points at an annotated tag, the comparision code can fail
as the resolved tag might not be the same sha.

Handle this by also resolving the SRCREV. We only need to do this if
they don't match in the first place for a minor performance win.

Also add a test for this.

(Bitbake rev: 136c06e251de68ed64355ec6b47a522ff3a372e3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-03 11:03:43 +01:00
Ross Burton
43a2be2897 bitbake: tests/fetch: add a test for download paths without a proper filename
For example the miniupnpd recipe has a SRC_URI like this:

    http://miniupnp.tuxfamily.org/files/download.php?file=${BP}.tar.gz

In this case the path is /files/download.php, which isn't useful when
the latest_upstream logic bails early if there is no version in the path.

The logic now also checks in the downloadfilename, so add a test that
this works as expected.

(Bitbake rev: fffbf5d5e1c8556cddf0794e0b303bb0106747a0)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Ross Burton
58c015cdd7 bitbake: tests/fetch: support setting PV in the wget fetcher
Some code paths in latest_versionstring() need PV to be set correctly.

(Bitbake rev: 0a9f90ff658e09feda63b398ec35715a65ff6193)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Ross Burton
63246241f9 bitbake: tests/fetch: use a namedtuple for the wget test data
Use a named tuple so the test can access named members instead of just
accessing the data via index, which is harder to understand.

(Bitbake rev: 4b15652c84b06f0506c757e2647875a9b1cc7bfe)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Ross Burton
c3fc67960e bitbake: bb/tests: use subtests when looping in a test case
Marking the test iterations as subtests means that when one fails, it
can identify clearly which iteration has failed.

(Bitbake rev: 52c55e681332d7cdbe06f3c9d9c8d77cb0cb93f6)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Christian Lindeberg
c00ad319d4 bitbake: fetch2/gomod: Fix mirroring problem
Build the 'downloadfilename' parameter by replacing path separators in
the module path like the git fetcher builds the mirror tar ball name.
Copy the downloaded file in the fetcher's unpack method like the crate
fetcher instead of calling the base fetcher's unpack method.

(Bitbake rev: 7762cea087597019460d66b04268757bd46befdf)

Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 11:19:33 +00:00
Richard Purdie
bdd2b0fee1 bitbake: fetch: Drop multiple branch/revision support for single git urls
We used to use this for bare clones where a single git url could handle multiple
revisions (which implied multiple branches).

We don't use this any more and I doubt we'd want to go back to it. If we
remove it, we can simplfy the looping in the code which seems desireable.

This patch does change the warning for missing branch parameters to a
error. The message has hinted about that for long enough.

Some test cases are removed since they are no longer needed.

(Bitbake rev: 2515fbd10824005fa7f34e87706000c079920366)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-20 13:50:40 +00:00
Richard Purdie
cc5e31ceab bitbake: tests/fetch: Fix typo in npm test
(Bitbake rev: 79b04f61236117d310c12c1b1378ae63afb931ff)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 11:46:31 +00:00
Richard Purdie
122de6fd0f bitbake: tests/fetch: Add git tag verification tests
Add tests for git tag verification in both standard and shallow clones.

(Bitbake rev: f47127066d67e2ad80974fa1e7c0fcc7409161af)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 11:46:31 +00:00
Richard Purdie
18a533bfe6 bitbake: fetch/git: Rework tag parameter handling
Currently bitbake disallows tag parameters along with revision parameters.
This isn't great since quite often, we'd like to verify that a given revision
does match some tag. At the same time we don't want to or need to access
the network to verify this, which normally a tag would require.

Rework the code so that tag and revisions can both be specified together.
Verify that any tag specified matches the revision in use at unpack time.

This means we can start requiring people to put tags in git SRC_URIs
when revisions are used, making review a little easier that it isn't
some random revision.

The test that is dropped looks like a different test but the comment
is a copy and paste error. The SRCREV/rev mismatch test remains, this
removes the rev and tag set test.

(Bitbake rev: d591d7633fe8d739ec00395920e44910b0b77e27)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 11:46:30 +00:00
Richard Purdie
e8e34a04ce bitbake: tests/fetch: Add missing branch parameter to tests
(Bitbake rev: fd01e8e3a5a757d5f506095fc1ac4e45d888ae78)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 11:46:30 +00:00
Patrik Nordvall
8180865c22 bitbake: fetch2/git: Restore escape quoting for the git url when used
This fixes a bug where escapes in the url path would not be properly
restored for the git commands in the git fetcher. For example, a
space which is encoded as '%20' was not properly encoded before the
clone command.

e.g.

SRC_URI="git://git.openembedded.org/bitbake%20example/bitbake;protocol=https"

resulted in
git clone 'https://git.openembedded.org/bitbake example/bitbake'

instead of
git clone 'https://git.openembedded.org/bitbake%20example/bitbake'

(Bitbake rev: be48024253b93215bb110cd1d05925e789ec9680)

Signed-off-by: Patrik Nordvall <patrik.nordvall95@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-18 11:03:17 +00:00
Peter Kjellerstedt
55e65e6d70 bitbake: tests/fetch: Verify regular expression "URLs" that contain a ?
A regular expression "URL" in PREMIRRORS and MIRRORS may contain a ? as
part of the regular expression. Make sure this does not cause
problems.

(Bitbake rev: 5af7fe4473cd7e75d4eb7f8b93c499bd157ff156)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-07 15:38:15 +00:00
Stefan Koch
543b1e6ffb bitbake: tests/fetch: Add an additional test case to check whether the fast fetch is shallow
(Bitbake rev: 16f1961e077c525ccfc12496a3deca944df89fc6)

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-06 11:16:46 +00:00
Stefan Koch
db267f2c94 bitbake: tests/fetch: Adapt test cases for fast shallow fetches
- Address the absence of an initial full bare clone
- Utilize the initial shallow clone
- Modify existing test cases for this behavior
- Remove incompatible test cases

(Bitbake rev: 599fedacd7782dcb52825c22200f35344c102548)

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-06 11:16:46 +00:00
Mark Hatle
8ed26544f9 bitbake: bitbake: runqueue: Verify mcdepends are valid
In order to avoid a potentially confusing backtrace, check that the mcdepend
is valid when we add it.

Add a test case to ensure invalid configurations are caught and trigger an
error.

[RP: Reworked test case to simplify and improve code]
(Bitbake rev: ff523497270f37b484b44a4445c2194791bcb6ff)

Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-03 21:38:57 +00:00
Louis Rannou
2ae32dae58 bitbake: bitbake: tests/data: add tests for variable flags
Check default flags are correctly returned by getVarFlags and check all flags
are returned when internalflags is True.

Check delVarFlags also removes default value.

Check all flags are removed after delVar.

Run the test with:
    $ bitbake-selftest -v bb.tests.data.TestFlags
    test_delflag (bb.tests.data.TestFlags.test_delflag) ... ok
    test_delvar (bb.tests.data.TestFlags.test_delvar) ... ok
    test_setflag (bb.tests.data.TestFlags.test_setflag) ... ok

    ----------------------------------------------------------------------
    Ran 3 tests in 0.000s

    OK

This is a test case for [YOCTO #15685]

(Bitbake rev: ff8cae735cf489373af1aac7ee233d7b82d483d3)

Signed-off-by: Louis Rannou <louis.rannou@non.se.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-13 22:24:27 +00:00
Stefan Herbrechtsmeier
851b24cf81 bitbake: tests: fetch: quote URI password as per RFC3986
The password must be quoted as per RFC3986 because it can contain
reserved characters.

(Bitbake rev: 5c53dbf0ad4385e1de7f2eef66565a1f05dae67a)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00