BB_ORIGENV is used as a fallback environment block. It is repeatedly
accessed inside the loop. Since it is a loop invariant, move it out of
the loop.
(Bitbake rev: 346a1a6c76d40458d7b4c116147ec4d371bee74a)
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", line 328, in checkstatus
with bb.utils.environment(**newenv):
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/contextlib.py", line 142, in __exit__
next(self.gen)
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/utils.py", line 1737, in environment
del os.environ[var]
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/os.py", line 695, in __delitem__
raise KeyError(key) from None
It is possible the wrapped function might change the environment which can lead
to KeyError issues like this. The key may no longer be in the original environment
so handle that case correctly and avoid the tracebacks.
(Bitbake rev: ae4ce4c21998bb2a12a1e1f0b8e0af8d218b87e2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some cases where the metadata needs to check if a variable
has any overrides set, even if they are currently inactive. That code
currently pokes into datastore internals. Add API instead to replace
and avoid that.
(Bitbake rev: c885e5542dcf760b8fc5881e385abb4a10020874)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Loggers created with a logging configuration file automatically get the
once filter applied to ensure that warnonce()/erroronce() only appear a
single time. The same filter needs to also be added to
bb.msg.logger_create() to ensure that manually created loggers have the
same behavior
(Bitbake rev: bd93f7baff9d56de37a1af474189e0888b82d475)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds a log level and logging function call to use it where the
warning or error will only be displayed once, regardless of how many
times the message is logged.
This has to be done either in the cooker or on the UI side. I've opted
for the UI side since display control is really a UI issue but it uses
a common library filter function to enable it which can be reused
elsewhere.
The knotty message displayed as the build summary is tweaked to
make sense when the numbers won't match since it will still count
the number of times it was logged and this is probably helpful
for debugging in some cases so I've deliberately left it that way.
(Bitbake rev: 7bd40e3003a043e3cb7efc276681054b563b5e7b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a recipe raises a BBHandledException, it means the error was already
shown to the user. Adding an additional one here isn't helpful. What is
helpful is to mention that parsing was halted.
Tweak the code to do this with improves the messages the user sees
and helps understand what happened.
(Bitbake rev: fdf6ebc8b603fcfd3ed7c64baf486a4adabd25be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The overrides warning message is meant to show filename and line
numbers but the variable names are incorrect and this wasn't working.
Fix it.
(Bitbake rev: 551c1cb20fc9b9d0dab5d830182c2bf626e72845)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Export ClientError and ServerError classes from bb.asyncrpc.serv as
required by lib/hashserv/server.py module.
Error:
Client ('192.168.2.184', 51142) connected
Error from client: module 'bb.asyncrpc' has no attribute 'ClientError'
Traceback (most recent call last):
File "/opt/bbhashserv/lib/bb/asyncrpc/serv.py", line 192, in handle_client
await client.process_requests()
File "/opt/bbhashserv/lib/hashserv/server.py", line 200, in process_requests
await super().process_requests()
File "/opt/bbhashserv/lib/bb/asyncrpc/serv.py", line 72, in process_requests
await self.dispatch_message(d)
File "/opt/bbhashserv/lib/hashserv/server.py", line 217, in dispatch_message
raise bb.asyncrpc.ClientError("Unrecognized command %r" % msg)
AttributeError: module 'bb.asyncrpc' has no attribute 'ClientError'
(Bitbake rev: 8aca1d341500af8bbe788801c2df3da9bbfacaed)
Signed-off-by: Daniel Gomez <daniel@qtec.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When building the Hash Equivalence server Dockerfile, some
dependencies are missing in order to run the hash server properly:
Traceback errors:
Traceback (most recent call last):
File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
ret = main()
File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
from . import server
File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
import bb.asyncrpc
ModuleNotFoundError: No module named 'bb'
Traceback (most recent call last):
File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
ret = main()
File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
from . import server
File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
import bb.asyncrpc
File "/opt/bbhashserv/lib/bb/__init__.py", line 128, in <module>
from bb import fetch2 as fetch
File "/opt/bbhashserv/lib/bb/fetch2/__init__.py", line 1930, in <module>
from . import git
File "/opt/bbhashserv/lib/bb/fetch2/git.py", line 70, in <module>
import bb.progress
File "/opt/bbhashserv/lib/bb/progress.py", line 14, in <module>
import bb.build
File "/opt/bbhashserv/lib/bb/build.py", line 27, in <module>
from bb import data, event, utils
File "/opt/bbhashserv/lib/bb/data.py", line 36, in <module>
from bb import data_smart
File "/opt/bbhashserv/lib/bb/data_smart.py", line 23, in <module>
import bb, bb.codeparser
File "/opt/bbhashserv/lib/bb/codeparser.py", line 26, in <module>
import codegen
ModuleNotFoundError: No module named 'codegen'
Traceback (most recent call last):
File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
ret = main()
File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
from . import server
File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
import bb.asyncrpc
File "/opt/bbhashserv/lib/bb/__init__.py", line 128, in <module>
from bb import fetch2 as fetch
File "/opt/bbhashserv/lib/bb/fetch2/__init__.py", line 1930, in <module>
from . import git
File "/opt/bbhashserv/lib/bb/fetch2/git.py", line 70, in <module>
import bb.progress
File "/opt/bbhashserv/lib/bb/progress.py", line 14, in <module>
import bb.build
File "/opt/bbhashserv/lib/bb/build.py", line 27, in <module>
from bb import data, event, utils
File "/opt/bbhashserv/lib/bb/data.py", line 36, in <module>
from bb import data_smart
File "/opt/bbhashserv/lib/bb/data_smart.py", line 23, in <module>
import bb, bb.codeparser
File "/opt/bbhashserv/lib/bb/codeparser.py", line 32, in <module>
from bb.pysh import pyshyacc, pyshlex
File "/opt/bbhashserv/lib/bb/pysh/pyshyacc.py", line 13, in <module>
import bb.pysh.pyshlex as pyshlex
File "/opt/bbhashserv/lib/bb/pysh/pyshlex.py", line 17, in <module>
from ply import lex
ModuleNotFoundError: No module named 'ply'
Traceback (most recent call last):
File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
ret = main()
File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
from . import server
File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
import bb.asyncrpc
File "/opt/bbhashserv/lib/bb/__init__.py", line 128, in <module>
from bb import fetch2 as fetch
File "/opt/bbhashserv/lib/bb/fetch2/__init__.py", line 1935, in <module>
from . import wget
File "/opt/bbhashserv/lib/bb/fetch2/wget.py", line 30, in <module>
from bs4 import BeautifulSoup
ModuleNotFoundError: No module named 'bs4'
(Bitbake rev: 0bd637acfcba5a44230c291889d2a5ff571cb8c6)
Signed-off-by: Daniel Gomez <daniel@qtec.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where a git url uses a tag instead of a full source revision, breakage
can currently occur in builds. Issues include:
* the revision being looked up in multiple tasks (fetch and unpack)
* the risk a different revision may be obtained in those tasks
* that some tasks may not be allowed to access the network
* that a revision may not be consistent throughout a given build
* rerunning a specific task may given inconsistent results
To fix this, stop the workers from cleaning out the source revision store. This
should only be done in the cooker itself (based on current policy).
Also, where the code "sees" an upstream access, mark the recipe as not to be
cached. The reparse re-triggers the upstream lookup by the server.
Add a test to ensure that if get_srcrev isn't called, the user is told they're
using a configuration that is known to break.
(Bitbake rev: 4b5eed1626709ef3dc06b32fd55d40a2a6edd179)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We started to require the branch parameter for git urls. This updates
the tests to match and quieten some noise.
(Bitbake rev: e5144887f2b6ae266f7b52151005619480e80ec6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously npm files that specify downloadfilename= in the SRC_URI
would be downloaded to the root of ${DL_DIR} rather than in the
${DL_DIR}/npm2 directory where all other npm files are downloaded.
This should make it simpler when setting up and configuring a
premirror with the downloaded npm packages.
(Bitbake rev: 73fa376d6502ab6f2cccfb25a1193d9b1c3c3bc8)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The solution implementated in commit 96c30007 (fetch2: fix
downloadfilename issue with premirror) missed two corner cases. The
first is if the basename of the original URI also appears somewhere
else in the replacement URI, in which case it would also be replaced.
The second is if the basename of the original URI partially matches
the basename of the replacement URI, in which case the inital part of
the basename from the replacement URI would be left behind.
The second case caused test_npm_premirrors_with_specified_filename to
fail.
The solution is to prefix the basename with a slash when matching to
avoid partial matches, and only replace the basename at the end of the
URI.
This also adds two test cases that test for these problems. Before
this they would give the following errors:
- ['file:///mirror/example/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
+ ['file:///mirror/some-example-1.0.0.tgz/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
? +++++ ++++++++++
- ['file:///mirror/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
+ ['file:///mirror/some-some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
? +++++
(Bitbake rev: 5924c6f007519cd8ea6cc8b316814d17b43048ca)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are two totally opposite use cases for how a premirror is
expected to behave in combination with specifying a downloadfilename=
parameter in the SRC_URI. On the one hand there is the expectation
that it works like any other mirror, which means the premirror is
expected to contain a file with the original name specified in the
SRC_URI. On the other hand there is the expectation that one can use
the artefacts downloaded by bitbake in ${DL_DIR} as a premirror, in
which case it is expected to contain a file with the name from the
downloadfilename= parameter.
The latter case has been how downloaded files have been handled until
commit 8a3ff9f3 (fetch2: fix premirror URI when downloadfilename
defined), where the fetcher was changed to store files as per the
first case. This is also when the test_npm_premirrors test case was
modified in commit 5ba191a0 (tests/fetch: add and fix npm tests) to
expect the first case.
The above change was later reverted in commit 96c30007 (fetch2: fix
downloadfilename issue with premirror). However the
test_npm_premirrors test case was not updated to match, and has been
failing ever since. This has probably gone unnoticed because the npm
related test cases require that npm is installed on the host.
This commit updates test_npm_premirrors to expect that premirrors use
the filenames specified by downloadfilename= as this matches the
current fetcher implementation and also is the most likely use case
for premirrors. It also tries to mimic how one typically might setup
the premirror directory by simply copying the download directory.
(Bitbake rev: 9e913ade70474aaeb928814d4763e7105569d63a)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This unifies the three separate implementations of git() from
GitMakeShallowTest, GitShallowTest and GitLfsTest into one
implementation in FetcherTest. It also makes use of this in
FetcherLocalTest and FetcherNetworkTest.
(Bitbake rev: 8c8b418c3e6f8ab2535ad405e8006ee1012f3484)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This changes:
test_foo .. skipped 'Network tests being skipped'
test_foo .. skipped 'npm not installed, tests being skipped'
to:
test_foo .. skipped 'network test'
test_foo .. skipped 'npm not installed'
to avoid double "skipped" in the output.
(Bitbake rev: e551328a4595c220b94ab5002d978ddb4deeebcf)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since commit 044fb04d (fetch2: Allow whitespace only mirror entries)
there is no need to separate the entries in mirror variables with
"\n".
(Bitbake rev: ed22b078aae25e804ca5edf062e2c905d3e49426)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Both test_crate_url and test_crate_url_multi require Internet access.
(Bitbake rev: 34e9daed6a7f766bb6e27c7e02c94b545f4791d8)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since commit 044fb04d (fetch2: Allow whitespace only mirror entries)
there is no need to separate the entries in PREMIRRORS with "\n".
(Bitbake rev: 341746c7f7b8e04d3fad51ec1b9e462ced355bfc)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The NIS can't work when network is dissable, so preserve network for it, the
error is like:
do_ypcall: clnt_call: RPC: Unable to send; errno = Network is unreachable
Note, enable nscd on the build machine might be a solution, but that isn't
reliable since it depends on whether the network function has been cached or
not.
(Bitbake rev: 4eafae7904bae6e5c6bc50356e8a9077f2e207fa)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This imports the crate fetcher from OE-Core to resolve various module issues
and adds some very very basic tests of that new fetcher.
(Bitbake rev: 1f06f326fa8b47e2a4dce756d57a9369a2225201)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an unexpected exception occurs in a setscene task, it is currently hidden
from the user and not recorded in any logs. This isn't helpful to debug
such failures.
Change the code so that even in the "silent" or "quiet" task case (setscene
tasks), a warning is shown with the traceback unless it was an "handled"
exception. This means the failing function can show it's own warning/error
instead if it wants to and then raise a handled event.
(Bitbake rev: 41dcdc61eb40def8c14a42e8d7bb9ce5a34afa57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These variables were removed from the codebase, update the docs to match.
(Bitbake rev: 32180d5057c818a69987aada482e82acf3c72ef2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The URL expected by git fetcher when the SSH protocol is used differs
from the one given by Git servers to be used with the `git clone`
command.
Add a note making this specificity known to users.
Cc: Quentin Schulz <foss+yocto@0leil.net>
(Bitbake rev: 81c09d434aed13fcfc4ba426ae6a0b849447eb5f)
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The different stamp policies were poor versions of the siggen code and task
hashes, predating it and being used by packaged staging. They had many
limitations, hence their replacement. I'm not aware of any users of that
code any more so I believe it and the assoicated stamp whitelist variable
can simply be removed.
(Bitbake rev: 98407efc8c670abd71d3fa88ec3776ee9b5c38f3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch changes behaviour. After this change any task which does not
have the network flag set will have networking disabled on systems that
support that (kernel version dependent).
Add a "network" task specific flag which then triggers networking to
be enabled for this task, it is otherwise disabled.
This needs to happen before we enter the fakeroot environment of the task
due to the need for the real uid/gid which we save in the parent process.
(Bitbake rev: 0746b6a2a32fec4c18bf1a52b1454ca4c04bf543)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a function which uses the unshare glibc call to disable networking
in the current process. This doesn't work on older distros/kernels
but will on more recent ones so for now we simply ignore the cases we
can't execute on. uid/gid can be passed in externally so this can
work with pseudo/fakeroot contexts.
(Bitbake rev: 9d6341df611a1725090444f6f8eb0244aed08213)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is deprecated in python 3.12 and Fedora 35 is throwing warnings so
move to the new functions.
(Bitbake rev: 68a18fbcb5959e334cf307d7fa8dc63832edb942)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These urls are no longer adding much to the test coverage but the intermittent
network issues connecting to them are painful. Drop the urls.
(Bitbake rev: bdf5739c5d831dc97a7d81568f94a0953c71017f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Always use a temporary file for the user config 'NPM_CONFIG_USERCONFIG'
because npm otherwise failed if configs and npmrc aren't set:
double-loading config "/dev/null" as "global", previously loaded as "user"
(Bitbake rev: 9f272ad7f76c1559e745e9af686d0a529f917659)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Quote destdir in run chmod command to support special characters in
package name and to avoid syntax error for packages like
'@(._.)/execute'.
(Bitbake rev: a701dfce3f0e74b4d7c687eeda83fe9c8e7240b1)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Internal bitbake shutdown timings must have changed recently as we're
seeing race issues on the autobuilder around the removal of the bitbake.lock
file. Improve the lockfile race code to cover bitbake's lockfile too
and use it in all the tests.
[YOCTO #14658]
[YOCTO #14652]
(Bitbake rev: bd1912bed64424f9fb28396b71bb49b6090ed087)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should not redirect stderr to stdout if we need to get separated
stdout and stderr contents from Popen.communicate() later.
(Bitbake rev: 1ecc1d9424877df89fcda2f23c306998998a65ff)
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the console output for signature dumps e.g. with bitbake-dumpsig
isn't deterministic. Add some sorting to improve that.
(Bitbake rev: a663440b1623f97c9c169df5566e429fbc932a53)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a parameter that limits sending Basic authentication in the
Authorization header to only the first host and not any that we're
redirected to. Ignoring potential security concerns, temporary AWS URLs
will reject any request that includes authentication details in both the
query parameters (from the redirect) and in the Authorization header.
Temporary AWS URLs are now being used for release assets from private
Github repositories. According to the previous discussion linked below,
they're also in use by bitbucket.
See also:
https://lore.kernel.org/bitbake-devel/CAC9ffDEuZL-k8199bUyN+8frjw6bg-g=vrumxxtvt+RVParQ8Q@mail.gmail.com/
(Bitbake rev: a6ab32013a4381a1b694ed46caf2c9da932644d0)
Signed-off-by: Justin Bronder <jsbronder@cold-front.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a parameter `striplevel` to the SRC_URI to strip NUMBER leading
components (levels) from file names on extraction.
For example, if the archive `archive.tar.gz` contains `some/file`,
the SRC_URI `https://.../archive.tar.gz;subdir=other;striplevel=1`
will extract `some/file` to `other/file`.
This is useful to extract archives to a specified directory instead of
the original root component of the archive. The feature is required for
the npm support. The npm package contents should reside in a subfolder
inside a npm archive (usually it is called package/). npm strips one
directory layer when installing the package.
(Bitbake rev: aa4926e5d9c92f33b4434e2da709ff0bf3049f5b)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The tar command and its arguments are repeated for many archive types in
the unpack function. Unify the common parts in a variable to prepare
further extension.
(Bitbake rev: a08e57c9eaec1d9740a96149bf4843e576da4e5c)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
