If a custom python module is added thru BBIMPORTS and it
uses typing(Any,Tuple,Union...), codeparser will fail because
inspect.py raises TypeError exception if the object is a
built-in module, class, or function.
(Bitbake rev: 0ecfd0b8540220633e71d24cd73cc5306863ae3c)
Signed-off-by: Pedro Silva Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A regular expression "URL" in PREMIRRORS and MIRRORS may contain a ? as
part of the regular expression. Make sure this does not cause
problems.
(Bitbake rev: 5af7fe4473cd7e75d4eb7f8b93c499bd157ff156)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This partial reverts commit a5d569c94700f04b8193c6bccae5af619931b00f
which changes decodeurl() to use the URI class to parse the URL instead
of doing it itself. While reusing code is generally a good idea, using
urllib.parse.urlparse() (which the URI class does) to parse the regular
expression "URLs" that are used in PREMIRRORS and MIRRORS does not work.
A regular expression URL containing https?://... would be silently
ignored, while a URL using a negative lookahead such as
git://(?!internal\.git\.server).*/.* would result in a cryptic error:
Exception: re.error: missing ), unterminated subpattern at position 0
The problem is that urllib.parse.urlparse() treats the ? as the start of
URL parameters and thus stops parsing whatever part of the URL it was
parsing. Restore the old function and use it in the PREMIRRORS and
MIRRORS code.
(Bitbake rev: f8a7712754e6d0199a0d227fca288307b935368d)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At the end of this function, self.overrides is cleared, which means we'll
construct a new self.overrides after this call. And a new self.overrides
will give out different expand_cache, so the old expand_cache should also
be cleared to avoid any wrong value.
Currently, there's a problem revealed by recent recipe specific virtual
provider patch. If we enable multilib and set "OVERRIDES:prepend" in local.conf,
things don't work.
Here's the error message:
ERROR: Nothing PROVIDES 'lib32-gcc-cross-x86_64'
Below are reproduce steps:
1. Add in local.conf the following lines:
MACHINE ?= "qemux86-64"
require conf/multilib.conf
MULTILIBS ?= "multilib:lib32"
DEFAULTTUNE:virtclass-multilib-lib32 ?= "core2-32"
OVERRIDES:prepend = "some-override:"
(Note that using :append and :remove also reproduces the issue.)
2. bitbake -n lib32-sysstat
(bitbake -n core-image-minimal also reproduces the issue)
The expandWithRefs calls getVar, which fills expand_cache. So when setting
OVERRIDES:prepend, this will fill the expand_cache. When overridevars are updated,
if we don't clear expand_cache, we'll retrieve wrong values.
Previously, things happened to work because there's a call to expand
PREFERRED_PROVIDER_virtual/${TARGET_PREFIX}gcc, which in turn expands
TARGET_VENDOR. Now what we expand is PREFERRED_PROVIDER_ virtual/cross-cc,
so the problem is revealed.
(Bitbake rev: 842e94234c9b9faf4e902721f516071f643e49c9)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When var matches __setvar_regexp__, which has the value as below,
__setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>:append|:prepend|:remove)(:(?P<add>[^A-Z]*))?$')
the keyword will not be __doc__ or __module__ and var will always
contain ":".
(Bitbake rev: 0af396917fe176df7b1ed0bd7c7267507ed12e03)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The non-greedy modifier suffix ? is not necessary here because } is
not in the character set [a-zA-Z0-9\-_+./~:].
(Bitbake rev: 80735719d0002c02b704a49604ddd82c461203a2)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously if the websockets module was not installed
bitbake would only print a warning and continue, resulting in a degraded user
experience due to inability to use the configured sstate server.
Let's consider that as fatal misconfiguration, so that users can address
the issue properly and not wonder why builds are taking forever.
(Bitbake rev: cfba2a9fca9dfa3b05ec9040fe0cb8143ac04af7)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- Address the absence of an initial full bare clone
- Utilize the initial shallow clone
- Modify existing test cases for this behavior
- Remove incompatible test cases
(Bitbake rev: 599fedacd7782dcb52825c22200f35344c102548)
Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When `ud.shallow == 1`:
- Prefer an initial shallow clone over an initial full bare clone,
while still utilizing any already existing full bare clones.
- If the Git error "Server does not allow request for unadvertised object"
occurs, the initial full bare clone is fetched automatically.
This may happen if the Git server does not allow the request
or if the Git client has issues with this functionality,
especially with the Git client from Ubuntu 20.04.
This improves:
- Resolve timeout issues during initial clones on slow internet connections
by reducing the amount of data transferred.
- Eliminate the need to use an HTTPS tarball `SRC_URI`
to reduce data transfer.
- Allow SSH-based authentication (e.g. cert and agent-based) when
using non-public repos, so additional HTTPS tokens may not be required.
(Bitbake rev: 457288b2fda86fd00cdcaefac616129b0029e1f9)
Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the fixture changes, the tests need to be tweaked unfotunately.
(Bitbake rev: 708abd1a8060684127acc7ce4142f05865005750)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to validate the toaster fixtures information against our current release
data to ensure it is correct when we release. Add a script we can use to
do this which the autobuilder can run to validate things.
[YOCTO #15516]
(Bitbake rev: 5b2d79ed505bbfa2fb2d355935e75199b7f2c37e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to avoid a potentially confusing backtrace, check that the mcdepend
is valid when we add it.
Add a test case to ensure invalid configurations are caught and trigger an
error.
[RP: Reworked test case to simplify and improve code]
(Bitbake rev: ff523497270f37b484b44a4445c2194791bcb6ff)
Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've been seeing intermittent failures on Ubuntu 22.04 in oe-selftest which
were problematic to debug. The failure was inside lock_timeout and once that was
identified and the backtrace obtained, the problem becomes clearer:
File "X/bitbake/lib/bb/server/process.py", line 466, in idle_thread_internal
retval = function(self, data, False)
File "X/bitbake/lib/bb/command.py", line 123, in runAsyncCommand
self.cooker.updateCache()
File "X/bitbake/lib/bb/cooker.py", line 1629, in updateCache
self.parser = CookerParser(self, mcfilelist, total_masked)
File "X/bitbake/lib/bb/cooker.py", line 2141, in __init__
self.bb_caches = bb.cache.MulticonfigCache(self.cfgbuilder, self.cfghash, cooker.caches_array)
File "X/bitbake/lib/bb/cache.py", line 772, in __init__
loaded += c.prepare_cache(progress)
File "X/bitbake/lib/bb/cache.py", line 435, in prepare_cache
loaded = self.load_cachefile(progress)
File "X/bitbake/lib/bb/cache.py", line 516, in load_cachefile
progress(cachefile.tell() + previous_progress)
File "X/bitbake/lib/bb/cache.py", line 751, in progress
bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize),
File "X/bitbake/lib/bb/event.py", line 234, in fire
fire_ui_handlers(event, d)
File "X/bitbake/lib/bb/event.py", line 210, in fire_ui_handlers
_ui_handlers[h].event.send(event)
File "X/bitbake/lib/bb/cooker.py", line 117, in send
str_event = codecs.encode(pickle.dumps(event), \'base64\').decode(\'utf-8\')
File "/usr/lib/python3.10/asyncio/sslproto.py", line 320, in __del__
_warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg
sw(msg.message, msg.category, msg.filename, msg.lineno,
File "X/bitbake/lib/bb/main.py", line 113, in _showwarning
warnlog.warning(s)
File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning
self._log(WARNING, msg, args, **kwargs)
File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
self.handle(record)
File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
self.callHandlers(record)
File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
self.emit(record)
File "X/bitbake/lib/bb/event.py", line 778, in emit
fire(record, None)
File "X/bitbake/lib/bb/event.py", line 234, in fire
fire_ui_handlers(event, d)
File "X/bitbake/lib/bb/event.py", line 197, in fire_ui_handlers
with bb.utils.lock_timeout(_thread_lock):
File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "X/bitbake/lib/bb/utils.py", line 1888, in lock_timeout
bb.server.process.serverlog("Couldn\'t get the lock for 5 mins, timed out, exiting. %s" % traceback.format_stack())
or put in simpler terms, whilst sending an event(), an unrelated warning
message happens to be triggered from asyncio:
/usr/lib/python3.10/asyncio/sslproto.py:320: ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x7f0e797d3100>
which triggers a second event() which can't be sent as we're already
in the critcal section and already hold the lock.
That warning is due to the version of asyncio used on Ubuntu 22.04 with
python 3.10 and that comined with timing issues explains why we don't
see it on other python versions or distros.
We can't handle the second event as the lock is there to serialise the
events. Instead, we queue the event and then process the queue later.
Add a new version of lock_timeout which allows us to handle the situation
more gracefully.
(Bitbake rev: 2c590ff1aff89d23b25ce808650f200013a1e6af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We never want to exit whilst holding these locks as it deadlocks all python
threads. Add signal blocking around the lock critical part so a signal
shouldn't cause such an exit.
(Bitbake rev: a097755c671e2b530dea6200a94b39fa9dca246c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should really try and take the lock in the try/finally block so that
in some rare cases such as badly timed interrupt/signal, we always release
the lock.
(Bitbake rev: a9eb8bf7174b6962b5ba07192fe95b8c7112d9d2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is an OpenEmbedded-specific variable that is no longer used[1].
[1] oe-core 1f56155e ("base: Switch to use addpylib directive and
BB_GLOBAL_PYMODULES")
(Bitbake rev: 34bfa55a3c14f5d1d6bb3e31b747906fa7cb99c3)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using BB_FETCH_PREMIRRORONLY we write to the datastore to disable the network.
This change needs to be undo when handling later urls, so operate on a copy of the
datastore to allow this.
Reported by Julian Haller <julian.haller@philips.com>
(Bitbake rev: 67a5ede8ae92ed7dcad29fd0dcfd62c6640b10b2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The minimum Python version required for Yocto 5.0 is 3.8 which causes
failure in poky/bitbake/lib/bblayers/query.py when listing layers by
using command "bitbake-layers show-recipes -f --bare --mc MC" for the
given multiconfig MC.
The reason for that failure is the use of "removeprefix" string method
which got introduced in Python 3.9.
This patch replaces the "removeprefix" method with an equivalent
solution supported by Python 3.8.
(Bitbake rev: 004cfdec1c865f2351bbac99acb3d63bfef9d380)
Signed-off-by: Joerg Schmidt <joerg.schmidt@garmin.com>
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Remove double imports
* Re-roder the imports as mentioned by pylint
(Bitbake rev: 6de536bbaee8cf7664c5702a96a5ca18ad09fb9d)
Signed-off-by: Michael Estner <michaelestner@web.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Check default flags are correctly returned by getVarFlags and check all flags
are returned when internalflags is True.
Check delVarFlags also removes default value.
Check all flags are removed after delVar.
Run the test with:
$ bitbake-selftest -v bb.tests.data.TestFlags
test_delflag (bb.tests.data.TestFlags.test_delflag) ... ok
test_delvar (bb.tests.data.TestFlags.test_delvar) ... ok
test_setflag (bb.tests.data.TestFlags.test_setflag) ... ok
----------------------------------------------------------------------
Ran 3 tests in 0.000s
OK
This is a test case for [YOCTO #15685]
(Bitbake rev: ff8cae735cf489373af1aac7ee233d7b82d483d3)
Signed-off-by: Louis Rannou <louis.rannou@non.se.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Variable flags have been fixed in commit
0329a7e3ac694737f2d2c1861f65492551360663 which introduces the
"_defaultval_flag_" prefix for default values.
This must not be ignored in delVarFlags and getVarFlags.
Fixes [YOCTO #15685]
(Bitbake rev: 2ee079fc1b7cf6d384ca17bd034b0a40461d9d18)
Signed-off-by: Louis Rannou <louis.rannou@non.se.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The URI path is already unquoted. Remove the unnecessary unquote
function calls for URI path values.
(Bitbake rev: 3de12bbc28b5a4189b849720735cf3e268d3941d)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the path variable from the fetch data instead of decoding the path
manually from the plain unexpanded url.
(Bitbake rev: ad3a29fa6ea53741d4e1786de35f8e7fc4292e7a)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetch data class already expands the type, host, path, user, pswd
and parm variables. The fetcher classes already expand the localfile
variable. The getVar function expands the returned string per default.
Remove unnecessary expand function calls to simplify the code.
(Bitbake rev: 1b1eb037b861fbf20491ac17e519e9eaf232b858)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the URI class to decode and encode an URL. Remove duplicate code and
unify the behavior.
(Bitbake rev: a5d569c94700f04b8193c6bccae5af619931b00f)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The password must be quoted as per RFC3986 because it can contain
reserved characters.
(Bitbake rev: 5c53dbf0ad4385e1de7f2eef66565a1f05dae67a)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Do not use upper case hostnames because the hostname is case insensitive
and maybe decoded into lower case to follow the common style.
(Bitbake rev: 554b7048412c4c67bf895a8b98822b54ac3a66db)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A file URI can't contain a user. Do not treat the @ as reserved
character for a file URI.
(Bitbake rev: 11cf4062f48536547a352e24d6e963d91fdd1190)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Old code that parse variable names in assignment commands behave differently for
variables that ends with special symbol for single-character variable names and
multi-character variable names. For example:
A+="1" # Change variable ${A}, '+' glued to '='
A+ = "1" # Change variable ${A+}
+="1" # Change variable ${+}, the '+' symbol not part of assignment operator
+ = "1" # Change variable ${+}
New code would always assume that '.=', '+=', and ':=' is assignment operator.
As result code like the following would raise parsing error
+="value"
While code with extra spaces would work as before
+ = "value" # Change variable ${+}
This change allow to catch issues in code that generate bitbake configuration
files in a manner like "echo ${VARNAME}+=${VALUE} >> conf/local.conf"
(Bitbake rev: 93059aad13a12cd69d86368795c88e5349197d5d)
Signed-off-by: Nikolai Merinov <n.merinov@inango-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This test covers the handling of unclosed functions.
It tests that both whitespace and tabs generate the
correct exception if added before a closing bracket.
Additionally that a residue blocks generates a error
is tested as well.
[YOCTO #15470]
(Bitbake rev: 29e67acb87ae76879efe9688a69c961a96df10f1)
Signed-off-by: Savvas Etairidis <falital@hotmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The string value "default" for the default multiconfig is confusing since an
empty string is used pretty much everywhere in the code. Remove the few
remaining references to that to standarise.
This affects the default value of BB_CURRENT_MC and does have an impact
on metadata, particulalry bitbake.conf in openembedded-core. That said, the
number of bugs we'll avoid by trying to make "default" back to "" within
bitbake's code make fixing those extremely worthwhile.
(Bitbake rev: 0fa0d8d764bbeb8a44c47f79d7b849068d565199)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
From git 2.48 release notes:
"""
When "git fetch $remote" notices that refs/remotes/$remote/HEAD is
missing and discovers what branch the other side points with its
HEAD, refs/remotes/$remote/HEAD is updated to point to it.
"""
This means with git 2.48 onwards, there is a mystery "HEAD" revision
appearing in some of our shallow clone tests. We can avoid this by
using the same canonicalization as used for the reference revisions.
This resolves autobuilder failures on the Fedora 40 workers.
(Bitbake rev: c83444d1210740e27b1744d3aa7c5cad4e28db2f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, providers are set on a global config basis. This change allows
for a select set of providers configured in BB_RECIPE_VIRTUAL_PROVIDERS to
be selected on a per recipe basis. This would allow for the selection of
virtual/cross-cc as gcc or clang for example in OE-Core.
DEPENDS and task flag [depends] values are processed.
(Bitbake rev: fb119c7888ae8a749aa824f8c51780176af077f9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
??= operator works for variable value by defining a "_defaultval" flag.
Use something similar for flags: For the default value of the flag
"flag_name", define a flag "_defaultval_flag_flagname" that is used when
reading flag_name but no other value has been set.
Fixes [YOCTO #15685]
(Bitbake rev: 0329a7e3ac694737f2d2c1861f65492551360663)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Reviewed-by: Ghislain Mangé <ghislain.mange@smile.fr>
Suggested-by: Ola Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Run the test with:
$ bitbake-selftest bb.tests.parse.ParseTest.test_parse_defaulttest
.
----------------------------------------------------------------------
Ran 1 test in 0.001s
OK
This is a test case for [YOCTO #15685]
(Bitbake rev: d12c80125d810df54379cc4f019a95ab9064ad2c)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Reviewed-by: Ghislain Mangé <ghislain.mange@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is the following warning when executing to bitbake linux-yocto:
bitbake/lib/bb/fetch2/__init__.py:464: DeprecationWarning: 'count' is passed as positional argument
This is because the 4th parameter of re.sub(pattern, repl, string, count=0, flags=0)
is a keyword parameter. We use keyword arguments for parameters that are not positional.
(Bitbake rev: 4bd62639f7023eec9f55ad7909fa9250538de936)
Signed-off-by: Bin Lan <bin.lan.cn@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for the Go mod h1 hash. The hash is based on the Go dirhash
package. The package defines hashes over directory trees and is uses for
Go mod files and zip archives.
(Bitbake rev: deefb01592f717efba68e3997fefd04dc7611d88)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The websockets library throws a number of exceptions which are currently
not caught leading to unhandled exceptions in the idle loop.
Fix this by catching them and reexposing them as a `ConnectionError`
which is the exception expected by users of `asyncrpc`.
(Bitbake rev: 41d62911a480283287265fe063696d2acd5904aa)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher support entries with an embedded checksum like 'sha256sum'
in the SRC_URI. It adds the parameter 'name' as prefix to the checksums
if the parameter is set. This behavior is unexpected and leads to hacks
in fetchers. Fallback to the checksum without the useless prefix and
set the parameter 'name' in the gomod fetcher unconditional.
(Bitbake rev: 7a86c5a20ea2586f1ae240613644e065e7b21683)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
