Some layers think they're going to be 'clever' and copy the values from
another layer, e.g. using ${LAYERSERIES_COMPAT_core}. The whole point of
this mechanism is to make it clear which releases a layer supports and
show when a layer master branch is bitrotting and is unmaintained.
Therefore add some code to avoid people doing this. I wish we didn't have
to but...
(Bitbake rev: 6709aedccbb2e7ddbb1b2e7e4893481a7b536436)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For many years OE-Core has injected it's own python modules into the
python namespace using an immediate expansion of a variable in base.bbclass.
It also added all entries from BBPATH to sys.path.
We really need this to become a first class citizen of the langauge, this new
addpylib directive allows that. Usage is of the form:
addpylib <directory> <namespace>
The namespace is imported and if there is an attribute BBIMPORT, that
list of names is iterated and imported too.
This mirrors what OE-Core has done for a long time with one difference in
implmentation, sys.path is only appended to. This means later imported
namespaces can't overwrite an earlier one and can't overwrite the main
python module space. In practice we've never done that and it isn't
something we should encourage or support.
The new directive is only applied for .conf files and not other filetypes
as it only makes sense in that context. It is also only allowed in the
"base configuration" context of cookerdata since adding it at the recipe
level wouldn't work properly due to the way it changes the global namespace.
At the same time, move the list of modules to place in the global namespace
into a BB_GLOBAL_PYMODULES variable. It is intended that only the core layer
should touch this and it is meant to be a very small list, usually os and sys.
BB_GLOBAL_PYMODULES is expected to be set before the first addpylib directive.
Layers adding a lib directory will now need to use this directive as BBPATH
is not going to be added automatically by OE-Core in future. The directives are
immediate operations so it does make modules available sooner than the current
OE-Core approach.
The new code appends to sys.path rather than prepends as core did, as
overwriting python standard library modules would be a bad idea and naturally
encouraging people to collaborate around our own core modules is desireable.
(Bitbake rev: afb8478d3853f6edf3669b93588314627d617d6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Get rid of the duplicate code and add extra check that the
locale en_US.UTF-8 is available on the system. This new helper
method is now located right above the method filter_environment()
which sets LC_ALL environment variable to 'en_US.UTF-8'.
[YOCTO #10165]
(Bitbake rev: a4ce040a6fd540a1cac52f808f909f9fcf8c961c)
Signed-off-by: Frank de Brabander <debrabander@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Starts the sync thread slightly earlier to give it some extra time to
dump out the caches while the main process tears down the parsing
processes
(Bitbake rev: 105f2897b0618713b036fc0f7a6e0f3e78d1707a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Uses an event to terminate the parser threads instead of a queue. This
is not only simpler, but saves about 500ms on shutdown time
(Bitbake rev: 2aed34e1d4bf24bba6263f168ff31b55b5fbe982)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to track siggen hash construction data can be useful for cache
debugging. For now, add an extra cache class which contains this information.
It can be enabled in the same way as the hob data cache through a feature flag
to cooker. This allows us to experiment with the data without carrying larger
patches around and ultimately may allow use to have a hash mismatch debugging
mode that is more easily enabled.
(Bitbake rev: 0736a8a03da8b774fafbd28f746bef4705378049)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The basehash data is really internal bitbake data and passing an object
directly is more efficient than creating and then extracting variables.
This will match the format of other data we may optionally wish to
store in the cache so more to the more efficient method. Nothing I
can see is using this data today (and nothing should be).
(Bitbake rev: e621093a1bf37cd75ede3fb77ab6845556870fc7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make a small change in the layout of the code in build_dependencies
which makes subsequent patches easier to read. No functionality change,
just moving the function definitions to the start of the function block.
(Bitbake rev: fff13b1e5e8397130b4378e0ba2301336ec651a2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the expand cache doesn't work for "parser" return types, which
is the main type used by the build_dependencies() call that we spend most
of the time in when parsing. Tweak the code to cache the unexpanded value
in the expand cache and hence allow reuse of the parser in other fast path
cases for small speed gains.
(Bitbake rev: b4a8e5071dbcba2217b79e83e08b275ffcbc0eef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you place a multiconfig which isn't enabled into an mcdepends you currently
get a traceback from runqueue. We can do better, add some code to tell the user
what happened in a more readable way without the traceback.
[YOCTO #14970]
(Bitbake rev: a4693b70764bb394ee2cf8dd12a5f6fce866008b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These are static regex compilations which don't change or a standard module
import (bb). There is noneed to declare them as global so drop the pointless
code which doesn't do anything.
(Bitbake rev: 09a4c159e3fd184f730821e7bd99916b0d28dc70)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code looks like it is from a time when the data module was executable.
Nobody does that now and this usage of pydoc is long since obsolete/broken
so clean up the code which doesn't do anything useful.
(Bitbake rev: 6f4ef770641a9657edaf8618f4c86dfb1116622c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
One section of the code is enforcing lowercase overrides, the other
is allowing numeric characters. We should be consistent with one or
the other.
(Bitbake rev: df5b3b841fd8d6a652d643e9ae2bba09d60043e0)
Signed-off-by: 张忠山 <zzs213@126.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some variables may be set as:
X = 1
as well the more usual
X = "1"
so add support to to_boolean to handle this case.
(Bitbake rev: ef9c033b011e68bbfedf7ddf118633c14388aaaf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If someone is unfortunate enough to run into override recursion issues
they're hard to debug with the existing message. We can at least show the
values that OVERRIDES takes to show there is some problem and aid debugging.
(Bitbake rev: 43035b75201616e7bfd680d3d15c5c0fc7c04eb6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python handles frozensets a little more optimally than normal sets. Once we
finish parsing, we don't edit this data so we can convert to them.
To do that, we need to stop changing them so process ignore_deps earlier
then we can freeze the data and keep it frozen.
This has the side effect that we need to be careful to sort the data
in some of the variables when calculating the hashes.
Overall this does seem to show a decent parsing time speed improvement
of 20-25% in a local test but this would be highly setup dependent.
Also ensure the sigdata can handle exported frozenset and make it import
back to them instead of sets.
(Bitbake rev: 19475627c363a52da49ec144422c87448ff2a6c5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We identified a use case where a native recipe (autoconf-native) was
rebuilt with no change in output yet the sstate for do_package tasks
wasn't being used.
The issue is that do_package tasks have a hard dependency on
pseudo-native:do_populate_sysroot. That task was one of the many
tasks being rehashed when autoconf-native's hash was changed.
If update_tasks processed a recipe before it had processed pseudo-native,
that recipe would be marked as not possible from sstate and would
run the full tasks.
The fix is to split the processing into two passes, first to handle
the existing covered/notcovered updates, then in the second pass,
check whether there are "harddep" issues.
This defers the do_package tasks until after pseudo-native is installed
from sstate as expected and everything works well again.
(Bitbake rev: e479d1e418a7d34f0a4663b4a0e22bb11503c8ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Even after enabling all our debugging, we had a reproducible test case
where sstate wasn't being reused and it was unclear from the logs why.
This patch adds debugging on the possible codepaths that were breaking
and allowed the issue to be debugged and fixed.
(Bitbake rev: 9233ad685b9b5e9eeb775fc71761712aaf0e876c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't test not using the cache and I'm not aware of anyone using this, it
would be hard to with modern bitbake.
Drop the conditional code and simply error if CACHE isn't set.
(Bitbake rev: 063ffe699bc5fc23174643dfedb66864cacfcff8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When setting up the worker we were transfering large amounts of data
which aren't needed until task execution time.
Defer the fakeroot and taskdeps data until they're needed for a specific
task. This will duplicate some information when executing different tasks
for a given recipe but as is is spread over the build run, it shouldn't
be an issue overall.
Also take the opportunity to clean up the silly length argument lists
that were being passed around at the expense of extra dictionary keys.
(Bitbake rev: 3a82acdcf40bdccd933c4dcef3d7e480f0d7ad3a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous cleanups meant that when the cooker was started, profiling
was always disabled as configuration was sent to the server later and this
was too late to profile the main loop.
Pass the "profile" option over the server commandline so that we can
profile cooker itself again, the setting can now take effect early enough.
(Bitbake rev: c97c1f1c127ef3f8fbbd1b4e187ab58bfb0a73e5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At this point users appear to all call add_info directly. Failing
to do that means the file dependency tracking code isn't active
so would cause problems. Therefore drop the unused function.
(Bitbake rev: 6b24efc0f4d19738d96754280e70bc493005167d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some parts of functions in Cache() were broken and unused, there was
also a totally unused function. This was historical as a result of the
cooker parsing process needing to handle cached entries in the main
thread but parsing actions in seperate processes.
Document the way it works, update the function name to be clear about
what it now does and drop the old code which was unused.
(Bitbake rev: af83ee32df85c8e4144f022a1f58493eb72cb18e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When 'NoCache' was written, databuilder/cookerdata didn't exist. It does
now and the recipe parsing functionality contained in NoCache clearly
belongs there, it isn't a cache function. Move those functions, renaming
to match the style in databuilder but otherwise not changing functionality
for now. Fix up the callers to match (which make it clear this is the right
move).
(Bitbake rev: 783879319c6a4cf3639fcbf763b964e42f602eca)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Just like download_submodule() does, fixed warings when run
bb.fetch2.Fetch([url], d) in process_submodules' function:
WARNING: grpc-native-1.50.0-r0 do_fetch: URL: gitsm://github.com/abseil/abseil-cpp.git;protocol=https;name=third_party/abseil-cpp;subpath=third_party/abseil-cpp does not set any branch parameter. The future default branch used by tools and repositories is uncertain and we will therefore soon require this is set in all git urls.
(Bitbake rev: 0ed7c75eb0508a1f699f47d7f22d559501865f61)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77 introduced regression
in submodules path parsing. As the result gitsm fetcher fails on each
submodule which name begins from the name of the parent repo which is
totally valid usecase [Yocto #14045] [1]
Fix the code to error out only if submodule's name is equal to parent
name but not if it's part of it.
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14045#c4
(Bitbake rev: 3ad27272c18f2bb9edd441f840167a3dabd5407b)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We dropped the update_data calls a while ago. Clean up the code
to match the reality and drop the remaining no-op pieces. Update
the comments to reflect the slowest operations and let the cookie
monster's spirit live on!
(Bitbake rev: 584989ed2b5af4e8799571dece0cf94f995ef14e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #14917]
Attempting to use a dictionary in a python code snippet for variable
assignment results in an error. For example attempting something such
as
IDX = "green"
VAL = "${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]}"
produces the error
expansion of VAL threw ExpansionError: Failure expanding variable
VAL, expression was ${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]}
which triggered exception SyntaxError: '{' was never closed (Var
<VAL>, line 1)
The existing __expand_python_regexp__, "\${@.+?}", will match the
first close curly bracket encountered, resulting in incomplete and
un-parsable code, and thus produce the error. We can correct this by
allowing a single depth of nested curly brackets in
__expand_python_regexp__ by using "\${@(?:{.*?}|.)+?}", which will
match up to and including the matching close curly bracket to the
open, '${@', curly bracket, even if there are one or more singly
nested curly brackets present. This change allows the usecase
described above to function.
This change can't be made on its own though. The old regex would, in
an obscure way, handle the case where a python snippet contained an
unexpandable variable. Since the unexpandable variable is in curly
brackets it would cause incomplete/un-parsable python code and thus
remain unparsed. So something like
VAL = "${@d.getVar('foo') + ${unsetvar}}"
would remain unparsed as the close curly bracket in "${unsetvar}"
would match and terminate the snippet prematurely. This quirk resulted
in the proper handling of python snippets with unexpanded
variables. With the change to __expand_python_regexp__ the full
snippet will match and be parsed, but to match the old/correct
behavior we would not want to parse it until ${unsetvar} can be
expanded. To ensure the old/correct behavior for python snippets with
unexpanded variables remains in place we add a check for unexpanded
variables in the python snippets before running them.
This handling of unparsed variables brings two benefits. The first we
now have an explicit check visible to all for unexpanded variables
instead of a somewhat hidden behavior. The second is that if there are
multiple python snippets the old behavior would run the code for each
but a single snippet with unexpanded variables would mean all snippets
would remain unparsed, meaning more and repeated processing at a later
time.
For example:
"${@2*2},${@d.getVar('foo') ${unsetvar}}"
old behavior would give:
"${@2*2},${@d.getVar('foo') ${unsetvar}}"
new behavior will give:
"4,${@d.getVar('foo') ${unsetvar}}"
The old behavior would calculate '2*2' but toss the result when the
second snippet would fail to parse resulting in future recalculations
(or fetching from cache), while the new behavior avoids this.
(Bitbake rev: 94e49b9b9e409c29eb04603b1305d96ebe661a4b)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
First, commit 81efd77987f6decf256967fa16521a40c14d3518 created a copy
of __expand_var_regexp__ and __expand_python_regexp__ when creating
the initial version of data_smart.py. A while later commit
db1c998b31da06d7f3eb09fc6f59a1915b7b549e dropped all references to
these variables from data.py.
This leaves us today with two versions of these variables in the
global scope. However, only those defined in data_smart.py are being
used, in data_smart.py.
Unfortunately there was no indication in the commit log for commit
db1c998b31da indicating why the variables were left in place despite
the functions referencing them were being removed. Additionally
data.py imports data_smart, thus the versions of __expand_var_regexp__
and __expand_python_regexp__ defined in data_smart.py would be used by
all bitbake code, beyond, potentially, some code in data.py which we
know has no references to these variables.
To remove any potential confusion around these variables drop the old
definitions from data.py.
(Bitbake rev: 60f43d0428d43c981b44b6c8d125f77440f6c8f9)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the client socket is closed, asyncio.StreamReader.readline() will
return an empty bytes object, not None.
This prevents multiple tracebacks being logged by bitbake-hashserv each
time bitbake is started and performs a connection check.
(Bitbake rev: 2d07f252704dff7747fa1f9adf223a452806717f)
Signed-off-by: Justin Bronder <jsbronder@cold-front.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This git configuration variable is deprecated in 2.36.0 onwards, so git
warns in the logs for every git call.
Luckily the default value has always been false[1], so we can just remove
this.
[ YOCTO #14939 ]
[1] aafe9fbaf4
(Bitbake rev: 8ad310633e0c5d5593631c1196cbdde30147efce)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add
a parameter to the commands in the tests to allow this to continue to work.
(Bitbake rev: 209f7ba352b60722830157054e3fc56cb9c693eb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test for special characters in user and password to qualify
decodeurl() inspired by a bug report describing that '=' signs in a
password was problematic.
Add a second test to qualify decodeurl() as related to the change in
commit 628c4bf6c89b [fetch2/__init__: handle @ in package names].
Relates to [YOCTO #14476]
(Bitbake rev: ee04cf09c7022168c035affa654773652a49793e)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the case where hashlib is not available, the try would fail and fall
through resulting in a backtrace on the usage of the 'sig'. The backtrace
itself was confusing and made it difficult to determine what went wrong.
Update the import to be in it's own try block with an appropriate
message to indicate what went wrong.
Note, the current version of ply all of this code has been restructured
so this is not applicable upstream.
Additionally, some versions of hashlib don't appear to implement the
second FIPS related argument. Detect this and support both versions.
(Bitbake rev: 484ab42f440070c0369b81f5c69da860fa47a798)
Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It can be used to enable the loopback interface, typically after calling
disable_network().
Also correct a typo in a debug message.
(Bitbake rev: 0d317209d4234c5f05a9fcdc13c52f502f104018)
Signed-off-by: Mattias Jernberg <mattias.jernberg@axis.com>
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output of runfetchcmd is always empty in this case, as
the test doesn't produce any output.
SSH either returns 0 or 1, which is handled via exceptions.
This means the current check is not only unnecessary but prevents the
function from working.
We can just assume that if we reach the end of the function that the
file exists and return True.
(Bitbake rev: d599af48635fab587e5b913591b95daf87b40080)
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
c212b0f3 change the debug log level unintentional when tryng to fix a knotty issue.
This will maintain the same debug log level 2 as before.
(Bitbake rev: 19f8265023281f3b1d5d0a02e47f8d7d08cfcc16)
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are many messed up calls for the debug log,
so is better to warm about this as they will not work
as expected.
The level need to be an integer and the msg a string.
(Bitbake rev: c1d9c1d25ce36848040dc0ce182835e497ccbb82)
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
f68682a7 ("logging: Make bitbake logger compatible with python logger")
replaced several .debug() calls to make them comply with the standard
python logging API, but a few were missed.
(Bitbake rev: eb25cd4d64b9a4e8e2b2ce7fbccc123d00b2fc2b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The condition will always evaluate to true and
thus is redundant.
(Bitbake rev: be1ee681e8a566564549068dcf90c95c36544815)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
local file fetches now validate checksums. The checksums for mirror
tarballs of repositories will not match so ignore these checksums.
(Bitbake rev: 6424f4b7e9c1ba8db81346e8b3a806dd035d4551)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The connect_unix() call had a bug where if a relative path to a socket
was passed (which the non-async client always does), and the current
working directory was changed after the initial call, it would fail to
reconnect if it became disconnected, since the socket couldn't be found
relative to the new current working directory.
To work around this, change the socket connection for UNIX domain
sockets to be synchronous and change current working before connecting.
This isn't ideal since the connection could block the entire event loop,
but in practice this shouldn't happen since the socket are local files
anyway.
Help debugging and resolving from Joshua Watt.
(Bitbake rev: 5964bb67bb20df7f411ee0650cf189504a05cf25)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If two recipes have conflicting checksums for a file, the code will currently
remove the existing file when a mismatch is downloaded, even if another task
successfully fetched it.
This changes the code to verify the checksum (if possible) before replacing
the file. This removes a potential race window and stops builds failing
everywhere from one incorrect checksum.
To make this work, we need to be able to override localpath and avoid
NoChecksum errors being logged.
(Bitbake rev: 4b8de2e7d12667d69d86ffe6e9f85a7932c4c9a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PN is correct here, bitbake has no knowledge of BPN.
This reverts commit d613e48c07d4b12219270c1359cbf2f390b848dd.
(Bitbake rev: cffcfacb747d41304c857b17bfea646e220b2389)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When checking for the non-existing file, BPN is actually the acutal recipe
name. And we should use BPN for the error message and it also fix the below
test when multilib is enabled.
$ oe-selftest -r bbtests.BitbakeTests.test_invalid_recipe_src_uri
(Bitbake rev: d613e48c07d4b12219270c1359cbf2f390b848dd)
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
