Adds a subcommand to clean unused outhash entries from the server based
on age
(Bitbake rev: 29c310be2c00d0e192923a4b90c60771ef0c620c)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a subcommand to invoke the remove API on the server
(Bitbake rev: a778b99e6a2a1a1e1b7eb26d48313fadcd34de54)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than outputting the string "None" for undefined variables, output
only a linefeed (the same as for variables that are defined to the empty
string).
(Bitbake rev: f3ba9c3726ec7b38b557100d8a2d4b6a1446a968)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an undefined variable or variable flag is specified, bitbake-getvar
will now fail with an error message indicating this.
The error can be supressed with --ignore-undefined, which matches the
previous behavior.
This also changes the errors related to specifying --flag or --unexpand
without --value so that they are sent to stderr rather than stdout.
(Bitbake rev: 136b8dda4e8b6f4d7e45a552c2d2e278b3ae1b7d)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It does not make any sense to get log output from bitbake-getvar when
the --value option is used as the log output is sent to stdout and thus
interferes with the output of the variable's value.
(Bitbake rev: 6b7883533af9c14d80a9f1ae5142644f155b5dee)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Initializing Tinfoil with setup_logging = False only has an effect when
recipe parsing is not needed. To make it work regardless of if --recipe
is used, manipulate the quiet parameter to Tinfoil.prepare() instead.
(Bitbake rev: 71ee69a20f21f3d37f4f060a7d8e87d9f1dc6aa1)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
declaring queue=b"" creates an object of types bytes().
bytes() is an immutable object, and therefore doing "self.queue = self.queue + r"
creates a new object containing "self.queue" concatenated with "r".
On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker,
so those copies significantly slow down the initialization of the bitbake-worker.
Rather use bytearray() which a mutable type, and use extend() to avoid copies.
In my test setup, byterray.extend() is 10.000 times faster than copying the queue,
for a queue size of 180MB.
(Bitbake rev: 2302b5316091dff189e6c3f546341b2274ed9d0a)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are times when it would be useful for code to know which layer
(or collection in old bitbake terms) it is contained within.
Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe
so that it is possible to determine this. To do it, we need to pass data
from the cooker into the recipe endpoints, since only the top level cooker
information knows about the layer structure which makes the patch a bit
painful.
The idea is that this would make layer overrides possible:
OVERRIDES .= ":layer-${FILE_LAYERNAME}"
which then opens possibilities like:
WARN_QA:append:layer-core = " patch-fuzz"
as an example where OE-Core could enable specific QA tests only for that
specific layer.
(Bitbake rev: 7090a14b0035842112d073acf7f2ed1a01fdeccf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So that OE-Core can depend on bb.event.check_for_interrupts(), bump our
verison number to a development series version.
(Bitbake rev: dea1b2f3fc31f28daed5da16e62da8895d6e5716)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the export/unexport/network flags are acted on only by presence
or lack of in the data store. This is deliberate due to performance reasons
since a given recipe shell task might export ~80-90 variables and this means
expanding flags for every one of them.
This does catch users unaware since values which expand to nothing don't work
e.g. ${@""} and setting values to "0" wouldn't work either.
The downside to this patch is slow down in parsing speed of around 4%.
The network flag is easier to change and doesn't affect performance, it was
made to operate the same as export/unexport for consitency reasons.
(Bitbake rev: 3d96c07f9fd4ba1a84826811d14bb4e98ad58846)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-getvar does not have a way to silence bitbake
server's logger and that makes the tool hard to use for
text processing. This is especially true when one wants to
get a bitbake value to be piped to some other utility and
instead we get uncontrolled logging messages or warnings
together with bitbake's variable value.
Example without quiet:
bitbake-getvar --value MACHINE
NOTE: Starting bitbake server...
qemux86-64
With quiet:
bitbake-getvar --value MACHINE --quiet
qemux86-64
(Bitbake rev: af354e975d0b4c26d0e91e3c82946b093bc11b45)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The thread/process locks we use translate to futexes in Linux. If a
process dies holding the lock, anything else trying to take the lock
will hang indefinitely. An example would be the OOM killer taking out
a parser process.
To avoid bitbake processes just hanging indefinitely, add a timeout to
our lock calls using a context manager. If we can't obtain the lock
after waiting 5 minutes, hard exit out using os._exit(1). Use _exit()
to avoid locking in any other places trying to write error messages to
event handler queues (which also need locks).
Whilst a bit harsh, this should mean we stop having lots of long running
processes in cases where things are never going to work out and also
avoids hanging builds on the autobuilder.
(Bitbake rev: d2a3f662b0eed900fc012a392bfa0a365df0df9b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
One of the challenges in maintaining the code is that it sometimes uses
a datacaches structure and sometimes a datastore. Rather than continue
the current dual API madness, have the worker contexts create a dummy
datacaches structure with the entries we need. Whilst this does need to
be kept in sync with the real structure, that doesn't change and this
allows the code to be simplified.
With this new approach, we can unify the stamps dependency code again.
(Bitbake rev: c6d325fc9b53e9d588ab273ee3c2a99a70fba42c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Get rid of the duplicate code and add extra check that the
locale en_US.UTF-8 is available on the system. This new helper
method is now located right above the method filter_environment()
which sets LC_ALL environment variable to 'en_US.UTF-8'.
[YOCTO #10165]
(Bitbake rev: a4ce040a6fd540a1cac52f808f909f9fcf8c961c)
Signed-off-by: Frank de Brabander <debrabander@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When setting up the worker we were transfering large amounts of data
which aren't needed until task execution time.
Defer the fakeroot and taskdeps data until they're needed for a specific
task. This will duplicate some information when executing different tasks
for a given recipe but as is is spread over the build run, it shouldn't
be an issue overall.
Also take the opportunity to clean up the silly length argument lists
that were being passed around at the expense of extra dictionary keys.
(Bitbake rev: 3a82acdcf40bdccd933c4dcef3d7e480f0d7ad3a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous cleanups meant that when the cooker was started, profiling
was always disabled as configuration was sent to the server later and this
was too late to profile the main loop.
Pass the "profile" option over the server commandline so that we can
profile cooker itself again, the setting can now take effect early enough.
(Bitbake rev: c97c1f1c127ef3f8fbbd1b4e187ab58bfb0a73e5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When 'NoCache' was written, databuilder/cookerdata didn't exist. It does
now and the recipe parsing functionality contained in NoCache clearly
belongs there, it isn't a cache function. Move those functions, renaming
to match the style in databuilder but otherwise not changing functionality
for now. Fix up the callers to match (which make it clear this is the right
move).
(Bitbake rev: 783879319c6a4cf3639fcbf763b964e42f602eca)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Plugins may want to use it (e.g. the layers-setup plugin that would
want to discover writer sub-plugins with it), and so it makes sense
to make tinfoil available a bit eariler.
(Bitbake rev: 2f6c7523a622f59ddf84a1a196927492bc5fa7a2)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have some confusion for users since some classes are meant to work
in the configuration space (or "globally") and some are meant to be
selected by recipes individually.
The cleanest way I could find to clarify this is to create "classes-global"
and "classes-recipe" directories which contain the approproate classes and
have bitbake switch scope between them at the appropriate point during
parsing. The existing "classes" directory is always searched as a fallback.
Once a class is moved to a specific directory, it will no longer be found
in the incorrect context. A good example from OE is that
INHERIT += "testimage"
will no longer work but
IMAGE_CLASSES += "testimage"
will, which makes the global scope cleaner by only including it where it
is useful and intended to be used (images).
(Bitbake rev: f33ce7e742f46635658c400b82558cf822690b5e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where copyright headers were not present, add them to make things
clear.
(Bitbake rev: 1aa338a216350a2751fff52f866039343e9ac013)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows OE to depend on the unihash copy functionality.
(Bitbake rev: 928aa3dc715d1d3be4b7a8d1247ea2ab5ee2d38e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the upcoming LTS, it is time we changed the bitbake version so
move to 2.0.
(Bitbake rev: 9a13bf8e20b1841ec99281d45be4c4fc1118438c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the remaining usage of "abort" in documentation, error
messages, and comments with halt/fail/exit as appropriate.
A couple of external Javascript API calls in Toaster remain, as
they cannot currently be changed.
(Bitbake rev: bc27762bf3ffb4a20b58eace5302438c4a526626)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where a git url uses a tag instead of a full source revision, breakage
can currently occur in builds. Issues include:
* the revision being looked up in multiple tasks (fetch and unpack)
* the risk a different revision may be obtained in those tasks
* that some tasks may not be allowed to access the network
* that a revision may not be consistent throughout a given build
* rerunning a specific task may given inconsistent results
To fix this, stop the workers from cleaning out the source revision store. This
should only be done in the cooker itself (based on current policy).
Also, where the code "sees" an upstream access, mark the recipe as not to be
cached. The reparse re-triggers the upstream lookup by the server.
Add a test to ensure that if get_srcrev isn't called, the user is told they're
using a configuration that is known to break.
(Bitbake rev: 4b5eed1626709ef3dc06b32fd55d40a2a6edd179)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The NIS can't work when network is dissable, so preserve network for it, the
error is like:
do_ypcall: clnt_call: RPC: Unable to send; errno = Network is unreachable
Note, enable nscd on the build machine might be a solution, but that isn't
reliable since it depends on whether the network function has been cached or
not.
(Bitbake rev: 4eafae7904bae6e5c6bc50356e8a9077f2e207fa)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch changes behaviour. After this change any task which does not
have the network flag set will have networking disabled on systems that
support that (kernel version dependent).
Add a "network" task specific flag which then triggers networking to
be enabled for this task, it is otherwise disabled.
This needs to happen before we enter the fakeroot environment of the task
due to the need for the real uid/gid which we save in the parent process.
(Bitbake rev: 0746b6a2a32fec4c18bf1a52b1454ca4c04bf543)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We occasionally see bitbake-worker failing and from the logs, an unpickle error
occurs. Add more debug so we can further debug this next time it fails.
[YOCTO #14595]
(Bitbake rev: fe8105cc06beca8240b76ea366a1eff5aa9c5412)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some debugging, BB_CURRENTTASK is set too late to be useful as
it isn't present in some event handlers for example. There is no
other way to know which task is actually running so set the value
earlier.
(Bitbake rev: 7daaaaa27f55b5a458656857c6d61a51b34a62fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the build is interrupted, handle the shutdown of pseudo even in this
case to avoid data corruption inside docker containers.
[YOCTO #14555]
(Bitbake rev: a2a04c6fe94bc56efcff299c669a151746e35916)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have a problem where pseudo server processes exist after bitbake exits
and hold the pseudo database in memory. In a docker container, the processes
will be killed as the container is destroyed with no warning and no opportunity
to write the data to disk. This leads to permissions/inode corruptions and
data loss.
Send a shutdown message to pseudo which in new versions of pseudo will flush
the database, thereby fixing some of the issues people using docker containers
see.
(Bitbake rev: a07a971b40acd3eee12e203d2cfa3e49f56109f6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We really do want to see those, as they tend to turn into
hard errors eventually, as what happened with collections
vs collections.abc in python 3.10.
(Bitbake rev: bc43fbb86361a21dc2d5deb910810c5a77fdabe8)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If bitbake-worker fails, return an error code showing that. Also
make the thread cleanup code explict in a finally clause as it would
otherwise hang.
[YOCTO #14393]
(Bitbake rev: 7e0af70fb53fb13f824ca954b8cc1dffee730233)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a library that implements file-like objects (similar to
gzip.GzipFile) that can stream to arbitrary compression programs. This
is utilized to implement a LZ4 and zstd compression API.
(Bitbake rev: 61c3acd058ea018696bd284b3922d0b458838d05)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've talked about having this for long enough. Add a command which queries a single
variable value with history. This saves "bitbake -e | grep" and avoids the
various pitfalls that has.
It also provides a neat example of using tinfoil to make such a query.
Parameters to limit the output to just the value, to limit to a variable flag
and to not expand the output are provided.
[YOCTO #10748]
(Bitbake rev: 4c1881b620e885f55d7772f8626b8a76c2828333)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the previous patch this code is now pointless as we'd have hit a TypeError
before now.
(Bitbake rev: 6301a99055c79d89b715f72182cd0ef1b781b89a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-server is spawned by process.py and passes the arguments it is
given to ProcessServer. There's some type confusion here:
bitbake-server is called with a string representation of the timeout,
which may be None. If the timeout is not set, pass 0 instead of None.
Inside bitbake-server a ProcessServer is created which expects the
timeout to be a float not a string, so always float() the value.
[ YOCTO #14350 ]
(Bitbake rev: c93ae1f861208f6d39fd15c84fbcd0e2b54331f5)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
