This alias is intended for managing specific local configs and it
is prompted by adding support for config fragments (in a separate
commit to oe-core); after some deliberation I concluded there should be
a separate tool, as bitbake-layers is already somewhat over-stuffed,
and this will give space for more build/conf/* operations in the future
that anyone can come up with (such as tweaking site-specific items
in site.conf etc.)
The alias completely reuses existing code via symlink and
the difference is in where it looks for plugins.
(Bitbake rev: ba90fe673aa87cb0cda9b2e465ebe2063551f527)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the new listtasks() function (introduced in 185c4b) to avoid
accessing a private variable.
Also use assertSequenceEqual() directly as we don't really care about
the actual type returned, only that it's a sequence.
(Bitbake rev: 452289efc6d3608ceca321286fd5bf417a6e1bed)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was suggested that using flags in fragment content to
mark fragment-specific metadata (such as descriptions and
dependencies) is prone to quiet regressions when fragments
are renamed or moved, and it clutters the fragment content.
With this change allowed fragment metadata variables must
be explicitly listed in a variable that is given to addfragment
directive, and parser will add the flag containing the fragment name
to them.
(Bitbake rev: ed9a3ca9426500511feb77f41a146953dbfe9af7)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It takes two parameters:
- location prefix for fragments
- name of variable that holds the list of enabled fragments, each of them prefixed
by layer id
Implementation of this directive essentially expands the fragment list
obtained from the variable into absolute fragment paths and hands them to the
implementation of 'require'.
(Bitbake rev: f687746703e7b096c5480668fd4f49bd4951182b)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is beneficial for config fragments, as their names
are specified via flags, and those names can include slashes:
BB_CONF_FRAGMENT_SUMMARY[init/systemd] = "This fragment enables systemd as an init manager"
(Bitbake rev: 80805988492b35593067230f68782a5687c8f557)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the user forgets to set MACHINE, bitbake just exits without
printing anything.
This is because BB_CONSOLELOG ends up with an unexpanded '${MACHINE}', which
bb.utils.mkdirhier tries to report using bb.fatal. But bb.fatal utilizes the
very logging infrastructure that this code was trying to setup.
(Bitbake rev: 7d3f3655b2f610f76898c84b8b97ef2e26529c41)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Analysis shows that "bitbake core-image-ptest-all" spends a lot of
time in scenequeue_updatecounters and much of it is rebuilding a set
which doens't change. Reorder the code to avoid that performance
glitch.
(Bitbake rev: 923c19b9713e398d8e66e6d4422dfd4c18a03486)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than looping through things we looped through on the previous execution,
start looping where we left off for setscene processing. This gives speed
improvements depending on the kind of build being executed.
(Bitbake rev: 00f4d932e3af0eeb333339cbe942010fc76dee0f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There have been complaints about the performance of large multiconfig builds
for a while. The key missing data point was that the builds needed to have large
overlaps in sstate objects. This can be simulated by building the same things with
just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of
deferred tasks.
The issue is that the expensive checks in the setscene loop were hit every time
through runqueue's execute function before the check on deferred tasks. This leads
to task execution starvation as that only happens once per iteration.
Move the skip check earlier in the function which speeds things up enormously
and should improve performance of such builds for users.
(Bitbake rev: 9c6c506757f2b3e28c8b20513b45da6b4659c95f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_unpack generates a 'Bitbake Fetcher Error: UnpackError' when
uncompressing a single 'file.zst', zstd command line returns the error
'unexpected end of file', and the input file 'file.zst' becomes empty.
The command line is 'zstd --decompress --stdout file.zst > file.zst'.
So the issue is that the input and output filenames are the same.
Fix the output filename when using zstd for a single file.
(Bitbake rev: 5881ca0e0d53937cd25a57ff682a3b7577b0b7f8)
Signed-off-by: Pascal Eberhard <pascal.eberhard@se.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removes the code in bitbake to show custom backtrace formatting for
exceptions. In particular, the bitbake exception code prints function
arguments, which while helpful is a security problem when passwords and
other secrets can be passed as function arguments.
As it turns out, the handling of the custom serialized exception stack
frames was pretty much made obsolete by d7db75020ed ("event/msg: Pass
formatted exceptions"), which changed the events to pass a preformatted
stacktrack list of strings, but the passing of the serialized data was
never removed.
Change all the code to use the python traceback API to format exceptions
instead of the custom code; conveniently traceback.format_exception()
also returns a list of stack trace strings, so it can be used as a drop
in replacement for bb.exception.format_exception()
(Bitbake rev: 2cda75a185aaf8f657f072dac34f8cef9d75f63a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There's no easy way to list all of the tasks in a recipe, you can either
look at __BBTASKS (internal variable, shouldn't be used) or iterate all
items in the datastore looking for variables with the 'task' flag set
(which is slow).
Solve this problem by adding a bb.build.listtasks() function that
returns an immutable copy of the __BBTASSK variable.
(Bitbake rev: 185c4b803962b20ba65a7d885dfe1a14e68736ef)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was added in 2.7.2. Since using inherit_defer is safer that inherit
when inheriting conditionally, move the instructions about that in
inherit_defer.
Fixes [YOCTO #15640].
Reported-by: Yoann Congal <yoann.congal@smile.fr>
(Bitbake rev: eb10df5a9619e243e28e0f4cd6122c24ed668f52)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the recipe provided by (-r, --recipe) is not found
tinfoil raises an exception that is not catched for
readability, example:
Traceback (most recent call last):
File "/.../poky/bitbake/bin/bitbake-getvar", line 45, in <module>
d = tinfoil.parse_recipe(args.recipe)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.../poky/bitbake/lib/bb/tinfoil.py", line 633, in parse_recipe
fn = self.get_recipe_file(pn)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/.../poky/bitbake/lib/bb/tinfoil.py", line 550, in get_recipe_file
raise bb.providers.NoProvider('Unable to find any recipe file matching "%s"' % pn)
bb.providers.NoProvider: Unable to find any recipe file matching "aaa"
(Bitbake rev: 06aa6c292813a28c84736193b550fb2d18884d43)
Signed-off-by: Talel BELHAJSALEM <bhstalel@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Testing shows the worst case CDN response time can be up to 100s. The wget fetcher
is used for accessing sstate from the CDN so increase our timeouts there to match
our worst case repsonse times.
(Bitbake rev: 7aca591529e115bc277f93811d8c586630acc8c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default pipe size is 64kb on builds, which can be inefficient
for larger log files from workers. Increase the pipe size to 512kb
since build systems have decent amounts of memory and this is a more
efficient way of batching the data.
Tweak the default read sizes to match the pipe size for efficiency.
Since the contstant is only present in python 3.10 onwards, add
some compatibility code.
(Bitbake rev: 69c14e46600ba5ae9703f67704ab2548875ae6d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are large amounts of data being transferred to the cooker
from the worker, recreating the bytearray becomes inefficient as it
happens for every pipesize block of data, defaulting to 64kb.
Instead we can use the deletion API for bytearrays to make this more
efficient and avoid the object recreation.
We noticed this with a strace ptest image taking days to complete the
build after having 6GB of data in the testimage log. Whilst there are
other issues there, making this code more efficient doesn't hurt.
(Bitbake rev: a4a72b7edb368f352784c856a647236a887010dd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
DANGLINGAPPENDS_WARNONLY is a very poorly designed interface and
is commonly abused. The challenge is that once it is set, by any
layer, it applies everywhere.
Some layers rely on this to get notification they need to update
bbappend files and having the behaviour change from inclusion of an
antisocial layer is not good.
In addition, showing warnings as an accepted thing on the console
devalues them and trains the user to ignore them. I want to steer us
away from this mindset.
We could extend the functionality and make it apply only to certain
layers, or only to certain appends but then we've basically re-invented
BBMASK.
Given all the above, we should drop support for BB_DANGLINGAPPENDS_WARNONLY
and direct anyone with issues to BBMASK instead.
https://lists.openembedded.org/g/openembedded-architecture/message/2029
[YOCTO #14870]
(Bitbake rev: fca9c9e3cb6f8e9f99bf51dc5e8a8d83f4c84c69)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a new tinfoil API "finalizeData" which does key expantion and
runs the anonymous functions to allow bitbake-getvar to have the same
output as "bitbake -e".
(Bitbake rev: 63db64ca98564420a378f14c337074ae06da8c63)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes what looked like a type since parseConfiguration.needconfig is
already set just after parseConfiguration() definition.
(Bitbake rev: d371df029ece3b9e600a89d08337c437a8ddbf63)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idle and main loops have socket select calls to know when to execute.
This means we can increase the normal timeout frequency since this is
just a fall back and have some small efficiency gains.
(Bitbake rev: 8d8e17af8619c976819170c9d5d9a686a666c317)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are no idle functions present, don't sent heartbeat events. These
are only meant to happen while builds are active.
(Bitbake rev: 9a2d5e63b07c3912838781776c61f0f1ac9640e1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Keeping this code separate just makes the code harder to understand,
merge them.
(Bitbake rev: e5ac26a0e1779df1da3277bf48899c8f7642f1f8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing some failures on hosts where slow "idle" loop iterations are
causing bitbake server timeouts. These seem to happen particularly in the
dump_signatures() function within runqueue.
That isn't entirely surprising since it creates a pool of threads to execute
work an at best can take around 10s to execture and return control backto the
main loop. On a slow system, it is understandable this can take longer,
particularly as these functions are creating large chunks of IO.
Since the work is being done in threads, we can launch them, return to idle
and check on the results periodically as they complete.
This should hopefully address some of the remaining timeout issues we see on
the autobuilder in oe-selftest sstate tests.
(Bitbake rev: e66f1b643b4b77404ba31f2704cda5af9bf00a57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lz4c has been considered deprecated by upstream since at least 2018 [1]
and has been disabled by default recently [2]. openSUSE Tumbleweed is
no longer packaging the deprecated version and others will likely
follow.
Luckily, we're not using any legacy-specific options and, going back as
far as Ubuntu 16.04/Fedora 25, both lz4 and lz4cli are installed as part
of the same package
1. https://github.com/lz4/lz4/pull/553
2. https://github.com/lz4/lz4/pull/1479
(Bitbake rev: 907472034b344e4eb73cfd43059a413469f52e1c)
Signed-off-by: Justin Bronder <jsbronder@cold-front.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
GNOME gitlab has occasional downtime which impacts bitbake-selftest
and causes autobuilder failures. Switch to our own mirror for test
purposes to avoid those issues.
(Bitbake rev: 91e268b11ed683bd197026f9b36001f6d54ee05c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
GNOME gitlab has occasional downtime which impacts bitbake-selftest
and causes autobuilder failures. Switch to our own mirror for test
purposes to avoid those issues.
(Bitbake rev: 008808755ed6cfeb6c41273e69ce718f0833c26c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The file may be a symlink when mirror from local disk, and the symlink will be
broken when the linked file is removed, use os.path.islink() to check and
remove the broken symlink.
(Bitbake rev: d0deb5fe6e7a84b07b49ef685b209729d2fdc780)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
BB_GIT_SHALLOW = "1"
BB_GENERATE_SHALLOW_TARBALLS = "1"
$ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall
The downloads/gitsmshallow_github.com.nemtrif.*.tar.gz won't be cleaned without
this fix.
(Bitbake rev: bab7a8970a0237a9d24217685a595e76a1336c07)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
$ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall
The downloads/git2/github.com.nemtrif.ftest won't be cleaned without this fix.
(Bitbake rev: 79f25fc5c1b8d0e08540f4aa07875309f5325f47)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The tmpdir such as downloads/tmplp3cnemv won't be removed without this fix.
(Bitbake rev: 2ba8d3214759142afc11f0a88d80eb30a8bcde3a)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are 14 lines can be removed, and can make it easy to maintain.
(Bitbake rev: 0ea2c1ac079d63349407a69172ff80cd9acc7252)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Libgcc is required as a runtime dependency. Without it, we get the
following error:
OSError: Error loading shared library libgcc_s.so.1: No such file or
directory
(Bitbake rev: 95e61f3dacacb3a001d9f0e2db4c4a2960d96640)
Signed-off-by: Jesse Riemens <riemensjesse@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an epilog to the help text that indicates the possible options for
the server address
(Bitbake rev: b6b703fce02057212ad11b1d1286c6178c533bad)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We really don't want to be skipping a test like this, drop the exception handling
code as in my testing we thankfully no longer seem to be seeing it. If we do again
we need to fix the underlying issue.
(Bitbake rev: b67547f890dc7f347ec05a97cbfb22d3703998cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix the exception handling in the cancel builds function.
This involved adding WebDriverException which sometimes occurs but
also correcting the other exception handlers to continue to increment
the timeout to avoid test hangs.
(Bitbake rev: e111a2bd4f7a8a4dc2c63e94e91ac6cacca95af8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Most of the tests that click on buttons need the DOM to stablise, including
any running JQuery code before the test can proceed. Add calls to do this
whenever we're about to click on an element.
(Bitbake rev: 0eb206b355248e2a874a62baec30025652f2a5a8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switch the clickable() calls to use the new element_clickable() function
which accepts a finder labmda function. This means if the element doesn't
yet exist, the code can rebuild the query and try again once a small
amount of time has elapsed.
There were a ton of timing related races around these element interactions
and this seemed to be the most robust way to address the issues.
The change also makes some of the elements slightly more specific so
the code can work effectively.
(Bitbake rev: 38643aadbb5a960004b886cf7709beaf2fc96652)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the element never exists, the timeout is never incremented and the test
hangs indefinitely. Fix the exception handling to avoid that and allow
the timeout to happen.
(Bitbake rev: 9eabe923d457bbce65227da4cd71c275c32108e6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes for the browser tests which:
* Add/improve waiting for specific elements
* Waiting for elements to be interactable, not just visible
* Hiding notification windows
* Using more specific asserts to obtain improved error messages
(Bitbake rev: 0c3f9bc2b9516d2af391f2e081280eb7fa62a05e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
