In interactive bitbake sessions it is obvious what tasks are running
when one of them hangs or otherwise takes a long time. However, in
non-interactive sessions (such as automated builds) bitbake just prints
a message saying that it is "still alive" with no clues as to what tasks
are active still.
By simply listing the active tasks when printing the keep alive message,
we don't need to parse the bitbake log to identify which of the tasks
is still active and has presumably hung.
(Bitbake rev: f9f57fb7d2c8a13df1eb9d5b9766f15e229dcf97)
(Bitbake rev: 30f6c3f175617beea8e8bb75dcf255611e3fc2fd)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Force a sync point for end-of build event handler force
the build's outcome status commit, to resolve a race
condition with the build completion takedown.
[YOCTO #14765]
(Bitbake rev: f76c13536e19d2401365c0fc240ef8dbcb3ee7e8)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Creating the new log instances triggers a lot of python logging overhead
in a commonly called function (about 600,000 for parsing OE-Core).
We only need the log functionality if we're parsing, not if we just hit
from the cache. Therefore defer the log setup overhead until we know it
is a cache miss.
Whilst this complicates the code slightly, the performance gain is worth
it as for parsing OE-Core we drop 60 million funciton calls (from 225
overall).
(Bitbake rev: ac868167ad854f9bb32dcb2e63528870547805a7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can call getVarFlags() instead of the multiple getVarFlag calls
which is a little more efficient. This reduces the number of overall
function calls so is good but probably isn't much faster (or slower).
(Bitbake rev: 505a4dd34e822bdf902d9b348dbcdf5b2c94e784)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code tries to expand missing entities when they're encountered in
python expressions. Looking at traces, these are often things which
would not be in the data store such as "bb".
Optimise to skip the data store queries for things we know will never
be there. The expansion cache usually covers these but skipping
entirely cuts a few million function calls parsing OE-Core.
(Bitbake rev: 1ae712635a2eef3ecbf541e1f7cc2eeb2f3799c9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the upcoming LTS, it is time we changed the bitbake version so
move to 2.0.
(Bitbake rev: 9a13bf8e20b1841ec99281d45be4c4fc1118438c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
connection.features.autocommits_when_autocommit_is_off
was deprecated in 3.0 and is no longer present in 3.2
(Bitbake rev: cc0f526fb4298349d0eea44c8f35d3dd226cc8d6)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake -e is not showing variable history correctly for all variables.
The issue was triggered by the change to knotty to default to enabling
variable tracking in the base datastore but that exposed another issue.
The real problem is that calling reset() inside cooker reverts to the
data tracking enabled by the UI for the base datastore, then turns
off tracking. In the case of the environment printing code, it needs
it to be left on.
Tweak the code to ensure data store tracking really is enabled.
The code here is clearly a bit of a mess but this at least fixes a
clear regression until more invasive improvements can be made.
(Bitbake rev: 9596bffc9903f023d1cc97ce531ddb1cf010c4d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Use max() to clamp progress to >= 0.
* Be consistent when evaluating self.quiet (treat it as a boolean).
(Bitbake rev: 160f71372ff93894d9314619e9d3b547c1f3cda3)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In commit 8055ec36 (knotty: Improve setscene task display) the setscene
tasks got their own line in the task output. However, the progress bar
code does not handle newlines in its widgets, so the length of the
setscene line was included when calculating how much space is available
for the progress bar of the running tasks, making it much too short.
Instead of trying to teach the progress bar code to handle newlines,
separate the output of the setscene tasks from the progress bar for the
real tasks.
(Bitbake rev: a41f7792f17acdba8c7ea83b79e413ae6a49da68)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use pluralise() to correct the grammar, and drop the colon at the end if
runnning in quiet mode.
(Bitbake rev: 57396289f935892390c11afe95f3eede28fd80e7)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "Parsing halted due to errors" is shown when user has old-style
variables requiring renaming. Make it clear that it isn't an additional
error, but a result of earlier errors.
(Bitbake rev: e81b6ae61dd8b5d0f77146263d79dfe66c66b060)
Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids backtraces when starting toaster or using bitbake in
remote mode.
(Bitbake rev: bf723f2cb5d288ca730e4f029110b36380420a01)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is general agreement the current display of the total setscene and
executed full tasks is confusing. Fixing it is tricky to be clear to new
users whilst not wasting screen real estate for experienced users and not
compromisng features like the progress bar.
As a compromise, move the setscene total to the previous line making
the status display a two line summary for interactive terminals. This
makes it clear what two of the numbers represent (setscene task) whilst
separating this out from the other items.
Also tweak the non-interactive output to use a comma as a separator,
the use of the forward slash was a poor choice.
(Bitbake rev: 8055ec360507e6a678ee5c4018ec1ab7f5a9cce5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the footer code doesn't quite handle multiline items correct.
Fix this to do so.
(Bitbake rev: 8eeccf73185d986c6abd6426b1d1e72da3a982df)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This implements support for sstate mirrors using ssh as transport
protocol.
(Bitbake rev: 0a3b5b3de7bcb1c5c3748cba42d394cc484e966b)
Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix absolute paths and paths containing the ":" character. Both is
necessary for supporting sstate mirrors via ssh (not implemented yet).
(Bitbake rev: df5505a1ba15524c3a185360d687854300aef342)
Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Support URLs like ssh://HOST/PATH. They were previously not recognized
due to a missing @ sign.
(Bitbake rev: a2aa18bd27dac8902e52b466cb7118f71367d3dc)
Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hashlib.md5() is not permitted on a FIPS enabled host system. This is due
to md5 not being an approved hash algorithm.
Instead use:
hashlib.new('MD5', usedforsecurity=False)
This is allowed, as it's clear the hash is used for a non-security purpose.
Note: utils.py version should never be used to verify file integrity, but
instead be used to identify if the file may have changed. sha256 should be
used for integrity purposes.
(Bitbake rev: af866dd077867cba0129757bfcc689551445e9d7)
Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
File "scripts/lib/checklayer/__init__.py", line 49, in _get_layer_collections
ldata.expandVarref('LAYERDIR')
File "build/bitbake/lib/bb/data_smart.py", line 1007, in expandVarref
if referrervalue and ref in referrervalue:
TypeError: argument of type 'bool' is not iterable
We inject True values as an internal sentinel to the datastore, fix
this codepath to handle it.
(Bitbake rev: 3b88562d87ac94725c1a683c859c2a6a3287d173)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Until now, if a renamed variable was found, bitbake exited
immediately if it was in a class, but continued after an error
message if the variable was in a recipe. This was caused by
cookerdata.py CookerDataBuilder::parseBaseConfiguration checking
a different DataSmart instance than the variable was set in.
To solve the issue, add a special variable and set it when we
find a renamed variable. Check for it in ast.py and bail out
if needed.
(Bitbake rev: d12400600e30549c88dc9e7883dc3d63b1dc1117)
Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the remaining usage of "abort" in documentation, error
messages, and comments with halt/fail/exit as appropriate.
A couple of external Javascript API calls in Toaster remain, as
they cannot currently be changed.
(Bitbake rev: bc27762bf3ffb4a20b58eace5302438c4a526626)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the "ABORT" action in BB_DISKMON_DIRS entries with "HALT".
In order to ease migration, code has been added to warn users to
update their configurations if the old name is used, as opposed to
to throwing an error.
(Bitbake rev: 11dc65dc077398ff9818060769c99c0090291186)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for custom strings in BB_RENAMED_VARIABLES and use this to show
that BB_STAMP_WHITELIST and BB_STAMP_POLICY are no longer supported.
(Bitbake rev: 0914011f7647571ab125bbddcd7d68e3da47226a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_HASHBASE_WHITELIST -> BB_BASEHASH_IGNORE_VARS
BB_HASHTASK_WHITELIST -> BB_TASKHASH_IGNORE_TASKS
the derived code variables basewhitelist and taskwhitelist have been
renamed to basehash_ignore_vars and taskhash_ignore_tasks,
respectively.
[RP: Added compatibility mapping code and tweaked naming]
(Bitbake rev: efaafc9ec2e8c0475e3fb27e877a1c0a5532a0e5)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change adds support for improving the user experience when variables
are renamed. This isn't as simple as it might first appear since some
bitbake variables are used through the environment before the datastore
exists, some are bitbake variables which we know about straight away
and some are metadata defined which we don't know about until later.
This patch adds support for handling these different cases, allowing a list
of bitbake renamed variables to be defined in bitbake itself and allows this
to be extended through the metadata using BB_RENAMED_VARIABLES.
In order to give the best feedback to the user, we default to turning on
variable history tracking in the base data store from knotty, which allows
filename and line number information to be shown.
(Bitbake rev: bd50a5d5e4b4fa90844464396887ebdff0d4e5f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a python function exec_func() calls a shell task, the logging wasn't working
correctly in all cases since the exception was turned into a BBHandledException()
and the logfile piece was lost which is handled at the top task level.
The easiest way to avoid this is to allow the ExecutionError exceptions to
be raised to a higher level, we don't need the traceback for them.
(Bitbake rev: 7cae11f558f9ff5fd05ef23b789aaef92fb5a327)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The changing of the environment inside the wget fetcher can race if
threading is used, such as with sstate in OE-Core. Abstract the function
so the environment can be correct before the function is called, removing
the race since the enviroment is then no longer changed.
(Bitbake rev: c73bb6023c73f003a160bb02aa4da1b580b86c23)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
