By switching from print() to bb.plain() to print failing task logs, we
allow them to be saved in BB_CONSOLELOG.
Fixes [YOCTO #15798]: This allows AB reproducibility test to save the
full log of the failing tasks and helps debugging.
(Bitbake rev: d3b3ad32da7c7ebf61814fc807f8667a37aa149b)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the user forgets to set MACHINE, bitbake just exits without
printing anything.
This is because BB_CONSOLELOG ends up with an unexpanded '${MACHINE}', which
bb.utils.mkdirhier tries to report using bb.fatal. But bb.fatal utilizes the
very logging infrastructure that this code was trying to setup.
(Bitbake rev: 7d3f3655b2f610f76898c84b8b97ef2e26529c41)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When tasks fail, it's very frustrating to have to scroll up to find the
log path(s). Many of us have the muscle memory to navigate to the 'temp'
directories under tmp/work/, but new users do not.
This change enhances the final summary to include log paths (reported
via bb.build.TaskFailed events). Here's an example:
NOTE: Tasks Summary: Attempted 856 tasks of which 853 didn't need to be rerun and 3 failed.
Summary: 3 tasks failed:
virtual:native:/home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch
log: /home/chris/repos/poky/build/tmp/work/x86_64-linux/ncurses-native/6.5/temp/log.do_fetch.1253462
/home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch
log: /home/chris/repos/poky/build/tmp/work/core2-64-poky-linux/ncurses/6.5/temp/log.do_fetch.1253466
virtual:nativesdk:/home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch
log: /home/chris/repos/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-ncurses/6.5/temp/log.do_fetch.1253467
Summary: There were 3 WARNING messages.
Summary: There were 6 ERROR messages, returning a non-zero exit code.
Each log is rendered as a clickable hyperlink in the terminal. See
https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda
(Bitbake rev: 2852a478ab03a482989c3a7e247860ab4f0e9f3e)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Red text on a black background can make it difficult for people with
visual impairments to read the text of error messages. Respect the
presence of a non-empty NO_COLOR environment variable as an indication
the user doesn't want colorization to be automatically enabled.
See: https://no-color.org/
(Bitbake rev: d9986c54cd3d67ed1f7cb636b17696c8d0d4db85)
Signed-off-by: Peter A. Bigot <pab@pabigot.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In runCommand() the send() and recv() can fail and raise
BrokenPipeError and EOFError exceptions when the bitbake-server is
unexpectedly terminated. In these cases a python traceback is
currently dumped. Similarly updateFromServer() which calls
runCommand() can also raise these and other exceptions, and currently
lacks proper exception handling resulting in python traceback.
We wrap calls to runCommand() and updateFromServer() in a try/except
block as well as improve the exception handling for updateToServer().
This along with the earlier commit which added text to the
BrokenPipeError and EOFError exceptions in runCommand() to indicate a
bitbake-server termination may have occurred, should improve the user's
ability to understand and handle these errors.
An easy way to trigger each of the runCommand() exceptions is to
'kill -9' bitbake-server before (causes EOFError) or after
(causes BrokenPipeError) the "Loading Cache" stage.
(Bitbake rev: 804d366ee3ddc0f37f0a6c712c8d42db45b119bc)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are three levels of shutdown, the initial "wait for current tasks",
then "stop current tasks" and "exit now". Change the code so that we don't
instantly exit after "stop current tasks" but allow the events to come
through from the server first. The new shutdown level allows that to then
be broken out of too.
(Bitbake rev: af38345b91cfc8a6bb5c38f5753255e7635bfe2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing failures where the UI hangs if the server disappears. Ping
the cooker/server if we've not had any events in the last minute so we can
check if it is still alive.
(Bitbake rev: 6567ad6181f9e39812097f0154647e4b38238fdd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are events queued and there is an exception in the main loop
of the UI code, it will print tracebacks on the console indefinitely.
Avoid that by improving the loop exit conditions.
(Bitbake rev: 2d0940b920a22b244f3ba6849c7cd019578386b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While the progress bar is good at conveying information about how much
of a task has executed, the elapsed time of the task is still very
much relevant to show.
(Bitbake rev: 41eeb4f34fb2306303f7688ec5e0ae965a573aa4)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The keep alive timeout is excessively long at 83 minutes (5000 seconds),
reduce this to 10 minutes: this should be long enough that it rarely
triggers in normal builds, but when it does it has useful information.
(Bitbake rev: 2e47346b95b09d7ab8f0603e2d62cfb549dc1f5c)
(Bitbake rev: dcf52157d3635925491783be656c6b76d1efe1a4)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In interactive bitbake sessions it is obvious what tasks are running
when one of them hangs or otherwise takes a long time. However, in
non-interactive sessions (such as automated builds) bitbake just prints
a message saying that it is "still alive" with no clues as to what tasks
are active still.
By simply listing the active tasks when printing the keep alive message,
we don't need to parse the bitbake log to identify which of the tasks
is still active and has presumably hung.
(Bitbake rev: f9f57fb7d2c8a13df1eb9d5b9766f15e229dcf97)
(Bitbake rev: 30f6c3f175617beea8e8bb75dcf255611e3fc2fd)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Use max() to clamp progress to >= 0.
* Be consistent when evaluating self.quiet (treat it as a boolean).
(Bitbake rev: 160f71372ff93894d9314619e9d3b547c1f3cda3)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In commit 8055ec36 (knotty: Improve setscene task display) the setscene
tasks got their own line in the task output. However, the progress bar
code does not handle newlines in its widgets, so the length of the
setscene line was included when calculating how much space is available
for the progress bar of the running tasks, making it much too short.
Instead of trying to teach the progress bar code to handle newlines,
separate the output of the setscene tasks from the progress bar for the
real tasks.
(Bitbake rev: a41f7792f17acdba8c7ea83b79e413ae6a49da68)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use pluralise() to correct the grammar, and drop the colon at the end if
runnning in quiet mode.
(Bitbake rev: 57396289f935892390c11afe95f3eede28fd80e7)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is general agreement the current display of the total setscene and
executed full tasks is confusing. Fixing it is tricky to be clear to new
users whilst not wasting screen real estate for experienced users and not
compromisng features like the progress bar.
As a compromise, move the setscene total to the previous line making
the status display a two line summary for interactive terminals. This
makes it clear what two of the numbers represent (setscene task) whilst
separating this out from the other items.
Also tweak the non-interactive output to use a comma as a separator,
the use of the forward slash was a poor choice.
(Bitbake rev: 8055ec360507e6a678ee5c4018ec1ab7f5a9cce5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the footer code doesn't quite handle multiline items correct.
Fix this to do so.
(Bitbake rev: 8eeccf73185d986c6abd6426b1d1e72da3a982df)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change adds support for improving the user experience when variables
are renamed. This isn't as simple as it might first appear since some
bitbake variables are used through the environment before the datastore
exists, some are bitbake variables which we know about straight away
and some are metadata defined which we don't know about until later.
This patch adds support for handling these different cases, allowing a list
of bitbake renamed variables to be defined in bitbake itself and allows this
to be extended through the metadata using BB_RENAMED_VARIABLES.
In order to give the best feedback to the user, we default to turning on
variable history tracking in the base data store from knotty, which allows
filename and line number information to be shown.
(Bitbake rev: bd50a5d5e4b4fa90844464396887ebdff0d4e5f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds a log level and logging function call to use it where the
warning or error will only be displayed once, regardless of how many
times the message is logged.
This has to be done either in the cooker or on the UI side. I've opted
for the UI side since display control is really a UI issue but it uses
a common library filter function to enable it which can be reused
elsewhere.
The knotty message displayed as the build summary is tweaked to
make sense when the numbers won't match since it will still count
the number of times it was logged and this is probably helpful
for debugging in some cases so I've deliberately left it that way.
(Bitbake rev: 7bd40e3003a043e3cb7efc276681054b563b5e7b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the changes to task accounting, bitbake doesn't show progress
when executing setscene tasks on the summary console output.
Change to show a progress within the setscene tasks and a progress
within the main tasks. I can't see any way to display this more easily
without confusing users.
[YOCTO #14586]
(Bitbake rev: 0244acb968eb593d2ad7bc6e52f222c2b1d39aa9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent fixes to merge setscene and normal task accounting in runqueue
fixed some display issues but broke the task numbering of setscene tasks.
Add new accounting methods to the stats structure specifically designed
for setscene. This accounts for the fact that setscene tasks can rerun
multiple times in the build.
Then use the new data in the UI to correctly display the numbers the
user wants to see to understand progress.
(Bitbake rev: ed7e2da88bf4b7bfc7ebfc12b9bd6c0fb7d8c1aa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "-l" command line options to enable specific logging domains wasn't
working with the switch to structured logging because they were only
being used to set the legacy logging domains. Fix this by implementing
the logic to parse the user options into the logging configuration.
(Bitbake rev: 005fc7a8c588d0b0bca382469645cbf481ad8e30)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After parseprogress.finish() it was intended to
set parseprogress to None, but a typo means this
is not happening.
(Bitbake rev: f504d6f6598f62aa20fbf69c30fea95569858edb)
Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new server startup code means exceptions can happen when we aren't
setup to show them to the user correctly, leading to ugly tracebacks.
Add in some special case handling of BBHandledException to at least
ensure that common case doesn't traceback and the user sees meaningful
output.
In the future, the logging setup can likely be improved, as can the way
runCommand handles exceptions, they all should likely become real
exceptions again on the UI side.
[YOCTO #14022]
[YOCTO #14033]
(Bitbake rev: 6059d0e77f60ddb679049bd34478f41b1ab7995d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Correctly import, and inherit functions, and variables.
Also fix some typos and remove some Python 2 code that isn't recognised.
(Bitbake rev: b0c807be5c2170c9481c1a04d4c11972135d7dc5)
Signed-off-by: Frazer Clews <frazerleslieclews@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The logging module provides a shutdown() function that does the same
thing in a much better way
(Bitbake rev: 970cd2fc4f0bbc93069dee5a15a608dd76081c67)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The BitBake.verbconsole needs to be treated like a console output logger
(meaning that the TerminalFilter attaches an InteractConsoleLogFilter to
it), even if it's not directly attached to the root 'BitBake' logger.
First, assign a special "is_console" property to the relevant handlers,
then look for the property in the handlers from the configuration object
return by bb.msg.setLoggingConfig(). Finally, pass the list of all
handlers to the TerminalFilter object; it doesn't care about the
difference between console and errconsole, so pass all the relevant
handlers as a list.
This fixes cases where the console output was corrupted when messages
were sent to the 'BitBake.verbconsole' handler.
(Bitbake rev: 2010be588c74a99256df7b565a309c84c2973546)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updates hash equivalence logging so that the interesting VERBOSE
messages are always logged to the consolelog file so that issues in
individual user builds can be diagnosed. The autobuilder logging config
then updates this so that they also are shown on stdout, since the
consolelog file is not capture there.
In order to facilitate this, 2 new logging handlers were added,
"BitBake.verbconsole" and "BitBake.verbconsolelog". Neither of these
handlers are attached to anything by default, but they will log any
messages that wouldn't otherwise be logged by the normal console or
consolelog handlers. Users can attach whatever loggers the desire to
this handler to get them to appear on the console or in the consolelog,
as demonstrated by the autobuilderlog.json file.
(Bitbake rev: 766587c0baaaeb5cda3e9454395edbb70e33f756)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds code to close all loggers when bitbake exits. This prevents
unclosed file ResourceWarnings. A form of this closing existed
previously, but was removed in the new logging code.
(Bitbake rev: b3f3779adf63c0d970462a558a6205da1d30c0ed)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sets up logging in knotty to use python's structured logging config and
the bb.msg.setLoggingConfig() helper to setup logging. This allows the
user to specify additional logging mechanism in a config file with
BB_LOGCONFIG
(Bitbake rev: 646a68a49364b50a42168b4b16308f7217eec0dc)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Passing around the log formatter variable was unnecessary since the log
levels of interest can be accesses as class members of
bb.msg.BBLogFormatter. Switching to do this will make using the
structured python logging much easier, since it can be difficult to
extract out the formatter for a specific handler.
(Bitbake rev: c1c867df24b4ef204027d485acac7c75c63f2bc0)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a comment section that can be easily uncommented to enable dumping
the logging tree. This module is extremely useful for debugging issued
with logging configuration
(Bitbake rev: 30461310915f911b80f92e03df694af7c1eb1f46)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Handles the log messages from the bitbake server with the specific
logger that the event originated from. This allows hierarchical logging
configurations to work as expected.
(Bitbake rev: 9624d42133e024fd044d0d089c7017ed53eed874)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This matches the other bb.command.Command* events and without it,
running `bitbake --revisions-changed` will hang indefinitely if there
are changed revisions.
(Bitbake rev: 40520d229c8ea51ee9784184ab5d13a82dd1eb61)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
amend the code to handle singleton comparisons properly so it only checks
if they only refer to the same object or not, and not bother
comparing the values.
(Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
removed unused imports which made the code harder to read, and slightly
but less efficient
(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating a new progress bar (using BBProgress), a colon was
appended to the supplied message. However, when updating the message,
no colon was appended.
Change this so that the colon is instead part of the widgets that make
up the progress bar so that it does not matter when and how the
message is updated, it always displays the same.
(Bitbake rev: 08f35c04f6e1ce4c4ca5c2bef4cd8a192e12e682)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the introduction of the hash equivalence server, the progress bar
for "Checking sstate mirror object availability" is shown repeatedly
while the tasks are being executed. If the footer is not hidden then,
it will be moved up one line every time, creating a messy interface.
(Bitbake rev: 56b5ec4c2b3e658e73ca6c3a12feeb96df0977fb)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen cases where a task can execute with a given pid, complete
and a new task can start using the same pid before the UI handler has
had time to adapt.
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/knotty.py", line 484, in main
helper.eventHandler(event)
File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/uihelper.py", line 30, in eventHandler
del self.running_tasks[event.pid]
KeyError: 13490
This means using pids to match up events on the UI side is a bad
idea. Change the code to use task ids instead. There is a small
amount of fuzzy matching for the progress information since there
is no task information there and we don't want the overhead of a task
ID in every event, however since pid reuse is unlikely, we can live
with a progress bar not quite working properly in a corner case like
this.
[YOCTO #13667]
(Bitbake rev: e427eafa1bb04008d12100ccc5c862122bba53e0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
$ rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p
Press the first Ctrl-C when the parsing process is at about 50%:
Keyboard Interrupt, closing down...
Then presss the second Ctrl-C:
File "/path/to/bitbake/bitbake/lib/bb/ui/knotty.py", line 619, in main
event = eventHandler.waitEvent(0.25)
File "/path/to/bitbake/lib/bb/server/process.py", line 591, in waitEvent
self.eventQueueNotify.wait(delay)
File "/usr/lib/python3.5/threading.py", line 549, in wait
signaled = self._cond.wait(timeout)
File "/usr/lib/python3.5/threading.py", line 297, in wait
gotit = waiter.acquire(True, timeout)
KeyboardInterrupt
Capture the second KeyboardInterrupt during stateShutdown is running can fix
the problem. There may be still tracebacks for the third KeyboardInterrupt, but
I'm leaning to not fix it since we aimed for supporting 2 KeyboardInterrupts
only.
(Bitbake rev: 8c26b451f22193ef1c544e2017cc84515566c1b8)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst this class has existed for years, it doesn't have any
users and has a questionable interface. Drop it to allow for further
simplification and changes.
(Bitbake rev: 3ab51764f7965d696bb2c5a872bf161473df4289)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.
(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.
The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.
The COPYING file is replaced with LICENSE.X files which contain the full
license texts.
(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CI systems like jenkins and buildbot will timeout applications which haven't had console output
in some period of time. Add 'keepalive' output to knotty which gives output every 5000s if not
other output was made and tasks are still running. This reduces some problems encountered
with our CI testing.
(Bitbake rev: aa4f31e5741dd98acec73f16f6028e52f4c22d6f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A task's runtime is currently printed in seconds. Change it to
include minutes and hours for easier reading.
(Bitbake rev: c593ae5ec9fecd4bde823948024e4d56314a60ce)
Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a task fails during build_targets(), we need to print out the log
lines as knotty does or the user will be missing information about the
failure.
(This should get some deeper refactoring, but now isn't the time for
that.)
(Bitbake rev: 24879df071d4803db3d39ae1d5cad852daa92f28)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If for example you run:
bitbake -r somefile.inc
rm somefile.inc
bitbake -e
bitbake will crash with an error about not being able to find somefile.inc. This
is because it tries to reparse the base config for the early getVariable requests
before it sees the updated missing -r option.
Send the updateConfig command earlier to avoid this.
(Bitbake rev: a38164620ebdc770690c5f39ff9ed69d3f82719e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
