When tasks are run with -v (verbose) on the bitbake commandline, shell
tasks print their stdout, python tasks do not.
This change redirects the python task's print output to an in memory
buffer. After the task is executed the output is printed to stdout via
the logger. This makes the python task behavior match the shell task
behavior when running with -v. The contents of the task's log files
remain unchanged after this change.
This approach should keep the correct order in most cases, however, if
the python task accesses the logger directly, that content will appear
before other output. On the other hand, this change should negate the
need for python tasks to access the logger directly.
Special care is taken to save/restore the existing stdout and stderr
and preventing sending output directly to the logger when there are
"recursive" calls, for instance when a python function calls a shell
function, avoiding printing things potentially out of order and/or
multiple times.
The logging-test.bb in meta-selftest can be used to review this
change. This has been tested with the full bblogging oeqa tests.
[Yocto #14544]
(Bitbake rev: 81a58647b2f4fc0a2589b2978fc9d81b2bfe6aec)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current method of passing either a task's datastore, or
dataCaches and a filename into the stamp functions is rather
horrible.
Due to the different contexts, fixing this is hard but we do control
the bitbake side of the API usage so we can migrate those to use other
functions and then only support a datastore in the public bb.build API
which is only called from task context in OE-Core.
(Bitbake rev: c79ecec580e4c2a141ae483ec0f6448f70593dcf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_setscene was from a different era before our modern setscene
per task code. It hasn't been used for years so remove some old
obsolete references to it.
(Bitbake rev: ef72282298f7c4db74383c23bb0251dd06d3c6d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We dropped the update_data calls a while ago. Clean up the code
to match the reality and drop the remaining no-op pieces. Update
the comments to reflect the slowest operations and let the cookie
monster's spirit live on!
(Bitbake rev: 584989ed2b5af4e8799571dece0cf94f995ef14e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is useful when debugging as it helps understand possible
race conditions between tasks of diferent recipes.
(Bitbake rev: 950a2ea4c91d6e13d7587104367fa85cc7efe01c)
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently have no API to be able to remove all the potential stamps of a
task. It is unusual to need to do this, particularly as you could race against
other things happening in the system but we do have a use case for this in
cleaning up sysroots in OE-Core. The alternative is to mess with CLEANMASK in
OE-Core but that is just going to add potential for errors.
We need the first part of the make_stamp() function so separate that out so
it can be called seperately.
(Bitbake rev: 4d671504a25863018ac51c21c005cef0a4d8f05c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an unexpected exception occurs in a setscene task, it is currently hidden
from the user and not recorded in any logs. This isn't helpful to debug
such failures.
Change the code so that even in the "silent" or "quiet" task case (setscene
tasks), a warning is shown with the traceback unless it was an "handled"
exception. This means the failing function can show it's own warning/error
instead if it wants to and then raise a handled event.
(Bitbake rev: 41dcdc61eb40def8c14a42e8d7bb9ce5a34afa57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some debugging, BB_CURRENTTASK is set too late to be useful as
it isn't present in some event handlers for example. There is no
other way to know which task is actually running so set the value
earlier.
(Bitbake rev: 7daaaaa27f55b5a458656857c6d61a51b34a62fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a race between sending the TaskFailed event which could trigger
the UI to look at the logs and flushing and closing the log files.
Reverse the order of the finally clause and the exception handling
to ensure we've handled the logfiles before sending the task events.
This should fix a race seen in bblogging.BitBakeLogging.test_python_exit_logging
(Bitbake rev: 032190aac31604d37740d8aecf6e74a5448de358)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we see things like:
Log data follows:
| DEBUG: Executing python function do_pythontest_exit
| DEBUG: Python function do_pythontest_exit finished
| ERROR: 1
| This is python stdout
Whilst after the change we see things like:
Log data follows:
| DEBUG: Executing python function do_pythontest_exit
| This is python stdout
| DEBUG: Python function do_pythontest_exit finished
| ERROR: 1
since the output is now correctly mixed with the log messages. In some cases the logging
tests indicate the output is being lost entirely which is bad for debugging and makes
things rather confusing.
(Bitbake rev: 8966b43761500e0505333d8c9a3f0f2c3dbe7559)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are circular task references, error on them rather than show
a recursion error. A simple reproducer is:
"""
do_packageswu () {
:
}
addtask do_packageswu after do_image_complete before do_image_qa
"""
into image_types.bbclass. There is code in runqueue to detect these but
we never get that far with the current codebase.
[YOCTO #13140]
(Bitbake rev: 339d4d6be515a71311b81fb9e99742af0d8a5130)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With "bitbake -v", for task failures you'd see the log output twice. Avoid
this by using the existing "did we print info" switch.
(Bitbake rev: e2c1afda4cb8023ed4ffeb5dc5bee4f0055659a8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a python task fails with sys.exit(), we currently see no TaskFailed event.
The high level code does detect the exit code and fail the task but it can
leave the UI inconsistent with log output.
Fix this be intercepting SystemExit explicitly. This makes python
task failures consistent with shell task failures.
(Bitbake rev: 9eee9fd4f2f96789ad2b037e74d561bdc1426856)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The point of the injected text is to identify where the function comes from. Using
the correct function name would therefore be better.
(Bitbake rev: 30c6ff8551c235254ab90663ab88f66bb0c71edb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently setting those flags to even the empty string "" causes it to be set,
which is contrary to the documentation. In a future version of BitBake, we'd
like to change the behavior so that setting the flag to "" does not set it.
This will allow conditionally setting noexec, using variable expansion or
inline Python.
I found no places in poky or meta-openembedded where this warning would trigger.
[YOCTO #13808]
(Bitbake rev: 1e7655c4f765ba7b4791c4cca048a69bf8d9c93d)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a new function which compares the stamp filename we want (including
taskhash) with what is in the stamp directory (using the clean mask).
This tells us which stamp files are stale and are due to be rerun.
(Bitbake rev: b126a53882d202e4df0f9661303355c9fe9ec80e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.
Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.
[RP: Small fix to ensure bb.debug calls bbdebug()]
(Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The levels of indirection to set these verbose logging options is rather
crazy. This attempts to turn things into two specific options with
much more specific meanings. For now its all still controlled by the
commandline verbose option and should funciton as previously, with
the addition that the BB_VERBOSE_LOGS option can now be task specific.
(Bitbake rev: 423c046f2173aaff3072dc3d0882d01b8a0b0212)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This shuts up both the DEBUG and EXIT handlers.
Also, remove an unneeded "set -o errtrace" (i.e. set -E),
since we no longer use an ERR trap.
(Bitbake rev: 89e851fa0403d1e98aeed69990101e3f84f0b283)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Leverage the comments that emit_var writes and the backtrace that
the shell func writes to generate an additional metadata-relative
backtrace. This will help the user troubleshoot shell funcs much
more easily.
Example:
| WARNING: /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955:171 exit 1 from 'exit 1'
| WARNING: Backtrace (BB generated script):
| #1: myclass_do_something, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955, line 171
| #2: do_something, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955, line 166
| #3: actually_fail, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955, line 153
| #4: my_compile_extra, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955, line 155
| #5: do_compile, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955, line 141
| #6: main, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.68955, line 184
|
| Backtrace (metadata-relative locations):
| #1: myclass_do_something, /home/laplante/repos/oe-core/meta/classes/myclass.bbclass, line 2
| #2: do_something, autogenerated, line 2
| #3: actually_fail, /home/laplante/repos/oe-core/meta/recipes-extended/libsolv/libsolv_0.7.14.bb, line 36
| #4: my_compile_extra, /home/laplante/repos/oe-core/meta/recipes-extended/libsolv/libsolv_0.7.14.bb, line 38
| #5: do_compile, autogenerated, line 3
ERROR: Task (/home/laplante/repos/oe-core/meta/recipes-extended/libsolv/libsolv_0.7.14.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 542 tasks of which 541 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/home/laplante/repos/oe-core/meta/recipes-extended/libsolv/libsolv_0.7.14.bb:do_compile
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
(Bitbake rev: ae1aa4ea79826c32b20e1e7abdf77a15b601c6f2)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Just use the |resolve| function which already takes care of it.
(Bitbake rev: 91b809a0902ffd42be4edf7f0a7d25e6d354d822)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PF is already captured, but it's a pain to try to parse out PN and PV,
which can be helpful to have available.
(Bitbake rev: 2885034f42a1dce7586e081a0461b8932a3593bf)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
removed unused imports which made the code harder to read, and slightly
but less efficient
(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen cases where a task can execute with a given pid, complete
and a new task can start using the same pid before the UI handler has
had time to adapt.
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/knotty.py", line 484, in main
helper.eventHandler(event)
File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/uihelper.py", line 30, in eventHandler
del self.running_tasks[event.pid]
KeyError: 13490
This means using pids to match up events on the UI side is a bad
idea. Change the code to use task ids instead. There is a small
amount of fuzzy matching for the progress information since there
is no task information there and we don't want the overhead of a task
ID in every event, however since pid reuse is unlikely, we can live
with a progress bar not quite working properly in a corner case like
this.
[YOCTO #13667]
(Bitbake rev: e427eafa1bb04008d12100ccc5c862122bba53e0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its hard to see what this exception adds in the current codebase. The logfile
attribute is effectively ignored, the exception doesn't serve a defined
purpose and mostly seems to be worked around.
Remove it entirely. If this does cause output problems, we'll figure
out better ways to address those.
(Bitbake rev: cfeffb602dd5319f071cd6bcf84139ec77f2d170)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A separate patch to base.bbclass (in poky) will add the OE_EXTRA_IMPORTS
variable. The contents are appended into OE_IMPORTS. This provides a
mechanism by which layers (in their layer.conf) can make custom progress
handlers available.
As a backup, individual recipes can inject progress handlers into
__builtins__.
Custom handlers are expected to have this __init__ signature:
def __init__(self, d, outfile=None, otherargs=None):
Recipes can then use the handlers like this:
do_task[progress] = "custom:mylayer.util.ProgressHandler[:args]"
The last part (everything after and including the second colon) is
optional. If provided, it is passed to HandlerClass's __init__ as
otherargs="args". Otherwise, otherargs=None.
(Bitbake rev: 20289d62c84c393990dd3deb0cca1b17c09092e6)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It seems context management support was half-implemented, but never
finished. For example, LogTee has __enter__ and __exit__ but they
haven't been exercised until now.
(Bitbake rev: bf522ad3e0c52cdb69b406226840d870ff4f2766)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.
(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.
(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.
The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.
The COPYING file is replaced with LICENSE.X files which contain the full
license texts.
(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This breaks with rm_work so disable the warning until we find a better
solution (and change the test accordingly too).
(Bitbake rev: 93e94c06baf013e3d072465a55bddd1fe61c0772)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
WARNING: elfutils: dependent task do_rm_work does not exist
is much less useful than
WARNING: elfutils: dependent task do_rm_work for do_deploy does not exist
(Bitbake rev: e034c6f75e3d7730ff16a8d1bd0cba03beda0af8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the missing bbverbnote fifo cmd for logging from shell.
(Bitbake rev: cf11fdb30c405e1a4521a7299f84816c0e13a881)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The following command is incorrect, but was ignored silently, that may suprise
users:
addtask task after task_not_existed
This patch can check and warn for it. It would be better to also check "before"
tasks, but there is no easier way to do it.
[YOCTO #13282]
(Bitbake rev: b1ad36169c2ad189d84c53c707b66c12d5ff812f)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously this information was simply discarded, which in some cases
resulted in generic, unhelpful failures.
With this change the user would see what the exception is, but
without ugly tracebacks or bulky, irrelevant information from the log file.
(Bitbake rev: 8da9242702fbef8cf3156b95a1076802e0f653c7)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the capability for tasks from different
multiconfigs to depend on one another.
These dependencies can be enabled using the following format:
task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on"
For the sake of simplicity consider the following example:
Assuming we have set up multiconfig builds, one for qemux86 and one for
qemuarm, named x86 and arm respectively.
Adding the following line to an image recipe (core-image-sato):
do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"
Would state that core-image-sato:do_image from x86 will depend on
core-image-minimal:do_rootfs from arm so it can be executed.
This patch makes modifications to:
- cooker: To glue both multiconfigs in one place and make sure
the dependencies can be provided.
- taskdata: To parse and add a new kind of dependency (mcdepends) to
the taskdata object.
- runqueue: To differentiate tasks from different multiconfigs,
add the specified dependencies to the corresponding tasks, and
create a working runqueue that contains tasks from both multiconfigs.
- siggen: To avoid looking for tasks from different multiconfigs on
objects where they dont belong.
The taskdata objects are still not aware of the concept of multiconfig,
so each object doesnt know which multiconfig its building, hence why
the mcdepends are added to all taskdata objects equally (we really
dont expect many of these), but the actual dependencies are added only
to the required tasks by the runqueue.
(Bitbake rev: da8cb8633504bdc815bdcefc538340b9bce5065d)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If for example you deltask do_build but it has recrdeps set, its confusing
to have that list returned when the task no longer exists (same would apply
to deps too if it was set after the deltask).
(Bitbake rev: b7a81cb91e82ba64b63c9153dc161a2ab4696715)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a utility function that gives the list of dependent tasks between
two specified tasks (just within the same recipe). This is intended to
be able to be executed from recipe context so it uses the datastore
rather than having access to the runqueue.
This will be used in OpenEmbedded-Core's populate_sdk_ext.bbclass to
get the list of tasks between do_image_complete and do_build.
(Bitbake rev: 433379bf12cf31fdf46defdf66695cf8be9994b1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
exc_func() fails with a hard to debug exception when the
function does not exist, for example due to a typo:
ERROR: ...: Traceback (most recent call last):
File "/work/bitbake/lib/bb/build.py", line 644, in exec_task
return _exec_task(fn, task, d, quieterr)
File "/work/bitbake/lib/bb/build.py", line 584, in _exec_task
exec_func(func, localdata)
File "/work/bitbake/lib/bb/build.py", line 198, in exec_func
cleandirs = flags.get('cleandirs')
AttributeError: 'NoneType' object has no attribute 'get'
There is code further down which will print a warning, but we don't
reach that unless we allow None as result of of d.getVarFlags() first.
The warning is further down intentionally and has to stay there, to
ensure that 'cleandirs' gets executed also for empty functions.
(Bitbake rev: 25df3db5eeda9dbf417e67e96845f376e5c8e6b3)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The API is required by the revised rm_work.bbclass implementation,
which needs to know all tasks that do_build depends so that it
can properly inject itself between do_build and those tasks.
The new API primarily hides the internal implementation of the "after"
and "before" dependency tracking. Because tasks defined as
precondition via "recrdeptask" may or may not be relevant (they are for
rm_work.bclass), the API also includes support for that.
There's no default value for including recrdeptasks, so developers
have to think about what they need.
(Bitbake rev: 9289ab40e77906e983a2f79cd7602ee95be5025a)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the UI shows X is building, possibly multiple times but doesn't
say which of the multilibs that might be. This adds a prefix to the task
name so the mulitconfig being built can be identified.
(Bitbake rev: d93ddd185dde53f50aea06ada094bb0d9b5ba4f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In certain circumstances it can be useful to get access to BitBake's
logger within a task; the main example is in OpenEmbedded's image
construction code where we want to be able to check the log file for
errors and warnings, but we don't want to see any errors or warnings
that were emitted through the logger; so we need a way to exclude those.
In order to do this, pass the logger object into the task via a
BB_TASK_LOGGER variable, and add a logging handler class to bb.utils
that can be added to it in order to keep a list of warnings/errors that
have been emitted.
(Bitbake rev: f1cd6fab604f14d8686b1d783cbfe012d923ee42)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.
Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)
(Bitbake rev: c19baa8c19ea8ab9b9b64fd30298d8764c6fd2cd)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
