Commit Graph

147 Commits

Author SHA1 Message Date
Richard Purdie
054875b222 bitbake: event/utils: Avoid deadlock from lock_timeout() and recursive events
We've been seeing intermittent failures on Ubuntu 22.04 in oe-selftest which
were problematic to debug. The failure was inside lock_timeout and once that was
identified and the backtrace obtained, the problem becomes clearer:

  File "X/bitbake/lib/bb/server/process.py", line 466, in idle_thread_internal
    retval = function(self, data, False)
  File "X/bitbake/lib/bb/command.py", line 123, in runAsyncCommand
    self.cooker.updateCache()
  File "X/bitbake/lib/bb/cooker.py", line 1629, in updateCache
    self.parser = CookerParser(self, mcfilelist, total_masked)
  File "X/bitbake/lib/bb/cooker.py", line 2141, in __init__
    self.bb_caches = bb.cache.MulticonfigCache(self.cfgbuilder, self.cfghash, cooker.caches_array)
  File "X/bitbake/lib/bb/cache.py", line 772, in __init__
    loaded += c.prepare_cache(progress)
  File "X/bitbake/lib/bb/cache.py", line 435, in prepare_cache
    loaded = self.load_cachefile(progress)
  File "X/bitbake/lib/bb/cache.py", line 516, in load_cachefile
    progress(cachefile.tell() + previous_progress)
  File "X/bitbake/lib/bb/cache.py", line 751, in progress
    bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize),
  File "X/bitbake/lib/bb/event.py", line 234, in fire
    fire_ui_handlers(event, d)
  File "X/bitbake/lib/bb/event.py", line 210, in fire_ui_handlers
    _ui_handlers[h].event.send(event)
  File "X/bitbake/lib/bb/cooker.py", line 117, in send
    str_event = codecs.encode(pickle.dumps(event), \'base64\').decode(\'utf-8\')
  File "/usr/lib/python3.10/asyncio/sslproto.py", line 320, in __del__
    _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
  File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg
    sw(msg.message, msg.category, msg.filename, msg.lineno,
  File "X/bitbake/lib/bb/main.py", line 113, in _showwarning
    warnlog.warning(s)
  File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "X/bitbake/lib/bb/event.py", line 778, in emit
    fire(record, None)
  File "X/bitbake/lib/bb/event.py", line 234, in fire
    fire_ui_handlers(event, d)
  File "X/bitbake/lib/bb/event.py", line 197, in fire_ui_handlers
    with bb.utils.lock_timeout(_thread_lock):
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "X/bitbake/lib/bb/utils.py", line 1888, in lock_timeout
    bb.server.process.serverlog("Couldn\'t get the lock for 5 mins, timed out, exiting. %s" % traceback.format_stack())

or put in simpler terms, whilst sending an event(), an unrelated warning
message happens to be triggered from asyncio:

/usr/lib/python3.10/asyncio/sslproto.py:320: ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x7f0e797d3100>

which triggers a second event() which can't be sent as we're already
in the critcal section and already hold the lock.

That warning is due to the version of asyncio used on Ubuntu 22.04 with
python 3.10 and that comined with timing issues explains why we don't
see it on other python versions or distros.

We can't handle the second event as the lock is there to serialise the
events. Instead, we queue the event and then process the queue later.

Add a new version of lock_timeout which allows us to handle the situation
more gracefully.

(Bitbake rev: 2c590ff1aff89d23b25ce808650f200013a1e6af)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-03 21:38:57 +00:00
Joshua Watt
29da7370d2 bitbake: Remove custom exception backtrace formatting
Removes the code in bitbake to show custom backtrace formatting for
exceptions. In particular, the bitbake exception code prints function
arguments, which while helpful is a security problem when passwords and
other secrets can be passed as function arguments.

As it turns out, the handling of the custom serialized exception stack
frames was pretty much made obsolete by d7db75020ed ("event/msg: Pass
formatted exceptions"), which changed the events to pass a preformatted
stacktrack list of strings, but the passing of the serialized data was
never removed.

Change all the code to use the python traceback API to format exceptions
instead of the custom code; conveniently traceback.format_exception()
also returns a list of stack trace strings, so it can be used as a drop
in replacement for bb.exception.format_exception()

(Bitbake rev: 2cda75a185aaf8f657f072dac34f8cef9d75f63a)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-28 00:06:24 +00:00
Robert Yang
3f41592920 bitbake: bitbake: event: Inject empty lines to make code match lineno in filename
So that we can get the correct error messages.

* In python 3.10.9, the error message was:
  ERROR: Unable to register event handler 'defaultbase_eventhandler':
    File "/path/to/poky/meta/classes-global/base.bbclass", line 4
      # SPDX-License-Identifier: MIT
             ^^^^^
  SyntaxError: invalid syntax

  This is hard to debug since the error line number 4 is incorrect, but nothing
  is wrong with the code in line 4.

* Now the error message and lineno is correct:
  ERROR: Unable to register event handler 'defaultbase_eventhandler':
    File "/path/to/poky/meta/classes-global/base.bbclass", line 256
      an error line
         ^^^^^
  SyntaxError: invalid syntax

And no impact on parsing time:
* Before:
$ rm -fr cache tmp; time bitbake -p
real    0m27.254s

* Now:
$ rm -fr cache tmp; time bitbake -p
real    0m27.200s

(Bitbake rev: c212933d9c786806852c87f188250a4f0a14c048)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-10 13:55:33 +00:00
Julien Stephan
8eeb58cf4a bitbake: bitbake: cooker: add a new function to retrieve task signatures
adding a new command in cooker to compute and get task signatures

this commit also add the associated command and event needed to get the
signatures using tinfoil

(Bitbake rev: 05c15162de90c41dad67e37a95ec9fdb440a7864)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-28 12:38:26 +01:00
Mingli Yu
066f44b161 bitbake: event: add bb.event.ParseError
Add bb.event.ParseError to let error-report can catch this kind of error.

(Bitbake rev: 316524ab59a5e738c25e062923ee5717d88ae5c7)

Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-13 12:01:45 +01:00
Richard Purdie
42c92cbc83 bitbake: event/cooker/runqueue: Add ability to interrupt longer running code
Bitbake is now able to understand when a UI wants it to stop the current
processing. There are some long running functions which currently have no
mechanism to interrupt them however.

This patch adds a call, bb.event.check_for_interrupts(d) which can be
placed in long running code and allows an internal state flag within
bitbake to be checked. If set, that flag will trigger an exit.

This means that Ctrl+C can be made to work in a variety of places where
it currently would not.

Long running event handlers in OE-Core can also then benefit from this
new approach with the addition of the function call as well.

(Bitbake rev: b7ed7e9a815c4e10447fd499508be3dbb47f06e8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-23 12:12:18 +00:00
Richard Purdie
a19687acd1 bitbake: lib/bb: Update thread/process locks to use a timeout
The thread/process locks we use translate to futexes in Linux. If a
process dies holding the lock, anything else trying to take the lock
will hang indefinitely. An example would be the OOM killer taking out
a parser process.

To avoid bitbake processes just hanging indefinitely, add a timeout to
our lock calls using a context manager. If we can't obtain the lock
after waiting 5 minutes, hard exit out using os._exit(1). Use _exit()
to avoid locking in any other places trying to write error messages to
event handler queues (which also need locks).

Whilst a bit harsh, this should mean we stop having lots of long running
processes in cases where things are never going to work out and also
avoids hanging builds on the autobuilder.

(Bitbake rev: d2a3f662b0eed900fc012a392bfa0a365df0df9b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-05 11:50:17 +00:00
Richard Purdie
6a6a5c01f7 bitbake: event: Always use threadlock
With the move to a server idle thread, we always need threading. The
existing accessor functions could end up turning this off!

I was going to hold the lock whilst changing it, check if the value
was already set, cache the result and also fix the event code to always
release the lock with a try/finally.

Instead, disable the existing functions and use a with: block
to handle the lock, keeping things much simpler.

(Bitbake rev: 645c9d3b50e55f69b222cc338373cdfd91d524ce)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-31 17:05:17 +00:00
Richard Purdie
cb8efd4d20 bitbake: event: Add enable/disable heartbeat code
Currently heartbeat events are always generated by the server whilst it is
active. Change this so they only appear when builds are running, which is
when most code would expect to be executed. This removes a number of races
around changes in the datastore which can happen outside of builds.

(Bitbake rev: 8c36c90afc392980d999a981a924dc7d22e2766e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-31 17:05:17 +00:00
Richard Purdie
907e91796e bitbake: event: builtins fix for 'd' deletion
I've been seeing event handlers where 'd' seems to disappear half way through
event handler execution. This is problematic when multiple threads are active
since this code assumes single threading.

The easiest fix is to change the handler function calls to contain d as a
parameter as we do elsewhere for other functions. This will break any non-text
handlers but I was only able to spot one of those in runqueue. It will also
break handlers than call functions that assume 'd' is in the global namespace
but those failures should be obvious and we can fix those to pass d around.

This solution avoids manipulating builtins which was always a horrible thing
to do anyway and solves the issue without needing locking, thankfully.

(Bitbake rev: 1e12f0a4b592dacd006d370ec29cd71d2a44312e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-29 00:07:06 +00:00
Mikko Rapeli
5ed5a3cc8a bitbake: event.py: ignore exceptions from stdout and sterr operations in atexit
When atexit functions run, stdout and stderr operations may fail, e.g.
when output is piped to less but has been exited by the user.
This removes error print from output of "bitbake -e sqlite3 | less"
if user presses "q" before bitbake has finished processing:

[Errno 32] Broken pipeError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/builder/src/poky/bitbake/lib/bb/event.py", line 135, in print_ui_queue
    sys.stdout.flush()

(Bitbake rev: 35167536c163eb6b7653cbcaad9f65b834d3e2f8)

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-11 10:30:56 +01:00
Scott Murray
4f77505d94 bitbake: lib/bb: Replace "ABORT" action in BB_DISKMON_DIRS
In line with the inclusive language migration defined at:

https://wiki.yoctoproject.org/wiki/Inclusive_language

replace the "ABORT" action in BB_DISKMON_DIRS entries with "HALT".
In order to ease migration, code has been added to warn users to
update their configurations if the old name is used, as opposed to
to throwing an error.

(Bitbake rev: 11dc65dc077398ff9818060769c99c0090291186)

Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-21 23:37:26 +00:00
Zygmunt Krynicki
50cd76495d bitbake: event: Fix typo "asynchronous" and "occasionally"
(Bitbake rev: bb60472ff41b583b7b3a65e7ddccd4554840952b)

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-20 16:45:24 +00:00
Richard Purdie
86ec266d91 bitbake: runqueue/event: Add an event for notifying of stale setscene tasks
Use the new functionality in build.py to identify stale setscene tasks
and send an event to the metadata listing them. The metadata then
has the option of performing cleanup operations if it thinks that
appropriate.

(Bitbake rev: ef8c980a3ae92c168b7ca16a4d19cd38a9574761)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23 22:51:25 +00:00
Richard Purdie
22d7989e8f bitbake: event: Fix another 'if d' test to test for None explictly
This should have been fixed in the previous patch, well spotted Chris!

(Bitbake rev: c3f6fee42bfa23f23f167cb29f0cfa05ac2fa197)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 21:38:32 +00:00
Richard Purdie
b1a5b43735 bitbake: event: Fix multiconfig event handler change performance regressions
There were two issues in this code, firstly the code could stack duplicates
in the variable, secondly, calling "if data" caused the datastore to compute
len(data) which is comparitively expensive. Checking "if data is not None"
is much much faster/cheaper.

The issue was clear from "bitbake -p -P" output where the time in register()
showed large amounts of time in the __len__ function of the datastore.

(Bitbake rev: 9298d08b7dcf9d79f54536f87686d65c4ad7deb9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Chen Qi
d3a0f074bc bitbake: event.py: fix regression about INVALIDCONF
A previous commit uses __BBHANDLERS_MC to record all handlers, but
it does not take into consideration of INVALIDCONF, thus causing regression.

We need to record the name before returned AlreadyRegistered, otherwise, when
reparsing due to INVALIDCONF, bascially all handlers are not called.

(Bitbake rev: f6fb6f22fa6ba1ffe121948d7f53ac724358491e)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Joshua Watt
40f5a289d5 bitbake: event: Fix broken builds when multiconfig has a hyphen in the name
5f7fdf7b2d ("bitbake: event: Prevent bitbake from executing event
handler for wrong multiconfig target") broke multiconfig builds contain
a hyphen, since it's attempt to use the multiconfig as part of a
function name and python functions are not allowed to contain a hyphen.

Rework the bitbake multiconfig test to test a multiconfig with a hyphen
and one with an underscore to validate this doesn't break in the future.

(Bitbake rev: c3168df330a4563cbd03ba74de55a22217d823ed)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-21 22:02:01 +00:00
Tomasz Dziendzielski
8b792d4f75 bitbake: event: Prevent bitbake from executing event handler for wrong multiconfig target
When multiconfig is used bitbake might try to run events that don't
exist for specific mc target. In cooker.py we pass
`self.databuilder.mcdata[mc]` data that contains names of events'
handlers per mc target, but fire_class_handlers uses global _handlers
variable that is created during parsing of all the targets.

This leads to a problem where bitbake runs event handler that don't
exist for a target or even overrides them - if multiple targets use
event handler with the same name but different code then only one
version will be executed for all targets.

See [YOCTO #13071] for detailed bug information.

Add mc target name as a prefix to event handler name so there won't be
two different handlers with the same name. Add internal __BBHANDLERS_MC
variable to have the handlers lists per machine.

(Bitbake rev: 5f7fdf7b2d8c59805c8ef4dae84f536baa5e172b)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-16 11:26:11 +00:00
Chris Laplante
77441a08d4 bitbake: compat.py: remove file since it no longer actually implements anything
Now that compat.py just imports Python standard library stuff, get rid
of the layer of indirection.

(Bitbake rev: e2be6defbb9fcf25f9df04c3b452d0dba48dfd03)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-26 09:05:38 +01:00
Richard Purdie
4a8b8a405d bitbake: event/ast: Add RecipePostKeyExpansion event
(Bitbake rev: 5c30cbe35e921f118e7da6b804af281627329d77)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-27 10:46:33 +01:00
Robert Yang
50524c152c bitbake: event: Remove duplicated items from close matches
It printed duplicated ones when there are multiple similar recipes in
differrent layers, for example, if python-lockfile in different layers,
and there is no python3-lockfile:

$ bitbake python3-lockfile
ERROR: Nothing PROVIDES 'python3-lockfile'. Close matches:
python-lockfile
python-lockfile
python3-aiofiles

Remove the duplicated ones to fix the problem.

(Bitbake rev: 5612192cec9f467e2ab5a86482cb34876d198bc6)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19 11:26:12 +00:00
Frazer Clews
fa5524890e bitbake: lib: amend code to use proper singleton comparisons where possible
amend the code to handle singleton comparisons properly so it only checks
if they only refer to the same object or not, and not bother
comparing the values.

(Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc)

Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19 13:31:05 +00:00
Frazer Clews
0ac5174c7d bitbake: lib: remove unused imports
removed unused imports which made the code harder to read, and slightly
but less efficient

(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)

Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19 13:31:05 +00:00
Richard Purdie
fb8fc5e78c bitbake: event/runqueue: Drop StampUpdate event, its pointless/unused
Whilst this class has existed for years, it doesn't have any
users and has a questionable interface. Drop it to allow for further
simplification and changes.

(Bitbake rev: 3ab51764f7965d696bb2c5a872bf161473df4289)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-15 10:28:12 +01:00
Richard Purdie
0a0cc5ff2e bitbake: event: Clear ui_queue after handling it
(Bitbake rev: 55631da1336589e583e8341a655179f7714ab3fe)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Richard Purdie
9501864db8 bitbake: bitbake: Strip old editor directives from file headers
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.

(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
cf9c0be3f6 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00
Richard Purdie
74feaddda3 bitbake: server/process: Various server startup logging fixes
There were various problems in the server startup loggin:

a) stdout/stderr were not being flushed before forking which could potentially
   duplicate output

b) there were separate buffers for stdout/stderr leading to confusing logs
   where the entries could be reordered. This was particularly confusing
   due to the separator the logs use to idendify new messages

c) an fd wasn't being closed during server startup meaning if the
   server failed to start, the closed fd wasn't detected as it was held
   open by the other reference

d) If the pipe was detected as being closed, the code incorrectly retried
   server startup

e) The event code would remap stdout/stderr without flushing them, leading
   to lose log messages

(Bitbake rev: 0594faa0b52ce5dbd948d836d88617d38d9862d1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-05 18:01:37 +01:00
Kai Kang
5581b374b9 bitbake: event.py: fix typo
(Bitbake rev: d49483eb733ee27325349246fa37a84140c0941d)

Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-14 16:32:48 +01:00
Richard Purdie
869ae2ce2a bitbake: events: Drop unused cfg property
The cfg property on events doesn't add much, all code appears to access
"data" at this point. Remove it to clean up the interface.

(Bitbake rev: bd4b9e4460b60f142c3bf346fb04e360e512eaee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-08 08:48:54 +00:00
Richard Purdie
a04ee42393 bitbake: event: Ensure we clean up loggers
Whilst we're likely exiting in this case, clean up the loggers we add
so that in the case of certain server retries there is no possibility
multiple loggers stack up.

(Bitbake rev: 25b7bf6672be66bcbfe5760610dce7d3e866cdcc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-09 12:24:40 +00:00
Richard Purdie
25e52d34d0 bitbake: event: Don't write duplicate logs to stdout and stderr in no UI case
This code would duplicate messages to stdout and stderr when no UI connected
and there were error level messages.

Rework the code so it either uses stderr (for errors and above) or
stdout for warnings/debug but not both for the same messages.

(Bitbake rev: 45cff5734ba2ba8c8d36d17d722a5804d39b258b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Paul Eggleton
d4c3ace097 bitbake: bitbake-diffsigs: fix regression after recent server changes
We were bridging the gap between the server and UI here by calling a
bb.siggen.find_siginfo, a function defined and set on that module from
the metadata. This worked from the UI side before but since the recent
server changes is no longer accessible. Create a new command so this can
execute on the server side and return the result by way of a new event.

(We're still running compare_sigfiles() on the signature generator but
that isn't quite the same thing and does still work.)

Fixes [YOCTO #11844].

(Bitbake rev: fdcea991baa4f83d9c98d468d7b49c8c388a4a15)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-27 15:14:20 +01:00
Paul Eggleton
21bb330f46 bitbake: lib/bb/event: refactor printing events
We really ought to have just one place where the string representation
of these events is produced. This doesn't take any real control away
from the UI - if an alternative representation is desired, that can
still be made.

(Bitbake rev: cb15db2a799be6d8eab9a2a43a9a573f89229cff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Richard Purdie
dd71707d5a bitbake: event/command: Allow UI to request the UI eventhander ID
The UI may want to change its event mask however to do this, it needs the
event handler's ID. Tweak the code to allow this to be stored and add
a command to query it.

Use the new command in the process server backend.

(Bitbake rev: f8cf2cb58b80ce74f756a11a9773b6b0e78d51ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Richard Purdie
35846a3961 bitbake: event: Queue offline events for the UI
Messages printed when no UI is connected (e.g. memres) are currently lost.
Use the existing queue mechanism to queue these until a UI attaches, then
replay them. This isn't ideal but better than the current situation of
losing them entirely.

(Bitbake rev: cb241fb8544dfb05646dbae1a1b04e17878a466c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08 13:29:54 +01:00
Richard Purdie
bca4396106 bitbake: bitbake: Add MultiConfigParsed event
There are some cases where the metadata needs to be aware a multiconfig build
is happening and have access to the multiconfig data stores to merge data into
the common build. This adds such an event allowing access to these datastores.

(Bitbake rev: 160e47f5df90850e64dcb857c81a5039abc9235f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08 13:29:54 +01:00
Jair Gonzalez
d591b00af0 bitbake: event: remove mapping for deleted event handlers
The current remove method for class event handlers does not update the
event mapping after an event handler is deleted from the main dictionary.

This change enhances the remove method by also updating the event mapping
accordingly.

This was detected after creating the bb.tests.event module.

[YOCTO #10368]

(Bitbake rev: 839f83dea1e99d1d182b138e5aea1df1c45a92fc)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23 14:14:17 +01:00
Paul Eggleton
b367cd91d1 bitbake: event: drop some unused events
These were all used by support code for the Hob UI which has been
removed - nothing will currently fire or respond to these events.

(Bitbake rev: 03ba8db0595723286ad462fa197f862e1efdf0de)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-22 09:23:38 +01:00
Richard Purdie
631229df72 bitbake: event/ast: Add RecipeTaskPreProcess event before task finalisation
There are various pieces of code which need to run after the tasks are
finalised but before bitbake locks in on the task dependencies. This
adds such an event so dependency changes in anonymous python can
be accounted for and acted upon by these specific event handlers.

(Bitbake rev: 4dcd0e53f5ff4bf4f2d6cbdc51ff33a5f5f206af)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:52:52 +00:00
Richard Purdie
840e221ea7 bitbake: event: Replace slightly elderly bbmake references!
bbmake hasn't existed for a while (a decade+) so we should
probably remove the references.

(Bitbake rev: 4170cf874850b950f31d2e36c895c110c0b096c3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:13 +00:00
Patrick Ohly
268bf22263 bitbake: monitordisk: add event
The current disk usage is interesting and may be worth logging over
time as part of the build statistics. Instead of re-implementing the
code and the configuration option (BB_DISKMON_DIRS), the information
gathered by monitordisk.py is made available to buildstats.bbclass via
a new event.

This has pros and cons:
- there is already a useful default configuration for "interesting" directories
- no code duplication
- on the other hand, users cannot configure recording separately from
  monitoring (probably not that important)

(Bitbake rev: f065ac17d0031dca6309ddbff18c8792630de865)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Patrick Ohly
083365143e bitbake: cooker process: fire heartbeat event at regular time intervals
The intended usage is for recording current system statistics from
/proc in buildstats.bbclass during a build and for improving the
BB_DISKMON_DIRS implementation.

All other existing hooks are less suitable because they trigger at
unpredictable rates: too often can be handled by doing rate-limiting
in the event handler, but not often enough (for example, when there is
only one long-running task) cannot because the handler does not get
called at all.

The implementation of the new heartbeat event hooks into the cooker
process event queue. The process already wakes up every 0.1s, which is
often enough for the intentionally coarse 1s delay between
heartbeats. That value was chosen to keep the overhead low while still
being frequent enough for the intended usage.

If necessary, BB_HEARTBEAT_EVENT can be set to a float specifying
the delay in seconds between these heartbeat events.

(Bitbake rev: 7cf22ea057d28c54bd98dc1ab7a43402a29ff1f5)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Aníbal Limón
924ca1037c bitbake: bb.event: fix infinite loop on print_ui_queue
If bitbake ends before _uiready and bb.event.LogHandler was add
to the bitbake logger it causes an infinite loop when logging
something.

The scenario is print_ui_queue is called at exit and executes
the log handlers [2] one of them is bb.event.LogHandler this handler
appends the same entry to ui_queue causing the inifine loop [3].

In order to fix a new copy of the ui_queue list is created when iterate
ui_queue.

[YOCTO #10399]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=10399#c0
[2] http://git.openembedded.org/bitbake/tree/lib/bb/event.py?id=41d9cd41d40b04746c82b4a940dca47df02514fc#n156
[3]
http://git.openembedded.org/bitbake/tree/lib/bb/event.py?id=41d9cd41d40b04746c82b4a940dca47df02514fc#n164

(Bitbake rev: 46fecca9d531a07788b5cac8b2dc6a8267d8b6d0)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-15 10:01:44 +01:00
Joshua Lock
1110dde73a bitbake: event: prevent unclosed file warning in print_ui_queue
Use logger.addHandler(), rather than assigning an array of Handlers
to the loggers handlers property directly, to avoid a warning from
Python 3 about unclosed files:

$ bitbake
Nothing to do.  Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.
WARNING: /home/joshuagl/Projects/poky/bitbake/lib/bb/event.py:143: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/joshuagl/Projects/poky/build/tmp/log/cooker/qemux86/20161004094928.log' mode='a' encoding='UTF-8'>
  logger.handlers = [stdout]

(Bitbake rev: 1e23b1f1a80066223b98e18b163840051ac74944)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-05 10:25:34 +01:00
Aníbal Limón
79012f943d bitbake: bb/event.py: fire_ui_handlers enable threading lock support
In some cases there is a need to fire bb events into multiple
python threads so locking is needed (writing to a fd/socket).

Adding a helper functions for disable/enable by request to avoid
overhead.

[YOCTO #10330]

(Bitbake rev: a583dc0b296415ec904c081c4de96ceef46732a8)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-05 10:10:11 +01:00
Ed Bartosh
d649c64beb bitbake: event.py: output errors and warnings to stderr
All logging messages are printed on stdout when processing
UI event queue. This makes it impossible to distinguish between
errors and normal bitbake output. Output to stderror or stdout
depending on log level should fix this.

(Bitbake rev: 56ac0d4c7a5f47aeb707b15a0c305d9f73aae945)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-15 12:15:07 +01:00
Elliot Smith
c471740f5b bitbake: cooker: add BuildInit event
In situations where a bitbake run fails before the build
properly starts and BuildStarted is fired, a UI has no way
to get at the targets passed to the build. This makes it
difficult for the UI to report on the targets which failed.

Fire a BuildInit event before running buildTargets() or
buildFile(). This enables a UI to capture targets passed to
buildTargets(), even if the build fails (e.g. the targets
themselves are invalid).

[YOCTO #8440]

(Bitbake rev: ac02fda870965bf7d44ff5688eda54d2d11ab9c7)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11 00:09:26 +01:00