currentAsyncCommand currently doesn't have any locking and we have
a conflict in "idle" conditions since the idle functions count needs
to be zero *and* there needs to be no active command.
Move the changes/checks of currentAsyncCommand to within the lock
and then we can add it to the condition for idle, simplifying some
of the code.
(Bitbake rev: b5215887d2f8ea3f28f1ebda721bd5b8f93ec7f3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We no longer need to abstract the inotify callback handler, remove the
abstraction and simplify/clean up the code.
(Bitbake rev: af4ccab8acc49e91bf7647f209d69f4858618466)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The thread/process locks we use translate to futexes in Linux. If a
process dies holding the lock, anything else trying to take the lock
will hang indefinitely. An example would be the OOM killer taking out
a parser process.
To avoid bitbake processes just hanging indefinitely, add a timeout to
our lock calls using a context manager. If we can't obtain the lock
after waiting 5 minutes, hard exit out using os._exit(1). Use _exit()
to avoid locking in any other places trying to write error messages to
event handler queues (which also need locks).
Whilst a bit harsh, this should mean we stop having lots of long running
processes in cases where things are never going to work out and also
avoids hanging builds on the autobuilder.
(Bitbake rev: d2a3f662b0eed900fc012a392bfa0a365df0df9b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When bitbake is off running heavier "idle" commands, it doesn't service it's
command socket which means stopping/interrupting it is hard. It also means we
can't "ping" from the UI to know if it is still alive.
For those reasons, split idle command execution into it's own thread.
The commands are generally already self containted so this is easier than
expected. We do have to be careful to only handle inotify poll() from a single
thread at a time. It also means we always have to use a thread lock when sending
events since both the idle thread and the command thread may generate log messages
(and hence events). The patch depends on previous fixes to the builtins locking
in event.py and the heartbeat enable/disable changes as well as other locking
additions.
We use a condition to signal from the idle thread when other sections of code
can continue, thanks to Joshua Watt for the review and tweaks squashed into this
patch. We do have some sync points where we need to ensure any currently executing
commands have finished before we can start a new async command for example.
(Bitbake rev: 67dd9a5e84811df8869a82da6a37a41ee8fe94e2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In preparation for adding splitting bitbakes work into two threads,
add locking around the idle functions list accesses.
(Bitbake rev: a9c63ce8932898b595fb7776cf5467d3c0afe4f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When idle handlers want to exit, returning "False" isn't very clear
and also causes challenges with the ordering of the removing the idle
handler and marking that no async command is running.
Use a specific class to signal the exit condition allowing clearer code
and allowing the async command to be cleared after the handler has been
removed, reducing any opportunity for races.
(Bitbake rev: 102e8d0d4c5c0dd8c7ba09ad26589deec77e4308)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the idle functions loop suffers a traceback, it is
silently dropped and there is no log message to say what happened.
This change at least means the traceback is in the cooker log, making
some debugging possible.
Add some logging to show when handlers are added/removed to allow
a better idea of what the server code is doing from the server log
file.
(Bitbake rev: 9cf3102dc36513124fe5ead2f1e448b51833b6ac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently heartbeat events are always generated by the server whilst it is
active. Change this so they only appear when builds are running, which is
when most code would expect to be executed. This removes a number of races
around changes in the datastore which can happen outside of builds.
(Bitbake rev: 8c36c90afc392980d999a981a924dc7d22e2766e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen cases where the bitbake.sock file appears to disappear but the
server continues to hold bitbake.lock. The most likely explaination is
that some previous build directory was moved out the way, a server there
kept running, eventually exited and removed the sock file from the wrong
directory.
To guard against this, save the inode information for the sock file and check
it before deleting the file. The new code isn't entirely race free but should
guard against what is a rare but annoying potential issue.
(Bitbake rev: b02ebbffdae27e564450446bf84c4e98d094ee4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Log when the socket file already exists and is removed before
recreating a new socket.
Log when unlinking the socket file failed.
(Bitbake rev: cfd7c9899f988bab6d9fe7bbfbdb60603fb5ed34)
Signed-off-by: Frank de Brabander <debrabander@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're struggling to understand how bitbake.sock can sometimes disappear
in live builds when we can't see where it could have been deleted.
This causes connection failures to the server and failed builds.
Add some extra debugging around the server log and client retry
log messages to give more information for the next time this issue
occurs.
(Bitbake rev: 376a516dc8c96727fd042ada65f803013601ee2d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We keep seeing server issues where the lockfile is present but we can't
connect to it. Reuse the lockfile debugging code from the server to
dump better information to the console from the client side when we
run into this issue. Whilst not pretty, this might give us a chance
of being able to debug the problems further.
(Bitbake rev: 22685460b5ecb1aeb4ff3436088ecdacb43044d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous cleanups meant that when the cooker was started, profiling
was always disabled as configuration was sent to the server later and this
was too late to profile the main loop.
Pass the "profile" option over the server commandline so that we can
profile cooker itself again, the setting can now take effect early enough.
(Bitbake rev: c97c1f1c127ef3f8fbbd1b4e187ab58bfb0a73e5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I realised only the first logging message was being displayed in a given
parsing process. The reason turned out to be the UI handler failing
with a "pop from empty list". The default handler was then lost and
no further messages were processed.
Fix this by catching the exception correctly in the connection writer code.
(Bitbake rev: d3e64f64525187f1409531a0bd99df576e627f7f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In theory this should have been worked around but is still occurring. Add
it to the list of things to ignore when bitbake is shutting down.
Traceback (most recent call last):
File "/usr/lib64/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 698, in startCallbackHandler
event = self.reader.get()
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 722, in get
res = self.reader.recv_bytes()
File "/usr/lib64/python3.9/multiprocessing/connection.py", line 221, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib64/python3.9/multiprocessing/connection.py", line 426, in _recv_bytes
return self._recv(size)
File "/usr/lib64/python3.9/multiprocessing/connection.py", line 384, in _recv
chunk = read(handle, remaining)
TypeError: an integer is required (got type NoneType)'
(Bitbake rev: 7a28ac4fe478bee1e52e84412da9626495f9c6c7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing UI deadlocks occasionally and this is possibly due to the
use of a daemonic thread in the UI event queue processing. This thread
could terminate holding a threading Lock() which would cause issues
for the process when exitting.
Change the shutdown process to handle this more cleanly.
(Bitbake rev: f5ad8349a5dbff9824a89f5708cfd011d61888c9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The open coded lock acquire/release in the UI event handler doesn't
cover the case an exception occurs and if one did, it could deadlock
the code. Switch to use 'with' statements which would handle this
possibility.
We have seen deadlocks in the UI at exit this so this removes a
possible cause.
(Bitbake rev: bd12792f28efd2f03510653ec947ebf961315272)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The python gc can trigger whilst we're holding the event stream lock
and when cleaning up objects, they can trigger warnings. This translates
into a new event which would then need the lock and we can deadlock.
Disable gc whilst we hold that lock to avoid this unfortunate and
problematic situation.
(Bitbake rev: 96a6303949cefd469bcf5ed250ff512271354357)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a parser process is terminated while holding a write lock, then it
will lead to a deadlock (see
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.terminate).
With SIGTERM, we don't want to terminate holding the lock. We also don't
want a SIGINT to cause a partial write to the event stream.
I tried using signal masks to avoid this but it doesn't work, see
https://bugs.python.org/issue47139
Instead, add a signal handler and catch the calls around the critical section.
We also need a thread lock to ensure other threads in the same process don't
handle the signal until all the threads are not in the lock.
(Bitbake rev: a40efaa5556a188dfe46c8d060adde37dc400dcd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This debug is useful but the cooker shutdown or post_serve() may have cleanup
left so run after those.
(Bitbake rev: 1463fc0448d1a6a7265806a4a8b165b610dfb43f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python scales badly when concatinating strings in loops. Most of these
references aren't problematic but at least one (in data.py) is probably
a performance issue as the issue is compounded as strings become large.
The way to handle this in python is to create lists which don't reconstruct
all the objects when appending to them. We may as well fix all the references
since it stops them being copy/pasted into something problematic in the future.
This patch was based on issues highligthted by a report from AWS Codeguru.
(Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the code can just show nothing as the exception if there was a double
fault, which in this code path is quite likely. This leads to an error log
which effectively says "it failed" with no information about how.
Improve things so we get a nice verbose traceback left in the logs/output
which is preferable to no logs.
(Bitbake rev: e5782b71647d1eb6de53bde7bc4f6019a5589f21)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the client process never sends cooker data, the server timeout will
be 0.0, not None. This will prevent the server from exiting, as it is
waiting for a new client. In particular, the client will disconnect with
a bad "INHERIT" line, such as:
INHERIT += "this-class-does-not-exist"
Instead of checking explicitly for None, check for a false value, which
means either 0.0 or None.
(Bitbake rev: 13e2855bff6a6ead6dbd33c5be4b988aafcd4afa)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen cases where an OOM error causes bitbake server to hang:
9171 02:21:09.127810 Command Completed
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/bin/bitbake-server", line 51, in <module>
bb.server.process.execServer(lockfd, readypipeinfd, lockname, sockname, timeout, xmlrpcinterface)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 550, in execServer
server.run()
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 108, in run
ret = self.main()
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 242, in main
ready = self.idle_commands(.1, fds)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 370, in idle_commands
bb.event.fire(heartbeat, self.cooker.data)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 216, in fire
fire_class_handlers(event, d)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 123, in fire_class_handlers
execute_handler(name, handler, event, d)
File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 93, in execute_handler
ret = handler(event)
File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 182, in defaultrun_buildstats
write_host_data(os.path.join(bsdir, "host_stats"), e, d, "interval")
File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 160, in write_host_data
output = subprocess.check_output(c.split(), stderr=subprocess.STDOUT, timeout=limit).decode('utf-8')
File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory
We need to wrap the calls in the same high level wrapper as idle function calls
and trigger an exit upon an unhandled exception.
(Bitbake rev: 74042b5b89d5a170013fc1a327ce3a6530fbf7d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-server is spawned by process.py and passes the arguments it is
given to ProcessServer. There's some type confusion here:
bitbake-server is called with a string representation of the timeout,
which may be None. If the timeout is not set, pass 0 instead of None.
Inside bitbake-server a ProcessServer is created which expects the
timeout to be a float not a string, so always float() the value.
[ YOCTO #14350 ]
(Bitbake rev: c93ae1f861208f6d39fd15c84fbcd0e2b54331f5)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are autobuilder logs where the server commands are failing
but we have no debug info in the server log. Improve this to try and
understand what is failing.
(Bitbake rev: 04d3a79226c9ea448b22f4efbab33876a72c9bdb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its hard to tell from the server logs whether commands complete or not
(or how long they take). Add extra info to allow more debugging of
server timeouts.
(Bitbake rev: 56285ada585ec1481449522282b335bcb5a2671e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Knowing which process printed which messages and the timestamp of the
message is useful for debugging, so add this. Ensure the log parsing
isn't affected by using search() instead of match().
(Bitbake rev: 1d043666710df1fa9d9586fd974c0371dd1514b0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Showing "leftover process" messages when a new server has started and is being
used by some UI is horrible. Compare the PID data from the lockfile to
avoid this (and the ton of confusing log data it generates).
Also, move the time.sleep() call to be after the first lock attempt, which
reduces noise in the logs significantly.
(Bitbake rev: ce1897a31afb5a14997bc3d2f459b90d43eecb7d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lsof/fuser error if the file doesn't exist. It can be deleted by something
else so ignore this if it happens and loop.
(Bitbake rev: b100d22ce37b7548b50e59a71802bcc903acd6ea)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cookerlog output goes to a file and its misleading to look at it and
not have it up to date with what the cooker is actually doing. Ensure
written data is flushed.
Ultimately this should be using python's logging but that is for another
day, we need simple fixes right now.
(Bitbake rev: d95c3dd2b8ac50423976a7baf0a51e9580871761)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The binary string printed into the output is ugly, parse this so the
linebreaks come out in the logs and make them much more readable (I
was misssing the information initially despite looking for it).
(Bitbake rev: c2dd8bb434d5738fedf9019651074b90affff3b2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With an invalid layer, the desired error output should be:
ERROR: The following layer directories do not exist:
ERROR: /this_path_does_not_exist
ERROR: Please check BBLAYERS in .../build-invalid-layer/conf/bblayers.conf
Instead we were met with a backtrace:
Traceback (most recent call last):
File "/scratch1/fray/xilinx/poky/bitbake/bin/bitbake", line 36, in <module>
cookerdata.CookerConfiguration()))
...
File "/scratch1/fray/xilinx/poky/bitbake/lib/bb/cookerdata.py", line 267, in parseBaseConfiguration
self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles)
File "/scratch1/fray/xilinx/poky/bitbake/lib/bb/cookerdata.py", line 358, in parseConfigurationFiles
sys.exit(1)
SystemExit: 1
(Bitbake rev: 3a2503c785a5cd9dca0dc68c3aec31b4bec7684b)
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new server startup code means exceptions can happen when we aren't
setup to show them to the user correctly, leading to ugly tracebacks.
Add in some special case handling of BBHandledException to at least
ensure that common case doesn't traceback and the user sees meaningful
output.
In the future, the logging setup can likely be improved, as can the way
runCommand handles exceptions, they all should likely become real
exceptions again on the UI side.
[YOCTO #14022]
[YOCTO #14033]
(Bitbake rev: 6059d0e77f60ddb679049bd34478f41b1ab7995d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I have a suspicion based on process traces that the flock() call is no
longer interrupted by SIGALRM and hence the timeout doesn't work. We
were relying on EINTR triggering around syscalls but python is likely
protecting us from that in modern versions.
Re-implement this code with a different mechanism which doesn't have
that potential issue.
(Bitbake rev: 8eb52afdfd4c3e6478d4f8cc56e99def3f1c924c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using sys.executable ensures we're using the same python binary for the server
as the client, which is probably advisable. "bitbake-server" is left as the process
name as its more distinctive in process listings.
(Bitbake rev: 387a339b330e8122a62a148249beb3f064dd4e3d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Correctly import, and inherit functions, and variables.
Also fix some typos and remove some Python 2 code that isn't recognised.
(Bitbake rev: b0c807be5c2170c9481c1a04d4c11972135d7dc5)
Signed-off-by: Frazer Clews <frazerleslieclews@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trying to have a new python process forked off an original doesn't work
out well and ends up having race issues. To avoid this, exec() a new
bitbake server process. This starts with a fresh python interpreter
and resolves various atexit and other multiprocessing issues once
and for all.
(Bitbake rev: 9501dd6fdd7a7c25cbfa4464cf881fcf8c049ce2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Dump info into the logs if there are extra threads left at process exit
time.
(Bitbake rev: 1c9496797b753e67351bd5cb98ef2b8e9435d51e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The first thing the UIs do is update the server config from the UI. We
can just rely upon that and start the server with a standard config,
removing the need to pass the confusing configuration object around
as well as configParams, which contains a similar copy of some of the
data.
This makes memory resident bitbake work the same way as the normal
mode, removing the opportunity for some class of bugs.
The xmlrpcinterface and server_timeout values are passed in at server
startup time now and there no longer a second option in the
configuration which is effective ignored once the server starts.
(Bitbake rev: 783a03330802e83c525c55522e3ee2a933bded3a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sock object isn't used client side so we can just created it in
the server process and save passing around the fd/object.
(Bitbake rev: ee5d2c92dcce89ccb701e028ffc6419eb315f5ce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we end up parsing the base configuration multiple times as
initially, the right settings haven't come from the UI. We can defer
this until later in startup using runCommand as a trigger.
The advantage to doing this is improved startup times and ultimately
we should be able to avoid the double parse of the base configuration.
(Bitbake rev: 3caa43b665604475d2c87ba505efb0b9fca9c2e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is the possibility data is being lost from the logfile due to
data buffering. Add in a couple of extra flush calls to ensure data
is being written out before the lock file is dropped.
Possible fix for [YOCTO #14000]?
(Bitbake rev: 596ea229a87d26a8e970c7ee77179519ad081fef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Further cleanup, just pass these settings directly.
(Bitbake rev: ac2284357f1fc7044dac9c146fad218fc9906412)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having the idle callbacks abstracted via the configuration object
makes no sense. Its like this for historical reasons from the
multiple server backends but we don't need this now so simplfy.
(Bitbake rev: e56c49717355c9493b07d5fc80981a95ad8a0ec8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
