This cleans up the socket keep alive into better class structured code
and adds cleanup of the open file descriptors upon shutdown.
(Bitbake rev: 77fd3a3a29a569e212374b27aea742ddbaafcdd5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the monitoring thread exits with some error code or runs indefinitely. Allow
closure of the pipe its monitoring to have the thread exit cleanly/silently.
(Bitbake rev: 930d077637928213e13a07c78fee3bf7a8c37ebf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Expose the underlying close() and fileno() methods which allow connection
monitoring and cleanup.
(Bitbake rev: f79187f4ebfad7969be47b429995e7f7a3e33c1e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code would duplicate messages to stdout and stderr when no UI connected
and there were error level messages.
Rework the code so it either uses stderr (for errors and above) or
stdout for warnings/debug but not both for the same messages.
(Bitbake rev: 45cff5734ba2ba8c8d36d17d722a5804d39b258b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were bridging the gap between the server and UI here by calling a
bb.siggen.find_siginfo, a function defined and set on that module from
the metadata. This worked from the UI side before but since the recent
server changes is no longer accessible. Create a new command so this can
execute on the server side and return the result by way of a new event.
(We're still running compare_sigfiles() on the signature generator but
that isn't quite the same thing and does still work.)
Fixes [YOCTO #11844].
(Bitbake rev: fdcea991baa4f83d9c98d468d7b49c8c388a4a15)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When wget is fetching a listing for a directory over FTP it writes to a
temporary file called .listing in the current directory. If there are many such
operations happening in parallel - for example during 'bitbake world -c
checkpkg' - then up to BB_NUMBER_THREADS instances of wget will be racing to
write to, read, and delete the same file.
This results in various failures such as the file disappearing before wget has
processed it or the file changing contents, which causes checkpkg to randomly
fail.
Mitigate the race condition by creating a temporary directory to run wget in
when doing directory listings.
[ YOCTO #11828 ]
(Bitbake rev: 91d4ca93df092cf86ab84faaa94cc66ff9f43057)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The warning message currently shown can occur more frequently than previously
if a previous bitbake server is shutting down and we're reconnecting to a new
server. Change it to a note message to match the higher level connection
logging retry messages and so as not to interfer with selftests.
(Bitbake rev: b7514340cd6a2753eb217b059229bb279c3849ec)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that there is only one server, this abstraction is no longer needed
and causes indrection/confusion. The server shutdown is also broken with
the cooker post_server calls happening too late, leading to "lock held"
warnings in the logs if PRServ is enabled.
Remove the abstraction and put the shutdown calls in the right order
with respect to the locking.
(Bitbake rev: c0ddde7cf680225127d6285685652b905ed176c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the sstate is accessed via HTTP, the existence check can fail due
to network issues, in which case bitbake silently continues without
sstate.
One such network issue is an HTTP server like Python's own SimpleHTTP
which closes the TCP connection despite an explicit "Keep-Alive" in
the HTTP request header. The server does that without a "close" in the
HTTP response header, so the socket remains in the connection cache,
leading to "urlopen failed: <urlopen error [Errno 9] Bad file
descriptor>" (only visible in "bitbake -D -D" output) when trying to
use the cached connection again.
The connection might also get closed for other reasons (proxy,
timeouts, etc.), so this is something that the client should be able
to handle.
This is achieved by checking for the error, removing the bad
connection, and letting the check_status() method try again with a new
connection. It is necessary to let the second attempt fail
permanently, because bad proxy setups have been observed to also lead
to such broken connections. In that case, we need to abort for real
after trying twice, otherwise a build would just hang forever.
[YOCTO #11782]
(Bitbake rev: 6fa07752bbd3ac345cd8617da49a70e0b2dd565f)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If BB_STRICT_CHECKSUMS is set to anything other than "1" i.e. we're not
going to raise an error, then fire an event so that scripts can listen
for it and get the checksums.
(Bitbake rev: 8b2ccb4b865f2df118ef668847df682a83f9c500)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If BB_STRICT_CHECKSUMS is set to "ignore" then don't display a warning
if no checksums are specified in the recipe. This is not intended to be
used from recipes - it is needed when we move to using more standard
code paths to fetch new files from scripts i.e. where we don't know what
the checksums are in advance.
(Bitbake rev: f15ca7339de8a448a93a14cf6130b3925178a920)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want this API to be easier to use, so add missing function
documentation to help with that.
(Bitbake rev: 3e0e002d6497caa987f327cd83ad4db82cca6810)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A common task for tinfoil-using scripts is to iterate over all recipes.
This isn't too difficult with the current API, but the pkg_* variables
are a little awkward and are really designed for bitbake's internal
usage - and it gets a bit more difficult when you want to access some of
the other information such as packages and rprovides. To resolve this,
create a new recipe info class and add an all_recipes() function to
generate this for all recipes. Also add a get_recipe_info() function to
get the information for a specific recipe (by PN).
(It might perhaps be suggested that we already have a structure similar
to this in the cache, however the one we add here is designed for
external use and allows the internal structures to change if needed
without affecting the API).
(Bitbake rev: 308994028e59735ca726c5d2c1f0f85baccfe89d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add access to fn_provides, packages, packages_dynamic and rproviders on
the recipecache object. This requires an additional corresponding
command plumbing to be added.
(Bitbake rev: 3df9b7c615174a6557581f3cd157842a28f6bb26)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Up to this point, if you wanted to run build tasks in the normal way
they get run from a python script, there was no other way than to shell
out to bitbake. Worse than that, you couldn't have tinfoil active during
that because only one bitbake instance could be running at once. As long
as we're prepared to handle the events produced, we can create a wrapper
around calling the buildTargets command. Borrow code from knotty to do
this in such a way that we get the expected running task display
(courtesy of TermFilter) and Ctrl+C handling.
(Bitbake rev: 43761eee756be52a1021be53a40dc591a6c35fa7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This isn't useful for knotty itself, but for use from tinfoil
in case we can't get access to either the console or errconsole, allow
either to be unspecified (None).
(Bitbake rev: 7544de437fc66b81502ecdb5db859182c45827cb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is useful for the caller to know whether the uihelper has handled the
event passed so that it can skip other event handling code if so.
(Bitbake rev: be498abfbbb19bdd31e5b53713a74049007e3737)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We really ought to have just one place where the string representation
of these events is produced. This doesn't take any real control away
from the UI - if an alternative representation is desired, that can
still be made.
(Bitbake rev: cb15db2a799be6d8eab9a2a43a9a573f89229cff)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make it easy to determine if recipes are parsed (and thus information
about available recipes is in memory).
(Bitbake rev: 7efde2df2ff25063d36ac015146f1975284a69ff)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python style recommends underscore based naming rather than camelCase,
and thus the former has been used for most of tinfoil's functions. Add
an underscored version of parseRecipes() for consistency and change the
one place we call it to use the new version.
(Bitbake rev: 821f6c41d850752d2bcc2ccd4f8e75b2897a0a3e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In OE's devtool we want to repeatedly run build_file() without showing
unnecessary messages and triggering buildhistory for each call.
build_file() is just a wrapper around the buildFile command. Change
the final "hidewarning" parameter of the buildFile command to "internal"
and have this call a new buildFileInternal() function without triggering
any of the normal build events, silencing the normal info messages from
the runqueue ("Executing RunQueue Tasks", "Tasks Summary" etc.) and
avoiding calling parseConfiguration() which we've already done at this
point.
(Bitbake rev: ba53e067a2d448dd63b4ca252557ce98aa8e6321)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
$ bitbake --observe-only
FATAL: '--observe-only' can only be used by UI clients connecting to a server.
And even:
$bitbake --observe-only -B localhost:-1
FATAL: '--observe-only' can only be used by UI clients connecting to a server.
This was beucase the brackets were in a wrong position.
(Bitbake rev: 4ffc91a2b3eb13e98078e6b1913f056a0c1797bc)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed 1:
$ . ../poky/oe-init-build-env-memres .
$ bitbake -m
$ bitbake -m # shutdown it again
$ bitbake -m
Starting bitbake server...
Terminated bitbake server.
It starts the server and kill it which is redundant.
$ bitbake -m
$ bitbake --status-only
Starting bitbake server...
It starts the server which is not what we need.
(Bitbake rev: 335fd0106359dddfe1eeb0c7d0bc03f8454b895c)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
$ bitbake --server-only -B localhost:-1
Infinite loop
Reconnecting to bitbake server...
[snip]
Break the loop when server-only can fix the problem.
(Bitbake rev: c4d5c3dab5b35f3c6713efad1c8a23229ce4e56b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no global var "configuration", so the old code hang at
self.bitbake_lock.write(), and nothing wrote to bitbake.lock. I didn't
figure out why it hang (but not print errors).
Reproducer:
$ bitbake -B localhost:-1 world -k
Check bitbake.log, there was nothing, now fixed.
(Bitbake rev: dbdd9010663cd8dcb328e1b6fb40e3a777293bc5)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This changes the way bitbake server works quite radically. Now, the
server is always a process based server with the option of starting
an XMLRPC listener on a specific inferface/port.
Behind the scenes this is done with a "bitbake.sock" file alongside
the bitbake.lock file. If we can obtain the lock, we know we need
to start a server. The server always listens on the socket and UIs
can then connect to this. UIs connect by sending a set of three file
descriptors over the domain socket, one for sending commands, one for
receiving command results and the other for receiving events.
These changes meant we can throw away all the horrid server abstraction
code, the plugable transport option to bitbake and the code becomes
much more readable and debuggable. It also likely removes a ton of
ways you could hang the UI/cooker in weird ways due to all the race
conditions that existed with previous processes.
Changes:
* The foreground option for bitbake-server was dropped. Just tail
the log if you really want this, the codepaths were complicated enough
without adding one for this.
* BBSERVER="autodetect" was dropped. The server will autostart and
autoconnect in process mode. You have to specify an xmlrpc server
address since that can't be autodetected. I can't see a use case
for autodetect now.
* The transport/servetype option to bitbake was dropped.
* A BB_SERVER_TIMEOUT variable is added which allows the server
to stay resident for a period of time after the last client
disconnects before unloading. This is used if the -T/--idle-timeout
option is not passed to bitbake.
This change is invasive and may well introduce new issues however I
believe the codebase is in a much better position for further
development and debugging.
(Bitbake rev: 72a3dbe13a23588e24c0baca6d58c35cdeba3f63)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if this code is used with something like oeqa's xml logging
it fails as sys.stdout is an io stream. Add in try/except to handle
this case.
Add a waitpid() call to remove a zombie whilst forking.
Also, append to the logfile, don't overwrite it (otherwise
debugging can be a real pain when the server is restarting for
unknown reasons).
(Bitbake rev: 4e780fa19fc720948cf73133f56f7d837ad9283a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Finding the top level build directory is currently hard and relies on
having a complete cooker being setup. Add a helper function which
does the same thing without all the extra overhead. This is needed
to be able to locate the bitbake lockfile and hence the socket
for connecting clients in the new server model.
(Bitbake rev: d196afe68032898c31a8599ca7d3ceba58d96b0a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In preparation for rewriting this code, expand the relatively useless
base classes into the code itself.
(Bitbake rev: a1c6151420d86bac658c08ae714647062edd6ef2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The UI may want to change its event mask however to do this, it needs the
event handler's ID. Tweak the code to allow this to be stored and add
a command to query it.
Use the new command in the process server backend.
(Bitbake rev: f8cf2cb58b80ce74f756a11a9773b6b0e78d51ee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix one incorrect indentation and several lines with trailing spaces.
(Bitbake rev: 4dbf4f1907a32f303c2e9acf2f27817cf3c9ec38)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To quote Paul:
Not that long ago we added a prefix to logged messages to allow us to see
where the message has been generated (recipe / task). This is undoubtedly
useful for errors and warnings, however, I'm not sure it's really appropriate
for bb.plain(). As an example, see the output for -c listtasks now:
...
NOTE: Executing RunQueue Tasks
nodejs-native-4.5.0-r0 do_listtasks: do_addto_recipe_sysroot
nodejs-native-4.5.0-r0 do_listtasks: do_build Default task for a recipe - depends on all other normal tasks required to 'build' a recipe
nodejs-native-4.5.0-r0 do_listtasks: do_checklicense
nodejs-native-4.5.0-r0 do_listtasks: do_checklicenseall
nodejs-native-4.5.0-r0 do_listtasks: do_checkpkg
...
This patch excludes PLAIN messages from this prefixing making the log output
neater.
[YOCTO #11457]
(Bitbake rev: 4a14b44b3e4fad3a3e5e53461aa8ba9929a515b8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When heartbeat event support was added it was only added to process.py. Add
it to server/xmlrpc too. There is duplicated code however since we're likely
to combine the server abstractions soon its not worth worrying about now.
This ensures the backends have the same event support.
[YOCTO #10741]
(Bitbake rev: 00bf2e60222767b4dee84fb3f958732a83544e80)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Messages printed when no UI is connected (e.g. memres) are currently lost.
Use the existing queue mechanism to queue these until a UI attaches, then
replay them. This isn't ideal but better than the current situation of
losing them entirely.
(Bitbake rev: cb241fb8544dfb05646dbae1a1b04e17878a466c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
You might think Queue.Queue.get(True, 0) would return an event immediately
if present and otherwise return. It doesn't, it immediately "times out"
and returns with nothing from the queue.
The behaviour we want is not to wait but return anything present which is
what .get(False) does so map to this.
This fixes some odd behaviour observed in some of the tinfoil selftests.
(Bitbake rev: 412bfab8721ea317898a1974f6a7a0d0bea763df)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The datastore can assume internal API, this just removes the function indirection
overhead involved in this very common codepath (800,000 calls in parsing OE-Core).
(Bitbake rev: 9a36531ed2b2881a65e5d39ee4b68d2bb392ed78)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently builds in multiple TMPDIRs with multiconfig can break
since the BuildStarted event is used to create directory strutures in several
cases (e.g. buildstats.bbclass) and there is only on BuildStarted event
generated in a multiconfig build.
We have two options, a) to add a new MultiConfigBuildStarted event which is
generated once per multiconfig, or b) allow multiple BuildStarted events.
Having reviewed the code and current users of BuildStarted, sending one event
per multiconfig seems like its the best way forward and the existing code looks
able to cope with the duplication of events. I did also check toaster and I think
that can handle this issue too (multiconfig builds may have other issues there).
I'm therefore proposing we send multiple BuildStarted events for multiconfig
and for consistency, send multiple BuildCompleted events too.
We need to ensure that BUILDNAME, BUILDSTART and DATE/TIME are set consistently
in all the different multiconfig datastores. These events can write to the
datastore so copies are not used. buildFile was also cleaned up to ensure it
uses the right datastore in various places.
(Bitbake rev: 0b00f0382780ab5390a5c3f756a9b4efafe0aec8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some cases where the metadata needs to be aware a multiconfig build
is happening and have access to the multiconfig data stores to merge data into
the common build. This adds such an event allowing access to these datastores.
(Bitbake rev: 160e47f5df90850e64dcb857c81a5039abc9235f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
'mirrortarball' is supposed to be a local variable to the function.
(Bitbake rev: a457cbfb1f20a47db3978290921d0708cd96bd70)
Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
