The provided packages by a skipped recipe are supposed to be listed in
SkippedPackage.rprovides, which is used when generating a meaningful
error message when a build fails because of a skipped package.
Previously this variable only contained the contents of ${RPROVIDES}.
However, most recipes don't define RPROVIDES, they define
RPROVIDES_<pkg> for each package they provide. Additionally, the recipe
provides the packages in PACKAGES without them being included in
${RPROVIDES}.
Before this change, having a runtime dependency on a skipped non-recipe
package would result in a build error stating that the build failed
because the package was skipped, but without providing any reason for
why it was skipped.
(Bitbake rev: efd026c26a377b826a49b945a8212bf7de8a480a)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Recent changes mean data might not be setup. If its not, avoid tracebacks.
(Bitbake rev: 3daff610d9f39d73c80c54d1df46f573666e20db)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Blocks SIGINT in the worker processes to prevent them from running the
parent process signal handler, which causes them to deadlock under
certain circumstances.
[YOCTO #14034]
(Bitbake rev: 9f4207f4b598f549cbd4159841c720276736f23b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the recent init changes, if a client disconnects before issuing a
command, the cooker can break in the reset handlers. Add some guards
in the code to prevent this.
(Bitbake rev: 12605e30e4c4e1ae6a67c97363b892ebf0b9566c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The original code saved BBFILES back to BBFILES without any changes which isn't
usefule, so remove that line. Now save prioritized BBFILES to
BBFILES_PRIORITIZED which can accelerate the query a lot for the one which
relies on it such as bb.utils.get_file_layer().
(Bitbake rev: 49bdb5dfa57b41b3ed399961e947c404f9195998)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Otherwise this can interfer with multiprocessing exit handling.
(Bitbake rev: b88816c4c84fa4f5ad39c263f5e75b96476e9768)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you make parsing fail (e.g. add something like:
X := "${@d.getVar('MCMACHINES').split()[1]}"
to meson.bbclass, then run "while true; do bitbake -g bash; done"
it will eventually hang. It appears the cancel_join_thread() call the
parsing failure triggers, breaks the results_queue badly enough that it
sits in read() indefintely (called from self.result_queue.get(timeout=0.25)).
The timeout only applies to lock aquisition, not the read call.
I've tried various other approaches such as using cancel_join_thread()
in other places but the only way things don't lock up is to avoid
cancel_join_thread() entirely for results_queue.
I do have a concern that this may adversely affect Ctrl+C handling
but equally, its broken now already and this appears to improve
things.
[YOCTO #14034]
(Bitbake rev: 9c61a1cc7be46c23da1f4ef3bee070fb83c4be57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During cooker shutdown, its possible the parser isn't cleaned up. Fix
this (which may partially explain why threads were left hanging around
at exit).
(Bitbake rev: 928609f30f3a20aaa2f88afc18044a4e10199488)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This could potentialy account for some of the missing thread cleanup
we're seeing.
(Bitbake rev: 8f2d690428de8934868b406b79c4699a8ebe902c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Calling sys.exit() in the middle of the code is rather antisocial. We catch
this in various places but we shouldn't have to. In all these cases we have
already sent events explaining to the user what happened. This means the
correct exception is BBHandledException.
The recent startup changes have moved the point a lot of this code gets
called to inside the UI, with memres it would have always been possible
from there anyway. This change makes things much more consistent.
(Bitbake rev: 91699f366d24480ff3b19faec78fb9f3181b3e14)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Only the env variables which were added to the datastore were being
updated. We need to update the whole copy, we just only trigger a
reparse if any of the variables making it into the datastore change.
This avoids a bug where variables such as DISPLAY in a new UI context
would break under memory resident bitbake.
(Bitbake rev: 4bb71b627767297269e762b414443e15e28bfac4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changes to configuration.env were not updating BB_ORIGENV, fix this.
(Bitbake rev: c5fbd8452f87e0a2d234eaf27d0450aacdeb8891)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The first thing the UIs do is update the server config from the UI. We
can just rely upon that and start the server with a standard config,
removing the need to pass the confusing configuration object around
as well as configParams, which contains a similar copy of some of the
data.
This makes memory resident bitbake work the same way as the normal
mode, removing the opportunity for some class of bugs.
The xmlrpcinterface and server_timeout values are passed in at server
startup time now and there no longer a second option in the
configuration which is effective ignored once the server starts.
(Bitbake rev: 783a03330802e83c525c55522e3ee2a933bded3a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we end up parsing the base configuration multiple times as
initially, the right settings haven't come from the UI. We can defer
this until later in startup using runCommand as a trigger.
The advantage to doing this is improved startup times and ultimately
we should be able to avoid the double parse of the base configuration.
(Bitbake rev: 3caa43b665604475d2c87ba505efb0b9fca9c2e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the eventlog is not handled correctly for memory resident
bitbake. Fix this by allowing adpations to configuration changes.
(Bitbake rev: f7d2c9116116659ea42260a3bb96dca100aadae7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than passing debug/verbose/debug_domains around, pass the
computed output of these. Ensure that the cooker sets the levels
to match the levels currently set in the UI and generally try and
make it easier to understand what the code is doing.
(Bitbake rev: f0535beecc692a6213be2a22f9eef5956450ecf8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The levels of indirection to set these verbose logging options is rather
crazy. This attempts to turn things into two specific options with
much more specific meanings. For now its all still controlled by the
commandline verbose option and should funciton as previously, with
the addition that the BB_VERBOSE_LOGS option can now be task specific.
(Bitbake rev: 423c046f2173aaff3072dc3d0882d01b8a0b0212)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hongxu Jia reported a problem where the bb_cache files were not always being
written out correctly. This was due to the sync thread being terminated
prematurely.
Whilst the preceeding changes mean the exit handler for this thread is now
correctly called since we switch to using sys.exit() instead of os._exit(),
this write can happen after we drop the bitbake lock, leading to potential
races. Avoid that headache by adding in explicit thread join() calls before
we drop the lock (which atexit or Finalize can't do).
(Bitbake rev: afd1900939f7b042297558f4cb01f50f3a299267)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Each run through the parser would leak a thread from the queue created to
shut the parser down. Close this down correctly and clean up the code flow
slightly whilst in the area, making sure this thread does shut down correctly
(we don't care if it loses data).
(Bitbake rev: 51ba35e9bbd4da8b5f3b3b5b8213cb634a6b0f06)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having the idle callbacks abstracted via the configuration object
makes no sense. Its like this for historical reasons from the
multiple server backends but we don't need this now so simplfy.
(Bitbake rev: e56c49717355c9493b07d5fc80981a95ad8a0ec8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"bitbake mc:arm:bash mc:arm:busybox"
works but
"bitbake multiconfig:arm:bash multiconfig:arm:busybox"
does not. The reason is the list is modified whilst iterating.
Don't do that.
[YOCTO #13607]
(Bitbake rev: cd041a78d96e656438d93fb1e288080b8a6fe8bd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids a traceback if an invalid multiconfig is referenced in the bitbake
commandline and tweaks the message to make it more understanable.
(Bitbake rev: f31d7d0ad57b0ecc2ae06ed4b547c98df2aaa1a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if all recipes in a layer are skipped, there are warnings that the
BBFILE_PATTERN_ entry didn't match anything. We probably shouldn't do this
for skipped recipes.
The current code is hard to understand, not least as it passes variables
which functions modify by reference rather than giving a return value.
Update calc_bbfile_priority() to return values rather than modifying them.
Refactor the code to try and make it clearer what its doing and fix the
skipped recipe issue by passing in the list of parsed files.
The function is complicated by the need to not rerun regex matching more
than we ever have to which complicates the flow, it would be easier if we
just reran operations multiple times.
(Bitbake rev: 969cb27b4d978551817612ff4558bec81cfb655c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that there is a cache object per multiconfig, it is not necessary
for each cache object to parse all other multiconfigs. Instead, each
cache now only parses the files for it's multiconfig.
(Bitbake rev: 3c5c7346adf4ca7ec761c08738f12401ba75b7c8)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Splits the parsing cache to maintain one cache per multiconfig instead
of one global cache. This is necessary now that the files and appends
can vary for each multiconfig. A bb.cache.MulticonfigCache
dictionary-like proxy object is created instead of a single
bb.cache.Cache object. This object will create and properly initialize
bb.cache.Cache object for each multiconfig, and each of these caches has
a dedicated cache file with a name based on the multiconfig.
(Bitbake rev: 5272f2489586479880ae8d046dfcdbe0963ee5bb)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Splits the cooker to track a collection per multiconfig instead of a
single collection for all multiconfigs. Practically speaking, this
allows each multiconfigs to each have different BBMASKs that apply to it
instead of each one using the mask specified in the base configuration.
(Bitbake rev: dd6d8eca2027f8d9be8a734a493227b440075e49)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cooker had a multiconfig parameter for the findProviders() and
findBestProviders() API, but it was being ignored.
(Bitbake rev: ea0b68ac2b77676ed1c63f0ee1ae5d300f2b4696)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If for example a tinfoil connection edits the datastore, a subsequent
connection can be "corrupted" by those changes. By setting the parse
status of the caches as False at exit, the behaviour becomes the same
as a newly setup server as a new data store is setup.
This avoids problems in tests when BB_SERVER_TIMEOUT is set as the
server is properly reset between connections.
[YOCTO #13812]
(Bitbake rev: e66759106e21da2b34a6cdec7aa681ad2204da54)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing, reset the loghandler when finished, else the messages
can be misleading.
(Bitbake rev: 7af80cd1dd577b05d39a3cc5d5c547a2549e39df)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing recipes its apparent the memory usage of bitbake rises linearly
with number of recipes parsed. It shouldn't.
Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the
dependency information left behind in siggen was the culprit. Add a new
method to allow us to drop this information. We don't need it after the recipe
has been parsed and hashes calculated (at runtime its different but only the
currently executing task would be in memory).
This should give signficant memory usage improvements for bitbake and that
in turn should help speed on more constrained systems, as well as when used in
multiconfig environments.
(Bitbake rev: 5d98d8e39bba42f458532b1eef3619f2321d8a2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removed the deprecated methods as it will only cause problems later on,
and since warn() just calls warning(), it shouldn't change anything
(Bitbake rev: a194f275235f22411cb2368f06a44f61ceb6a0f3)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
amend the code to handle singleton comparisons properly so it only checks
if they only refer to the same object or not, and not bother
comparing the values.
(Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
removed unused imports which made the code harder to read, and slightly
but less efficient
(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When there are many watched files, keeping track of them using lists
is suboptimal. Using sets improves the performance considerably.
(Bitbake rev: 1e96df260e47d160dbd36bfc92c31ef06266f662)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This should have been removed together with expanded_data in commit
e3694e73 (cooker/command: Drop expanded_data).
(Bitbake rev: 33197db8abf82be240d7c1c5c9d2484a08a90849)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The hash server process is terminated and waited on with join(), so it
should not be a daemon. Daemonizing it cause races with the server
cleanup, especially in the selftest because the process may not have
terminated and cleanup up its socket before the test cleanup runs and
tries to do it.
[YOCTO #13542]
(Bitbake rev: 7c829675581818f92d57056b57fbd3880829b6bd)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reworks the hash equivalence server to address performance issues that
were encountered with the REST mechanism used previously, particularly
during the heavy request load encountered during signature generation.
Notable changes are:
1) The server protocol is no longer HTTP based. Instead, it uses a
simpler JSON over a streaming protocol link. This protocol has much
lower overhead than HTTP since it eliminates the HTTP headers.
2) The hash equivalence server can either bind to a TCP port, or a Unix
domain socket. Unix domain sockets are more efficient for local
communication, and so are preferred if the user enables hash
equivalence only for the local build. The arguments to the
'bitbake-hashserve' command have been updated accordingly.
3) The value to which BB_HASHSERVE should be set to enable a local hash
equivalence server is changed to "auto" instead of "localhost:0". The
latter didn't make sense when the local server was using a Unix
domain socket.
4) Clients are expected to keep a persistent connection to the server
instead of creating a new connection each time a request is made for
optimal performance.
5) Most of the client logic has been moved to the hashserve module in
bitbake. This makes it easier to share the client code.
6) A new bitbake command has been added called 'bitbake-hashclient'.
This command can be used to query a hash equivalence server, including
fetching the statistics and running a performance stress test.
7) The table indexes in the SQLite database have been updated to
optimize hash lookups. This change is backward compatible, as the
database will delete the old indexes first if they exist.
8) The server has been reworked to use python async to maximize
performance with persistently connected clients. This requires Python
3.5 or later.
(Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 94c0c7f15c7a6244a8576ed948ffc21afb96ba82.
This ignores the layer priority, making the issue much worse.
E.g. I'm seeing a lot of failures caused by missing users, because
base-passwd bbappends applied in unexpected order caused different
passwd.master to be found in re-ordered FILESPATH.
(Bitbake rev: 2dc862237dba82da37c8ac9289e0a21409b1305c)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Thanks to wildcards in bbappend filenames, it's possible to have
multiple bbappends that apply to the same recipe in the same directory.
In order to get sstate hits between different workspaces, we want to
apply those bbappend files in a consistent order. Since readdir()
returns files in a non-deterministic order between workspaces (based on
inode number and/or time of creation), we'll need to sort its result in
order to have any consistency.
(Bitbake rev: 94c0c7f15c7a6244a8576ed948ffc21afb96ba82)
Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The information of recipe-depends.dot is misleading.
e.g.
$ grep xz recipe-depends.dot | grep bzip2
"bzip2" -> "xz"
"xz" -> "bzip2"
Users would wonder why they get some circular dependency.
The information is derived from removing the task names
of task-depends.dot. It's not giving people any additonal
information, and it's misleading.
So we remove the generation of this file.
(Bitbake rev: 4c484cc01e3eee7ab2ab0359fd680b4dbd31dc30)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At exit the hashserv code was causing tracebacks as join() wasn't
being called from the thread that started the process. Ensure that
the hashserver is started from the pre_serve hook which is the
final thread the cooker runs in. This avoids the traceback at the
expense of some horrific poking into data stores which will ultimately
need improving through a proper API.
(Bitbake rev: 05888700e5f6cba48a26c8a4c447634a28e3baa6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
$ rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p
Press *one* Ctrl-C when the parsing process is at about 50%, then the processes
are not exited:
Keyboard Interrupt, closing down...
Timeout while waiting for a reply from the bitbake server
It hangs at process.join(), according to:
https://docs.python.org/3.7/library/multiprocessing.html
Cleanup the queue before call process.join() can fix the problem.
(Bitbake rev: 3eddfadd19b2ce4c061861abf0c340e3825b41ff)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were hard to debug lockups when trying to use threading to start
hashserv as a thread. Switch to multiprocessing which doesn't show the
same locking problems.
(Bitbake rev: be23d887c8e244f1ef961298fbc9214d0fd0968a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful, particularly in the local developer model of usage, for
bitbake to start and stop a hash equivalence server on local port,
rather than relying on one being started by the user before the build.
The new BB_HASHSERVE variable supports this.
The database handling is moved internally into the hashserv code so that
different threads/processes can be used for the server without errors.
(Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have no BBMULTICONFIG set but set mcdepends, they're currently
ignored. We can handle them correctly with this small tweak.
(Bitbake rev: 578f0c02f6a13f4315e7c2ce8b5e876dd2025055)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows "multiconfig:" targets to continue to work by internally
mapping them to the new "mc:" naming, allowing older builds to work
as before.
(Bitbake rev: c4d90890547af642e99cc541af3415df3559563e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After real world use its clear the "multiconfig:" prefix to multiconfig tasks,
whilst clear, is also clumbersome. Switch to use the short version instead.
mcdepends will continue to work with "multiconfig:" for now as well. The commandline
will only accept mc: going forward.
[YOCTO #11168]
(Bitbake rev: 821daf093b76504067a8b77dfa4b181af6ec92b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support to the 'bitbake -e' command so that it can display the base
environment for a multiconfig. It was previously possible to get the
base environment for the main environment by running "bitbake -e", but
there was no support for getting the base environment for a multiconfig
without specifying a recipe. A user can now print the base environment
for the multiconfig "foo" by running:
$ bitbake -e multiconfig:foo
(Bitbake rev: 3d657af8a6120193d45d01968605b30075a56198)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
