A minor performance optmisation to keep lists smaller when running large
builds. We can do this since once a task has been built, we don't need
to worry about it. This improves a major bottleneck that shows up on
performance profile charts in dryruns.
(Bitbake rev: cd6b89230823707c3c9bb9e6883bf5a971916581)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some runqueue code paths where the unihash cache would not be
saved where for example only parsing or an occurred. Save the cache at the
end of runqueue generation to ensure entries are cached.
(Bitbake rev: 9eee0d36870c11dd303894a6151c33a83bd3a1bc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to set the setscene tasklist before we call into the
taskhash/unihash code else the behaviour is inconsistent.
Avoid reporting hashes for non setscene tasks since we'd never
query that.
(Bitbake rev: 419a7840b8627278db694029c25df00214d01d96)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently runqueue will rerun setscene tasks multiple times as hashes
change. This has caused numerous problems since a setscene task may
become "unavailable" for some future signature combination and the code
then can't easily "unskip" tasks its already passed into the execution
queue.
At least for now, only run setscene once and assume they're equivalent
at that point. In practise that has been much more stable in testing.
Tweak the test to match the change in behaviour.
(Bitbake rev: 4205a3ef23834f317642bba155d67cd772176fb6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tasks were not migrating consistently, particularly:
* if a task was rehashed which had already run
* if a task which was valid became invalid due to a rehash
We need to always run the migration code for rehashed tasks and then
reprocess them for hash validity. This means rearranging the code.
It also means several tests are no longer correct and can't be written
correctly to work on all possible workflows so those are removed.
(Bitbake rev: 8443989ee41e9b162972935513e437b5c66ea74d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
According to sstate_checkhashes which is defined in sstate.bbclass, the
currentcoun should be a number (0, not None).
Fixed:
$ bitbake base-files -Sprintdiff
> bb.plain("Sstate summary: Wanted %d Found %d Missed %d Current %d (%d%% match, %d%% complete)" % (total, len(found), len(missed), currentcount, match, complete))
TypeError: %d format: a number is required, not NoneType
(Bitbake rev: 45cb73e2846eaffe8964a573875f54808e8f3633)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reworks the hash equivalence server to address performance issues that
were encountered with the REST mechanism used previously, particularly
during the heavy request load encountered during signature generation.
Notable changes are:
1) The server protocol is no longer HTTP based. Instead, it uses a
simpler JSON over a streaming protocol link. This protocol has much
lower overhead than HTTP since it eliminates the HTTP headers.
2) The hash equivalence server can either bind to a TCP port, or a Unix
domain socket. Unix domain sockets are more efficient for local
communication, and so are preferred if the user enables hash
equivalence only for the local build. The arguments to the
'bitbake-hashserve' command have been updated accordingly.
3) The value to which BB_HASHSERVE should be set to enable a local hash
equivalence server is changed to "auto" instead of "localhost:0". The
latter didn't make sense when the local server was using a Unix
domain socket.
4) Clients are expected to keep a persistent connection to the server
instead of creating a new connection each time a request is made for
optimal performance.
5) Most of the client logic has been moved to the hashserve module in
bitbake. This makes it easier to share the client code.
6) A new bitbake command has been added called 'bitbake-hashclient'.
This command can be used to query a hash equivalence server, including
fetching the statistics and running a performance stress test.
7) The table indexes in the SQLite database have been updated to
optimize hash lookups. This change is backward compatible, as the
database will delete the old indexes first if they exist.
8) The server has been reworked to use python async to maximize
performance with persistently connected clients. This requires Python
3.5 or later.
(Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We only have hash equivalence for setscene tasks so only query the server
for those, reducing the number of connections needed.
(Bitbake rev: 22082c7b3ca0cffcedb7d1d8c6681d35286376db)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Specifying the force flag will now cause runall and runonly to
invalidate the tasks before running them. This allows a --runall or
--runonly to force the tasks to run, even if they would have otherwise
been skipped, e.g.:
bitbake -f --runall fetch
Will run all do_fetch tasks even if they wouldn't be necessary (for
example, skipped by setscene)
(Bitbake rev: 71e52d3822016027106f2a2e74b8dfdf20f5dc1e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than repeatedly calling mc_from_tid() do this in the parent,
removing around a million function calls. Takes time spent in this
function from 40s to 36s.
(Bitbake rev: 28b3f0d8867804799420689c314ac4a8f01efb8c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are other data structures which can be reprocessed at the same
time as holdoff_tasks, further improving build efficiency in various
places.
(Bitbake rev: 02090b3456b7a2de12e72dfeaabfd3b631609924)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't need to process the holdoff task list until we're executing tasks
which saves some data manipulation, at the cost of some data structures
not being correct at all times. This saves significant amounts of time
in various profile charts of larger builds.
(Bitbake rev: 270f076111b12eab358417b0c4cf9c70d7cc787a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've observed do_package and do_package_setscene running in parallel. The
reason is that holdoff_tasks wasn't getting updated. Looking at the code, it
would seem the reason is that the task was in pending_migrations and hence
changed wasn't set and holdoff_tasks wasn't updated.
Fix this. It only affects builds with rehashing enabled.
(Bitbake rev: e26e61e84575669bd223f6ab316798097ed95ec8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake <target> --runonly=fetch
failed as the target_tids list included entries which were no longer targeted
task ids. Fix this.
(Bitbake rev: 94e848ae6544e628a19cb97115279b0b1678967c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This doesn't appear to have ill effects right now but there is a correctness
issue which this so fix it.
(Bitbake rev: a5e084a266f63c2fd370122327615e49beaeb94e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was overwriting data in the parent which caused all kinds of
odd/weird failures.
(Bitbake rev: 4c5aeb424247a9d0c907524ffacd9c61fcdc0852)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the task hashes change we need to ensure the stampcache is cleared out
else tasks don't rerun when they should as we're basing decisions on stale
cache data.
(Bitbake rev: 08962092d3bb7887d82f97d442a6103c0677eae7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We weren't marking this special case of setscene task as buildable leading to
runqueue task failures.
(Bitbake rev: 930efbc563443d82df8d692bb8ff172ca2bae192)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Further testing shows we should test some extra datastructures to help pinpoint logic
errors more precisely. This adds some further data structure sanity checks.
(Bitbake rev: 83c4370b25c3a14cc946965c5c5f83ea28f488a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This debug statement could result in a long list of tasks which when
repeatedly sent over our IPC, slowed down the builds immensely. Remove
it in favour of other more targeted debugging added recently, bringing
back some lost performance, particularly on builds with large numbers
of tasks.
(Bitbake rev: 85fe627fdb6510f0942917964386fad9d8c479c8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The event from the task notifiing of hash equivalency should only be processed
when the task completes. This can otherwise result in a race where a dependent
task may run before the original task completes causing various failures.
To make this work reliably, the code had to be restructured quite a bit.
(Bitbake rev: 1bf5be46f92f125193638cf41ff207d68f592259)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The changed_setscene variable here is just odd and not needed. Worse,
it could prevent some tasks from being removed from the holdoff tasks
list. The list is being rebuilt and should work as intended just from
the other data, this is a leftover from previous versions of the code
as far as I can tell.
(Bitbake rev: 030b9f2b3ce6ed40e79304eb0ffee6c6613f43be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst we had good runqueue failure mode debug, it hadn't adapted to the
scenequeue changes. Run the scenequeue sanity tests at the end of
a build and output the results regardless of whether all setscene tasks
completed or not. This *massively* improves the ability to debug runqueue
problems.
(Bitbake rev: b9b2177473c0b95a23bd519a201e1d2ba101c6c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Looking at the profile information, a lot of time is being spent in
next_buildable_task. This is probably due to the generator expressions
not working well with the empty test.
The easiest way to improve things is to switch to using set manipulations.
We also don't need to update self.buildable the way the original code did
as we don't rely on that anywhere.
(Bitbake rev: 3bcf9ad4964b7e42d1a02ce231e9db42a81ead2a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If tasks are in the covered list of tasks for a given setscene task,
it needs to wait for those to complete before we can start.
(Bitbake rev: fdee640c26750b852eb68f5c80437377aa300ed8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous tasks_covered and tasks_notcovered were basically unstable
data structures. We couldn't always tell whether tasks should be covered
or not when trying to repair the sturcture if sstate tasks reran.
In the end its simpler to throw the lists away and rebuild them based upon
current data rather than trying to patch it adhoc. This turns out to be
simpler and much more reliable and I've much more confidence in this code.
(Bitbake rev: 52ee2ba2c617d928569f5afa404925c8b6f317bc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to copy this set, not modify the original else all kinds
of weird and bad things break, mostly from circular references.
We'll not go into how much sleep I lost tracking down the fallout
from this.
(Bitbake rev: 49927546d2b306830c98f6f9da4a6ad828f6a3a6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently siggen uses the format "<filename>.<taskname>" for referencing tasks
whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the
separator everywhere.
This is an API breaking change since the cache is affected, as are siginfo files
and any custom signature handlers such as those in OE-Core.
Ultimately this will let us clean up and the accessor functions from runqueue,
removing all the ".rsplit(".", 1)[0]" type code currently all over the place.
Once a standard is used everwhere we can update the code over time to be more
optimal.
(Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function uses an old API which uses offsets into lists as a communication
mechanism. Update the API to use "tid" which is used universally in runqueue now.
We can also add kwargs support to the funciton definition to drop some of the
backwards compaiblility hoops we had to jump though with different function
argument combinations.
(Bitbake rev: dc23550047e5078da491ce9a6f30989cb5260df6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful, particularly in the local developer model of usage, for
bitbake to start and stop a hash equivalence server on local port,
rather than relying on one being started by the user before the build.
The new BB_HASHSERVE variable supports this.
The database handling is moved internally into the hashserv code so that
different threads/processes can be used for the server without errors.
(Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst this isn't strictly necessary, its helpful if the log output is
consistent and its also helpful if bugs either appear or don't appear
for a specific configuration. Ensuring the various iterations we make
are deterministic (sorted) helps with this.
(Bitbake rev: 6a901bb904a97ca90d88be2c6901d3d32346282f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a compelling usecase for tasks being able to notify runqueue
that their "unihash" has changed. When this is recieved, the hashes of
all subsequent tasks should be recomputed and their new hashes checked
against existing setscene validity. Any newly available setscene tasks
should then be executed.
Making this work effectively needs several pieces. An event is added
which the cooker listen for. If a new hash becomes available it can
send an event to notify of this.
When such an event is seen, hash recomputations are made. A setscene
task can't be run until all the tasks it "covers" are stopped. The
notion of "holdoff" tasks is therefore added, these are removed from
the buildable list with the assumption that some setscene task will
run and cover them.
The workers need to be notified when taskhashes change to update their
own internal siggen data stores. A new worker command is added to do this
which will affect all newly spawned worker processes from that worker.
An example workflow which tests this code is:
Configuration:
BB_SIGNATURE_HANDLER = "OEEquivHash"
SSTATE_HASHEQUIV_SERVER = "http://localhost:8686"
$ bitbake-hashserv &
$ bitbake automake-native
$ bitbake autoconf-native automake-native -c clean
$ bitbake m4-native -c install -f
$ bitbake automake-native
with the test being whether automake-native is installed from sstate.
(Bitbake rev: 1f630fdf0260db08541d3ca9f25f852931c19905)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to preserve unihash task hashes between runs. Use the new SimpleCache
class to create such a class within the signature generator which is loaded
at init time and saved when builds complete. The default is unpopulated but
metadata sig handlers can populate this cache.
(Bitbake rev: 1f326f2c29c2664a5daaeeb0c1fd332630efbdba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than a special copy of the data structure which we change, compute
the logic using set operations from other data we have. This means
we can add tasks back into the scenequeue without having to worry about
reversing operations on this variable with all the potential bugs that
might involve.
(Bitbake rev: b707d0cbc25fa336a1e95ff588f1ea37eee063eb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you specify both setscene and non-setscene tasks on the commandline, the
non-setscene tasks could be missed, e.g. "bitbake X:do_patch X:do_populate_sysroot"
and do_patch would fail to run.
Fix the problem in runqueue and add a testcase.
(Bitbake rev: 75292fdec5d9c0b5b3c554c4b7474a63656f7e12)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if a multiconfig build contains different configurations which
have overlapping sstate artefacts, it will build them multiple times.
This is clearly suboptimal and not what users want/expect.
This adds code to detect this and stall all but one of the setscne tasks
so that once its built, it can be found by the other tasks.
We take care to iterate the multiconfigs in order so try and avoid
dependency loops. We also match on PN+taskname+taskhash since this is
what we know sstate in OE-Core would use. There are some tasks even within
a multiconfig which match hashes (mostly do_populate_lic tasks) but those
have a much higher chance of circular dependency so aren't work attempting
to optimise.
If a deadlock does occur the build will be slower but there is code to
unbreak such a deadlock so it hopefully doens't break anything.
Comments are injected into the test tasks so they have different task
hashes and a new test for this optimisation is added.
(Bitbake rev: a75c5fd6d4ec56836de0be2fe679c81297a080ad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pull the common pieces of the hash verification code into a single function
and reduce code duplication.
(Bitbake rev: d0c39e05cef841c6f29cc6c919df6cbf271a9bda)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix some unwanted extra indentation.
(Bitbake rev: 460a5c2e3e1d72f2da16fbc96832fadc82e72c52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was left from when task IDs complicated the code, simplify.
(Bitbake rev: ae36b5c693bb9f13c88199e78e3c31616852eafb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This combines the scqenequeue and normal task execution into one function
and simplifies the state engine accordingly.
This is the final set of cleanup to fully merge things without adding the
extra noise to the previous commits.
(Bitbake rev: 56f3396d8c7cfbebd175877c9d773e4e35f8dea1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since there is now parallel execution of setscene and normal tasks, the way
setscenewhitelist handling worked can't function the way it did. Paul and I
never liked its error output anyway.
This code tries a different approach, checking the task at execution time
but printing the uncovered task list.
This code may need improvement after real world usage but can
work with the new task flows.
(Bitbake rev: a08d8ba5f5194a09391b1904ee31c04c5f0b1e28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is the serious functionality change in this runqueue patch series of
changes.
Rather than two phases of execution, the scenequeue setscene phase, followed
by normal task exeuction, this change allows them to execute in parallel
together.
To do this we need to handle marking of tasks as covered/uncovered in a piecemeal
fashion on a task by task basis rather than in a single function.
The code will block normal task exeuction until any setcene task which could
cover that task is executed and its status is known. There is a slight
optimisation which could be possible here at the risk of races but that
doesn't seem worthwhile.
The state engine isn't entirely cleaned up in this commit (see FIXME) and
the setscenewhitelist functionality is broken by it (see following patches)
however its good enough to test with normal workflows.
(Bitbake rev: 58b3f0847cc2d47e76f74d59dcbbf78fe41b118b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It wasn't clear whether the variable contained just setscene covered
tasks or all covered tasks. We need both sets of data so lets just have
two clearly named variables.
(Bitbake rev: a9fb55627762e7c8b3df30b335ad0b2f1adc080e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code for setting up buildable tasks can be simplified.
(Bitbake rev: ce3cd2df5b034f8dbdcf9834e8b9a393b6b01aad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its now clear a variable is pointless, remove it and tweak the logic
so the data structure of the existing variable matches what we need.
(Bitbake rev: c257c7b93b86dd794d31307e820215301c7ccf3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Simplfy some looping code which no longer has any purpose.
(Bitbake rev: 01dfc37095e5c661f275917d22aa1c1ad7f24d8d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some extra comments to build_scenequeue_data() and fix the debug code
so it actually works.
(Bitbake rev: 8ea6d8193fc89b4596da69e400fbc50e5a443f9f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing code to compute the 'unskippable' setscene task list is overcomlicated,
so replace it with something functionally equivalent but simpler and more efficient.
We don't need to process all chains, just the 'top' ones to the first setscene tasks.
This also makes the code more readable.
(Bitbake rev: 06982c82f10cbdbea0b601e5cf0450a2a99c14c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to have a list of all the tasks a given setscene task covers
and we can easily generate this data whilst doing other data processing.
This is used in later changes to runqueue rather than trying to compute it
on the fly which is difficult.
(Bitbake rev: 63ddc2fec40bd1b456702b97091f9dc5ef70a941)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst this class has existed for years, it doesn't have any
users and has a questionable interface. Drop it to allow for further
simplification and changes.
(Bitbake rev: 3ab51764f7965d696bb2c5a872bf161473df4289)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
