As the logic in bitbake improves, the logic in the tests needs to as well.
Afer we built a task for the first time, allow its setscene hash verification
status to change, mirroring what would happen in a multiconfig build.
(Bitbake rev: 27ec2e69ab3e32972caf8b072b2945736696d83d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if a multiconfig build contains different configurations which
have overlapping sstate artefacts, it will build them multiple times.
This is clearly suboptimal and not what users want/expect.
This adds code to detect this and stall all but one of the setscne tasks
so that once its built, it can be found by the other tasks.
We take care to iterate the multiconfigs in order so try and avoid
dependency loops. We also match on PN+taskname+taskhash since this is
what we know sstate in OE-Core would use. There are some tasks even within
a multiconfig which match hashes (mostly do_populate_lic tasks) but those
have a much higher chance of circular dependency so aren't work attempting
to optimise.
If a deadlock does occur the build will be slower but there is code to
unbreak such a deadlock so it hopefully doens't break anything.
Comments are injected into the test tasks so they have different task
hashes and a new test for this optimisation is added.
(Bitbake rev: a75c5fd6d4ec56836de0be2fe679c81297a080ad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pull the common pieces of the hash verification code into a single function
and reduce code duplication.
(Bitbake rev: d0c39e05cef841c6f29cc6c919df6cbf271a9bda)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix some unwanted extra indentation.
(Bitbake rev: 460a5c2e3e1d72f2da16fbc96832fadc82e72c52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was left from when task IDs complicated the code, simplify.
(Bitbake rev: ae36b5c693bb9f13c88199e78e3c31616852eafb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the merge of the scenequeue with real tasks, this now confuses the
statistics. The real tasks are the definitive progress so monitor only
those.
(Bitbake rev: 20956b508a082224139c8f56b68299edff6e0443)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need some tests for runqueue, its been something which has been hard to test
for a long time. Add some dummy metadata to allow this, mirroring the OE
structure in spirit.
(Bitbake rev: 37564d7440c5d7aa05ec537f3b79026b1c83bb68)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This combines the scqenequeue and normal task execution into one function
and simplifies the state engine accordingly.
This is the final set of cleanup to fully merge things without adding the
extra noise to the previous commits.
(Bitbake rev: 56f3396d8c7cfbebd175877c9d773e4e35f8dea1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since there is now parallel execution of setscene and normal tasks, the way
setscenewhitelist handling worked can't function the way it did. Paul and I
never liked its error output anyway.
This code tries a different approach, checking the task at execution time
but printing the uncovered task list.
This code may need improvement after real world usage but can
work with the new task flows.
(Bitbake rev: a08d8ba5f5194a09391b1904ee31c04c5f0b1e28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is the serious functionality change in this runqueue patch series of
changes.
Rather than two phases of execution, the scenequeue setscene phase, followed
by normal task exeuction, this change allows them to execute in parallel
together.
To do this we need to handle marking of tasks as covered/uncovered in a piecemeal
fashion on a task by task basis rather than in a single function.
The code will block normal task exeuction until any setcene task which could
cover that task is executed and its status is known. There is a slight
optimisation which could be possible here at the risk of races but that
doesn't seem worthwhile.
The state engine isn't entirely cleaned up in this commit (see FIXME) and
the setscenewhitelist functionality is broken by it (see following patches)
however its good enough to test with normal workflows.
(Bitbake rev: 58b3f0847cc2d47e76f74d59dcbbf78fe41b118b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It wasn't clear whether the variable contained just setscene covered
tasks or all covered tasks. We need both sets of data so lets just have
two clearly named variables.
(Bitbake rev: a9fb55627762e7c8b3df30b335ad0b2f1adc080e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code for setting up buildable tasks can be simplified.
(Bitbake rev: ce3cd2df5b034f8dbdcf9834e8b9a393b6b01aad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its now clear a variable is pointless, remove it and tweak the logic
so the data structure of the existing variable matches what we need.
(Bitbake rev: c257c7b93b86dd794d31307e820215301c7ccf3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Simplfy some looping code which no longer has any purpose.
(Bitbake rev: 01dfc37095e5c661f275917d22aa1c1ad7f24d8d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some extra comments to build_scenequeue_data() and fix the debug code
so it actually works.
(Bitbake rev: 8ea6d8193fc89b4596da69e400fbc50e5a443f9f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing code to compute the 'unskippable' setscene task list is overcomlicated,
so replace it with something functionally equivalent but simpler and more efficient.
We don't need to process all chains, just the 'top' ones to the first setscene tasks.
This also makes the code more readable.
(Bitbake rev: 06982c82f10cbdbea0b601e5cf0450a2a99c14c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to have a list of all the tasks a given setscene task covers
and we can easily generate this data whilst doing other data processing.
This is used in later changes to runqueue rather than trying to compute it
on the fly which is difficult.
(Bitbake rev: 63ddc2fec40bd1b456702b97091f9dc5ef70a941)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst this class has existed for years, it doesn't have any
users and has a questionable interface. Drop it to allow for further
simplification and changes.
(Bitbake rev: 3ab51764f7965d696bb2c5a872bf161473df4289)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also move the scheduler init over, apart for the builtable tasks part which need
to remain called later.
(Bitbake rev: ad30a16cd30f9eab0224eb271f98f9a24516b621)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cleanup to the _execute_runqueue logic to reduce indentation, drop the
dummy executor class concept and prepare for further changes.
(Bitbake rev: 726e3c61a69fef16e605ba9b911a17cd99f1a2c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace the remains of the Tasks and Scenequeue Tasks classes with simple
function calls. Also drop the dummy version of the execution class to
simplify further changes as its not needed.
(Bitbake rev: 33805394310046cd58c2194f6d063b3946811014)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some tasks were not being marked as covered/notcovered since internal
calls were being made without using the external call points.
Fix the accounting issues by using the correct external call points.
(Bitbake rev: fe0a7be03e8baed22f6b0915cd5f7956ba3fbf83)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Merge the unique functions from the Tasks and Scenequeue Tasks classes
into the common base class.
(Bitbake rev: 7539fe22bc831bb835901e3aca77985ab4ebc4c7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a seperate stats class for scenequeue tasks and move the setup
into the base class. Update references accordingly.
(Bitbake rev: 32f39bbd5d3b7394689da9ba05be2c15b4523b27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In preparation for merging the setscene and normal task execution,
uniquely namespace the scenequeue specific functions.
For the one shared function, add the "sq_live" variable so we know
which functions to send the results to.
(Bitbake rev: 2cbe9399902ba67dca566c7344b2247412cf4d5c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For ease of refactoring, move this code to its own separate function
until it becomes clear what we should do with it.
(Bitbake rev: 4b96b204f986dd62fba485876b7208665c14268d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The function is only used by setscene code so the parameter is pointless,
remove it.
(Bitbake rev: b52dbf5e9cb327f8434213d286ad333f5dbad1d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the bulk of the scenequeue data generation to its own function
allowing for refactoring of the code.
Create the start of an object to represent this data.
(Bitbake rev: 68326e0426f25a1bbfd5ae3aa278656a3744053e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the removal of the setcene verify code, this additional code block
is also now unneeded since tasks can't be forced at this point in the code
any move. This effectively reverts f21910157d873c030b149c4cdc5b57c5062ab5a6.
(Bitbake rev: 4514fe4f045d595cc9b938f9326f66f2b3e99f71)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Nothing in OE-Core uses this and hasn't since 2017. It wasn't needed by core
metadata since the switch to recipe specific sysroots.
Since this function would be hard to implement with the planned changes to
runqueue, drop it which allows simplification and further code cleanup.
(Bitbake rev: 5deaa5df730a8a846f3192b4a639b7a2a72c1b71)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Work off a copy of the 'buildable' class variable, allowing easier
future code changes.
(Bitbake rev: e851169acfebba404514135bf512e6f045739a13)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that runqueue optimises based on task hash, we need to ensure
tasks have unique hashes even in the simplest siggen mode. Use
the task name to calculate a unique hash.
This fixes runqueue tests when hash optimisations are added.
(Bitbake rev: 8ede873ef4ef492fbaf01474685c1ca8b34d80d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the unihash changes the default signature handler didn't work. Tweak it
to adapt to those changes (allowing the runqueue tests to work).
(Bitbake rev: 7d486d3fb7176a3486f3f2484457724d7185df58)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The provides and rprovides had been set for skipped recipes, packages are
similar to them (all of them provide something), so also set it. This makes it
easier to figure out the RDEPENDS issues, for example, lmsensors
(lmsensors_3.5.0.bb) RRECOMMENDS lmsensors-config-fancontrol
(lmsensors-config_1.0.bb), but lmsensors-config is skipped for some reasons,
then if we run:
$ bitbake lmsensors
ERROR: Nothing RPROVIDES 'lmsensors-config-fancontrol' (but /path/to/lmsensors_3.5.0.bb RDEPENDS on or otherwise requires it)
NOTE: Runtime target 'lmsensors-config-fancontrol' is unbuildable, removing...
Missing or unbuildable dependency chain was: ['lmsensors-config-fancontrol']
ERROR: Required build target 'lmsensors' has no buildable providers.
Missing or unbuildable dependency chain was: ['lmsensors', 'lmsensors-config-fancontrol']
We had no way to know who rprovides lmsensors-config-fancontrol, we can figure
it out by bitbake/contrib/dump_cache.py after this patch.
(Bitbake rev: 9cf7a5e5a28e676427970a821893e9d930973969)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whether skip or not, they are always set, so move the lines ahead to avoid
duplicated lines.
(Bitbake rev: c1a8ebb8f83e5108b667f291c924fc2fbd2ac769)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an option to skip _setscene only if they would normally be
executed, without ignoring sstate completely.
Previously, '--no-setscene' would allow a build that completely ignored
sstate and _setscene tasks, and '--setscene-only' would allow a build
that only ran _setscene tasks, but there was no option do a build that
would respect tasks previously restored from sstate and build everything
else. Now one can run:
bitbake --setscene-only IMAGE; bitbake --skip-setscene IMAGE
which is functionally equivalent to:
bitbake IMAGE
The indented use is to allow a build to complete successfully in the
presence of _setscene task failures by splitting apart the two phases
e.g.:
(bitbake -k --setscene-only IMAGE || true) && bitbake --skip-setscene IMAGE
(Bitbake rev: 813ba5b7c13b573a0b813b628a819bdbf0627540)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the server shuts down 'cleanly' due to some issue, the socket will close.
A recently reported example was an invalid PRSERV being set. Doing this silently
and without changing the retries count will case the server startup to loop
infinitely.
Change the code so it triggers the usual retries note messages and times
out eventually pointing the user at the cooker log file.
[YOCTO #12984]
(Bitbake rev: bb696636ef0c59f9e9640bb9460e7cce323cc785)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the UI hasn't started, no messages are printed to the console
until the server starts. This is confusing, particularly if the server
never starts. Flush the UI queue through the simply handler upon connection
retry so the user sees the messages they're supposed to be seeing.
Also point the user at the logfile for hints about why this may be.
(Bitbake rev: 4b9ab675cebb427ab8ad0c56c7b37eed50a2a39e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
test_git_submodule_dbus_broker, test_git_submodule_CLI11, test_git_submodule_update_CLI11,
test_git_submodule_aktualizr and test_git_submodule_iotedge try to access the network via
git clone, which fails when there is no network available. Add the relevant skip tag.
(Bitbake rev: 9b0538753da0514e6518723dac537007abf7a649)
Signed-off-by: Luca Boccassi <luca.boccassi@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* this is just another test case for issue already fixed in:
commit fef56d28c3efec4876c379898cbc4d4c65303aee
Author: Alexander Kanavin <alex.kanavin@gmail.com>
Date: Sun Feb 24 21:07:28 2019 +0100
Subject: bitbake: fix version comparison when one of the versions ends in .
* The TypeError is triggered not by '.' at the end, but from the extra
numberic component in one of the versions.
* When one version has fewer elements, it's extended by another (0, None)
element where 0 means numeric component. Then the result cannot be
decided by comparing the types (oa < ob, ob > oa) and it continues
to compare values (ca < cb) which fails when one of them is the None
from (0, None) appended before.
======================================================================
ERROR: test_vercmpstring (bb.tests.utils.VerCmpString)
----------------------------------------------------------------------
Traceback (most recent call last):
File "lib/bb/tests/utils.py", line 32, in test_vercmpstring
result = bb.utils.vercmp_string('1a', '1a1')
File "lib/bb/utils.py", line 131, in vercmp_string
return vercmp(ta, tb)
File "lib/bb/utils.py", line 123, in vercmp
r = vercmp_part(va, vb)
File "lib/bb/utils.py", line 112, in vercmp_part
elif ca < cb:
TypeError: '<' not supported between instances of 'NoneType' and 'int'
----------------------------------------------------------------------
Ran 3 tests in 0.002s
(Bitbake rev: 9767fffe3115a1f1afa3c6a2b39720fefb8dc4d5)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Check existence of all the bblayer direcotories at once and print them
all, so if there are multiple nonexistent directories, user does not
have to correct bblayers.conf and restart bitbake multiple times.
[YOCTO #11647]
(Bitbake rev: 19291f7c4d17086ebb6a7b80c3cb06333d7fd55b)
Signed-off-by: Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have no BBMULTICONFIG set but set mcdepends, they're currently
ignored. We can handle them correctly with this small tweak.
(Bitbake rev: 578f0c02f6a13f4315e7c2ce8b5e876dd2025055)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A separate patch to base.bbclass (in poky) will add the OE_EXTRA_IMPORTS
variable. The contents are appended into OE_IMPORTS. This provides a
mechanism by which layers (in their layer.conf) can make custom progress
handlers available.
As a backup, individual recipes can inject progress handlers into
__builtins__.
Custom handlers are expected to have this __init__ signature:
def __init__(self, d, outfile=None, otherargs=None):
Recipes can then use the handlers like this:
do_task[progress] = "custom:mylayer.util.ProgressHandler[:args]"
The last part (everything after and including the second colon) is
optional. If provided, it is passed to HandlerClass's __init__ as
otherargs="args". Otherwise, otherargs=None.
(Bitbake rev: 20289d62c84c393990dd3deb0cca1b17c09092e6)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It seems context management support was half-implemented, but never
finished. For example, LogTee has __enter__ and __exit__ but they
haven't been exercised until now.
(Bitbake rev: bf522ad3e0c52cdb69b406226840d870ff4f2766)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
