As the logic in bitbake improves, the logic in the tests needs to as well.
Afer we built a task for the first time, allow its setscene hash verification
status to change, mirroring what would happen in a multiconfig build.
(Bitbake rev: 27ec2e69ab3e32972caf8b072b2945736696d83d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if a multiconfig build contains different configurations which
have overlapping sstate artefacts, it will build them multiple times.
This is clearly suboptimal and not what users want/expect.
This adds code to detect this and stall all but one of the setscne tasks
so that once its built, it can be found by the other tasks.
We take care to iterate the multiconfigs in order so try and avoid
dependency loops. We also match on PN+taskname+taskhash since this is
what we know sstate in OE-Core would use. There are some tasks even within
a multiconfig which match hashes (mostly do_populate_lic tasks) but those
have a much higher chance of circular dependency so aren't work attempting
to optimise.
If a deadlock does occur the build will be slower but there is code to
unbreak such a deadlock so it hopefully doens't break anything.
Comments are injected into the test tasks so they have different task
hashes and a new test for this optimisation is added.
(Bitbake rev: a75c5fd6d4ec56836de0be2fe679c81297a080ad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pull the common pieces of the hash verification code into a single function
and reduce code duplication.
(Bitbake rev: d0c39e05cef841c6f29cc6c919df6cbf271a9bda)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix some unwanted extra indentation.
(Bitbake rev: 460a5c2e3e1d72f2da16fbc96832fadc82e72c52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was left from when task IDs complicated the code, simplify.
(Bitbake rev: ae36b5c693bb9f13c88199e78e3c31616852eafb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Briefly:
Brazil no longer observes DST.
'zic -b slim' outputs smaller TZif files; please try it out.
Palestine's 2019 spring-forward transition was on 03-29, not 03-30.
Changes to future timestamps
Brazil has canceled DST and will stay on standard time indefinitely.
(Thanks to Steffen Thorsen, Marcus Diniz, and Daniel Soares de
Oliveira.)
Predictions for Morocco now go through 2087 instead of 2037, to
work around a problem on newlib when using TZif files output by
zic 2019a or earlier. (Problem reported by David Gauchard.)
Changes to past and future timestamps
Palestine's 2019 spring transition was 03-29 at 00:00, not 03-30
at 01:00. (Thanks to Sharef Mustafa and Even Scharning.) Guess
future transitions to be March's last Friday at 00:00.
Changes to past timestamps
Hong Kong's 1941-06-15 spring-forward transition was at 03:00, not
03:30. Its 1945 transition from JST to HKT was on 11-18 at 02:00,
not 09-15 at 00:00. In 1946 its spring-forward transition was on
04-21 at 00:00, not the previous day at 03:30. From 1946 through
1952 its fall-back transitions occurred at 04:30, not at 03:30.
In 1947 its fall-back transition was on 11-30, not 12-30.
(Thanks to P Chan.)
Changes to past time zone abbreviations
Italy's 1866 transition to Rome Mean Time was on December 12, not
September 22. This affects only the time zone abbreviation for
Europe/Rome between those dates. (Thanks to Stephen Trainor and
Luigi Rosa.)
Changes affecting metadata only
Add info about the Crimea situation in zone1970.tab and zone.tab.
(Problem reported by Serhii Demediuk.)
Changes to code
zic's new -b option supports a way to control data bloat and to
test for year-2038 bugs in software that reads TZif files.
'zic -b fat' and 'zic -b slim' generate larger and smaller output;
for example, changing from fat to slim shrinks the Europe/London
file from 3648 to 1599 bytes, saving about 56%. Fat and slim
files represent the same set of timestamps and use the same TZif
format as documented in tzfile(5) and in Internet RFC 8536.
Fat format attempts to work around bugs or incompatibilities in
older software, notably software that mishandles 64-bit TZif data
or uses obsolete TZ strings like "EET-2EEST" that lack DST rules.
Slim format is more efficient and does not work around 64-bit bugs
or obsolete TZ strings. Currently zic defaults to fat format
unless you compile with -DZIC_BLOAT_DEFAULT=\"slim\"; this
out-of-the-box default is intended to change in future releases
as the buggy software often mishandles timestamps anyway.
zic no longer treats a set of rules ending in 2037 specially.
Previously, zic assumed that such a ruleset meant that future
timestamps could not be predicted, and therefore omitted a
POSIX-like TZ string in the TZif output. The old behavior is no
longer needed for current tzdata, and caused problems with newlib
when used with older tzdata (reported by David Gauchard).
zic no longer generates some artifact transitions. For example,
Europe/London no longer has a no-op transition in January 1996.
Changes to build procedure
tzdata.zi now assumes zic 2017c or later. This shrinks tzdata.zi
by a percent or so.
Changes to documentation and commentary
The Makefile now documents the POSIXRULES macro as being obsolete,
and similarly, zic's -p POSIXRULES option is now documented as
being obsolete. Although the POSIXRULES feature still exists and
works as before, in practice it is rarely used for its intended
purpose, and it does not work either in the default reference
implementation (for timestamps after 2037) or in common
implementations such as GNU/Linux (for contemporary timestamps).
Since POSIXRULES was designed primarily as a temporary transition
facility for System V platforms that died off decades ago, it is
being decommissioned rather than institutionalized.
New info on Bonin Islands and Marcus (thanks to Wakaba and Phake
Nick).
(From OE-Core rev: bbbb985808e5c301cdb7fdb1ff677706e99b4785)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Due to improved validation of sysctl settings in recent kernels (5.2+, but
also stable kernels like 4.19.53), systemd will log an error message like
systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument
during boot. Backport the bugfix from the systemd master.
(From OE-Core rev: 289a29fc687d1224097f553a6f8c7f1baf903cae)
Signed-off-by: Matthias Schiffer <matthias.schiffer@ew.tq-group.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure that we don't create an image test data symlink named
".testdata.json" when IMAGE_LINK_NAME is empty.
(From OE-Core rev: 97e1af51814c63963dc6eee003e0cf0e4dead024)
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure that we don't create a symlink named ".manifest" if IMAGE_LINK_NAME
is empty.
(From OE-Core rev: 267697f5e1c931e39fd81dd1b14691e364be64f3)
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The options in ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS} are already passed
via ${CC}/${CXX} and there is no reason to pass them a second time. Thus
we can remove MESON_TOOLCHAIN_ARGS. And when it is removed, the other
MESON_*_ARGS variables revert to the standard CFLAGS, CXXFLAGS and
LDFLAGS, so just use them directly instead.
Apart from the obvious improvement with not passing a lot of options
twice, this also solves a problem where -pie would be passed on the
command line in a way that it would prevent building any dynamic
libraries using meson if using a toolchain that is not built with
--enable-default-pie and if security_flags.inc is used.
(From OE-Core rev: 300f4ac59d4b96fc25a40565b22441b51ab08ede)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The patching to make the new libtool work (from 2008) is no longer needed.
(From OE-Core rev: 4210fafa851d011023f5a58ed3887148168f861c)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
OE no longer ships a git snapshot of e2fsprogs,
so use the new upstream default now.
(From OE-Core rev: f5edce401cfb31ebd0200adaba9a201caf7ea705)
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are several packages which all provide /usr/lib/sendmail
when lsb is enabled. So use alternative to manage it.
(From OE-Core rev: 0a433802a9b7a41ab6732bc1915571b7a190985f)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before commit[80d17497b7 mdadm: improve the run-ptest],
the mdadm ptest just run some tests without
"--keep-going" option. After the option added in
commit 80d17497b7, all test cases have chance to
be called.
But the logic in mdadm upstream commit
[e2a8e9d tests: wait for complete rebuild in integrity checks]
will make the test enter infinite loop especially in
qemu env as commit e2a8e9d update the logic from
"check wait" to "check state 'U*'" for testcase
tests/01r5integ and tests/01raid6integ. Considering all
other cases still use "check wait" logic, so revert e2a8e9d
to make tests/01r5integ and tests/01raid6integ also use
"check wait" logic to avoid the infinite loop.
[YOCTO #13368]
(From OE-Core rev: b507793f4adde0c60588b06520d7d0b70acfd82a)
Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Debian 10 is the new stable release and is being tested on the autobuilder, so
add this to the supported distribution list.
[ YOCTO #13432 ]
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the merge of the scenequeue with real tasks, this now confuses the
statistics. The real tasks are the definitive progress so monitor only
those.
(Bitbake rev: 20956b508a082224139c8f56b68299edff6e0443)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need some tests for runqueue, its been something which has been hard to test
for a long time. Add some dummy metadata to allow this, mirroring the OE
structure in spirit.
(Bitbake rev: 37564d7440c5d7aa05ec537f3b79026b1c83bb68)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This combines the scqenequeue and normal task execution into one function
and simplifies the state engine accordingly.
This is the final set of cleanup to fully merge things without adding the
extra noise to the previous commits.
(Bitbake rev: 56f3396d8c7cfbebd175877c9d773e4e35f8dea1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since there is now parallel execution of setscene and normal tasks, the way
setscenewhitelist handling worked can't function the way it did. Paul and I
never liked its error output anyway.
This code tries a different approach, checking the task at execution time
but printing the uncovered task list.
This code may need improvement after real world usage but can
work with the new task flows.
(Bitbake rev: a08d8ba5f5194a09391b1904ee31c04c5f0b1e28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is the serious functionality change in this runqueue patch series of
changes.
Rather than two phases of execution, the scenequeue setscene phase, followed
by normal task exeuction, this change allows them to execute in parallel
together.
To do this we need to handle marking of tasks as covered/uncovered in a piecemeal
fashion on a task by task basis rather than in a single function.
The code will block normal task exeuction until any setcene task which could
cover that task is executed and its status is known. There is a slight
optimisation which could be possible here at the risk of races but that
doesn't seem worthwhile.
The state engine isn't entirely cleaned up in this commit (see FIXME) and
the setscenewhitelist functionality is broken by it (see following patches)
however its good enough to test with normal workflows.
(Bitbake rev: 58b3f0847cc2d47e76f74d59dcbbf78fe41b118b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It wasn't clear whether the variable contained just setscene covered
tasks or all covered tasks. We need both sets of data so lets just have
two clearly named variables.
(Bitbake rev: a9fb55627762e7c8b3df30b335ad0b2f1adc080e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code for setting up buildable tasks can be simplified.
(Bitbake rev: ce3cd2df5b034f8dbdcf9834e8b9a393b6b01aad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its now clear a variable is pointless, remove it and tweak the logic
so the data structure of the existing variable matches what we need.
(Bitbake rev: c257c7b93b86dd794d31307e820215301c7ccf3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Simplfy some looping code which no longer has any purpose.
(Bitbake rev: 01dfc37095e5c661f275917d22aa1c1ad7f24d8d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some extra comments to build_scenequeue_data() and fix the debug code
so it actually works.
(Bitbake rev: 8ea6d8193fc89b4596da69e400fbc50e5a443f9f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing code to compute the 'unskippable' setscene task list is overcomlicated,
so replace it with something functionally equivalent but simpler and more efficient.
We don't need to process all chains, just the 'top' ones to the first setscene tasks.
This also makes the code more readable.
(Bitbake rev: 06982c82f10cbdbea0b601e5cf0450a2a99c14c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to have a list of all the tasks a given setscene task covers
and we can easily generate this data whilst doing other data processing.
This is used in later changes to runqueue rather than trying to compute it
on the fly which is difficult.
(Bitbake rev: 63ddc2fec40bd1b456702b97091f9dc5ef70a941)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst this class has existed for years, it doesn't have any
users and has a questionable interface. Drop it to allow for further
simplification and changes.
(Bitbake rev: 3ab51764f7965d696bb2c5a872bf161473df4289)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also move the scheduler init over, apart for the builtable tasks part which need
to remain called later.
(Bitbake rev: ad30a16cd30f9eab0224eb271f98f9a24516b621)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cleanup to the _execute_runqueue logic to reduce indentation, drop the
dummy executor class concept and prepare for further changes.
(Bitbake rev: 726e3c61a69fef16e605ba9b911a17cd99f1a2c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace the remains of the Tasks and Scenequeue Tasks classes with simple
function calls. Also drop the dummy version of the execution class to
simplify further changes as its not needed.
(Bitbake rev: 33805394310046cd58c2194f6d063b3946811014)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some tasks were not being marked as covered/notcovered since internal
calls were being made without using the external call points.
Fix the accounting issues by using the correct external call points.
(Bitbake rev: fe0a7be03e8baed22f6b0915cd5f7956ba3fbf83)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Merge the unique functions from the Tasks and Scenequeue Tasks classes
into the common base class.
(Bitbake rev: 7539fe22bc831bb835901e3aca77985ab4ebc4c7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a seperate stats class for scenequeue tasks and move the setup
into the base class. Update references accordingly.
(Bitbake rev: 32f39bbd5d3b7394689da9ba05be2c15b4523b27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In preparation for merging the setscene and normal task execution,
uniquely namespace the scenequeue specific functions.
For the one shared function, add the "sq_live" variable so we know
which functions to send the results to.
(Bitbake rev: 2cbe9399902ba67dca566c7344b2247412cf4d5c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For ease of refactoring, move this code to its own separate function
until it becomes clear what we should do with it.
(Bitbake rev: 4b96b204f986dd62fba485876b7208665c14268d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The function is only used by setscene code so the parameter is pointless,
remove it.
(Bitbake rev: b52dbf5e9cb327f8434213d286ad333f5dbad1d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the bulk of the scenequeue data generation to its own function
allowing for refactoring of the code.
Create the start of an object to represent this data.
(Bitbake rev: 68326e0426f25a1bbfd5ae3aa278656a3744053e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the removal of the setcene verify code, this additional code block
is also now unneeded since tasks can't be forced at this point in the code
any move. This effectively reverts f21910157d873c030b149c4cdc5b57c5062ab5a6.
(Bitbake rev: 4514fe4f045d595cc9b938f9326f66f2b3e99f71)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Nothing in OE-Core uses this and hasn't since 2017. It wasn't needed by core
metadata since the switch to recipe specific sysroots.
Since this function would be hard to implement with the planned changes to
runqueue, drop it which allows simplification and further code cleanup.
(Bitbake rev: 5deaa5df730a8a846f3192b4a639b7a2a72c1b71)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Work off a copy of the 'buildable' class variable, allowing easier
future code changes.
(Bitbake rev: e851169acfebba404514135bf512e6f045739a13)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that runqueue optimises based on task hash, we need to ensure
tasks have unique hashes even in the simplest siggen mode. Use
the task name to calculate a unique hash.
This fixes runqueue tests when hash optimisations are added.
(Bitbake rev: 8ede873ef4ef492fbaf01474685c1ca8b34d80d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the unihash changes the default signature handler didn't work. Tweak it
to adapt to those changes (allowing the runqueue tests to work).
(Bitbake rev: 7d486d3fb7176a3486f3f2484457724d7185df58)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The provides and rprovides had been set for skipped recipes, packages are
similar to them (all of them provide something), so also set it. This makes it
easier to figure out the RDEPENDS issues, for example, lmsensors
(lmsensors_3.5.0.bb) RRECOMMENDS lmsensors-config-fancontrol
(lmsensors-config_1.0.bb), but lmsensors-config is skipped for some reasons,
then if we run:
$ bitbake lmsensors
ERROR: Nothing RPROVIDES 'lmsensors-config-fancontrol' (but /path/to/lmsensors_3.5.0.bb RDEPENDS on or otherwise requires it)
NOTE: Runtime target 'lmsensors-config-fancontrol' is unbuildable, removing...
Missing or unbuildable dependency chain was: ['lmsensors-config-fancontrol']
ERROR: Required build target 'lmsensors' has no buildable providers.
Missing or unbuildable dependency chain was: ['lmsensors', 'lmsensors-config-fancontrol']
We had no way to know who rprovides lmsensors-config-fancontrol, we can figure
it out by bitbake/contrib/dump_cache.py after this patch.
(Bitbake rev: 9cf7a5e5a28e676427970a821893e9d930973969)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
