Removes the keys() function from the MulticonfigCache. This appears to
be a leftover from before the class inherited from collections.Mapping,
is now unnecessary, and was outright incorrect.
(Bitbake rev: 5f37b6d2829fcac1f16602d9697f8bfbcb65ff62)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Globbing in file:// urls is terminally broken. Currently when its used, the
file checksum code is basically bypassed. This means changes to the source
files don't change the task checksum, the task doesn't rebuild when the
inputs change and things generally break.
To make globbing work generically, we'd have to scan the file system for
all possible matches to the glob and log whether they exist or not. We can't
simply log the files which exist, we have to also know which files could
later exist and influence the choice of file so we know when to reparse.
For a simple file://xxx/*, this could be done but for bigger patterns,
it becomes much more problemtic. We already support file://xxx/ in urls.
So, lets decide we'll not support globs in file://urls. Worse case users
can put files in a directory and reference that, moving files into place
if needed.
Remove all the glob special cases (see the comments if anyone doesn't
believe this is terminally broken) and error to the user if they have
such urls.
(Bitbake rev: 0c9302d950c6f37bfcc4256b41001d63f668bdf7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The layout of the cache data hasn't changed, but the cache has now been
split into different files for multiconfig. If a user pulls in these
changes, it's possible that their base cache will still contain the
combined multiconfig cache entries, which are now unexpected and
generate errors like:
Unexpected multiconfig: foo
Bumping the version fixes this since the old cache data won't be
considered valid anymore.
(Bitbake rev: 1082188ce633ec6891c961d726f584b3f1259941)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The virtualfn variable is not defined, the filename is what should be
shown instead.
(Bitbake rev: 1f9d2c21db3a1ad2ab13dfebd2f8e9a7c3682ee2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is expected that load_cachfile() returns an integer indicating how
many entries were loaded from the cache. In the event the cache needs to
be rebuilt, 0 must be returned to prevent python from attempting to add
an None and an integer together.
(Bitbake rev: 3459d98fbc280637ecb36961bda8436818ee51e5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that there is a cache object per multiconfig, it is not necessary
for each cache object to parse all other multiconfigs. Instead, each
cache now only parses the files for it's multiconfig.
(Bitbake rev: 3c5c7346adf4ca7ec761c08738f12401ba75b7c8)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improves the logging of Cache objects by prefixing the log messages with
the multiconfig name of the cache, so as to distinguish between multiple
instances of the class. Also adds a more log messages.
(Bitbake rev: 74fd10b33c66f4142d6eff6531200f7620a06ae0)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Splits the parsing cache to maintain one cache per multiconfig instead
of one global cache. This is necessary now that the files and appends
can vary for each multiconfig. A bb.cache.MulticonfigCache
dictionary-like proxy object is created instead of a single
bb.cache.Cache object. This object will create and properly initialize
bb.cache.Cache object for each multiconfig, and each of these caches has
a dedicated cache file with a name based on the multiconfig.
(Bitbake rev: 5272f2489586479880ae8d046dfcdbe0963ee5bb)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Splits the cooker to track a collection per multiconfig instead of a
single collection for all multiconfigs. Practically speaking, this
allows each multiconfigs to each have different BBMASKs that apply to it
instead of each one using the mask specified in the base configuration.
(Bitbake rev: dd6d8eca2027f8d9be8a734a493227b440075e49)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some companies are using large numbers of patch files in SRC_URI.
Rightly or wrongly that exposes a performance problem where the code
does not handle the large string manipulations in a way which works
efficienty in python.
This is a modified version of a patch from z00539568
<zhangyifan46@huawei.com153340508@qq.com which addresses the performance
problem. I modified it to use a more advanced regex, retain the "*" check
and cache the regex.
[YOCTO #13824]
(Bitbake rev: c07f374998903359ed55f263c86466d05aa39b68)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
removed unused imports which made the code harder to read, and slightly
but less efficient
(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These messages spam the logs for no good reason, they were useful for debugging
a particular problem long ago but are distracting noise now. Disable them.
(Bitbake rev: 1a9247c468cf09da60e5d396ccb81e950841c99e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently siggen uses the format "<filename>.<taskname>" for referencing tasks
whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the
separator everywhere.
This is an API breaking change since the cache is affected, as are siginfo files
and any custom signature handlers such as those in OE-Core.
Ultimately this will let us clean up and the accessor functions from runqueue,
removing all the ".rsplit(".", 1)[0]" type code currently all over the place.
Once a standard is used everwhere we can update the code over time to be more
optimal.
(Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds a simple version of the MultiProcessCache which can be used to
save and load cache data, useful for a new usecase we have in
sigdata/runqueue.
(Bitbake rev: 19a6e35600ae6d2d1bcecca6e68ab8c37674774e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So that people or other tools can easily know which one is being used, just
like what we did for run.do_task and log.do_task, otherwise, we have no way
to know it. I usually use "ls -t", but it isn't reliable since the one which
is being used may not the latest one.
(Bitbake rev: cf286dff653eed542bf347ca46234c224944d5b0)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The provides and rprovides had been set for skipped recipes, packages are
similar to them (all of them provide something), so also set it. This makes it
easier to figure out the RDEPENDS issues, for example, lmsensors
(lmsensors_3.5.0.bb) RRECOMMENDS lmsensors-config-fancontrol
(lmsensors-config_1.0.bb), but lmsensors-config is skipped for some reasons,
then if we run:
$ bitbake lmsensors
ERROR: Nothing RPROVIDES 'lmsensors-config-fancontrol' (but /path/to/lmsensors_3.5.0.bb RDEPENDS on or otherwise requires it)
NOTE: Runtime target 'lmsensors-config-fancontrol' is unbuildable, removing...
Missing or unbuildable dependency chain was: ['lmsensors-config-fancontrol']
ERROR: Required build target 'lmsensors' has no buildable providers.
Missing or unbuildable dependency chain was: ['lmsensors', 'lmsensors-config-fancontrol']
We had no way to know who rprovides lmsensors-config-fancontrol, we can figure
it out by bitbake/contrib/dump_cache.py after this patch.
(Bitbake rev: 9cf7a5e5a28e676427970a821893e9d930973969)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whether skip or not, they are always set, so move the lines ahead to avoid
duplicated lines.
(Bitbake rev: c1a8ebb8f83e5108b667f291c924fc2fbd2ac769)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After real world use its clear the "multiconfig:" prefix to multiconfig tasks,
whilst clear, is also clumbersome. Switch to use the short version instead.
mcdepends will continue to work with "multiconfig:" for now as well. The commandline
will only accept mc: going forward.
[YOCTO #11168]
(Bitbake rev: 821daf093b76504067a8b77dfa4b181af6ec92b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.
(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.
(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.
The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.
The COPYING file is replaced with LICENSE.X files which contain the full
license texts.
(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove a deprecated warning and stop using our own deprecated API!
(Bitbake rev: 83ece2c6f4b000e906fec9148f25bd1dff66cfb0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Print message when cachefile is found/not can help debug.
* Update "Using cache in" to "Cache dir:" since it was the same as the debug
message of "codeparser & file checksum caches", which caused confusion. And
whether the cache file will be used or not is still unknown at that time, so
just print the cache dir.
(Bitbake rev: c8d3a2016f432e8ed9e99d9c28850149ab6fd6d8)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cache code currently inserts PN into the package list if it isn't already
present. Whilst this ensures that the package list contains something which is
important for native recipes that don't set PACKAGES, it causes confusing
behaviour where a normal recipe doesn't have PN in PACKAGES: for example adding
dhcp to IMAGE_INSTALL will parse successfully but fail at rootfs time as the
dhcp recipe doesn't generate a dhcp package.
Solve this by only adding PN to the cache's package list if the package list is
empty. This results in the package list for recipes such as DHCP being correct,
but native recipes continue to have just PN in the list as before.
[ YOCTO #5533 ]
(Bitbake rev: df31a88786ce5bd7708ff14e1379dc2a58a8c0cf)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure we handle bb.utils.contains_any() as separate items, rather than
how we handle contains() where every item must be in the list.
Additionally, enable handling bb.utils.filter() which for the purposes
of looking at dependencies is the same as contains_any().
Additionally bump the codeparser cache and recipe cache versions to
invalidate the user's existing caches (ensuring that the changes take
effect and avoiding "taskhash mismatch" errors respectively).
(Bitbake rev: 496e3c84820a2a889d99d3604659e47a550941d5)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.
Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)
(Bitbake rev: c19baa8c19ea8ab9b9b64fd30298d8764c6fd2cd)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To accommodate the OpenEmbedded layer index recipe parsing, we have to
have the ability to pass in a custom config datastore since it
constructs a synthetic one. To make this possible after the multi-config
changes, rename the internal _load_bbfile() function to parse_recipe(),
make it a function at the module level (since it doesn't actually need
to access any members of the class or instance) and move setting
__BBMULTICONFIG inside it since other code will expect that to be set.
Part of the fix for [YOCTO #10192].
(Bitbake rev: 5b3fedfe0822dd7effa4b6d5e96eaf42669a71df)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the notion of supporting multiple configurations within
a single build. To enable it, set a line in local.conf like:
BBMULTICONFIG = "configA configB configC"
This would tell bitbake that before it parses the base configuration,
it should load conf/configA.conf and so on for each different
configuration. These would contain lines like:
MACHINE = "A"
or other variables which can be set which can be built in the same
build directory (or change TMPDIR not to conflict).
One downside I've already discovered is that if we want to inherit this
file right at the start of parsing, the only place you can put the
configurations is in "cwd", since BBPATH isn't constructed until the
layers are parsed and therefore using it as a preconf file isn't
possible unless its located there.
Execution of these targets takes the form "bitbake
multiconfig:configA:core-image-minimal core-image-sato" so similar to
our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND.
Implementation wise, the implication is that instead of tasks being
uniquely referenced with "recipename/fn:task" it now needs to be
"configuration:recipename:task".
We already started using "virtual" filenames for recipes when we
implemented BBCLASSEXTEND and this patch adds a new prefix to
these, "multiconfig:<configname>:" and hence avoid changes to a large
part of the codebase thanks to this. databuilder has an internal array
of data stores and uses the right one depending on the supplied virtual
filename.
That trick allows us to use the existing parsing code including the
multithreading mostly unchanged as well as most of the cache code.
For recipecache, we end up with a dict of these accessed by
multiconfig (mc). taskdata and runqueue can only cope with one recipecache
so for taskdata, we pass in each recipecache and have it compute the result
and end up with an array of taskdatas. We can only have one runqueue so there
extensive changes there.
This initial implementation has some drawbacks:
a) There are no inter-multi-configuration dependencies as yet
b) There are no sstate optimisations. This means if the build uses the
same object twice in say two different TMPDIRs, it will either load from
an existing sstate cache at the start or build it twice. We can then in
due course look at ways in which it would only build it once and then
reuse it. This will likely need significant changes to the way sstate
currently works to make that possible.
(Bitbake rev: 5287991691578825c847bac2368e9b51c0ede3f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than passing in a datastore to build on top of, use the data builder
object in the cache and base the parsed recipe from this. This turns
things into proper objects building from one another rather than messy
mixes of static and class functions.
This sets things up so we can support parsing and building multiple
configurations.
(Bitbake rev: fef18b445c0cb6b266cd939b9c78d7cbce38663f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some cases we want to parse recipes without any cache
setup or involvement. Split out the standalone functions into
a NoCache variant which the Cache is based upon, setting the scene
for further cleanup and restructuring.
(Bitbake rev: 120b64ea6a0c0ecae7af0fd15d989934fa4f1c36)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather that the current mix of static and class methods, refactor
so that the cache has the databuilder object internally. This becomes
useful for the following patches for multi config support.
It effectively completes some of the object oriented work we've been
working towards in the bitbake core for a while.
(Bitbake rev: 7da062956bf40c1b9ac1aaee222a13f40bba9b19)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Needing to access these static methods through a class doesn't
make sense. Move these to become module level standalone functions.
(Bitbake rev: 6d06e93c6a2204af6d2cf747a4610bd0eeb9f202)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Simple refactoring to allow for multiconfig support.
(Bitbake rev: 266b848da40904446eb1d084bbdc5307a9b45197)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some reason the data written in this way is coming back out the
files out of order. I've not been able to simplify the test case to a
point where this was standalone reproducible. Simplify the code and
write out the cache files sequentially since this seems to avoid the
errors and makes the code more readable.
(Bitbake rev: 14ec47f5f0566dbd280fae8a03160c8500ad3929)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen cache corruption where the pairs come out in a different
order to the way we saved them for unknown reasons. Add better sanity
checking to give a more user friendly error rather than a crash/traceback.
Also allows the system to reparse and recover.
(Bitbake rev: 4be4a15491530bd6dc018033ad3d4b2562ab6e23)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since we no longer have random data like version fields in these structures
and we can assume any extra cache data subclasses our class, simplify the
code.
This is mostly reindenting after removal of the pointless type checks.
(Bitbake rev: 5eb36278ac9975de1945f6da8161187320d90ba7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Firstly, don't store the versions fields in memory in the cache objects
data store. This just complicates the code for no good reason.
Secondly, write the version fields to all cache files, not just the
core one. This makes everything consistent and easier.
(Bitbake rev: cb666262b2f986b5d9331dfb30458ef1a151fa4d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an "extras" cache file is corrupted, the system would not notice
and later fail with errors about missing entries. Add a test for this
which means we can fall back to re-parsing in those cases.
[YOCTO #9902]
(Bitbake rev: 51843d8f2bbe2e54db7593ca61984abe70423ef6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Otherwise you can look at the log and wonder why parsing isn't happening
when it really is due to other code paths.
(Bitbake rev: b48d95677a4d285a77cda2892179965f7f8f06dd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.
(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some tweaks to use python 3 syntax in a python 2 compatible way.
(Bitbake rev: 322949c77dbaa4db01b5a43d85b39a2af67ba7b2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes it possible to prevent a recipe to be cached, and thus,
parsed every time.
Use with care.
[YOCTO #8853]
(Bitbake rev: 78335c1fbe5266116700c2413aac28b00423a75b)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If no cache file name is given a default from class variable is used,
like before.
(Bitbake rev: 2602a312818f564961de7dfa63c429d45ff9e5ac)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
