process.signal_received is a list for signum and not iterable,
change a suitable method to handle sig.
(Bitbake rev: bfc53b190bd2530c2bfcea0690127d7eff620f45)
Signed-off-by: Yang Xu <yang.xu@mediatek.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runtaskdep data in siginfo files was written out with full paths to
the bb files, matching bitbake's internal "unique key" ID for recipes/tasks.
When originally implemented this made sense.
Over time, the main use for the data in siginfo files has become to match
against other siginfo files to debug changes of hash calcuations. The
recipename data is not useful for this as the siginfo filenames use PN
instead which can often be derived from the recipe filename but not always.
It is time to throw away the 'tid' data format and switch over the use a
hybrid PN form which includes the multiconfig. That can be easily stripped
off in the find_siginfo code in oe-core.
The other purpose of having a sortable dependency ID is retained and the
multiconfig needs to be included to allow the taskhashes to be processed
and calculated correctly. PN is meant to be unique between recipes, only
one would ever be built so using PN in this location is fine.
The one risk of this change is there isn't any compatibility to the old
format. I'm not convinced we should spend time complicating the code
with it. This change will change the taskhashes everywhere so the only
mixing of old and new siginfo files will be either through hash equivalence
or through users using the tool against old and new info files manually
which will give some weird output but it should be clear they're in
different formats as there would be large paths from the old files not
present in the new ones.
We have options to add backwards compatibility if some issue is found
to need that.
(Bitbake rev: 637933e2e5a59228a8d17aae4160551cab5f2f61)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update test_python_snippet_function_reference to not require the 'func'
flag, now that we know the real function will be returned for context
functions without the flag.
(Bitbake rev: 83f41281ec3d9b4327ffc8e2312e1fb8f53cbf02)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We previously checked for the existence of the 'func' flag to determine
if we should avoid looking up in the metadata. This was done to ensure
the user gets the function for 'def' python functions rather than their
string contents. We can sidestep the metadata lookup and check our
function context directly, instead.
(Bitbake rev: 6cac1eac51efa9a54e8457f60ea1ea0e604c50b7)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This test makes sure that '${@eval()}' calls the eval builtin, even if an 'eval' variable is defined in the metadata.
(Bitbake rev: e9150447738a48f772240874b3512b08e982b19b)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids the need to hardcode a list of python builtins. This also slightly changes behavior, in a case like `${@eval("3")}`, this will ensure we always call the builtin, even if the metadata has an 'eval' variable defined.
(Bitbake rev: 9976ae50677b333d646ca0fd395468bd2301d03f)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that any change to the presence of builtins in inline
python execs will be noticed.
(Bitbake rev: ee22d3d51c60db2da97422b2be1e42239b7a2324)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a test for an issue seen long ago, to avoid regressions, where a
reference to a def'd function in the metadata would return the string
value from the metadata rather than the function in inline python.
(Bitbake rev: 9f7cb22febd557817c164e25a93f5660e9c06358)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set the maxsplit value to match the expected number of variables.
This also avoids an unnecessary split as the parameters are in the form 'key=value'
and the 'value' could contain the '=' character.
(Bitbake rev: 3b17a7ed9bf6cd2808946c2d9c3ed9961af11f19)
Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using multiconfig to target baremetal pieces of the system and building
corresponding toolchains for them results in hundreds and hundreds of
"Deferring %s after %s" and "Deferred task %s now buildable".
To clean up the output and to reduce risk of missing important warnings,
convert these notice messages to debug messages.
(Bitbake rev: 64bc00a46d1aacc23fe7e8d9a46a126f3a4bc318)
Signed-off-by: Denys Dmytriyenko <denis@denix.org>
Signed-off-by: Denys Dmytriyenko <denys@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is currently hard to tell when bitbake is throttling task execution
due to system pressure changes. Add notes to the console output to make
this clearer, only generating output when the values change.
(Bitbake rev: a6056599922fb2fe3f54c5c86ac7ea604f469adc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is unclear from the server logs when parsing starts. We know that timeouts
sometimes happen when parsing but it is unclear where in the code the delays
are from. Adding this debug message to the server log should help narrow
that down.
(Bitbake rev: a5c145f436d68f090b113cfb9b82857adc95b546)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To learn more about the server timeout issues, be clear in the error
message about which command is showing the timeout. It is currently
unclear if this is the original command or a ping to the server.
(Bitbake rev: ac3cd866274f67b29eff89e393132bdabf76dbfd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"fetch2/npmsw: Add support for the new format of the shrinkwrap file"
added support for the new format shrinkwrap files but this regressed
our tests which still use the old format.
Similar to how npm handles this, support both for now until we can
migrate our tests.
(Bitbake rev: 9941b480a0e2a8b57f2ed069cd583f2784394a2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Packages like @(._.)/execute causes problems because they generate names
that are not supported by yocto
(Bitbake rev: b2e1be67d2acca27451bed59874bc1c2a7ec44a6)
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Npm is a package manager that has its own manner to handle installation of packages.
But it is not yocto friendly, for instance NPM fetch dependencies in the middle of compilation.
The shrinkwrap file changed its format over npm versions, but npm does not version
this file, so we can use it properly.
The actual changes make NPM depencies work with the actual shrinkwrap format.
(Bitbake rev: 19b9f7f0f451a636f3fdcdc1bb283ab431ede612)
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are several multiconfigs in play for example a non-multiconfig with
a task with one hash and then three multiconfigs for the same task, different
architectures but the same hash (different to the non-mc), the three mcs
will be deferred until after the non-mc task but then will all run together
and race against each other.
Change the code to re-enable deferred tasks one at a time. This way, if they do
race, they won't run in parallel against each other.
(Bitbake rev: 9523e28658ad7fb446645b590608dfac2812afd3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default, pytest captures all stdout and exposes it using its built-in
fixtures (capsys, caplog etc), so stdout does not support getvalue().
To support running tests using both unittest and pytest, use assertLogs
to capture logging and assert on the log output instead.
(Bitbake rev: 2d28caa01bab9540d2bbaf713ae3e5c563d003f5)
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
git config returns an error code if user.email or user.name are not set,
so bb.process.run raises an ExecutionError, failing the test.
Improve the logic by catching the ExectionError and using this to set a
default value.
Change a direct invocation of 'git init' to use self.git_init so that
it follows the code path above.
Remove the related comment from the README now that git sets up the user
details automatically.
(Bitbake rev: 0c7079d0609a13713b890a9675abd9fba032f199)
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TestCase.assertRaisesRegexp was renamed to assertRaisesRegex in Python
3.2, so rename to fix a warning during test execution.
(Bitbake rev: 9bc2ca3197bff7a5ef6e0ece6bfef0cda68b808e)
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
pytest test collection attempts to collect all classes containing
"Test". TestTimeout is not a test class so add the __test__ = False
attribute to indicate this.
(Bitbake rev: b5455b98093a3eaf122ebe3d29c3bef949fd2440)
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I'd tested target configurations but in real world use, it became clear
the layername functionality in worker context was failing for virtual
class extensions. Fix this.
(Bitbake rev: 888778ebfa85677ff36065dfcd0f8a7684edfc80)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Include the hashfn (the value of BB_HASHFILENAME) in the task dependency
data. This allows tasks to get a specific unique hash for dependent
tasks when one is available.
(Bitbake rev: 4dbecf6059e495246267b09d0f43086d51e6df2c)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes a crash when the server process attempts to check the PID of the
lock file that resulted because an integer (os.getpid()) was attempting
to be concatenated to a string
(Bitbake rev: 5d499682a0a739b5269247a8f6dbb874e3eec456)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the user configures the TMPDIR variable in a way that it contains $HOME (in local.conf), the DataSmart (d) object's PATH variable will also contain $HOME.
As the NpmEnvironment patches HOME with a temporary directory, further evaluation of PATH will become invalid an the build breaks with an error (npm not found).
As countermeasure PATH is evaluated *before* patching HOME.
(Bitbake rev: 893f3b116b628457fe0bfaf4b8394b1cd05a90a5)
Signed-off-by: Arno Baumfalk <a.baumfalk@astro-kom.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are times when it would be useful for code to know which layer
(or collection in old bitbake terms) it is contained within.
Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe
so that it is possible to determine this. To do it, we need to pass data
from the cooker into the recipe endpoints, since only the top level cooker
information knows about the layer structure which makes the patch a bit
painful.
The idea is that this would make layer overrides possible:
OVERRIDES .= ":layer-${FILE_LAYERNAME}"
which then opens possibilities like:
WARN_QA:append:layer-core = " patch-fuzz"
as an example where OE-Core could enable specific QA tests only for that
specific layer.
(Bitbake rev: 7090a14b0035842112d073acf7f2ed1a01fdeccf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have a host where the git config for initial branch
name is something else than 'master' the unittest will fail
as they assume the default branch name is 'master'. Fix this
by explicitly set the intial branch name at 'git init'.
(Bitbake rev: 5bcd213c23da30a84baba88b775b1740e6bb77d0)
Signed-off-by: Peter Bergin <peter.bergin@windriver.com>
Signed-off-by: Peter Bergin <peter@berginkonsult.se>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current debug lists all the potential search paths but not the end result
which isn't helpful for debugging. Tweak to include the end result.
(Bitbake rev: a98b8dbb5fbd937a9530793485cf7c68fdd75cc2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To avoid confusion, rename the local fetcher localpaths function to
something more specific so it doesn't appear similar to the localpaths
top level function.
(Bitbake rev: ada5ea712cc42fff8c55bd261ee6d7579efd4a68)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Code changes for FILE_LAYERNAME exposed lifecycle issues around the collections
object in Cooker which only appeared in devtool usage in eSDK. Move
the collections setup to an earlier stage after parsing completes to
avoid any kind of race around it.
Also stop overwriting the code variable in MatchFiles. Ultimately we
need to combine these codepaths but that is for another patch.
(Bitbake rev: 27b872ed4fbe73b3b61e14cb885bb7c16c039cdb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The crate fetcher handles a crate with a name that matches the recipe's
name specially by placing the unpacked code in the current directory
(which typically is ${S}) rather than together with the sources for the
other crates. This broke when the URI names for all crates were changed
recently to include the version in the name.
Correct the crate fetcher to test against ${BP} instead of ${BPN}.
Also add a test case to the selftests to avoid this breaking again.
[Yocto #15012]
(Bitbake rev: 9af52b748357d0c843ce2507ce4d119fd9c37008)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recommendation from server maintainers is that the https protocol
is both faster and more reliable than the dedicated git protocol at this point.
Switch to it where possible.
(Bitbake rev: 879b830fcc60a022682ef22f58c04e7e07d03c63)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It can be hard to tell why bitbake does some things slowly. Log the changes
to the base configuration and parse cache status so that it becomes clear
from the logs when the cache invalidation causes a slowdown.
(Bitbake rev: 6e99d89f3c00a5f53c24d687eaef24f52fe0ef99)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Recipes containing both git and npmsw sources in the SRC_URI fail
during fetch from the shrinkwrap. It seems that when the fetcher is
fetching from the shrinkwrap, the SRCREV variable has been deleted but it
till ends up fetching from the git source resulting in an error because
SRCREV is undefined. The root cause of this is that the Fetch class defaults
to urls from the SRC_URI when the urls parameter contains an empty list. This
patch will ensure that Fetch is not instantiated if the urls list is empty.
(Bitbake rev: e602963dfd505eef08702366383358d29ee20c4d)
Signed-off-by: Svend Meyland Nicolaisen <public@smn.dk>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current PSI check calculation does not take into consideration
the possibility of the time interval between last check and current
check being much larger than 1s. In fact, the current behavior does
not match what the manual says about BB_PRESSURE_MAX_XXX, even if
the value is set to upper limit, 1000000, we still get many blocks
on new task launch. The difference between 'total' should be divided
by the time interval if it's larger than 1s.
(Bitbake rev: b4763c2c93e7494e0a27f5970c19c1aac66c228b)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update regex pattern to allow variable flag name with a single character.
Regression tests have also been updated in `bb.parse` and
`bin/bitbake-selftest -k ParseTest` has been successfully executed.
Eliminate a trailing space as well.
(Bitbake rev: bb9e523291a3cad6e1596ee6a1e715b5e5feba8f)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add bb.event.ParseError to let error-report can catch this kind of error.
(Bitbake rev: 316524ab59a5e738c25e062923ee5717d88ae5c7)
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is common for rust packages to depend on different versions of the
same crate. E.g.
| crate://crates.io/windows_x86_64_msvc/0.42.2 \
| crate://crates.io/windows_x86_64_msvc/0.48.0 \
Identification only by the plain crate name makes the sha256sum
ambiguous
| SRC_URI[windows_x86_64_msvc.sha256sum] = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0"
| SRC_URI[windows_x86_64_msvc.sha256sum] = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a"
and requires lot of manual work to fix the SRC_URI list.
Making the 'name' property unique by appending the version allows
direct copy & paste of reported sha256sum errors to complete the
crates list.
(Bitbake rev: ae2efb05196f9e29ef56ad9a84e2eae5fbdd8030)
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of only display the first and stop.
For recipe (crago based) that can contains several artifacts
to fetch with their checksum, it will particularly handy
to display all of missing one.
An example of error message would be
NOTE: Executing Tasks
ERROR: zvariant-3.12.0-r0 do_fetch: Missing SRC_URI checksum, please add those to the recipe:
SRC_URI[anyhow-1.0.70.sha256sum] = "7de8ce5e0f9f8d88245311066a578d72b7af3e7088f32783804676302df237e4"
SRC_URI[chrono-0.4.24.sha256sum] = "4e3c5919066adf22df73762e50cffcde3a758f2a848b113b586d1f86728b673b"
SRC_URI[serde-1.0.158.sha256sum] = "771d4d9c4163ee138805e12c710dd365e4f44be8be0503cb1bb9eb989425d9c9"
SRC_URI[system-deps-1.3.2.sha256sum] = "0f3ecc17269a19353b3558b313bba738b25d82993e30d62a18406a24aba4649b"
ERROR: zvariant-3.12.0-r0 do_fetch: Bitbake Fetcher Error: BBFetchException('There was some missing checksums in the recipe')
ERROR: Logfile of failure stored in: /home/jenkins/yocto-poky-master/poky/build/tmp/work/core2-64-poky-linux/zvariant/3.12.0-r0/temp/log.do_fetch.1025
ERROR: Task (/home/jenkins/yocto-poky-master/poky/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb:do_fetch) failed with exit code '1'
(Bitbake rev: dafa07c080e05975b6319b5adf78a9691c6b6643)
Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
Set EXCLUDE_FROM_WORLD = "0" in recipe_A:
$ bitbake world -g
Check pn-buildlist, the recipe_A won't be built, this patch fixes the problem:
EXCLUDE_FROM_WORLD = "1": Not build in world
EXCLUDE_FROM_WORLD = "0": Add back to world build
(Bitbake rev: 3f4ede2d67a2d75d3faf8887f90371bd7554d492)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have an obtuse rare and intermittent bitbake fetch failure. Understanding
the value of ud.localpath at the time of failure would be helpful to debug it
further. Tweak the debugging to provide this in the output.
(Bitbake rev: 935ab36f64d25c09f83209cd369714c2bc89aa7f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "git clone /path/to/git/objects_symlink" couldn't work after the following
change:
6f054f9fb3
But repo command manages the git repo as symlinks, so check whether the objects
is an symlink to fix the problem:
* Nothing is changed if git/objects is not a symlink
* Use "git clone file://" when git/objects is a symlink
(Bitbake rev: a0d8108eba8d542707740d00c66c1c5f5b963f18)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
