We had an issue where a webserver serving sstate had filesystem issues so
would accept connections but effectively not do anything with them. This
causes bitbake to hang whilst processing things like sstate objects inside
the checkstatus() calls. It can be replicated by setting up a server like:
socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null
and pointing SSTATE_MIRRORS in OE at that address.
Adding a timeout to the checkstatus calls of 30s means that whilst the
system will pause, it will then continue and not hang entirely. Since there
isn't a large transfer here, 30s should be a reasonable response time after
which we should fall back to building things ourselves.
[YOCTO #13716]
(Bitbake rev: 92bb53adf47b95182e763a41cd6f4f920250e7b7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
github is dropping support for git protocol in Git urls. Add code to remap
this to https in a way that could be used in older bitbake versions.
(Bitbake rev: 44320ce7a2bfd5126de2f83a1a1102a3e4485f69)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that ":" is a valid character in variable key names, it needs to be
allowed by the variable expansion code too, to match.
(Bitbake rev: 14ae61205111383d5f609519c02476925184f6d1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is becomming increasingly clear we need to find a way to show what
is/is not an override in our syntax. We need to do this in a way which
is clear to users, readable and in a way we can transition to.
The most effective way I've found to this is to use the ":" charater
to directly replace "_" where an override is being specified. This
includes "append", "prepend" and "remove" which are effectively special
override directives.
This patch simply adds the character to the parser so bitbake accepts
the value but maps it back to "_" internally so there is no behaviour
change.
This change is simple enough it could potentially be backported to older
version of bitbake meaning layers using the new syntax/markup could
work with older releases. Even if other no other changes are accepted
at this time and we don't backport, it does set us on a path where at
some point in future we could
require a more explict syntax.
I've tested this patch by converting oe-core/meta-yocto to the new
syntax for overrides (9000+ changes) and then seeing that builds
continue to work with this patch.
(Bitbake rev: aa9f7b80cfdb1119050af469a07ebd949829026c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If shell function name starts with 'python' or 'fakeroot' parser wrongly
assumes it's python/fakeroot function.
[YOCTO #14204]
Use regex lookahead assertions to check if 'python' expression is
followed by whitespace or '(' and if 'fakeroot' is followed by
whitespace.
(Bitbake rev: 9bddcd4824ed94915a2b7cf96a13d4ada01c7f9d)
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the selected version if not available, bitbake will happily attempt
to build something else. This should be a loud warning not a small note.
(Bitbake rev: 78cd63285713fde59506eb2e71a7b7ee59a594ff)
(Bitbake rev: a1987f240cdedc6a44bf255a3a76879aa4cd1cd3)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Gitsm fetcher crashes when cloning a repository that contains LFS files.
This happens because the unpack method is called during download, but the
submodules have not been downloaded yet at this point.
This issue was introduced in this
commit: 977b7268bf
[YOCTO #14283]
(Bitbake rev: e05d79a6ed92c9ce17b90fd5fb6186898a7b3bf8)
Signed-off-by: Niels Avonds <niels@codebits.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In a multiconfig situation there are circumstances where firstly, tasks
are deferred when they shouldn't be, then later, tasks can end up as
both covered and not covered.
This patch fixes two related issues. Firstly, the stamp validity checking
is done up front in the build and not reevaulated. When rebuilding the
deferred task list after scenequeue hash change updates, we need therefore
need to check if a task was in notcovered *or* covered when deciding to
defer it. This avoids strange logs like:
NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene)
NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch
where tasks have run but are then deferred.
Since we're recalculating the whole list, we also need to clear it before
iterating to rebuild it. By ensuring covered tasks aren't added to the
deferred queue, the covered + notcovered issue should also be avoided.
in the task deadlock forcing code.
[YOCTO #14342]
(Bitbake rev: 32183076cb54f6d247471429cfd99491e41a1cab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-server is spawned by process.py and passes the arguments it is
given to ProcessServer. There's some type confusion here:
bitbake-server is called with a string representation of the timeout,
which may be None. If the timeout is not set, pass 0 instead of None.
Inside bitbake-server a ProcessServer is created which expects the
timeout to be a float not a string, so always float() the value.
[ YOCTO #14350 ]
(Bitbake rev: 95cc877f3bf5ef2513d7af0ae424f78ec7c24f7d)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c93ae1f861208f6d39fd15c84fbcd0e2b54331f5)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The provided packages by a skipped recipe are supposed to be listed in
SkippedPackage.rprovides, which is used when generating a meaningful
error message when a build fails because of a skipped package.
Previously this variable only contained the contents of ${RPROVIDES}.
However, most recipes don't define RPROVIDES, they define
RPROVIDES_<pkg> for each package they provide. Additionally, the recipe
provides the packages in PACKAGES without them being included in
${RPROVIDES}.
Before this change, having a runtime dependency on a skipped non-recipe
package would result in a build error stating that the build failed
because the package was skipped, but without providing any reason for
why it was skipped.
(Bitbake rev: 7b7d7b02faedb603d81144a134e80027e4019ab0)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing
parser shutdown as default in case of build abort.
In this case bitbake sometimes hangs after facing error during parsing,
waiting for child processes to finish. Killing it then will spawn zombie
processes.
Thus we force the shutdown after catching an exception.
(Bitbake rev: 5d02c98489d3a5836676b9c3fb3bd0157449db2b)
Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 915330e1dbae1ee8fd9a0358decf2c294f771961)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen hard to debug issues where a task ends up in both the
covered and notcovered list. Add a sanity check to ensure if this
happens in future, we see it in the logs.
(Bitbake rev: 947c2ad65e4c59d01c603f767f11f9aa86b69264)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6e001410854792f9bb66a0409a2ac176171b0507)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen occasional issues where linux-yocto:do_compile_kernelmodules would
run without do_shared_workdir running before it. do_shared_workdir is an
setscene task but never has an sstate object generated so it will always
rerun. This should not happen since compile_kernemodules should only
execute if a setscene that depends on it didn't run and that should trigger
do_shared_workdir not to be marked as covered.
The issue is that build-appliance-image:do_package is one of the tasks which
covers linux-yocto:do_compile_kernelmodules but it is also a noexec task
and has a dependecy on pseudo-native:do_populate_sysroot.
In the problem case, pseudo-native:do_populate_sysroot is unavailable but
marked as covered since it is noexec. The "harddeps" code then also marks it
as notcovered. No task should ever be both covered and notcovered and this
is where the problems come from.
The solution is for the harddeps code only to to fail tasks if they've not
already been handled in some way. The code is assuming code couldn't have
handled revdeps at this point but we now have clear evidence they can.
(Bitbake rev: 0a61f86a8da5356ce0d894ef2ffb96ac6a219db5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f66556bbb38449789ceea2fd105e9f68df7fb660)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For a while I've been puzzled as to why debug logging from runqueue wouldn't
appear on the console with -DD. The logic in the bbdebug handling is inverted
so fix it and now we see the expected messages from runqueue with -D and -DD.
This should then let us debug other issues using those log messages.
(Bitbake rev: 7a6517243dbfecfd78f82f6709fbeaab2fb03e2c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With introduction of python3targetconfig class in core this is no longer
needed.
(Bitbake rev: 0a3bf681530bd63fc0036ca81ef868ab53fde56c)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 47b64cfacd7c498ef9ed5486d117f2d69a39f225)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Insert an explicit pass to fetch all blobs needed by Git LFS, during the
fetch() function. This avoids the default behavior of Git LFS to wait
until 'git checkout' to begin downloading the blobs pointed to by LFS records.
Network access is not allowed at that point in the recipe's lifecycle.
[YOCTO #14191]
(Bitbake rev: c73f8f2f4a6491c6bea54839630af6994c27ad24)
Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0efac66043662e7a2027192f50e92e982db2ba1c)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The get_hash() function is used to decide if the base configuration has changed
and hence whether a reparse is required. The vardepvalue flag's value was not
expanded but it is often used in contexts like:
METADATA_REVISION = "${@base_detect_revision(d)}"
METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}"
which in it's unexpanded form means reparsing doesn't happen when it should
as the data appears unchanged. Update get_hash to expand the values of
vardepvalue so reparsing works as expected. This avoids basehash mismatch
errors such as the one recently caused by using METADATA_REVISION in poky.conf's
DISTRO_VERSION variable. The issue there could be exposed by a recipe using
DISTRO_VERSION with the sequence:
bitbake os-release
<change the revision of the metadata with a dummy commit>
bitbake os-release -C install
which was caused because METADATA_REVISION changed but the metadata didn't reparse.
(Bitbake rev: cb7277e7bb3a440968fdb918d56fe8fa17bca2e6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On startup, bitbake spawns a cooker daemon and waits for
it's acknowledgement signal. If the acknowledgement
doesn't happen in time,the bitbake object will fail to
initialize and exit.
The error that occurs in this case isn't handled by
the existing try - catch block because SystemExit inherits
from a different base Exception class.
This commit adds SystemExit to the list of expected bitbake
server startup errors.
[YOCTO #13993]
(Bitbake rev: fec2b85689bba1d26ad6f376bc11cc29bb27cbe5)
Signed-off-by: Stacy Gaikovaia <stacy.gaikovaia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the Toaster documentation links to the new
Yocto Project URLs.
[YOCTO #14092]
(Bitbake rev: 9345d257ced432adc2d16af20ace58cc7c086aab)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster has been failing to start correctly when in interactive mode. The
issue is due to setEventMask being called (which triggers parsing) before
the environment has been sent from the UI over to the server. This means
PATH isn't setup, which causes the sanity checks on HOSTTOOLS to fail
in base.bbclass.
The fix is to ensure the environment is sent to the server before
other commands are run.
The pain in debugging this highlights other improvements to the logging
are needed.
[YOCTO #14079]
Reviewed-by: Tim Orling <timothy.t.orling@linux.intel.com>
Tested-by: Tim Orling <timothy.t.orling@linux.intel.com>
(Bitbake rev: a8efff5c83cd5a25f4b6720e6414a7aa35d04bc7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster directly supports the last two stable branches
of Yocto Project. With "Gatesgarth " being released, it
is time to replace "Zeus".
[YOCTO #14086]
(Bitbake rev: de578cd8f8b582c7bea83c75d06b7f667c07578a)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After parseprogress.finish() it was intended to
set parseprogress to None, but a typo means this
is not happening.
(Bitbake rev: f504d6f6598f62aa20fbf69c30fea95569858edb)
Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit replaces the instances where escaped double quotes
are used to support SRC_URI url containing spaces with the more
pythonic shlex.quote().
(Bitbake rev: 4f9ba9c794de55bea0343267467bddea99844374)
Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are autobuilder logs where the server commands are failing
but we have no debug info in the server log. Improve this to try and
understand what is failing.
(Bitbake rev: 04d3a79226c9ea448b22f4efbab33876a72c9bdb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The server may be displaying useful information for the user through log
messages so we should display anything that has been sent. Its either this
or expecting every UI to implement this code around every command call
which isn't good API.
[YOCTO #14054]
(Bitbake rev: 64ae9d7e2fad804dd9e12706c6d76b4b22f9586b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previous changes allowed BBHandledException to be detected but not exceptions
which inherit from it. Fix this. The code really needs totally reworking
to preserve the exceptions.
[YOCTO #14054]
(Bitbake rev: ef762d92df6c2554c6248e80212f984d9ec4c651)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that the DocBook files are removed, we can rename the top level
Makefile.
(Bitbake rev: a7c47f1eac8caac607a2b5f12d07235dff4d740f)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
The BitBake documentation was migrated to Sphinx. Let's remove the
deprecated DocBook files.
(Bitbake rev: 427721d8ff2c8e1db8cb490074f2eed88d03852a)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
From experience the body takes the whole space anyway and the text stays
within the screen boundaries by default, no need to make the min-width
640px then.
(Bitbake rev: d52190ea426d961f609c657dcb403baf59352969)
Signed-off-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is similar to this change in yocto-docs:
9e468274eaad (docs: ref-manual: ref-variables: add links to terms in glossary)
(Bitbake rev: ebdeef2c185465ac9f7d7f01ae9e8531355b9e70)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Recent changes mean data might not be setup. If its not, avoid tracebacks.
(Bitbake rev: 3daff610d9f39d73c80c54d1df46f573666e20db)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We try to write sstate with group-write permissions so that sstate-cache
can be shared between multiple users. However the siginfo files are
created in various tasks which may set their own umask (such as
do_populate_sysroot, 0022). This results in no group write permission
on the intermediate directories, which is fatal when sharing a cache.
Fix this by wrapping the siginfo mkdir in a umask change to 0002.
(Bitbake rev: 75d9ef04a908e366633b255d23ab3275f6860d3a)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a umask context manager which can be used to temporarily change the
umask in a 'with' block.
(Bitbake rev: 6c601e68a27e1c60b04c2a61830d1812cc883e09)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implements backslashes in local filenames.
A typical usecase for such a filename is a systemd unit.
Example: `dev-disk-by\x2dlabel-FOO.device`
(Bitbake rev: 14a35f273b579d5cd5fd92765b89c28f870dd577)
Signed-off-by: Leif Middelschulte <leif.middelschulte@klsmartin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The introspected API for GtkTreeModel and friends had some unexpected
quirks which have now been fixed, see[1] for details. However, for
example Ubuntu 20.04 has the fixed GTK but not an updated pygobject which
means taskexp raises an exception on startup.
Solve by manually looking at what functions are present and calling the
right one.
[ YOCTO #14055 ]
[1] 9cdbc56fba
(Bitbake rev: ac7d1114a7e99e6efd6a37b03d170faf678513fb)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 4fca9a07f2d6b0544977112672b786982d7bb8f2.
The default python handler is to ignore SIGPIPE errors yet by adding
this line, we see:
2020-09-23 22:10:20,325 - oe-selftest - INFO - recipetool.RecipetoolTests.test_recipetool_appendfile_subdir (subunit.RemotedTestCase)
2020-09-23 22:10:20,325 - oe-selftest - INFO - ... FAIL
2020-09-23 22:10:20,325 - oe-selftest - INFO - 11: 30/44 227/415 (14.85s) (recipetool.RecipetoolTests.test_recipetool_appendfile_subdir)
2020-09-23 22:10:20,325 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 226, in test_recipetool_appendfile_subdir
_, output = self._try_recipetool_appendfile('selftest-recipetool-appendfile', '/usr/share/selftest-replaceme-subdir', self.testfile, '', expectedlines, ['testfile'])
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 85, in _try_recipetool_appendfile
return self._try_recipetool_appendcmd(cmd, testrecipe, expectedfiles, expectedlines)
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 49, in _try_recipetool_appendcmd
bbappendfile = self._check_bbappend(testrecipe, recipefile, self.templayerdir)
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/devtool.py", line 186, in _check_bbappend
result = runCmd('bitbake-layers show-appends', cwd=self.builddir)
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/commands.py", line 201, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake-layers show-appends' returned non-zero exit status 141:
NOTE: Reconnecting to bitbake server...
where status 141 is SIGPIPE.
We expect SIGPIPE not to be raised like this. I tried the commands in the original
commit to pipe through less with the change reverted and I don't see the error
shown. I therefore believe this is safe to revert.
(Bitbake rev: 94a18f1f2e51dd5d3928e6e0c17f2c19c8b5db49)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change improves performance by reducing runtime about 33% for typical
inputs. (using test_clean_basepath_performance)
It is also easier to read, and slightly more resilient to future changes since it doesn't
mention 'virtual' anymore.
(Bitbake rev: 27b53186fa67d281d29b2f8e15bcff8dc2557b8a)
Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com>
Co-Developed-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com>
Signed-off-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While discussing with Richard we thought these might help document
and safeguard the basic requirements of clean_basepath.
A 'bonus' performance testcase is added but commented out since its
runtime is long and test machine specific. It is intended for developers
to test before and after their changes to the target function as a due
diligence verification.
(Bitbake rev: ee41549f26952d5f7af19a9b3d8a8b969866e2ef)
Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
