Gitsm fetcher crashes when cloning a repository that contains LFS files.
This happens because the unpack method is called during download, but the
submodules have not been downloaded yet at this point.
This issue was introduced in this
commit: 977b7268bf
[YOCTO #14283]
(Bitbake rev: e05d79a6ed92c9ce17b90fd5fb6186898a7b3bf8)
Signed-off-by: Niels Avonds <niels@codebits.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In a multiconfig situation there are circumstances where firstly, tasks
are deferred when they shouldn't be, then later, tasks can end up as
both covered and not covered.
This patch fixes two related issues. Firstly, the stamp validity checking
is done up front in the build and not reevaulated. When rebuilding the
deferred task list after scenequeue hash change updates, we need therefore
need to check if a task was in notcovered *or* covered when deciding to
defer it. This avoids strange logs like:
NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene)
NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch
where tasks have run but are then deferred.
Since we're recalculating the whole list, we also need to clear it before
iterating to rebuild it. By ensuring covered tasks aren't added to the
deferred queue, the covered + notcovered issue should also be avoided.
in the task deadlock forcing code.
[YOCTO #14342]
(Bitbake rev: 32183076cb54f6d247471429cfd99491e41a1cab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-server is spawned by process.py and passes the arguments it is
given to ProcessServer. There's some type confusion here:
bitbake-server is called with a string representation of the timeout,
which may be None. If the timeout is not set, pass 0 instead of None.
Inside bitbake-server a ProcessServer is created which expects the
timeout to be a float not a string, so always float() the value.
[ YOCTO #14350 ]
(Bitbake rev: 95cc877f3bf5ef2513d7af0ae424f78ec7c24f7d)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c93ae1f861208f6d39fd15c84fbcd0e2b54331f5)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The provided packages by a skipped recipe are supposed to be listed in
SkippedPackage.rprovides, which is used when generating a meaningful
error message when a build fails because of a skipped package.
Previously this variable only contained the contents of ${RPROVIDES}.
However, most recipes don't define RPROVIDES, they define
RPROVIDES_<pkg> for each package they provide. Additionally, the recipe
provides the packages in PACKAGES without them being included in
${RPROVIDES}.
Before this change, having a runtime dependency on a skipped non-recipe
package would result in a build error stating that the build failed
because the package was skipped, but without providing any reason for
why it was skipped.
(Bitbake rev: 7b7d7b02faedb603d81144a134e80027e4019ab0)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing
parser shutdown as default in case of build abort.
In this case bitbake sometimes hangs after facing error during parsing,
waiting for child processes to finish. Killing it then will spawn zombie
processes.
Thus we force the shutdown after catching an exception.
(Bitbake rev: 5d02c98489d3a5836676b9c3fb3bd0157449db2b)
Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 915330e1dbae1ee8fd9a0358decf2c294f771961)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen hard to debug issues where a task ends up in both the
covered and notcovered list. Add a sanity check to ensure if this
happens in future, we see it in the logs.
(Bitbake rev: 947c2ad65e4c59d01c603f767f11f9aa86b69264)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6e001410854792f9bb66a0409a2ac176171b0507)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen occasional issues where linux-yocto:do_compile_kernelmodules would
run without do_shared_workdir running before it. do_shared_workdir is an
setscene task but never has an sstate object generated so it will always
rerun. This should not happen since compile_kernemodules should only
execute if a setscene that depends on it didn't run and that should trigger
do_shared_workdir not to be marked as covered.
The issue is that build-appliance-image:do_package is one of the tasks which
covers linux-yocto:do_compile_kernelmodules but it is also a noexec task
and has a dependecy on pseudo-native:do_populate_sysroot.
In the problem case, pseudo-native:do_populate_sysroot is unavailable but
marked as covered since it is noexec. The "harddeps" code then also marks it
as notcovered. No task should ever be both covered and notcovered and this
is where the problems come from.
The solution is for the harddeps code only to to fail tasks if they've not
already been handled in some way. The code is assuming code couldn't have
handled revdeps at this point but we now have clear evidence they can.
(Bitbake rev: 0a61f86a8da5356ce0d894ef2ffb96ac6a219db5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f66556bbb38449789ceea2fd105e9f68df7fb660)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For a while I've been puzzled as to why debug logging from runqueue wouldn't
appear on the console with -DD. The logic in the bbdebug handling is inverted
so fix it and now we see the expected messages from runqueue with -D and -DD.
This should then let us debug other issues using those log messages.
(Bitbake rev: 7a6517243dbfecfd78f82f6709fbeaab2fb03e2c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With introduction of python3targetconfig class in core this is no longer
needed.
(Bitbake rev: 0a3bf681530bd63fc0036ca81ef868ab53fde56c)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 47b64cfacd7c498ef9ed5486d117f2d69a39f225)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Insert an explicit pass to fetch all blobs needed by Git LFS, during the
fetch() function. This avoids the default behavior of Git LFS to wait
until 'git checkout' to begin downloading the blobs pointed to by LFS records.
Network access is not allowed at that point in the recipe's lifecycle.
[YOCTO #14191]
(Bitbake rev: c73f8f2f4a6491c6bea54839630af6994c27ad24)
Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0efac66043662e7a2027192f50e92e982db2ba1c)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The get_hash() function is used to decide if the base configuration has changed
and hence whether a reparse is required. The vardepvalue flag's value was not
expanded but it is often used in contexts like:
METADATA_REVISION = "${@base_detect_revision(d)}"
METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}"
which in it's unexpanded form means reparsing doesn't happen when it should
as the data appears unchanged. Update get_hash to expand the values of
vardepvalue so reparsing works as expected. This avoids basehash mismatch
errors such as the one recently caused by using METADATA_REVISION in poky.conf's
DISTRO_VERSION variable. The issue there could be exposed by a recipe using
DISTRO_VERSION with the sequence:
bitbake os-release
<change the revision of the metadata with a dummy commit>
bitbake os-release -C install
which was caused because METADATA_REVISION changed but the metadata didn't reparse.
(Bitbake rev: cb7277e7bb3a440968fdb918d56fe8fa17bca2e6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On startup, bitbake spawns a cooker daemon and waits for
it's acknowledgement signal. If the acknowledgement
doesn't happen in time,the bitbake object will fail to
initialize and exit.
The error that occurs in this case isn't handled by
the existing try - catch block because SystemExit inherits
from a different base Exception class.
This commit adds SystemExit to the list of expected bitbake
server startup errors.
[YOCTO #13993]
(Bitbake rev: fec2b85689bba1d26ad6f376bc11cc29bb27cbe5)
Signed-off-by: Stacy Gaikovaia <stacy.gaikovaia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the Toaster documentation links to the new
Yocto Project URLs.
[YOCTO #14092]
(Bitbake rev: 9345d257ced432adc2d16af20ace58cc7c086aab)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster has been failing to start correctly when in interactive mode. The
issue is due to setEventMask being called (which triggers parsing) before
the environment has been sent from the UI over to the server. This means
PATH isn't setup, which causes the sanity checks on HOSTTOOLS to fail
in base.bbclass.
The fix is to ensure the environment is sent to the server before
other commands are run.
The pain in debugging this highlights other improvements to the logging
are needed.
[YOCTO #14079]
Reviewed-by: Tim Orling <timothy.t.orling@linux.intel.com>
Tested-by: Tim Orling <timothy.t.orling@linux.intel.com>
(Bitbake rev: a8efff5c83cd5a25f4b6720e6414a7aa35d04bc7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster directly supports the last two stable branches
of Yocto Project. With "Gatesgarth " being released, it
is time to replace "Zeus".
[YOCTO #14086]
(Bitbake rev: de578cd8f8b582c7bea83c75d06b7f667c07578a)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After parseprogress.finish() it was intended to
set parseprogress to None, but a typo means this
is not happening.
(Bitbake rev: f504d6f6598f62aa20fbf69c30fea95569858edb)
Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit replaces the instances where escaped double quotes
are used to support SRC_URI url containing spaces with the more
pythonic shlex.quote().
(Bitbake rev: 4f9ba9c794de55bea0343267467bddea99844374)
Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are autobuilder logs where the server commands are failing
but we have no debug info in the server log. Improve this to try and
understand what is failing.
(Bitbake rev: 04d3a79226c9ea448b22f4efbab33876a72c9bdb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The server may be displaying useful information for the user through log
messages so we should display anything that has been sent. Its either this
or expecting every UI to implement this code around every command call
which isn't good API.
[YOCTO #14054]
(Bitbake rev: 64ae9d7e2fad804dd9e12706c6d76b4b22f9586b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previous changes allowed BBHandledException to be detected but not exceptions
which inherit from it. Fix this. The code really needs totally reworking
to preserve the exceptions.
[YOCTO #14054]
(Bitbake rev: ef762d92df6c2554c6248e80212f984d9ec4c651)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Recent changes mean data might not be setup. If its not, avoid tracebacks.
(Bitbake rev: 3daff610d9f39d73c80c54d1df46f573666e20db)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We try to write sstate with group-write permissions so that sstate-cache
can be shared between multiple users. However the siginfo files are
created in various tasks which may set their own umask (such as
do_populate_sysroot, 0022). This results in no group write permission
on the intermediate directories, which is fatal when sharing a cache.
Fix this by wrapping the siginfo mkdir in a umask change to 0002.
(Bitbake rev: 75d9ef04a908e366633b255d23ab3275f6860d3a)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a umask context manager which can be used to temporarily change the
umask in a 'with' block.
(Bitbake rev: 6c601e68a27e1c60b04c2a61830d1812cc883e09)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implements backslashes in local filenames.
A typical usecase for such a filename is a systemd unit.
Example: `dev-disk-by\x2dlabel-FOO.device`
(Bitbake rev: 14a35f273b579d5cd5fd92765b89c28f870dd577)
Signed-off-by: Leif Middelschulte <leif.middelschulte@klsmartin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The introspected API for GtkTreeModel and friends had some unexpected
quirks which have now been fixed, see[1] for details. However, for
example Ubuntu 20.04 has the fixed GTK but not an updated pygobject which
means taskexp raises an exception on startup.
Solve by manually looking at what functions are present and calling the
right one.
[ YOCTO #14055 ]
[1] 9cdbc56fba
(Bitbake rev: ac7d1114a7e99e6efd6a37b03d170faf678513fb)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change improves performance by reducing runtime about 33% for typical
inputs. (using test_clean_basepath_performance)
It is also easier to read, and slightly more resilient to future changes since it doesn't
mention 'virtual' anymore.
(Bitbake rev: 27b53186fa67d281d29b2f8e15bcff8dc2557b8a)
Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com>
Co-Developed-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com>
Signed-off-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While discussing with Richard we thought these might help document
and safeguard the basic requirements of clean_basepath.
A 'bonus' performance testcase is added but commented out since its
runtime is long and test machine specific. It is intended for developers
to test before and after their changes to the target function as a due
diligence verification.
(Bitbake rev: ee41549f26952d5f7af19a9b3d8a8b969866e2ef)
Signed-off-by: Jean-Francois Dagenais <jeff.dagenais@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Microsoft's TFS VCS system allows for spaces in a git repository url.
An example of a valid url is:
ssh://tfs-my-company.org:22/tfs/My Projects/FooBar
This commit adds support for such urls by implementing two changes.
Firstly, when bitbake makes a git command line call the url is
surrounded by quotes so that the url, regardless of spaces, is
treated as one argument. Secondly, additional parsing of various
filepath variables, which are based off of the url, are now
completed with any spaces in the url replaced with underscores.
(Bitbake rev: eb38b6f0935763f7ba19e5618f376fcae1dac41a)
Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using localhost for direct builds on host is fine. A case with a
misbehavior has been sighted on a Docker build. Even when the host
supports IPv6, but Docker is not configured correspondingly - some
versions of the asyncio Python module seem to misbehave and try to
use IPv6 where it's not supported in the container. This happens at
least on some Ubuntu 18.04 based containers, resolving the IP
explicitly appears to be the fix.
(Bitbake rev: 0e20f91c11afdc17ea776aa02e0cc8b0d59a23d4)
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Blocks SIGINT in the worker processes to prevent them from running the
parent process signal handler, which causes them to deadlock under
certain circumstances.
[YOCTO #14034]
(Bitbake rev: 9f4207f4b598f549cbd4159841c720276736f23b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're either hitting rate limiting with freedesktop.org or the
servers have intermittent network connections. Use our own mirror
of these repositories instead.
(Bitbake rev: a1b7ab5c9d5e64969f5ca0e41c0ac13c723e3761)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing leaking open socket connections when errors occur and
tinfoil is in use. Improve the exception handling so the sockets
are closed even if exceptions occur, allowing more robust behaviour
when things go wrong.
(Bitbake rev: cefbec9ff47ca973a74ec7300cd736f3e0f0bce0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the recent init changes, if a client disconnects before issuing a
command, the cooker can break in the reset handlers. Add some guards
in the code to prevent this.
(Bitbake rev: 12605e30e4c4e1ae6a67c97363b892ebf0b9566c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The following code costs a lot of time when there are lot of layers and recipes:
for collection in collections:
collection_res[collection] = d.getVar('BBFILE_PATTERN_%s' % collection) or ''
My build has more than 100 layers and 3000 recipes, which calls d.getVar() 300K
(3000 * 100) times and makes 'bitbake-layers show-recipes' very slow, add a
keyword argument to get_file_layer() can fix the problem, it can save about 90%
time in my build (6min -> 40s).
(Bitbake rev: f08a6601c9bb09622855d62e1cedb92fafd2f71d)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This can make "$ bitbake-layers show-recipes" save about 60% time (14min ->
6min) in my build (more than 3000 recipes)
The command "bitbake-layers show-recipes" calls bb.utils.get_file_layer() with
each recipe, and get_file_layer() compare the file with each item in BBFILES
which makes it very time consuming when there are a lot of recipes and items in
BBFILES. So use BBFILES_PRIORITIZED and exit when file is matched, it doesn't
make sense to go on the loop when file is matched.
And use fnmatchcase to replace of fnmatch since the comparison should be
case-sensitive.
(Bitbake rev: 8d64181d29dc262e066a6114dd51e5f2d04f47de)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The original code saved BBFILES back to BBFILES without any changes which isn't
usefule, so remove that line. Now save prioritized BBFILES to
BBFILES_PRIORITIZED which can accelerate the query a lot for the one which
relies on it such as bb.utils.get_file_layer().
(Bitbake rev: 49bdb5dfa57b41b3ed399961e947c404f9195998)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Otherwise this can interfer with multiprocessing exit handling.
(Bitbake rev: b88816c4c84fa4f5ad39c263f5e75b96476e9768)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you make parsing fail (e.g. add something like:
X := "${@d.getVar('MCMACHINES').split()[1]}"
to meson.bbclass, then run "while true; do bitbake -g bash; done"
it will eventually hang. It appears the cancel_join_thread() call the
parsing failure triggers, breaks the results_queue badly enough that it
sits in read() indefintely (called from self.result_queue.get(timeout=0.25)).
The timeout only applies to lock aquisition, not the read call.
I've tried various other approaches such as using cancel_join_thread()
in other places but the only way things don't lock up is to avoid
cancel_join_thread() entirely for results_queue.
I do have a concern that this may adversely affect Ctrl+C handling
but equally, its broken now already and this appears to improve
things.
[YOCTO #14034]
(Bitbake rev: 9c61a1cc7be46c23da1f4ef3bee070fb83c4be57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During cooker shutdown, its possible the parser isn't cleaned up. Fix
this (which may partially explain why threads were left hanging around
at exit).
(Bitbake rev: 928609f30f3a20aaa2f88afc18044a4e10199488)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its hard to tell from the server logs whether commands complete or not
(or how long they take). Add extra info to allow more debugging of
server timeouts.
(Bitbake rev: 56285ada585ec1481449522282b335bcb5a2671e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Knowing which process printed which messages and the timestamp of the
message is useful for debugging, so add this. Ensure the log parsing
isn't affected by using search() instead of match().
(Bitbake rev: 1d043666710df1fa9d9586fd974c0371dd1514b0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Showing "leftover process" messages when a new server has started and is being
used by some UI is horrible. Compare the PID data from the lockfile to
avoid this (and the ton of confusing log data it generates).
Also, move the time.sleep() call to be after the first lock attempt, which
reduces noise in the logs significantly.
(Bitbake rev: ce1897a31afb5a14997bc3d2f459b90d43eecb7d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
