The purpose of ensuring 'incremental fetch' is not easy to see from
the codes. So add comments to explain this.
(Bitbake rev: 8b890b87e30cd05ec92ed71ee3691a47b4d77253)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We had issue when BB_ALLOWED_NETWORKS is set and `bitbake grpc-native -c fetch'
failed even with all contents available in PREMIRRORS.
Add a test case to ensure no regression in the future.
(Bitbake rev: 80c91ceb81b1cae203067af58d3f1fe9c619ae83)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For gitsm recipes, it's possible that some URL is used more than once.
e.g.,
A -> B:rev1 (B is a submodule of A)
A -> C (C is a submodule of A)
C -> B:rev2 (B is a submodule of C)
A anc C are both using B as submodules, but on different revs.
Now if we have:
B:rev1 -> D
B:rev2 -> E
Then, the mirror will not be fully used.
Say we have all repo mirrors for A, B, C, D, E, then in theory it's not
necessary to reach out to any network for downloading. But it's not the
case. After downloading B(rev1) and its submodule D from mirrors, the fetch
process continues to download C, thus B(rev2) and E. Now it finds that B
needs an update because its submodule E needs an update. Of course this is
true because E is not downloaded yet. Now the problem comes to whether to
use mirror or not. The git.py defines try_premirror to return 'False' when
the ud.clonedir exists. As B has been cloned, the ud.clonedir exists and
try_mirror returns False, resulting in not using mirror and going to upstream
directly.
We can see that the mirrors are not fully used. This is usually not problem,
as the cost is only some network download. But in case the following two
settings are there, we get errors.
BB_NO_NETWORK = "0"
BB_ALLOWED_NETWORKS = "*.some.allowed.domain"
In such case, the gitsm recipe A will fail to fetch. Note that all contents
that A needs are in mirrors and now it's failing to fetch. This is unexpected.
Note that the different revs of the same repo in gitsm recipe is not the only
way to reveal this problem. For example, there might be a recipe call B that
uses B:rev3. Check the protobuf and grpc recipes as an example.
For now, we can use the following steps to reproduce this issue. To be clear,
the grpc recipe in meta-oe is now 1.60.0.
1. Add in local.conf:
DL_DIR = "${TOPDIR}/downloads-premirror"
bitbake grpc -c fetch
2. Comment out the DL_DIR setting in local.conf and add the following lines:
PREMIRRORS:append = " \
git://.*/.* git://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
gitsm://.*/.* gitsm://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
"
3. Set BB_NO_NETWORK = "1" and then 'bitbake grpc -c fetch'.
This command succeeds and this shows that the premirror holds everything we need.
4. Add the following lines and then 'bitbake grpc -c fetch'.
BB_NO_NETWORK = "0"
BB_ALLOWED_NETWORKS = "*.some.domain"
After step 4, the error message is as below:
ERROR: grpc-1.60.0-r0 do_fetch: The URL: 'gitsm://github.com/protocolbuffers/protobuf.git;protocol=https;name=third_party/protobuf;subpath=third_party/protobuf;nobranch=1;lfs=True;bareclone=1;nobranch=1' is not trusted and cannot be used
This patch fixes this problem by handling this corner case, that is, if the URL is
not trusted from the settings of BB_NO_NETWORK and BB_ALLOWED_NETWORKS, then we should
try premirrors because trying to reach upstream is destined to fail.
(Bitbake rev: e1be272ad105b47d3131b77168d9172386993fcb)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fetch from premirror few times to emulate multiple machines sharing same
clonedir or few rebuilds of the package from (pre)mirror
Regression test for [Yocto #15369]
(Bitbake rev: 7fcbac574c68f16b95ab7abb2874931d168d3c9e)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix the issue with using of (pre)mirror in case if clonedir exists but
outdated.
Previous version of the code fetched new mirror content into FETCH_HEAD
instead of branch which caused refetch from the upstream. Add new remote
add fetch from it instead so the ref can be found by "_contains_ref"
Fixes [Yocto #15369]
(Bitbake rev: 69588e2a5c7c200e47b02b2391498dcb72388bd2)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When git is configured with safe.bareRepository=explicit [1], the
git-make-shallow fails miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].
It also seems that it is being rolled out in some environments as a
default for users.
In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.
The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.
Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: 7c63989db4590564516ed150930f4e2fa503e98f)
Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When git is configured with safe.bareRepository=explicit [1], the
bitbake selftests fail miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].
It also seems that it is being rolled out in some environments as a
default for users.
In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.
The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.
Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: a45e14a7343e36101e45639931322e5649587f57)
Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When git is configured with safe.bareRepository=explicit [1], the
bitbake git fetcher fails miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].
It also seems that it is being rolled out in some environments as a
default for users.
In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.
The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.
While at it, fix one open-coded invocation of git that wasn't using
ud.basecmd
Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: 5f3b1d8dc9ee70e707536bd75ee845b547440c97)
Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hash Equivalence server performs unconditional insert also of duplicate
hash entries. This causes excessive error log entries in Postgres.
Rather ignore the duplicate inserts.
The alternate behavior should be isolated to the postgres
engine type.
(Bitbake rev: e8d2d178d0fe96f9d6031c97328e8be17d752716)
Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds API to query if unihashes are known to the server in parallel
(Bitbake rev: 7e2479109b40ce82507f73b4f935903f7f79fb06)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implements a new API called get_unihashes() that allows for querying
multiple unihashes in parallel.
The API is also reworked to make it easier for derived classes to
interface with the new API in a consistent manner. Instead of overriding
get_unihash() to add custom handling for local hash calculating (e.g.
caches) derived classes should now override get_cached_unihash(), and
return the local unihash or None if there isn't one.
(Bitbake rev: 6faf48c09a4003a31b32e450779fb8ac9cc5e946)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implements a Client Pool derived from the AsyncRPC client pool that
allows querying for multiple equivalent hashes in parallel
(Bitbake rev: ba4c764d8061c7b88cd4985ca493d6ea6e317106)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an abstract base class that can be used to implement a pool of
client connections. The class implements a thread that runs an async
event loop, and allows derived classes to schedule work on the loop and
wait for the work to be finished.
(Bitbake rev: f113456417f9ac0a4b44b291a6e22ea8219c3a5f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds API to check if the server is aware of the existence of a given
unihash. This can be used as an optimization for sstate where a client
can query the hash equivalence server to check if a unihash exists
before querying the sstate cache. If the hash server isn't aware of the
existence of a unihash, then there is very likely not a matching sstate
object, so this should be able to significantly cut down on the number
of negative hits on the sstate cache.
(Bitbake rev: cfe0ac071cfb998e4a1dd263f8860b140843361a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the _execute() helper to execute queries. This helper does the
logging of the statement that was being done manually everywhere.
(Bitbake rev: 0409a00d62f45afb1b172acbcea17bf17942e846)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for removing unused unihashes from the database. This is
done using a "mark and sweep" style of garbage collection where a
collection is started by marking which unihashes should be kept in the
database, then performing a sweep to remove any unmarked hashes.
(Bitbake rev: 433d4a075a1acfbd2a2913061739353a84bb01ed)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Coming from a kernel background, I was thinking along the lines of
dmesg -n <integer>
for loglevel adjustments. So I tried various large and small and
even zero number values with no luck before getting frustrated and
opening up the python.
Let us save others the frustration and give a hint what the args it
expects should look like.
(Bitbake rev: df184b2a4e80fca847cfe90644110b74a1af613e)
Signed-off-by: Paul Gortmaker <paulg@kernel.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Create an ncurses version of the GTK app "taskexp.py".
* Add these additional features:
- Sort tasks in recipes by their dependency order
- Print individual and/or recipe-wide dependencies to a file
- Add a wild card filter
- Show the target recipes on BOLD
* Provide a GUI self test
* Provide a non-ncurses self test for ptest
(Bitbake rev: f49bec66ad51c8cddeceafbbe2445c46e396ee8b)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"bitbake world -n --setscene-only" shows poor performance as the numbers of tasks
increases. Analysys shows this is due to the "deferred" hard dependencies being
repeatedly processed which doesn't allow much forward progress in overall task
execution.
To avoid this, mark when it has been done and don't reprocess until dependencies
are updated. We have to be careful as we've seen bugs where these dependencies
aren't processed at the right time. They also have to be reproceseed during task
migration/rehashing so the code has to "self heal" the data structures.
(Bitbake rev: e5609bac06c17dabcf6286b47b1a3f19f5a1160f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A quick profile of bitbake world showed 147 million calls to taskname_from_tid().
The next_buildable_task function is performance senstive so move the call
inside the if block to reduce the number of calls and speed the code up.
(Bitbake rev: 8b332c16a7b6b85c5cbe1919dd8cae45fda6adf9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that runqueue performance profiling works again we can see a lot
of time is lost in build_taskdepdata. Whilst we can't compute that
in advance, we can compute the individual entries.
Therefore put a cache in place to compute those and save overhead in
starting up tasks.
(Bitbake rev: c4519b542702ba25023e53d77b275a6fa571ec50)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python 3 interprets string literals as Unicode strings, and therefore
\s is treated as an escaped Unicode character which is not correct.
Declaring the RegEx pattern as a raw string instead of unicode is
required for Python 3.
(Bitbake rev: a5f1bc69ef2e456bd163303a07cfb73825b01576)
Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure the message matches the filenames the code actually uses.
(Bitbake rev: deb7db2e2b125c6a6732db4f185f4de5926494fd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the "--user-agent" paramater in the wget base command to
perform all wget commands with this parameter, because a few
HTTP servers block requests with the default wget user agent.
For example, "hg.openjdk.org" never send a response to requests
have been sent with wget:
wget https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2
Resolving hg.openjdk.org (hg.openjdk.org)... 23.54.129.73
Connecting to hg.openjdk.org (hg.openjdk.org)|23.54.129.73|:443... connected.
HTTP request sent, awaiting response...
(Bitbake rev: d6fa261a9603677f0b3abbd309c1ca6073b63f4c)
Signed-off-by: Thomas Perrot <thomas.perrot@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the idle/main loop was added, we didn't include profiling information
for it. There is a performance issue in there, add logging for it.
(Bitbake rev: d8d5cd43a60560f67e86f4f625113b0f73b944c0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* github no longer provides support for subversion clients:
https://docs.github.com/en/enterprise-server@3.11/get-started/working-with-subversion-on-github/support-for-subversion-clients
it was shut down on 2024-01-08:
https://github.blog/2023-01-20-sunsetting-subversion-support/
and this test was now failing with:
======================================================================
ERROR: test_external_svn (bb.tests.fetch.SVNTest.test_external_svn)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/OE/layers/bitbake/lib/bb/tests/fetch.py", line 1287, in test_external_svn
fetcher.download()
File "/OE/layers/bitbake/lib/bb/fetch2/__init__.py", line 1896, in download
raise FetchError("Unable to fetch URL from any source.", u)
bb.fetch2.FetchError: Fetcher failure for URL: 'svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2'. Unable to fetch URL from any source.
Stdout:
Fetch svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2
Failed to fetch URL svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2, attempting MIRRORS if available
Fetcher failure: Fetch command export PSEUDO_DISABLED=1; /usr/bin/env svn --non-interactive --trust-server-cert co --no-auth-cache -r 2 file:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project/trunk@2 trunk failed with exit code 1, output:
A trunk/README.md
U trunk
Checked out revision 2.
svn: warning: W205011: Error handling externals definition for 'trunk/bitbake':
svn: warning: W170013: Unable to connect to a repository at URL 'https://github.com/PhilipHazel/pcre2.git'
svn: E205011: Failure occurred processing one or more externals definitions
in the rare cases where subversion was still installed on the host
running bitbake-selftest :).
to avoid this use still alive repository from https://svn.apache.org/
and pick something rather small and only the trunk subdirectory which
is fast to fetch:
svn co https://svn.apache.org/repos/asf/serf/trunk
takes just 2 sec here
adjust expected dir/file to use "protocols/fcgi_buckets.h" instead of
"trunk/README"
(Bitbake rev: a735898abcf056f897c9350bb128a5637e6b4617)
Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* when comparing 2 tmp/stamps/*do_configure.sigdata* I got
TypeError("filename must be a str or bytes object, or a file")
but both files I was comparing were
Zstandard compressed data (v0.8+), Dictionary ID: None
according to "file"
with TypeError catched to show which file it failed to open I got better
error which shows it was trying to read "do_prepare_recipe_sysroot.sigdata"
file now and after a while you might notice that it's not just the expected
file, but a dict with 'path', 'sstate', 'time'.
Fix that in bitbake-diffsigs but keep the TypeError and add OSError
in case it will eventually walk on file which isn't zstd compressed
pipecompress throws CompressionError.
ERROR: Failed to open {'path': '5.15.do_prepare_recipe_sysroot.sigdata.99b12a401341a0df7c3553cb00c87a7674295496bd5c25ed71764ee0d0fb8eb8', 'sstate': False, 'time': 1707136354.991718}: filename must be a str or bytes object, or a file
Traceback (most recent call last):
File "bitbake/bin/bitbake-diffsigs", line 192, in <module>
output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, recursecb, color=color)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/siggen.py", line 1039, in compare_sigfiles
recout = recursecb(dep, a[dep], b[dep])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/bin/bitbake-diffsigs", line 102, in recursecb
out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/siggen.py", line 857, in compare_sigfiles
raise err
File "bitbake/lib/bb/siggen.py", line 853, in compare_sigfiles
with bb.compress.zstd.open(a, "rt", encoding="utf-8", num_threads=1) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/compress/zstd.py", line 12, in open
return bb.compress._pipecompress.open_wrap(ZstdFile, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/compress/_pipecompress.py", line 59, in open_wrap
raise TypeError("filename must be a str or bytes object, or a file")
TypeError: filename must be a str or bytes object, or a file
* if I replace zstd file with just plaintext it fails with:
$ echo foo > foo
$ echo foo > bar
$ bitbake-diffsigs foo bar
zstd: /*stdin*\: unsupported format
ERROR: Process died with 1
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'>
with this change it shows the name of the file which it failed to uncompress:
$ bitbake-diffsigs foo bar
zstd: /*stdin*\: unsupported format
ERROR: Failed to open sigdata file 'foo': Process died with 1
ERROR: Process died with 1
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'>
(Bitbake rev: f3f843a4fd6e0a9e8f6edef5dd3cf1fce29c50ba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've added a new statement, inherit_defer, so bump the version so this
can be checked.
(Bitbake rev: 191e6eb2bceb467c97e315301f1f64722cf0e976)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for an inherit_defer statement which works as
inherit does but is only evaulated at the end of parsing.
This allows conditional expressions to be evaulated 'late' meaning changes
to PACKAGECONFIG from bbappends can be applied for example. This addresses
a usability/confusion issue users have with the current conditional
inherit mechanism since they don't realise the condition has to be
expanded immediately with the current model.
There is a commented out warning we could enable in future which
warns about the use of a conditional inherit that isn't deferred.
There is a behaviour difference in the placement of the inherit,
particularly around variables set using ?= which means wen swapping
from one to the other, caution must be used as values can change.
(Bitbake rev: 5c2e840eafeba1f0f754c226b87bfb674f7bea29)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where chains of RDEPENDS are multiple levels deep, the runall code was not
accounting for this and recursing deeply enough to gather all dependencies.
Fix this by iterating over the result until no more dependencies are found.
Tested-by: Jonas Gorski <jonas.gorski@bisdn.de>
Reported-by: Jonas Gorski <jonas.gorski@bisdn.de>
(Bitbake rev: 966f25dfc23a6d17b2b6d3e0100e9ae264f99025)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously printdiff code would iterate over tasks that were reported as invalid or absent,
trying to follow dependency chains that would reach the most basic invalid items in the tree.
While this works in tightly controlled local builds, it can lead to bizarre reports
against industrial-sized sstate caches, as the code would not consider whether the
overall target can be fulfilled from valid sstate objects, and instead report
missing sstate signature files that perhaps were never even created due to hash
equivalency providing shortcuts in builds.
This commit reworks the logic in two ways:
- start the iteration over final targets rather than missing objects
and try to recursively arrive at the root of the invalid object dependency.
A previous version of this patch relied relies on finding the most 'recent'
signature in stamps or sstate in a different function later, and recursively
comparing that to the current signature, which is unreliable on real world caches.
- if a given object can be fulfilled from sstate, recurse only into
its setscene dependencies; bitbake wouldn't care if dependencies
for the actual task are absent, and neither should printdiff
I wrote a recursive function for following dependencies, as
doing recursive algorithms non-recursively can result in write-only
code, as was the case here.
[YOCTO #15289]
(Bitbake rev: aadeca63da5d96160ce4d6d71da556e2e033f9b7)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In runCommand() the send() and recv() can fail and raise
BrokenPipeError and EOFError exceptions when the bitbake-server is
unexpectedly terminated. In these cases a python traceback is
currently dumped. Similarly updateFromServer() which calls
runCommand() can also raise these and other exceptions, and currently
lacks proper exception handling resulting in python traceback.
We wrap calls to runCommand() and updateFromServer() in a try/except
block as well as improve the exception handling for updateToServer().
This along with the earlier commit which added text to the
BrokenPipeError and EOFError exceptions in runCommand() to indicate a
bitbake-server termination may have occurred, should improve the user's
ability to understand and handle these errors.
An easy way to trigger each of the runCommand() exceptions is to
'kill -9' bitbake-server before (causes EOFError) or after
(causes BrokenPipeError) the "Loading Cache" stage.
(Bitbake rev: 804d366ee3ddc0f37f0a6c712c8d42db45b119bc)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Doing builds on systems with limited resources, or with high demand
package builds such as chromium it isn't uncommon for the OOM Killer
to be triggered and for bitbake-server to be selected as the process
to be killed. When the bitbake-server does terminate unexpectedly due
to the OOM Killer or otherwise, this currently results in a generic
python traceback with little indication as to what has failed.
Here we trap and raise the exceptions while extending the exception
text in runCommand() to make it clear that this is most likely caused
by the bitbake-server unexpectedly terminating.
Callers of runCommand() should be updated to properly handle the
BrokenPipeError and EOFError exceptions to avoid printing a python
traceback, but even if they don't, the added text in the exceptions
should provide some hints as to what might have caused the failure.
(Bitbake rev: 5ff62b802f79acc86bbd6a99484f08501ff5dc2d)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have two classes, both of which set EXPORT_FUNCTIONS for the same funciton
and a standard funciton definition for the function that is exported, the export
function can sometimes overwrite the standard one.
The issue is that the internal flag the code uses isn't ovweritten if the variable
is giving a new value. Fix the issue by using a comment in the code that is injected
so that we know if it is ours or not.
Also add some testing for EXPORT_FUNCTIONS, not perfect but a start.
(Bitbake rev: 66306d5151acb0a26a171c338d8f60eb9eb16c6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, there was duplicate clutter in the output, particularly if the
URI points to sstate cache items:
bb.fetch2.FetchError: Fetcher failure for URL: {uri}. URL {uri} doesn't work
(Bitbake rev: 61537b8a98b963e4af265e046d41407b32fa5935)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add test_lineno_in_eventhandler to test lineno in eventhandler.
(Bitbake rev: 4e5de537bebb68180c5755858c81b095eb9ae2f6)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So that we can get the correct error messages.
* In python 3.10.9, the error message was:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 4
# SPDX-License-Identifier: MIT
^^^^^
SyntaxError: invalid syntax
This is hard to debug since the error line number 4 is incorrect, but nothing
is wrong with the code in line 4.
* Now the error message and lineno is correct:
ERROR: Unable to register event handler 'defaultbase_eventhandler':
File "/path/to/poky/meta/classes-global/base.bbclass", line 256
an error line
^^^^^
SyntaxError: invalid syntax
And no impact on parsing time:
* Before:
$ rm -fr cache tmp; time bitbake -p
real 0m27.254s
* Now:
$ rm -fr cache tmp; time bitbake -p
real 0m27.200s
(Bitbake rev: c212933d9c786806852c87f188250a4f0a14c048)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Check some rows are visible in table instead of compare table row to
row_to_show, because recipe image table sometime doesn't display full avaiblable
images
(Bitbake rev: 1e2e5927ef7a8adfd3d0a3be1c75b4aa410d9908)
Signed-off-by: Alassane Yattara <alassane.yattara@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bump the version to 2.7.1 for the find_siginfo changes.
(Bitbake rev: 03995e16bf7186f5368f772f617d563f4d280641)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Even with the reworked printdiff code, sstate which is heavily used in parallel
can throw races at the tests: if a new matching, but otherwise unrelated
sstate signature appears between writing out local stamps and listing
matching sstate files, then that signature will be deemed 'the latest'
and the actual local stamp will be discarded. This change ensures
the scenario does not happen.
It also makes use of the reworked find_siginfo(), particularly the 'sstate'
entry in returned results.
(Bitbake rev: c8574b796dabb69699c70540dd95a44d8f7388ab)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
