A race condition has happened where the exiting server removed the PID
file between the existence check and the removal, resulting in a
FileNotFoundError exception.
The fix is to ignore the FileNotFoundError exception, the existence
check is now redundant so remove it to simplify.
Fixes [YOCTO #14341]
(Bitbake rev: 40d00bf9308e0bf73a00134a99a012a292daa1c5)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently when git fetches fail, it destroys all the existing local clone data.
For large repositories this can introduce long build delays when for example,
you just typo'd the git revision hash.
The git fetcher should be able to recover most directories so when the fetch is
for a git repo, avoid removing things unless clean is explicitly called
(e.g. a -c cleanall task).
(Bitbake rev: 1b3cd039fe19b24bd4be9a0202a98cdcbb0e9443)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for asyncrpc servers to send connection headers to clients
on connection. Since this is a breaking protocol change, clients must
opt-in to expect headers from the server, corresponding to a version
bump in the client protocol.
(Bitbake rev: 1cb2b8be6cc5269553f549285592e47b7d29db03)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a LayerBranch (a specific layer at a specific release) does not
exist in the layerindex database ("Layerindex Metadata"), the dependency
would throw a backtrace. Instead fail early and provide an error message.
Since layerindexlib will also check the local layers, inform the user that
a local checkout might resolve the situation. Recommend that they reach
out to the layer maintainers and layer index admins to properly fix it for
everyone.
While we are here, remove some trailing whitespace.
[YOCTO #15365]
[YOCTO #13954]
(Bitbake rev: 96cbe8f87209a927c157ebcf469f8b9d54fcf92e)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current implementation only performs a git lfs fetch alongside of a
regular git fetch. This causes issues when the downloaded revision is
already part of the fetched repository (e.g. because of moving back in
history or the updated revision already being part of the repository at
the time of the initial clone).
Fix this by explicitly checking whether the required LFS objects are
available in the downloade directory before confirming that a downloaded
repository is up-to-date.
This issue previously went unnoticed as git lfs would silently fetch the
missing objects during the `unpack` task. With network isolation turned
on, this no longer works, and unpacking fails.
(Bitbake rev: cfae1556bf671acec119a6c8bbc4b667a856b9ae)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a very rare case where the maxval is improperly computed
initially for cache loading progress, and the value will go over.
Explanation from bitbake/lib/bb/cache.py:736 in MulticonfigCache:__init__:progress()
# we might have calculated incorrect total size because a file
# might've been written out just after we checked its size
In that case, progressbar will receive a value over the initial maxval.
This results in a ValueError stack trace as well as bitbake returning 1.
Traceback (most recent call last):
File ".../poky/bitbake/lib/bb/ui/knotty.py", line 736, in main
cacheprogress.update(event.current)
File ".../poky/bitbake/lib/progressbar/progressbar.py", line 256, in update
raise ValueError('Value out of range')
ValueError: Value out of range
This fix mirrors the behavior of MulticonfigCache and accepts the new
value as the new maxval. This is also what the percentage printout
is doing in bitbake/lib/progressbar/progressbar.py:191 in ProgressBar:percentage()
I encountered this issue randomly while working on a project with
VSCode saving files while commands where fired.
Note: This file is a fork from python-progressbar. It hasn't been
refreshed in 8 years. We did only two commits, 5 years ago with minor
modifications. This new change is also not how the upstream project is
behaving.
(Bitbake rev: 7cea7f7a87da041fc1ad370c5c3d15aabad3a0d4)
Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some ditros don't enable /proc/pressure and it tends to be those which we
see bitbake timeout issues on, seemingly as load gets too high and the bitbake
processes don't get scheduled in for minutes at a time.
Add support for stopping running extra tasks if the system load average goes
above a certain threshold by setting BB_LOADFACTOR_MAX.
The value used is scaled by CPU number, so a value of 1 would be when
the load average equals the number of cpu cores of the system, under one
only starts tasks when the load average is below the number of cores.
This means you can centrally set a value such as 1.5 which will then
scale correctly to different sized machines with differing numbers
of CPUs.
The pressure regulation is probably more accurate and responsive, however
our graphs do show singificant load spikes on some workers and this
patch is aimed at trying to avoid those.
Pressure regulation is used where available in preference to this load
factor regulation when both are set.
(Bitbake rev: 14a27306f6dceb4999c2804ccae5a09cc3d8dd49)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Re-enable connection pooling in case `postgresql+psygopg` driver
is used. Async connection pooling is supported in psycopg 3 [psycopg]
driver in SQLAlchemy. Allow the connection pool to grow to
arbitrary size.
(Bitbake rev: 4fe05513b5314c201725e3f8ad54f58d70c56258)
Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 987ab2a446.
There's been a report that this breaks downloads from Jfrog Artifactory
as self.user_agent is set to 'Mozilla Firefox', and when Artifactory
sees that, it sends a response tailored for showing in an interactive browser
(which in my opinion it has every right to).
If we're using wget, we should say so via wget's default; handling uncooperative
servers should be done on per-recipe basis, and ideally with tickets
to admins of those servers.
(Bitbake rev: feef5cd12e877f42ffcace168d44b0e6eb80a907)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before, everything up to the last slash was removed when extracting the
names of the tags. This would lead to that a tag such as "agent/11.0.0"
would be incorrectly identified as "11.0.0", which would then be treated
as a correct version matching "^(?P<pver>\d+(\.\d+)+)".
(Bitbake rev: 8b21024b9966d5158ac4a77e87ffb935c2a57764)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases, the version found by latest_versionstring() may be higher
than the real version. Make it possible to specify a maximum version so
that this case can be detected.
(Bitbake rev: 9134d4777109bc78410c3e641420d9a78b485e33)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The purpose of ensuring 'incremental fetch' is not easy to see from
the codes. So add comments to explain this.
(Bitbake rev: 8b890b87e30cd05ec92ed71ee3691a47b4d77253)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We had issue when BB_ALLOWED_NETWORKS is set and `bitbake grpc-native -c fetch'
failed even with all contents available in PREMIRRORS.
Add a test case to ensure no regression in the future.
(Bitbake rev: 80c91ceb81b1cae203067af58d3f1fe9c619ae83)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For gitsm recipes, it's possible that some URL is used more than once.
e.g.,
A -> B:rev1 (B is a submodule of A)
A -> C (C is a submodule of A)
C -> B:rev2 (B is a submodule of C)
A anc C are both using B as submodules, but on different revs.
Now if we have:
B:rev1 -> D
B:rev2 -> E
Then, the mirror will not be fully used.
Say we have all repo mirrors for A, B, C, D, E, then in theory it's not
necessary to reach out to any network for downloading. But it's not the
case. After downloading B(rev1) and its submodule D from mirrors, the fetch
process continues to download C, thus B(rev2) and E. Now it finds that B
needs an update because its submodule E needs an update. Of course this is
true because E is not downloaded yet. Now the problem comes to whether to
use mirror or not. The git.py defines try_premirror to return 'False' when
the ud.clonedir exists. As B has been cloned, the ud.clonedir exists and
try_mirror returns False, resulting in not using mirror and going to upstream
directly.
We can see that the mirrors are not fully used. This is usually not problem,
as the cost is only some network download. But in case the following two
settings are there, we get errors.
BB_NO_NETWORK = "0"
BB_ALLOWED_NETWORKS = "*.some.allowed.domain"
In such case, the gitsm recipe A will fail to fetch. Note that all contents
that A needs are in mirrors and now it's failing to fetch. This is unexpected.
Note that the different revs of the same repo in gitsm recipe is not the only
way to reveal this problem. For example, there might be a recipe call B that
uses B:rev3. Check the protobuf and grpc recipes as an example.
For now, we can use the following steps to reproduce this issue. To be clear,
the grpc recipe in meta-oe is now 1.60.0.
1. Add in local.conf:
DL_DIR = "${TOPDIR}/downloads-premirror"
bitbake grpc -c fetch
2. Comment out the DL_DIR setting in local.conf and add the following lines:
PREMIRRORS:append = " \
git://.*/.* git://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
gitsm://.*/.* gitsm://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
"
3. Set BB_NO_NETWORK = "1" and then 'bitbake grpc -c fetch'.
This command succeeds and this shows that the premirror holds everything we need.
4. Add the following lines and then 'bitbake grpc -c fetch'.
BB_NO_NETWORK = "0"
BB_ALLOWED_NETWORKS = "*.some.domain"
After step 4, the error message is as below:
ERROR: grpc-1.60.0-r0 do_fetch: The URL: 'gitsm://github.com/protocolbuffers/protobuf.git;protocol=https;name=third_party/protobuf;subpath=third_party/protobuf;nobranch=1;lfs=True;bareclone=1;nobranch=1' is not trusted and cannot be used
This patch fixes this problem by handling this corner case, that is, if the URL is
not trusted from the settings of BB_NO_NETWORK and BB_ALLOWED_NETWORKS, then we should
try premirrors because trying to reach upstream is destined to fail.
(Bitbake rev: e1be272ad105b47d3131b77168d9172386993fcb)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fetch from premirror few times to emulate multiple machines sharing same
clonedir or few rebuilds of the package from (pre)mirror
Regression test for [Yocto #15369]
(Bitbake rev: 7fcbac574c68f16b95ab7abb2874931d168d3c9e)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix the issue with using of (pre)mirror in case if clonedir exists but
outdated.
Previous version of the code fetched new mirror content into FETCH_HEAD
instead of branch which caused refetch from the upstream. Add new remote
add fetch from it instead so the ref can be found by "_contains_ref"
Fixes [Yocto #15369]
(Bitbake rev: 69588e2a5c7c200e47b02b2391498dcb72388bd2)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When git is configured with safe.bareRepository=explicit [1], the
bitbake selftests fail miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].
It also seems that it is being rolled out in some environments as a
default for users.
In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.
The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.
Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: a45e14a7343e36101e45639931322e5649587f57)
Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When git is configured with safe.bareRepository=explicit [1], the
bitbake git fetcher fails miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].
It also seems that it is being rolled out in some environments as a
default for users.
In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.
The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.
While at it, fix one open-coded invocation of git that wasn't using
ud.basecmd
Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: 5f3b1d8dc9ee70e707536bd75ee845b547440c97)
Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hash Equivalence server performs unconditional insert also of duplicate
hash entries. This causes excessive error log entries in Postgres.
Rather ignore the duplicate inserts.
The alternate behavior should be isolated to the postgres
engine type.
(Bitbake rev: e8d2d178d0fe96f9d6031c97328e8be17d752716)
Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds API to query if unihashes are known to the server in parallel
(Bitbake rev: 7e2479109b40ce82507f73b4f935903f7f79fb06)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implements a new API called get_unihashes() that allows for querying
multiple unihashes in parallel.
The API is also reworked to make it easier for derived classes to
interface with the new API in a consistent manner. Instead of overriding
get_unihash() to add custom handling for local hash calculating (e.g.
caches) derived classes should now override get_cached_unihash(), and
return the local unihash or None if there isn't one.
(Bitbake rev: 6faf48c09a4003a31b32e450779fb8ac9cc5e946)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implements a Client Pool derived from the AsyncRPC client pool that
allows querying for multiple equivalent hashes in parallel
(Bitbake rev: ba4c764d8061c7b88cd4985ca493d6ea6e317106)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an abstract base class that can be used to implement a pool of
client connections. The class implements a thread that runs an async
event loop, and allows derived classes to schedule work on the loop and
wait for the work to be finished.
(Bitbake rev: f113456417f9ac0a4b44b291a6e22ea8219c3a5f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds API to check if the server is aware of the existence of a given
unihash. This can be used as an optimization for sstate where a client
can query the hash equivalence server to check if a unihash exists
before querying the sstate cache. If the hash server isn't aware of the
existence of a unihash, then there is very likely not a matching sstate
object, so this should be able to significantly cut down on the number
of negative hits on the sstate cache.
(Bitbake rev: cfe0ac071cfb998e4a1dd263f8860b140843361a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the _execute() helper to execute queries. This helper does the
logging of the statement that was being done manually everywhere.
(Bitbake rev: 0409a00d62f45afb1b172acbcea17bf17942e846)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for removing unused unihashes from the database. This is
done using a "mark and sweep" style of garbage collection where a
collection is started by marking which unihashes should be kept in the
database, then performing a sweep to remove any unmarked hashes.
(Bitbake rev: 433d4a075a1acfbd2a2913061739353a84bb01ed)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Create an ncurses version of the GTK app "taskexp.py".
* Add these additional features:
- Sort tasks in recipes by their dependency order
- Print individual and/or recipe-wide dependencies to a file
- Add a wild card filter
- Show the target recipes on BOLD
* Provide a GUI self test
* Provide a non-ncurses self test for ptest
(Bitbake rev: f49bec66ad51c8cddeceafbbe2445c46e396ee8b)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"bitbake world -n --setscene-only" shows poor performance as the numbers of tasks
increases. Analysys shows this is due to the "deferred" hard dependencies being
repeatedly processed which doesn't allow much forward progress in overall task
execution.
To avoid this, mark when it has been done and don't reprocess until dependencies
are updated. We have to be careful as we've seen bugs where these dependencies
aren't processed at the right time. They also have to be reproceseed during task
migration/rehashing so the code has to "self heal" the data structures.
(Bitbake rev: e5609bac06c17dabcf6286b47b1a3f19f5a1160f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A quick profile of bitbake world showed 147 million calls to taskname_from_tid().
The next_buildable_task function is performance senstive so move the call
inside the if block to reduce the number of calls and speed the code up.
(Bitbake rev: 8b332c16a7b6b85c5cbe1919dd8cae45fda6adf9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that runqueue performance profiling works again we can see a lot
of time is lost in build_taskdepdata. Whilst we can't compute that
in advance, we can compute the individual entries.
Therefore put a cache in place to compute those and save overhead in
starting up tasks.
(Bitbake rev: c4519b542702ba25023e53d77b275a6fa571ec50)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python 3 interprets string literals as Unicode strings, and therefore
\s is treated as an escaped Unicode character which is not correct.
Declaring the RegEx pattern as a raw string instead of unicode is
required for Python 3.
(Bitbake rev: a5f1bc69ef2e456bd163303a07cfb73825b01576)
Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure the message matches the filenames the code actually uses.
(Bitbake rev: deb7db2e2b125c6a6732db4f185f4de5926494fd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the "--user-agent" paramater in the wget base command to
perform all wget commands with this parameter, because a few
HTTP servers block requests with the default wget user agent.
For example, "hg.openjdk.org" never send a response to requests
have been sent with wget:
wget https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2
Resolving hg.openjdk.org (hg.openjdk.org)... 23.54.129.73
Connecting to hg.openjdk.org (hg.openjdk.org)|23.54.129.73|:443... connected.
HTTP request sent, awaiting response...
(Bitbake rev: d6fa261a9603677f0b3abbd309c1ca6073b63f4c)
Signed-off-by: Thomas Perrot <thomas.perrot@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the idle/main loop was added, we didn't include profiling information
for it. There is a performance issue in there, add logging for it.
(Bitbake rev: d8d5cd43a60560f67e86f4f625113b0f73b944c0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* github no longer provides support for subversion clients:
https://docs.github.com/en/enterprise-server@3.11/get-started/working-with-subversion-on-github/support-for-subversion-clients
it was shut down on 2024-01-08:
https://github.blog/2023-01-20-sunsetting-subversion-support/
and this test was now failing with:
======================================================================
ERROR: test_external_svn (bb.tests.fetch.SVNTest.test_external_svn)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/OE/layers/bitbake/lib/bb/tests/fetch.py", line 1287, in test_external_svn
fetcher.download()
File "/OE/layers/bitbake/lib/bb/fetch2/__init__.py", line 1896, in download
raise FetchError("Unable to fetch URL from any source.", u)
bb.fetch2.FetchError: Fetcher failure for URL: 'svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2'. Unable to fetch URL from any source.
Stdout:
Fetch svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2
Failed to fetch URL svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2, attempting MIRRORS if available
Fetcher failure: Fetch command export PSEUDO_DISABLED=1; /usr/bin/env svn --non-interactive --trust-server-cert co --no-auth-cache -r 2 file:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project/trunk@2 trunk failed with exit code 1, output:
A trunk/README.md
U trunk
Checked out revision 2.
svn: warning: W205011: Error handling externals definition for 'trunk/bitbake':
svn: warning: W170013: Unable to connect to a repository at URL 'https://github.com/PhilipHazel/pcre2.git'
svn: E205011: Failure occurred processing one or more externals definitions
in the rare cases where subversion was still installed on the host
running bitbake-selftest :).
to avoid this use still alive repository from https://svn.apache.org/
and pick something rather small and only the trunk subdirectory which
is fast to fetch:
svn co https://svn.apache.org/repos/asf/serf/trunk
takes just 2 sec here
adjust expected dir/file to use "protocols/fcgi_buckets.h" instead of
"trunk/README"
(Bitbake rev: a735898abcf056f897c9350bb128a5637e6b4617)
Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* when comparing 2 tmp/stamps/*do_configure.sigdata* I got
TypeError("filename must be a str or bytes object, or a file")
but both files I was comparing were
Zstandard compressed data (v0.8+), Dictionary ID: None
according to "file"
with TypeError catched to show which file it failed to open I got better
error which shows it was trying to read "do_prepare_recipe_sysroot.sigdata"
file now and after a while you might notice that it's not just the expected
file, but a dict with 'path', 'sstate', 'time'.
Fix that in bitbake-diffsigs but keep the TypeError and add OSError
in case it will eventually walk on file which isn't zstd compressed
pipecompress throws CompressionError.
ERROR: Failed to open {'path': '5.15.do_prepare_recipe_sysroot.sigdata.99b12a401341a0df7c3553cb00c87a7674295496bd5c25ed71764ee0d0fb8eb8', 'sstate': False, 'time': 1707136354.991718}: filename must be a str or bytes object, or a file
Traceback (most recent call last):
File "bitbake/bin/bitbake-diffsigs", line 192, in <module>
output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, recursecb, color=color)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/siggen.py", line 1039, in compare_sigfiles
recout = recursecb(dep, a[dep], b[dep])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/bin/bitbake-diffsigs", line 102, in recursecb
out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/siggen.py", line 857, in compare_sigfiles
raise err
File "bitbake/lib/bb/siggen.py", line 853, in compare_sigfiles
with bb.compress.zstd.open(a, "rt", encoding="utf-8", num_threads=1) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/compress/zstd.py", line 12, in open
return bb.compress._pipecompress.open_wrap(ZstdFile, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/compress/_pipecompress.py", line 59, in open_wrap
raise TypeError("filename must be a str or bytes object, or a file")
TypeError: filename must be a str or bytes object, or a file
* if I replace zstd file with just plaintext it fails with:
$ echo foo > foo
$ echo foo > bar
$ bitbake-diffsigs foo bar
zstd: /*stdin*\: unsupported format
ERROR: Process died with 1
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'>
with this change it shows the name of the file which it failed to uncompress:
$ bitbake-diffsigs foo bar
zstd: /*stdin*\: unsupported format
ERROR: Failed to open sigdata file 'foo': Process died with 1
ERROR: Process died with 1
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'>
(Bitbake rev: f3f843a4fd6e0a9e8f6edef5dd3cf1fce29c50ba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've added a new statement, inherit_defer, so bump the version so this
can be checked.
(Bitbake rev: 191e6eb2bceb467c97e315301f1f64722cf0e976)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for an inherit_defer statement which works as
inherit does but is only evaulated at the end of parsing.
This allows conditional expressions to be evaulated 'late' meaning changes
to PACKAGECONFIG from bbappends can be applied for example. This addresses
a usability/confusion issue users have with the current conditional
inherit mechanism since they don't realise the condition has to be
expanded immediately with the current model.
There is a commented out warning we could enable in future which
warns about the use of a conditional inherit that isn't deferred.
There is a behaviour difference in the placement of the inherit,
particularly around variables set using ?= which means wen swapping
from one to the other, caution must be used as values can change.
(Bitbake rev: 5c2e840eafeba1f0f754c226b87bfb674f7bea29)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where chains of RDEPENDS are multiple levels deep, the runall code was not
accounting for this and recursing deeply enough to gather all dependencies.
Fix this by iterating over the result until no more dependencies are found.
Tested-by: Jonas Gorski <jonas.gorski@bisdn.de>
Reported-by: Jonas Gorski <jonas.gorski@bisdn.de>
(Bitbake rev: 966f25dfc23a6d17b2b6d3e0100e9ae264f99025)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously printdiff code would iterate over tasks that were reported as invalid or absent,
trying to follow dependency chains that would reach the most basic invalid items in the tree.
While this works in tightly controlled local builds, it can lead to bizarre reports
against industrial-sized sstate caches, as the code would not consider whether the
overall target can be fulfilled from valid sstate objects, and instead report
missing sstate signature files that perhaps were never even created due to hash
equivalency providing shortcuts in builds.
This commit reworks the logic in two ways:
- start the iteration over final targets rather than missing objects
and try to recursively arrive at the root of the invalid object dependency.
A previous version of this patch relied relies on finding the most 'recent'
signature in stamps or sstate in a different function later, and recursively
comparing that to the current signature, which is unreliable on real world caches.
- if a given object can be fulfilled from sstate, recurse only into
its setscene dependencies; bitbake wouldn't care if dependencies
for the actual task are absent, and neither should printdiff
I wrote a recursive function for following dependencies, as
doing recursive algorithms non-recursively can result in write-only
code, as was the case here.
[YOCTO #15289]
(Bitbake rev: aadeca63da5d96160ce4d6d71da556e2e033f9b7)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In runCommand() the send() and recv() can fail and raise
BrokenPipeError and EOFError exceptions when the bitbake-server is
unexpectedly terminated. In these cases a python traceback is
currently dumped. Similarly updateFromServer() which calls
runCommand() can also raise these and other exceptions, and currently
lacks proper exception handling resulting in python traceback.
We wrap calls to runCommand() and updateFromServer() in a try/except
block as well as improve the exception handling for updateToServer().
This along with the earlier commit which added text to the
BrokenPipeError and EOFError exceptions in runCommand() to indicate a
bitbake-server termination may have occurred, should improve the user's
ability to understand and handle these errors.
An easy way to trigger each of the runCommand() exceptions is to
'kill -9' bitbake-server before (causes EOFError) or after
(causes BrokenPipeError) the "Loading Cache" stage.
(Bitbake rev: 804d366ee3ddc0f37f0a6c712c8d42db45b119bc)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Doing builds on systems with limited resources, or with high demand
package builds such as chromium it isn't uncommon for the OOM Killer
to be triggered and for bitbake-server to be selected as the process
to be killed. When the bitbake-server does terminate unexpectedly due
to the OOM Killer or otherwise, this currently results in a generic
python traceback with little indication as to what has failed.
Here we trap and raise the exceptions while extending the exception
text in runCommand() to make it clear that this is most likely caused
by the bitbake-server unexpectedly terminating.
Callers of runCommand() should be updated to properly handle the
BrokenPipeError and EOFError exceptions to avoid printing a python
traceback, but even if they don't, the added text in the exceptions
should provide some hints as to what might have caused the failure.
(Bitbake rev: 5ff62b802f79acc86bbd6a99484f08501ff5dc2d)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have two classes, both of which set EXPORT_FUNCTIONS for the same funciton
and a standard funciton definition for the function that is exported, the export
function can sometimes overwrite the standard one.
The issue is that the internal flag the code uses isn't ovweritten if the variable
is giving a new value. Fix the issue by using a comment in the code that is injected
so that we know if it is ours or not.
Also add some testing for EXPORT_FUNCTIONS, not perfect but a start.
(Bitbake rev: 66306d5151acb0a26a171c338d8f60eb9eb16c6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, there was duplicate clutter in the output, particularly if the
URI points to sstate cache items:
bb.fetch2.FetchError: Fetcher failure for URL: {uri}. URL {uri} doesn't work
(Bitbake rev: 61537b8a98b963e4af265e046d41407b32fa5935)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
