The switch to using json has messed up the type handling as the code
does assume that set()s are present. Add a decoder to reconstruct
the set() objects. Also fix the change of tuples to lists for the
file checksums and fix an existing type bug where dicts insteads of
lists was used.
Drop some old siginfo format handling code which is now long since
obsolete.
(Bitbake rev: 2d704842c0928f8dbe78fd081042aa7280af96be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen races where the socket may be gone but the server is still writing
out it's database. Handle that case too to avoid cleanup tracebacks.
[YOCTO #14440]
(Bitbake rev: b9e4fb843cb9d3a4d4404af093a781fab5520465)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
npmsw(and maybe other fetchers) may try to download same files simulatenously.
if a premirror is set, download method tries premirror location first. If file
is not found, done is set to false. In the meantime, other fetchers may download
the files required and verify_donestamp might return true.
In this scenerio, fetcher doesn't try to download again and fails. Do not
attempt to verify_donestamp if done is not set. By this way, we ensure that
fetcher attempts to do another download after premirror failure.
(Bitbake rev: 4fa25714916e84f99ecd22cb76cb5adada01e5e8)
Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If multiple npmsw fetchers are trying to download the same npm file, one of them
can try to download the file while other is calling verify. npmsw methods gets
called without holding the lock, which causes race conditions in fetching and
verification etc. Lock the lockfile before calling proxy fetcher methods.
(Bitbake rev: fa39e6689d0f0fff772e1c81682698f4b1587b8a)
Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since OE is about to change to zstd compression of sstate, it would make it
timely to convert the siginfo files from pickle which isn't reproducible
to json which is both reproducible and also human readable. At the same time
add zstd compression. This makes the siginfo files smaller, reprodubicle
and easier to debug.
Backwards compatibility mixing the two formats hasn't been supported since
in reality if sstate changes at the same time, files will be in one format
or the new one but comparing mixed formats won't make much sense.
Since json doesn't support sets, we translate them into lists in the files
themselves. We only use sets in bitbake since it makes things easier in
the internal code, sorted lists are fine for the file format.
[YOCTO #13973]
(Bitbake rev: 22c18494c9072788e6e26eb73de70378ae5c5bf5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code works fine if we pass a file descriptor in and we need to
do this from the siggen code so add that as a valid input.
(Bitbake rev: fc8d74b7de576005ecf67920501dc7e694880fda)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some debugging, BB_CURRENTTASK is set too late to be useful as
it isn't present in some event handlers for example. There is no
other way to know which task is actually running so set the value
earlier.
(Bitbake rev: 7daaaaa27f55b5a458656857c6d61a51b34a62fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The checkstatus semantics are unconventional, so document them.
(Bitbake rev: efb949020215b580e7dc3694c377b18df2fd7e9c)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevents `ResourceWarning: unclosed event loop` warnings when using the
synchronous client and python exits
(Bitbake rev: 8b95972bc04ce52a98c7780184af15a5e95f987b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
https has been the preferred way to access websites for many years now.
Change all of the URLs with a _working_ https server/certificate to use
that URL.
(Bitbake rev: da543cdaf88a387675e25d3555765f1146e4105e)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the changes to task accounting, bitbake doesn't show progress
when executing setscene tasks on the summary console output.
Change to show a progress within the setscene tasks and a progress
within the main tasks. I can't see any way to display this more easily
without confusing users.
[YOCTO #14586]
(Bitbake rev: 0244acb968eb593d2ad7bc6e52f222c2b1d39aa9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a configuration npmrc per npm environment to avoid repeated
creation of the same configuration file. Create the file via python to
avoid multiple npm config calls and add the ability to pass a file
path instead of a temporary file.
Deprecate the npm configs argument of the run function. The configs
should be passed to npm environment or as command specific arguments.
(Bitbake rev: 2c2df49b06a2bad7a5b8872a9998338a4660498f)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Delay directory restore and set execute/search directory mode bits in
unpack to support npm archives with a missing search directory mode.
(Bitbake rev: 60cbd34d3da8f0f523281aad7eec93eec9cd4db8)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cached revisions which are used to decide if a repository doesn't
need to be updated are misleading when used in conjunction with mirror
tarballs and can cause partial fetches to happen, resulting in unpack
errors as repositories were not fetched.
A concrete example: edk2-firmware in meta-arm is at version 202102
(ef91b0). This is built on the autobuilder so the source mirror contains
the repository as a mirror tarball. If I build edk2-firmware 202102 the
gitsm fetcher will initially download the top-level repository and then
iterate into the submodules to also fetch those repositories, including
cmocka from cryptomilk.org. edk2-firmware will then unpack and build
successfully.
I then update edk2-firmware to 202105 (e1999b) and build it.
Gitsm.needs_update() starts by calling Git.needs_update() which returns
False, as the mirror tarball contains this revision. It then looks at
the "nuggets" which are SRCREVs it has fetched before. The mirror
tarball itself contains the nugget for e1999b as this has been built on
the autobuilder, so needs_update return False, no more fetching is done,
and the build proceeds to unpack.
However, as part of the 202105 upgrade the URL of the cmocka submodule
changed, and this new repository was never fetched. This means that
unpack fails as one of the required git repositories isn't available.
The nugget codepaths appear to be an attempt at optimising the fetch
process, but have demonstratable failure cases. Just removing them
entirely solves the edk2-firmware example, and all of the fetcher test
cases still pass.
(Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If runfetchcmd() fails with bb.process.NotFoundError, the message output
is simply "Fetch command" which doesn't really explain what the problem
is.
Add "not found" to clarify what happened.
(Bitbake rev: 8de9dc02ed6a73b47f2ab10be30d1aed7954bc72)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set a prefix so the temporary directories are identifable.
(Bitbake rev: c3440b82cbe9c317f9961d61e12ea37fc9541ce0)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoids a warning from Python as \s isn't a valid escape.
(Bitbake rev: be39c2e926beebe78030fbe26c6737f08f960fcb)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When writing functions into shell scripts, write then in a deterministic
order. This is unlikely to affect anything at runtime but it does change
the signatures of the generated useradd postinst scripts in OE-Core and is
a good thing to be consistent about in general.
(Bitbake rev: 8a181dc8f3c8c9c9885ea3011cb234321a296d92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need a command genetating an event to test through the tinfoil API. The
current test has IO load issues so add a dummy version which won't have
the IO constraints.
(Bitbake rev: a144178584394ea0700ffc2f2bfac94e8690effc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The diff output isn't deterministic at the moment as the sets can have differing
ordering. Sort the output so it is consistent.
(Bitbake rev: 117830c1d7ef3e53052fa326e1ca62c5c3946c45)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a race between sending the TaskFailed event which could trigger
the UI to look at the logs and flushing and closing the log files.
Reverse the order of the finally clause and the exception handling
to ensure we've handled the logfiles before sending the task events.
This should fix a race seen in bblogging.BitBakeLogging.test_python_exit_logging
(Bitbake rev: 032190aac31604d37740d8aecf6e74a5448de358)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we see things like:
Log data follows:
| DEBUG: Executing python function do_pythontest_exit
| DEBUG: Python function do_pythontest_exit finished
| ERROR: 1
| This is python stdout
Whilst after the change we see things like:
Log data follows:
| DEBUG: Executing python function do_pythontest_exit
| This is python stdout
| DEBUG: Python function do_pythontest_exit finished
| ERROR: 1
since the output is now correctly mixed with the log messages. In some cases the logging
tests indicate the output is being lost entirely which is bad for debugging and makes
things rather confusing.
(Bitbake rev: 8966b43761500e0505333d8c9a3f0f2c3dbe7559)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes the peg revision functionality we use in the svn fetcher is
not the correct option. Add a parameter nopegrevision which can be added
to disable this behaviour.
[YOCTO #6258]
(Bitbake rev: ea26682b706f655a8e418f56bebe742e2d130a1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a "None" value gets into the varlist, it doesn't display properly.
Ensure we don't add one to have the exception display properly.
(Bitbake rev: ee26e258888114143e66330c256b5bfe7d071c53)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix:
lib/bb/tests/fetch.py:1288: DeprecationWarning: invalid escape sequence
for several lines of the fetch tests.
(Bitbake rev: 9d84fd557a3fcbae2cdd70b24e69325ad737a01e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
os.rename will overwrite the destination file if present so we can use this
instead of the process call overhead.
(Bitbake rev: b3cccaa6a896c41d8c9be5eebc327f726542d16b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a multiconfig starts with a digit, users would see pages of
errors as we use the multiconfig as a python function name prefix
and python functions cannot start with a digit. We could avoid doing
that but it is easier just to ask users to name multiconfigs not
starting with digits.
This tweak ensures the user sees an easier to understand error.
(Bitbake rev: f9cddaeef35b2ea0dadf717101ed896f6b857abd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make usage of Bitbake's s3 fetcher more flexible with different AWS profiles
and switch between profiles as export of the AWS_PROFILE environment variable
(Bitbake rev: 0f35dc4dfc829fe9c51c239d15567f86c5c14e58)
Signed-off-by: Valentin Danaila <drlv85@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This becomes a hard error in python 3.10.
(Bitbake rev: ae219e1f7460077f4492b31ac91cef4cf9b17277)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These no longer even work, and it's much better to just see all warnings
and fix them as they happen.
(Bitbake rev: 62d96ea2afbb0cfdb0d1f932dc96643e7bcd7f26)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a potential race over the mirror tarballs where a partial git repo
could be extracted causing fetcher failures if the tarball is being rewritten
whilst another build accesses it.
Create the mirror tarball atomically to avoid this.
[YOCTO #14441]
(Bitbake rev: 3250bc950c56bd7dd2114df26e5a8e13b04ceac8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent fixes to merge setscene and normal task accounting in runqueue
fixed some display issues but broke the task numbering of setscene tasks.
Add new accounting methods to the stats structure specifically designed
for setscene. This accounts for the fact that setscene tasks can rerun
multiple times in the build.
Then use the new data in the UI to correctly display the numbers the
user wants to see to understand progress.
(Bitbake rev: ed7e2da88bf4b7bfc7ebfc12b9bd6c0fb7d8c1aa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher would fail with:
Could not find a fetcher which supports the URL: ftps://...
(Bitbake rev: 9e56710c7203b1ec6cbefa758c81b69b697fe1a4)
Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The hash equivalence server has had the option to support a read-only
upstream server for some time now when launched as a standalone program,
but there was no way to set the upstream when using a locally started
server. Add a new variable called BB_HASHSERVE_UPSTREAM that can be used
to specify an upstream server when a local hash equivalence server is
used (e.g. BB_HASHSERVE is "auto")
(Bitbake rev: 250fa17f1391ff1ee01ab9b51d2a4f9aa35c1d1e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we parallelised normal and setscene tasks, the task stats
handling was left separate pending further thought. We had to remove
handling of the setscene tasks from the UI in order to maintain
consistent task numbering.
Currently, "0 of 0" tasks can be shown as setscene tasks execute
until the first normal task runs.
The only use left for sq_stats is in the active task numbers which
we can use the length of sq_ive for instead. We can therefore
drop it and return stats in all cases. This removes the "0 of 0" task
problem since the stats in all normal and setscene tasks matches.
[YOCTO #14479]
(Bitbake rev: eae6e947e37e18cded053814bd2a268b44fb25cd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is no BBLAYERS set in bblayers.conf show a more helpful
error and exit.
[YOCTO #14340]
(Bitbake rev: 97183e10faf9862b5d9489d6e2c27ac77c3b697d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the core layer is missing from bblayers.conf, the message the user sees is
hard to understand. Improve it.
[YOCTO #14340]
(Bitbake rev: 5815a7258ebb8a989e0c6f5798853559d9413f02)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Users sometimes put ${S} references in ${SRC_URI} without realising this can be
problematic. Improve the error messages if they accidentally do.
[YOCTO #11593]
(Bitbake rev: 89e0b19ec0b245a6cd414088904c91808e8814ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are circular task references, error on them rather than show
a recursion error. A simple reproducer is:
"""
do_packageswu () {
:
}
addtask do_packageswu after do_image_complete before do_image_qa
"""
into image_types.bbclass. There is code in runqueue to detect these but
we never get that far with the current codebase.
[YOCTO #13140]
(Bitbake rev: 339d4d6be515a71311b81fb9e99742af0d8a5130)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With "bitbake -v", for task failures you'd see the log output twice. Avoid
this by using the existing "did we print info" switch.
(Bitbake rev: e2c1afda4cb8023ed4ffeb5dc5bee4f0055659a8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the caller is piping the logs, they likely don't want them in the error exception
as well. This removes duplicate output from the build output allowing the UI level
controls on whether to show logs to work correctly.
(Bitbake rev: fc58ad84a9deb2620ad90611684dad65dafedb11)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
