The right hand side of dependencies in the task dependency file generated
by bitbake -g was missing multiconfig prefixes, corrupting the data. Fix
this.
[YOCTO #14621]
(Bitbake rev: c1938abf51b57938a21948bb414ad0467e4368d9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1d5ca721040c5e39aefa11219f62710de6587701)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous fix introduced a race where the queue might not be empty
but all the parser processes have exited. Handle this correctly to avoid
occasional errors.
(Bitbake rev: 8eaddb92a5fd14de6b5995aa92a6eed03b90a252)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8e7f2b6500e26610f52d128b48ca0a09bf6fb2cb)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If one of the parse threads disappears during parsing for some reason, bitbake
currently hangs. Avoid this (and zombie threads hanging around) by joining()
threads which have exited.
(Bitbake rev: 920111a330be59e5be2068a8f1a9edcbc6c14402)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit dc86a533d951d13643ce446533370da804782afc)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous fix for runall option handling had a small bug in it, it
didn't clear the originally processed task list which meant it was running
too many tasks. Fix this so the list is reset and rebuild correctly.
(Bitbake rev: 693eec8edf8d3b2b01c53be6776213cccd797485)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 87c9e120897ed04dfc64d4752fc602f9bfcb8645)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runbuild option handling in runqueue was flawed as items deleted from the
main task list may be dependencies and hence cause index errors.
Rather than modify runtaskentries straight away, compute a new shorted list
and use that as an input to the second phase. This avoids the need to add tasks
back to the list meaning delcount can be simplifed to a simple counter.
The second use case in runonly doen't re-add items so doesn't have this
issue.
(Bitbake rev: cc2e9c4800a8dfde24b3b5fa7184d0bb6398d4fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3428e3c54eb5cc03ff96f9cee6dc839afee7a419)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
vcs.pcre.org was a redirect to github which we use for subversion testing.
With the protocol changes at github and the removal of the redirect, use a
direct address for github.
(Bitbake rev: 85eb90edb4b912b4befb10128d60d342d0525eb3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6230ca71eb7eb2a6db162e28a01727d00af5299b)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the user specify an invalid upstream hash equivalence server in
BB_HASHSERVE_UPSTREAM notify the user that we can't connect the server.
(Bitbake rev: 7561fdc23f1aff370ead2abc5747c3a1c8b4ae4d)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit be45aeb9a84f30c28711e87e2d2a4a86320a8d94)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Username or password replacements in URIs were being appended rather than
replaced in mirror url remapping. Fix this and add a test case.
[YOCTO #13823]
(Bitbake rev: 85e7af227a48faec65838dcb7e73b17344bb2a0d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 66ad58bb87e5158aced572be4f1d5726bc97fcce)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We had an issue where a webserver serving sstate had filesystem issues so
would accept connections but effectively not do anything with them. This
causes bitbake to hang whilst processing things like sstate objects inside
the checkstatus() calls. It can be replicated by setting up a server like:
socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null
and pointing SSTATE_MIRRORS in OE at that address.
Adding a timeout to the checkstatus calls of 30s means that whilst the
system will pause, it will then continue and not hang entirely. Since there
isn't a large transfer here, 30s should be a reasonable response time after
which we should fall back to building things ourselves.
[YOCTO #13716]
(Bitbake rev: ba97caa58efe25bb62d2378fa52d21b6a6aa446c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit ae2b34285f8b3a1a3067c5e9b5d29e32e68c75f1.
Accidentally applied to the wrong branch.
(Bitbake rev: 1ac73638c1504cf2aa7f13257396aad617f25e8f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Operations like XXX:append += "YYY" are almost always wrong and this
is a common mistake made in the metadata. Show warnings for these usages
with a view to making it a fatal error eventually.
(Bitbake rev: ae2b34285f8b3a1a3067c5e9b5d29e32e68c75f1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The following commit to fix [Yocto #13039] causes regression of
the behavior of PREMIRRORS.
"bitbake: fetch2: fix premirror URI when downloadfilename defined"
Take meta-openembedded/meta-networking/recipes-protocols/freediameter/freediameter_1.4.0.bb
as an example.
SRC_URI = "\
http://www.freediameter.net/hg/${fd_pkgname}/archive/${PV}.tar.gz;downloadfilename=${fd_pkgname}-${PV}.tar.gz \
...
"
With the above commit, it now tries to fetch 1.4.0.tar.gz instead of
freeDiameter-1.4.0.tar.gz. This makes https://downloads.yoctoproject.org/mirror/sources
not work for freediameter, as it holds freeDiameter-1.4.0.tar.gz.
The commit above tries to avoid fetching from invalid url such as:
https://<some_mirror>/1.4.0.tar.gz/freeDiameter-1.4.0.tar.gz.
And its solution is to make basename to be 1.4.0.tar.gz, thus causing the
regression.
This patch fixes the above regression. For Yocto #13039, it now tries
to fetch from url: https://<some_mirror>/freeDiameter-1.4.0.tar.gz.
(Bitbake rev: 78949cf3fd31d8a408e93af7e27bcf26ae7942f4)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 96c30007dc0b32eee2b15771daec7948bc9bfd97)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test case test_fetch_premirror_use_downloadfilename_to_fetch to ensure
that 'downloadfilename' is used when fetching from premirror.
Although the other two previous test cases, test_fetch_premirror_specify_downloadfilename_regex_uri
and test_fetch_premirror_specify_downloadfilename_specific_uri already
implicitly contain such verification, we still need to add a very clear
case to ensure no regression.
(Bitbake rev: 057cbba6b7ade134e4fa3584b9e896be025a6f46)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 20aabc3d53f69949810ecf02295725db947ffef8)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When downloadfilename is specified, it is used to fetch from premirror.
So fix the test cases accordingly.
(Bitbake rev: af573273e4a5b73550af9639da18906f13bfa1a9)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3b4d2e3b5024324058360a2a28f33c34114218d0)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
github is dropping support for git protocol in Git urls. Add code to remap
this to https in a way that could be used in older bitbake versions.
(Bitbake rev: f19eefdaa5b43460f00d79d002f96112a6aa3c9a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen races where the socket may be gone but the server is still writing
out it's database. Handle that case too to avoid cleanup tracebacks.
[YOCTO #14440]
(Bitbake rev: 36b1b4c4fcee9dde628c7113203939730ab12ae5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b9e4fb843cb9d3a4d4404af093a781fab5520465)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have a choice of policy with hashequivalence - whether to reduce
sstate duplication in the sstate feed to a minimum or have maximal
sstate reuse from the user's perspective.
The challenge is that non-matching outhashes are generated due to
determinism issues, or due to differences in host gcc version,
architecture and so on and the question is how to reconcile then.
The approach before this patch is that any new match is added and
matches can update. This has the side effect that a queried value
from the server can change due to the replacement and you may not
always get the same value from the server. With the client side
caching bitbake has, this can be suboptimal and when using the
autobuilder sstate feed, it results in poor artefact reuse.
This patch switches to the other possible behaviour, once a hash is
assigned, it doesn't change. This means some sstate artefacts may be
duplicated but dependency chains aren't invalidated which I suspect
may give better overall performance.
Update the tests to match the new behaviour.
(Bitbake rev: 20d6ac753efa364349100cdc863e5eabec8e5b78)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes the hashequivalence server to resolve the diverging report race
error. This error occurs when the same task(hash) is run simultaneous on
two different builders, and then the results are reported back but the
hashes diverge (e.g. have different outhashes), and one outhash is
equivalent to a hash and another is not. If taskhash was not originally
in the database, the client will fallback to using the taskhash as the
suggested unihash and the server will see reports come in like:
taskhash: A
unihash: A
outhash: B
taskhash: C
unihash: C
outhash: B
taskhash: C
unihash: C
outhash: D
Note that the second and third reports are the same taskhash, with
diverging outhashes.
Taskhash C should be equivalent to taskhash (and unihash) A because they
share an outhash B, but the server would not do this when tasks were
reported in the order shown.
It became clear while trying to fix this that single large table to
store all reported hashes was going to make these updates difficult
since updating the unihash of all entries would be complex and time
consuming. Instead, it makes more sense to split apart the database into
two tables: One that maps taskhashes to unihashes and one that maps
outhashes to taskhashes. This should hopefully improve the parsing query
times as well since they only care about the taskhashes to unihashes
table, at the cost of more complex INNER JOIN queries on the lesser used
API.
Note this change does delete existing hash equivlance data and starts a
new database table rather than converting existing data.
(Bitbake rev: dff5a17558e2476064e85f35bad1fd65fec23600)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevents `ResourceWarning: unclosed event loop` warnings when using the
synchronous client and python exits
(Bitbake rev: 8b95972bc04ce52a98c7780184af15a5e95f987b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
https has been the preferred way to access websites for many years now.
Change all of the URLs with a _working_ https server/certificate to use
that URL.
(Bitbake rev: da543cdaf88a387675e25d3555765f1146e4105e)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the changes to task accounting, bitbake doesn't show progress
when executing setscene tasks on the summary console output.
Change to show a progress within the setscene tasks and a progress
within the main tasks. I can't see any way to display this more easily
without confusing users.
[YOCTO #14586]
(Bitbake rev: 0244acb968eb593d2ad7bc6e52f222c2b1d39aa9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a configuration npmrc per npm environment to avoid repeated
creation of the same configuration file. Create the file via python to
avoid multiple npm config calls and add the ability to pass a file
path instead of a temporary file.
Deprecate the npm configs argument of the run function. The configs
should be passed to npm environment or as command specific arguments.
(Bitbake rev: 2c2df49b06a2bad7a5b8872a9998338a4660498f)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Delay directory restore and set execute/search directory mode bits in
unpack to support npm archives with a missing search directory mode.
(Bitbake rev: 60cbd34d3da8f0f523281aad7eec93eec9cd4db8)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cached revisions which are used to decide if a repository doesn't
need to be updated are misleading when used in conjunction with mirror
tarballs and can cause partial fetches to happen, resulting in unpack
errors as repositories were not fetched.
A concrete example: edk2-firmware in meta-arm is at version 202102
(ef91b0). This is built on the autobuilder so the source mirror contains
the repository as a mirror tarball. If I build edk2-firmware 202102 the
gitsm fetcher will initially download the top-level repository and then
iterate into the submodules to also fetch those repositories, including
cmocka from cryptomilk.org. edk2-firmware will then unpack and build
successfully.
I then update edk2-firmware to 202105 (e1999b) and build it.
Gitsm.needs_update() starts by calling Git.needs_update() which returns
False, as the mirror tarball contains this revision. It then looks at
the "nuggets" which are SRCREVs it has fetched before. The mirror
tarball itself contains the nugget for e1999b as this has been built on
the autobuilder, so needs_update return False, no more fetching is done,
and the build proceeds to unpack.
However, as part of the 202105 upgrade the URL of the cmocka submodule
changed, and this new repository was never fetched. This means that
unpack fails as one of the required git repositories isn't available.
The nugget codepaths appear to be an attempt at optimising the fetch
process, but have demonstratable failure cases. Just removing them
entirely solves the edk2-firmware example, and all of the fetcher test
cases still pass.
(Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If runfetchcmd() fails with bb.process.NotFoundError, the message output
is simply "Fetch command" which doesn't really explain what the problem
is.
Add "not found" to clarify what happened.
(Bitbake rev: 8de9dc02ed6a73b47f2ab10be30d1aed7954bc72)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set a prefix so the temporary directories are identifable.
(Bitbake rev: c3440b82cbe9c317f9961d61e12ea37fc9541ce0)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoids a warning from Python as \s isn't a valid escape.
(Bitbake rev: be39c2e926beebe78030fbe26c6737f08f960fcb)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When writing functions into shell scripts, write then in a deterministic
order. This is unlikely to affect anything at runtime but it does change
the signatures of the generated useradd postinst scripts in OE-Core and is
a good thing to be consistent about in general.
(Bitbake rev: 8a181dc8f3c8c9c9885ea3011cb234321a296d92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need a command genetating an event to test through the tinfoil API. The
current test has IO load issues so add a dummy version which won't have
the IO constraints.
(Bitbake rev: a144178584394ea0700ffc2f2bfac94e8690effc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The diff output isn't deterministic at the moment as the sets can have differing
ordering. Sort the output so it is consistent.
(Bitbake rev: 117830c1d7ef3e53052fa326e1ca62c5c3946c45)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a race between sending the TaskFailed event which could trigger
the UI to look at the logs and flushing and closing the log files.
Reverse the order of the finally clause and the exception handling
to ensure we've handled the logfiles before sending the task events.
This should fix a race seen in bblogging.BitBakeLogging.test_python_exit_logging
(Bitbake rev: 032190aac31604d37740d8aecf6e74a5448de358)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we see things like:
Log data follows:
| DEBUG: Executing python function do_pythontest_exit
| DEBUG: Python function do_pythontest_exit finished
| ERROR: 1
| This is python stdout
Whilst after the change we see things like:
Log data follows:
| DEBUG: Executing python function do_pythontest_exit
| This is python stdout
| DEBUG: Python function do_pythontest_exit finished
| ERROR: 1
since the output is now correctly mixed with the log messages. In some cases the logging
tests indicate the output is being lost entirely which is bad for debugging and makes
things rather confusing.
(Bitbake rev: 8966b43761500e0505333d8c9a3f0f2c3dbe7559)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes the peg revision functionality we use in the svn fetcher is
not the correct option. Add a parameter nopegrevision which can be added
to disable this behaviour.
[YOCTO #6258]
(Bitbake rev: ea26682b706f655a8e418f56bebe742e2d130a1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a "None" value gets into the varlist, it doesn't display properly.
Ensure we don't add one to have the exception display properly.
(Bitbake rev: ee26e258888114143e66330c256b5bfe7d071c53)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix:
lib/bb/tests/fetch.py:1288: DeprecationWarning: invalid escape sequence
for several lines of the fetch tests.
(Bitbake rev: 9d84fd557a3fcbae2cdd70b24e69325ad737a01e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
os.rename will overwrite the destination file if present so we can use this
instead of the process call overhead.
(Bitbake rev: b3cccaa6a896c41d8c9be5eebc327f726542d16b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a multiconfig starts with a digit, users would see pages of
errors as we use the multiconfig as a python function name prefix
and python functions cannot start with a digit. We could avoid doing
that but it is easier just to ask users to name multiconfigs not
starting with digits.
This tweak ensures the user sees an easier to understand error.
(Bitbake rev: f9cddaeef35b2ea0dadf717101ed896f6b857abd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make usage of Bitbake's s3 fetcher more flexible with different AWS profiles
and switch between profiles as export of the AWS_PROFILE environment variable
(Bitbake rev: 0f35dc4dfc829fe9c51c239d15567f86c5c14e58)
Signed-off-by: Valentin Danaila <drlv85@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
