There are several dependencies which recipes have which are not needed
at runtime, only at build time. Extend the list of these from bison to
include quilt, patch, meson, autoconf, automake and ninja which should
reduce the amount of data being included in native sysroots.
This speeds up and reduces the sstate needed for image builds for example.
If this change breaks recipes, it probably means they're missing an explicit
DEPENDS on something in this list which is a bug in the recipe.
(From OE-Core rev: 8e26252b45b7660c7c67c702411bdec187a76ffc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have a choice of policy with hashequivalence - whether to reduce
sstate duplication in the sstate feed to a minimum or have maximal
sstate reuse from the user's perspective.
The challenge is that non-matching outhashes are generated due to
determinism issues, or due to differences in host gcc version,
architecture and so on and the question is how to reconcile then.
The approach before this patch is that any new match is added and
matches can update. This has the side effect that a queried value
from the server can change due to the replacement and you may not
always get the same value from the server. With the client side
caching bitbake has, this can be suboptimal and when using the
autobuilder sstate feed, it results in poor artefact reuse.
This patch switches to the other possible behaviour, once a hash is
assigned, it doesn't change. This means some sstate artefacts may be
duplicated but dependency chains aren't invalidated which I suspect
may give better overall performance.
Update the tests to match the new behaviour.
(Bitbake rev: 20d6ac753efa364349100cdc863e5eabec8e5b78)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes the hashequivalence server to resolve the diverging report race
error. This error occurs when the same task(hash) is run simultaneous on
two different builders, and then the results are reported back but the
hashes diverge (e.g. have different outhashes), and one outhash is
equivalent to a hash and another is not. If taskhash was not originally
in the database, the client will fallback to using the taskhash as the
suggested unihash and the server will see reports come in like:
taskhash: A
unihash: A
outhash: B
taskhash: C
unihash: C
outhash: B
taskhash: C
unihash: C
outhash: D
Note that the second and third reports are the same taskhash, with
diverging outhashes.
Taskhash C should be equivalent to taskhash (and unihash) A because they
share an outhash B, but the server would not do this when tasks were
reported in the order shown.
It became clear while trying to fix this that single large table to
store all reported hashes was going to make these updates difficult
since updating the unihash of all entries would be complex and time
consuming. Instead, it makes more sense to split apart the database into
two tables: One that maps taskhashes to unihashes and one that maps
outhashes to taskhashes. This should hopefully improve the parsing query
times as well since they only care about the taskhashes to unihashes
table, at the cost of more complex INNER JOIN queries on the lesser used
API.
Note this change does delete existing hash equivlance data and starts a
new database table rather than converting existing data.
(Bitbake rev: dff5a17558e2476064e85f35bad1fd65fec23600)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevents `ResourceWarning: unclosed event loop` warnings when using the
synchronous client and python exits
(Bitbake rev: 8b95972bc04ce52a98c7780184af15a5e95f987b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When ptest-runner times out or otherwise fails, it tries to
call ptest-runner-collect-system-data, so install the script.
The script currently calls dmesg, df, free (which are provided
by busybox, etc.) and pstree (which is a sub-package of psmisc).
Add pstree as an RDEPENDS.
(From OE-Core rev: 4e6be3fb521b23cfc175d0c09725bcc3ebbc73b2)
Signed-off-by: Tim Orling <timothy.t.orling@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
libcidn has been dropped since glibc 2.28
(From OE-Core rev: cf83790728ad569af01300f793754c0108c78b4e)
Signed-off-by: Fred Liu <yclw3d2y@live.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After upgrade to 2.13.0, for arm32, with DEBUG_BUILD enabled, lttng-ust
build failed with error:
| /path/to/tmp/work/cortexa15t2hf-neon-poky-linux-gnueabi/lttng-ust/2_2.13.0-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi/../../libexec/arm-poky-linux-gnueabi/gcc/arm-poky-linux-gnueabi/11.2.0/ld: ../../../src/lib/lttng-ust/.libs/liblttng-ust.so: undefined reference to `_uatomic_link_error'
| collect2: error: ld returned 1 exit status
| Makefile:399: recipe for target 'test_ust_error' failed
| make[3]: *** [test_ust_error] Error 1
The problem has reported to upstream, and upstream suggests to use
-DUATOMIC_NO_LINK_ERROR for the failure case, refer [1].
[1]https://lists.lttng.org/pipermail/lttng-dev/2021-September/030056.html
(From OE-Core rev: 722d58bccd5616b3b3364845bbf1121d3cc0c3cd)
Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
https has been the preferred way to access websites for many years now.
Change all of the URLs with a _working_ https server/certificate to use
that URL.
(From yocto-docs rev: c77868c780df94356d5f21453f80ace073fade9b)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
SVG directly included in EPUB output has multiple issues,
in particular font size and alignment ones (tested on two
EPUB readers). Instead, using PNG, generated from SVG when available
as the primary format for images. GIF and JPEG are fine too.
(From yocto-docs rev: ff3876ca4910cf61bc25bfeb07e65ad6f6a93b48)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
html output is great when directly using SVG
epub output has to use exported PNG because of issues using SVG directly
latexpdf output cannot use SVG, it needs exported PDF instead
This adds rules to generate PNG and PDF formats from new SVG sources.
Newly added diagrams have be stored in svg/ subdirectories
so that PNG and PDF files generated from SVG can be "gitignored"
without ignoring the original PNG diagrams that have no SVG source.
Note: had to remove the dependency to "Makefile" in the final "catch-all" target,
otherwise it was also catching the SVG to PNG and SVG to PDF targets.
(From yocto-docs rev: cd9bbd41d323bc124057396f142010509c65bbaf)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Modern Debian based distros (such as Ubuntu) now
use "apt" instead of "apt-get"
Also make sure "apt" is invoked with root privileges (mandatory)
(From yocto-docs rev: 380ef2474f3ecc2bce83f11cd838d054bef56d22)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes font inconsistencies, mixing a ``text`` string
with another string with the regular font, where the whole string
makes more sense with a single font.
The result is also much simpler and will make future
searches easier.
(From yocto-docs rev: e6d1c166a97a1b6bb42ca49dfcbd3d39f8eadc16)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
https has been the preferred way to access websites for many years now.
Change all of the URLs with a _working_ https server/certificate to use
that URL.
(Bitbake rev: da543cdaf88a387675e25d3555765f1146e4105e)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
https has been the preferred way to access websites for many years now.
Change all of the URLs with a _working_ https server/certificate to use
that URL.
(From meta-yocto rev: 89de71da1c1e46d42a5c786bcac76462aa9c31e0)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the build is interrupted, handle the shutdown of pseudo even in this
case to avoid data corruption inside docker containers.
[YOCTO #14555]
(Bitbake rev: a2a04c6fe94bc56efcff299c669a151746e35916)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the changes to task accounting, bitbake doesn't show progress
when executing setscene tasks on the summary console output.
Change to show a progress within the setscene tasks and a progress
within the main tasks. I can't see any way to display this more easily
without confusing users.
[YOCTO #14586]
(Bitbake rev: 0244acb968eb593d2ad7bc6e52f222c2b1d39aa9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a configuration npmrc per npm environment to avoid repeated
creation of the same configuration file. Create the file via python to
avoid multiple npm config calls and add the ability to pass a file
path instead of a temporary file.
Deprecate the npm configs argument of the run function. The configs
should be passed to npm environment or as command specific arguments.
(Bitbake rev: 2c2df49b06a2bad7a5b8872a9998338a4660498f)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Delay directory restore and set execute/search directory mode bits in
unpack to support npm archives with a missing search directory mode.
(Bitbake rev: 60cbd34d3da8f0f523281aad7eec93eec9cd4db8)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cached revisions which are used to decide if a repository doesn't
need to be updated are misleading when used in conjunction with mirror
tarballs and can cause partial fetches to happen, resulting in unpack
errors as repositories were not fetched.
A concrete example: edk2-firmware in meta-arm is at version 202102
(ef91b0). This is built on the autobuilder so the source mirror contains
the repository as a mirror tarball. If I build edk2-firmware 202102 the
gitsm fetcher will initially download the top-level repository and then
iterate into the submodules to also fetch those repositories, including
cmocka from cryptomilk.org. edk2-firmware will then unpack and build
successfully.
I then update edk2-firmware to 202105 (e1999b) and build it.
Gitsm.needs_update() starts by calling Git.needs_update() which returns
False, as the mirror tarball contains this revision. It then looks at
the "nuggets" which are SRCREVs it has fetched before. The mirror
tarball itself contains the nugget for e1999b as this has been built on
the autobuilder, so needs_update return False, no more fetching is done,
and the build proceeds to unpack.
However, as part of the 202105 upgrade the URL of the cmocka submodule
changed, and this new repository was never fetched. This means that
unpack fails as one of the required git repositories isn't available.
The nugget codepaths appear to be an attempt at optimising the fetch
process, but have demonstratable failure cases. Just removing them
entirely solves the edk2-firmware example, and all of the fetcher test
cases still pass.
(Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If runfetchcmd() fails with bb.process.NotFoundError, the message output
is simply "Fetch command" which doesn't really explain what the problem
is.
Add "not found" to clarify what happened.
(Bitbake rev: 8de9dc02ed6a73b47f2ab10be30d1aed7954bc72)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set a prefix so the temporary directories are identifable.
(Bitbake rev: c3440b82cbe9c317f9961d61e12ea37fc9541ce0)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoids a warning from Python as \s isn't a valid escape.
(Bitbake rev: be39c2e926beebe78030fbe26c6737f08f960fcb)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit 282d596b8 added a fix for image.bbclass related to QEMU
dependencies, such fix made the older logic borrowed from
image.bbclass incompatible.
Update the dependency logic to reflect new changes, we no longer
need to add the do_addto_recipe_sysroot dependency specifically.
(From OE-Core rev: 4663f06c81285aeab8e8d33ec5338d7854b9dd9f)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandro@enedino.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
autobuilder.yoctoproject.org URLS no longer work. Update them to a
working location.
(From OE-Core rev: 56f7bac1f0d1ced41e6908706be27149aa7b87e2)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Multilib changes RECIPE_SYSROOT which can make the value in PSEUDO_IGNORE_PATHS
incorrect. Add the correct value, which fixes races over files in the sysroot.
[YOCTO #14581]
(From OE-Core rev: 64003e5e1b51c0cd561681b1ac13293546b8182b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On an aarch64 build host, vendor is found to be "unknown", on x86 systems
it is "pc". This filters through to the PLATFORM tag in target rpms.
We saw reproducibility test failures where the PLATFORM tags in noarch
rpms were changing depending upon which host built them. Forcing the
vendor value to a consistent one makes things deterministic.
(From OE-Core rev: f6434075b2bdfc23c683d22281b674b1e6abde77)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We hardlink some files into the build, such as licence files in
do_populate_lic tasks. Depending on the umask that the source tree
was checked out with, the group permissions would vary. This
results in inconsistent task outhashes.
Avoid this by ignoring the group/other bits unless we're under
pseudo context.
Bump the ABI numbers to ensure we don't see cache corruption from
earlier builds.
(From OE-Core rev: 2bd9c806de7e5197168360e3bd1d223a04a92291)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The upstream qemu recipe uses host's pkg-config files as a solution to
detecting host's SDL. This has a side effect of using other host
libraries that are later queried by the configure script. This can get
into a situation when the host provides libnfs (for example) and because
later this dependency is not in place anymore, qemu will fail at
runtime.
This change adds a PACKAGECONFIG definition for libnfs that is disabled
by default, in turn disabling the pkgconfig autodetection in configure.
(From OE-Core rev: 42b364a25fdbc987c85dd46b8427045033924d99)
Signed-off-by: Andrei Gherzan <andrei.gherzan@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
