Add tests to verify that SRC_URI dependency sniffing works correctly.
(From OE-Core rev: 394b98f7d77c199a4a022447ec5d722ffb7d1741)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* The kernel-fitimage class adds a do_assemble_fitimage_initramfs task
regardless of INITRAMFS_IMAGE_BUNDLE setting, which in some cases can
result in that task running after do_uboot_assemble_fitimage and
overwriting the u-boot-spl.dtb file with the pristine version (without
public key). Fix this by making do_uboot_assemble_fitimage dependant
on both do_assemble_fitimage_* tasks, regardless of the aforementioned
setting.
* Adjust 'type' and 'os' on the U-boot fitimage its script so that
mkimage/dumpimage can recognize them.
* Move the deployment of the u-boot-spl-nodtb files outside of
concat_spl_dtb_helper(), so that we can better isolate the scenarios
of creating an (unsigned) U-Boot fitimage versus also signing it. This
prevents some stale files from being deployed in the images directory.
* Remove any u-boot-fitImage and u-boot-its files from build tree, in
case the build tree is being reused across bitbake calls.
(From OE-Core rev: dc26d35e0935f30af55a3d2cb5c501d1b5c35437)
Signed-off-by: Klaus Heinrich Kiwi <klaus@linux.vnet.ibm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Derived from the similar kernel fitImage sign testcase, the U-Boot
fitImage testcases exercises the following fitimage.FitImageTest
scenarios:
* test_uboot_fit_image - create unsigned U-Boot fitImage
* test_uboot_sign_fit_image - create unsigned U-Boot fitImage in
addition to signed Kernel fitImage
* test_sign_standalone_uboot_fit_image - Create signed U-Boot fitImage
without a Kernel fitImage
* test_sign_cascaded_uboot_fit_image - Create and sign U-Boot and
Kernel fitImages
(From OE-Core rev: e71e4c617568496ae3bd6bb678f97b4f73cb43d8)
Signed-off-by: Klaus Heinrich Kiwi <klaus@linux.vnet.ibm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have a working theory that IO queues on the autobuilder are impacting
runtime testing under qemu, particularly async writes which inice does not
influence. We already pass the snapshot option to qemu which copies the
image and runs out of the copy. Add in the ability to copy the image to
a specificed location which can be a tmpfs. This means that writes to the
image would no longer be blocked by other writes to disk in the system.
Preliminary tests show that this does improve the qemu errors at the expense
of sometimes showing qemu startup timeouts as on a loaded system with a large
test image, it can take longer than 120s to copy the image to tmpfs. Having
a most consistent failure mode for loaded tests is probably desireable though.
(From OE-Core rev: fd1c26ab426c3699ffd8082b83d65a84c8eb8bff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If oe-selftest is run with -j, the output to stdout/stderr is being
lost at present. Capture this and display it upon test failure. We
have code that previously tried to enable this but it wasn't functioning
correctly. This should give more usable error reports on the autobuilder.
This code will mix stdout and stderr as the output is streamed from the test
server without markup. This is most in keeping with subunit/testools though
and the easiest way to handle the various challenges here as far as I can
see.
(From OE-Core rev: 6a954ce5834c8026adecff8478c3d827640bc647)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Each time I look at this code I get confused about what the different
variables represent. Rename a few of them to better indicate what they
represent.
(From OE-Core rev: e39d97c0b191add9281bac463ca059685288c81a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since pyc cache files are really system specific, no real reason to copy or
generate them during the eSDK build process. Also generating them has the
possibility of re-using inodes that pseudo may have been tracking, leading
a build failure.
(From OE-Core rev: ce8eba263647ae63a722122e28f26af46ae083a0)
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The run-postinsts package runs post installation scripts
on target if packages request delayed post installations. When
no delayed post installations are found the sysV style scripts
are disabled for the package and hence it did not run on sysV
based systems. However, the package provides systemd service
as well which still ran on systems based on systemd even when
no post installations were found.
Rather than disabling/masking scripts for different initialization
managers we now simply remove/uninstall the run-postinsts package
when no post installations are found to be delayed till runtime.
This is also more aligned with the function (_uninstall_unneeded)
this functionality is triggered through.
(From OE-Core rev: 627fb3181edd71502fbdf96549c41b2dea027250)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For the purpose of image construction using du on a rootfs directory isn't
entirely satisfactory. Bare "du" will report the actual disk usage so file
systems which can compress the data will report less than the actual space
required. Using "du --apparent-size" will report the actual space used, but as
this simply sums the bytes used for content across an entire file system can
result in significant under-reporting due to block size overhead.
Attempt to solve these problems by implementing our own function to calculate
how large a rootfs will be. This function handles hardlinks correctly but
rounds up all sizes to multiples of the block size (currently, 4KB is the
hard-coded block size).
(From OE-Core rev: 6ca53ad7b26ee2b4e6d2c121c6f6d6eed7f6b56f)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current test code contains a subtle race. It is building an ext4,
then trying to reuse that image in a wic build for the same recipe.
Whilst this happened to work, the code is within its rights to clean
up the output before the wic code runs.
Avoid this by using separate image targets instead.
(From OE-Core rev: 590398080e81fb5e2f81e12b8900858837dfe25f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CVE_VERSION_SUFFIX in "patch" to treat version string with suffix "pX"
or "patchX" as patched release.
also update testcases to cover this changes and set CVE_VERSION_SUFFIX
for sudo.
(From OE-Core rev: 8076815fc2ffc8f632e73527ce2b7d158a29e9ea)
Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The hardcoded path issue was fixed, drop the exclusion.
(From OE-Core rev: 58e02c83dcf4d6c40c206a0371ef5d7f8db2b272)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The meson determinism issue is the same frozenset issue we encountered with
python itself. Remove the problematic pyc file until upstream work out the
best way to address the issue (as with core python).
(From OE-Core rev: ad00107dc02b3211f5d1a7fe889f538ee2bc064d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This test is to keep track of recipes that do not have a HOMEPAGE or DESCRIPTION,
with exceptions to recipes whose SRC_URI are not linked to external sources.
Results of this test: it take about 2 to 3 minutes for this test to complete execution
depending on the laptop used, Recipes that didn't have a homepage or description were found,
and using list data structure instead of set() returned an ordered list of recipes that
didn't have a homepage. Hence it's easier to view recipes with missing homepage and description.
[YOCTO #13471]
(From OE-Core rev: 5e0cfce7a126c194bfb7855952355c62eae3831a)
Signed-off-by: Dorinda Bassey <dorindabassey@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
go-dep was an effort for dependency management before go modules, which
since 2020 has been deprecated in favor of go modules. Since its not
developed any longer and go mdules is officially supported, this should
be retired from OE-core as well.
(From OE-Core rev: 1e7ed44d87034446f1d07692c9378c3b0a8a9dd3)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Cc: Otavio Salvador <otavio.salvador@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have fixes for perf reproducibility, so we can drop it from the
exclusion list.
(From OE-Core rev: 2989779fa244ec2c2c714c27f67753eefeeaf4af)
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some platforms do not use ttyS* for their serial consoles (e.g., qemuarm
and qemuarm64). The hardcoding of this can cause issues. Modify
runqemu to use the serial consoles defined in SERIAL_CONSOLES instead of
hardcoding.
(From OE-Core rev: 9dea4cd2f9f46ab3a75562639a22d8f56b4d26af)
Signed-off-by: Jon Mason <jon.mason@arm.com>
Change-Id: I746d56de5669c955c5e29d3ded70c0a4d3171f17
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a recipe that explicitly searches /usr/include, and use that in
oe-selftest to verify that host include paths are correctly causing
build failures.
(From OE-Core rev: b3e3eba796b843021b264f0e98dc30f983775d58)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit adds an optional layerid argument which can
be passed to the bitbake-layers create-layer command.
This allows for creation of a layer with a layer id
different to that of the layer's name. The default
behaviour of the command where the layer's id is set
to the layer's name is still retained.
(From OE-Core rev: 8f896bd9b34f19535838757c8f3049bae438e4fe)
Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The help message for the optional argument of setting
the priority of the new layer was incorrect.
(From OE-Core rev: d10d928c6030951994bc2404cde78137c86917a4)
Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under systemd weston's socket is in /run, not in /run/user/0.
(From OE-Core rev: ea7dfd857ce3779745dccda0cbfd93b5e5de1a36)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are build paths which are in generated sources injected
into ${PN}-src. These are in generated files and don't influence the
binary output. As such we can correct them at do_package time by
tweaking in an appropriately injected function. This fixes
the reproducbility of the recipe.
(From OE-Core rev: cc1de818247c9ae329fdc0536658b3968a4caea2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a patch adding sorting to a couple of points in the Makefiles
which removes most of the determinism issues in ltp.
Build swapon before the main build to ensure libswapon.o is built
deterministically as it races with swapoff.
All issues reported on the upstream mailing list.
(From OE-Core rev: 0f51f9a37e5d058bce28cfe7b9a32a895f83c091)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change adds core-image-weston to the reproducible build test.
For this we also need to update the static-group and static-passwd.
Besides the world package build which includes the wayland package,
this adds a wayland/weston-based image to the reproducible build test.
(From OE-Core rev: ffc4de47988ccf7568eecc8a27e8964beeaaacfb)
Signed-off-by: Jan-Simon Moeller <jsmoeller@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a hack to hardcode in specific rpaths which we then remove,
allowing the build to be reproducible.
Strip build patches out of one of the test scripts too.
(From OE-Core rev: 18299a114c66280ba2d00becc3fae235d3a6cbd7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a patch to address two determinism issues and allow reproducible
builds.
Also strip full paths we don't need out of some ptest files,
we can use the installed binaries.
(From OE-Core rev: 14e7cc6c6040b983024bec56fbd32d9101507ff0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When debugging why a single recipe doesn't reproduce, its a pain
to wait for the world to rebuild from scratch. Update the selftest
to allow this to be configured, for example you could set
targets as ['perf'] and sstate_targets as ['virtual/kernel']
and then it should only be rebuilding perf in the test rather than
things like the toolchain (parts of the kernel may be unavoiable
as they're not in sstate).
Can be run as:
OEQA_DEBUGGING_SAVED_OUTPUT=/tmp/perf-diffoscope oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds
to save diffoscope output.
(From OE-Core rev: 132a17d02f29711572e14a2f38a841323fbb6df6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
LTO likely doesn't buy us much here, disable it to allow the binaries
to be reproducible.
(From OE-Core rev: b4b5e52b5bd98deb0855700e4aae6228d52e5a83)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add sorting to the globbing within the Makefile to make the output
reproducible.
(From OE-Core rev: c43ae151f572786818fe048233b4bbfd6b0ba2cf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove hardcoding the build configuration into the help/version output
from swig to make the binaries reproducible.
(From OE-Core rev: 7ed28ae9717ea9dad4e131012186d5f08e8f0bec)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add sorting to wildcard expansion in the makefile to make builds
reproducible.
(From OE-Core rev: 5541ba76ccc0c416f315bc0dc14a20a33059bd5f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The library hardcodes paths to the python library internally and currently
these are build paths. Fix this to use the correct target path and
fix reproducibility in the process.
(From OE-Core rev: 4c190e8c9a2b53de3def1065fb6e479b72d040dc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to use CFLAGS with the correct WORKDIR in them, replace those
in the sysroot file with the ones appropriate to the current recipe.
(From OE-Core rev: 45edf189961aff1858be9bb7b63116073c0a0c10)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Configure was swallowing our cflags meaning the resulting binaries were
not reproducible. Tweak configure not to do that and fix reproducibility.
(From OE-Core rev: 1395269a6fae5cec01d721e2ad52bd7cb8fc38ce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a configuration option to pass in srcdir, removing hard coded
build paths from the binaries.
(From OE-Core rev: d7e92e3a22f0c87aff4f452b51f50ec417d2949b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The compressed docs contained a timestamp of the original file which
meant the SDE clamping during package creation didn't work. The
benefits of compression are minor, decompress the files to avoid
the reproducibility issues.
(From OE-Core rev: a3b59a67bea16899b57a0d187120c988495da4d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Distros may not use shadow for login manager
secondly login --help returns 1 only with shadow provided login
utility, with busybox it returns normal 0, the testcase clearly
assumes shadow being part of image, therefore add needed
checks to ensure the pre-requisites are met
(From OE-Core rev: 0820aeba74869bf6e85b200f39bc610a43ab9a7a)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Calling sync between each file compare is horrible performance wise
as we compare thousands of files. We don't care about IO latency here
so disable.
(From OE-Core rev: 25f78abd8bbeb201fd9452e7983e015027954948)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't use tbe BUILDINFO line of host information in the Makefile
so remove it for reproducibility.
(From OE-Core rev: a9742595fa90d4977fdd8129a4fe4932ddb96a18)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't want to encide BUILD_ROOT into target packages. This is used
for build time tests but in our case those would be on target anyway
do use the target paths.
(From OE-Core rev: 2818dc90bd62487879e7da2c68357b39282415b0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
config_data was including a build system path to pkg-config, fix
this.
(From OE-Core rev: 7b8f56224c12a40f4f0aca3f612198deb933af4d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
configure inspects the host's /etc/group for these configuration
options, fix this to the correct values by using configure options.
(From OE-Core rev: f16f9c727569414cd52862dcba18d8e423f4e961)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* not sure if there are some valid use-cases for missing manifest, but
recently I'm seeing increasing number of build failures where something
from native recipe is missing (seen it with pseudo, autoconf, nodejs
recently) and the only indication that something is wrong (before showing
sometimes misleading error like:
recipe-sysroot-native/usr/bin/node: No such file or directory
is this warning:
NOTE: Running task 7844 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_prepare_recipe_sysroot)
NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Started
WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
NOTE: Running task 7845 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_unpack)
NOTE: recipe nodejs-12.20.2-r0: task do_unpack: Started
WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Succeeded
if I rebuild that native dependency, then it gets fixed and I don't
see these failures in clean builds (as without sstate and with empty
TMPDIR), only in incremental builds
* but if there isn't valid reason for missing manifest file, then I think
it would be better to error early (or even bb.fatal())
(From OE-Core rev: 95607a26854d873399d2b9d7e5fcbffc0cbdba4c)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
rpm packaging has shown good numbers wrt reproducible builds as well
when testing core-image-minimal locally:
Reproducibility summary for rpm: same=4671 different=0 missing=0 total=4671
Thus enable it in the autobuilder to be more widely tested.
(From OE-Core rev: 84d3a90557444d8cd83d780b1c3f2b278ab07af2)
(From OE-Core rev: 85b9dd01f2e1d5429f8da0c48a25ec034acebf67)
Signed-off-by: Jan-Simon Möller <jsmoeller@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
