Previously if BUILD_REPRODUCIBLE_BINARIES was set to 0, the system would
fall back and select the default epoch (April 2011), but still perform
the reproducible build actions. This resulted in binaries that had an
unusually old date.
Simplify the functions and remove the anonymous python as no longer
necessary.
Also improve the documentation to better explain what the class is doing
and how a recipe can override the behavior if necessary.
(From OE-Core rev: 814bedacac9f5c343c9888c0b14649189a84f817)
Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1976013b026cfba94de32a13e994d92d7e9e39e5)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This sstate task is only needed when depended upon, it can be skipped
if there are no tasks running that directly depend upon it.
This reduced the number of sstate tasks in something like an image
build.
(From OE-Core rev: 884e44701ada57abe4d8ad9ece424435be25c6a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 601cee016da5c7505915e26641a085714de175ce)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating packages we build them with --clamp-mtime and use
SOURCE_DATE_EPOCH as the maximum mtime. This makes the end packages
reproducible. The data stored in sstate for do_package and the package
task doesn't benefit from this though and have varying timestamps.
This means their outhash varies and means hash equivalance isn't
effective at all and doesn't work as intended/desired.
We could create the sstate archives with the same clamping however
that would lead to different results depending on whether a task was
installed from sstate or not. Making that differ is a path to madness.
It also wouldn't fix the outhash of the task to be determninistic
without clamping of the date in the hash calculation code.
Instead, iterate over the files in sstate output and clamp them at
the code level. This isn't ideal but does make the file timestamps
determnistic everywhere and means we don't have to change the hash
calculation code.
This issue can be clearly seen looking at the do_package outhash for
a recipe which you then re-run the package task for after adding
something like whitespace to the install task. The outhash shouldn't
change but currently does.
(From OE-Core rev: 06b8f2a5a24be1a87f0eaf29fdba719ebe3bb06e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c3b3cc4745811b48b9193f83889946b2e1788932)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch is a fixup for 676757f "sstate: fix touching files inside pseudo"
running the 'id' command inside the sstate_unpack_package
function shows that this funcion run inside the pseudo:
uid=0(root) gid=0(root) groups=0(root)
The check for [ -w ${SSTATE_PKG} ] and [ -O ${SSTATE_PKG}.siginfo ]
will always return true and the touch can fail when the real user
don't have permission or in readonly filesystem.
As the documentation refers:
- the file test operator "-w" check if the file has write permission
(for the user running the test).
- the file test operator "-O" check if you are owner of file
We can avoid this test running the touch and mask any return errors
that we have.
(From OE-Core rev: 29fc85997ade490ae46ffca37ef8e1a56957c876)
(From OE-Core rev: e7d94a9cc5ab1b2c5d160fd06d643a4bc3409d26)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5b9210d66c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At the last nas outage, we realised that we don't have good mirrors of the
uninative tarball if our main system can't be accessed. kernel.org mirrors
some Yocto Project data so we've ensured uninative is there. Add the appropriate
mirror url to make use of that.
(From OE-Core rev: ebeb1458c7f24cd97978beb7cddf814cae43c6a2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update piglit to latest git revision and update the branch name,
since the original one is no longer updated. Make sure the VK
tests are only enabled if VK is also enabled in PACKAGECONFIG,
and that this is opt-in, otherwise older systems fail to build.
Cherry picked from squashed commits:
eb3a8d4c7b ("piglit: upgrade to latest revision")
a27b06f73a ("piglit: upgrade to latest revision")
bb091bc0be ("piglit: upgrade to latest revision")
394746d1cb ("piglit: upgrade to latest revision")
5aec8cff94 ("piglit: upgrade to latest revision")
fc4c82773d ("piglit: fix reproducibility")
6fbec0f12a ("piglit: update to latest revision")
8d23a0d498 ("piglit: upgrade to latest revision")
5144d515fe ("piglit: upgrade to latest revision")
dd085bd577 ("piglit: upgrade to latest revision")
9ba6df1b2c ("piglit: upgrade to latest revision")
1ccd71eb3e ("piglit: upgrade to latest revision")
(From OE-Core rev: 65aa83e25059b29563a45eacdad8178bc7a07fe4)
Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Cc: Anuj Mittal <anuj.mittal@intel.com>
Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
Cc: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add fcntl64 wrapper which hopefully fixes issues seen in findutils and the find
command in the libtool removal code when built with LFS compile flags on Gentoo.
(From OE-Core rev: c87d2dd8ec10d8164b0bff4307ea66f41b3360d4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f26867fe4daec7299f59a82ae4a0d70cceb3e082)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pulls in:
pseudo_db: Flush DB if there is a shutdown request
fcntl: Add support for fcntl F_GETPIPE_SZ and F_SETPIPE_SZ (test fix)
(From OE-Core rev: 649333a0d1bd5be4d5fb8a494e06686e08d291be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0882095d608ce3abbcc9814517434c21ea549063)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since file-native is ASSUME_PROVIDED magic.mgc is not being staged. As
a result diffoscope-native is failing with:
magic.MagicException: b'could not find any valid magic files!
Fix this by adding dependency on file-replacement-native
(From OE-Core rev: dcd8294f826f6e061cdd01c6c3594789ed46732e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the missing rdepends to fix below error:
# python3
[snip]
>>> import magic
[snip]
ModuleNotFoundError: No module named 'ctypes'
ModuleNotFoundError: No module named 'tempfile'
(From OE-Core rev: ba5562d34653fa6b5819dbc8ca80a42167c38c96)
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 115791844124bdddfbaec9d75bb887ef35c41f20)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffoscope changed the --exclude-directory-metadata option to require an
argument.
Add a test to validate that diffoscope is functioning as
expected to ensure that future upgrades do not unintentionally break
the reproducibility tests.
[YOCTO #14025]
(From OE-Core rev: a1087756221b8fcfe9c6903e678920f96e12292e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ea8fbcb7978ce48d7a9a83143d09402329535f86)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We had an issue where a webserver serving sstate had filesystem issues so
would accept connections but effectively not do anything with them. This
causes bitbake to hang whilst processing things like sstate objects inside
the checkstatus() calls. It can be replicated by setting up a server like:
socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null
and pointing SSTATE_MIRRORS in OE at that address.
Adding a timeout to the checkstatus calls of 30s means that whilst the
system will pause, it will then continue and not hang entirely. Since there
isn't a large transfer here, 30s should be a reasonable response time after
which we should fall back to building things ourselves.
[YOCTO #13716]
(Bitbake rev: c0348de8121c3a842bf44906f7e2f79e93f7275b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Following the scripted conversion adding branches to git://
SRC_URI entries, add the remaining references, mainly in the selftests
and recipetool.
(From OE-Core rev: 38fc0807eea14dc12610da4ba73c082d5a4b0744)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5340c0d688036c1be6c938f05d8a8c1e3b49ec38)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This update was made with the convert-scruri.py script in scripts/contrib
This script handles two emerging issues:
1. There is uncertainty about the default branch name in git going forward.
To try and cover the different possible outcomes, add branch names to all
git:// and gitsm:// SRC_URI entries.
2. Github are dropping support for git:// protocol fetching, so remap github
urls as needed. For more details see:
https://github.blog/2021-09-01-improving-git-protocol-security-github/
(From OE-Core rev: 827a805349f9732b2a5fa9184dc7922af36de327)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This script handles two emerging issues:
1. There is uncertainty about the default branch name in git going forward.
To try and cover the different possible outcomes, add branch names to all
git:// and gitsm:// SRC_URI entries.
2. Github are dropping support for git:// protocol fetching, so remap github
urls as needed. For more details see:
https://github.blog/2021-09-01-improving-git-protocol-security-github/
(From OE-Core rev: 904bdbab712e2f3c332c4d85d08ed83c957e249f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
vcs.pcre.org was a redirect to github which we use for subversion testing.
With the protocol changes at github and the removal of the redirect, use a
direct address for github.
(Bitbake rev: fa471399d41efdf61e95e0be541b45f0621756f3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6230ca71eb7eb2a6db162e28a01727d00af5299b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix:
lib/bb/tests/fetch.py:1288: DeprecationWarning: invalid escape sequence
for several lines of the fetch tests.
(Bitbake rev: bd8883d756328ca4c8f6bf97f77e17133a6bfb45)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9d84fd557a3fcbae2cdd70b24e69325ad737a01e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
github is dropping support for git protocol in Git urls. Add code to remap
this to https in a way that could be used in older bitbake versions.
(Bitbake rev: c222eddcebe892ae209aea7776cfc1147ac1df6e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
License-Update: new firmware files, copyright years, file names
(From OE-Core rev: f0d69f801d37ddb9bb82b2b5178f31538e5690cd)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit bfceaba4f38771047dfdfdfdbf16b794006dfd78)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changed the following line from:
[drm] Cannot find any crtc or sizes - going 1024x768 > [drm] Cannot find any crtc or sizes
This will expand the coverage of the failure to also cover the case when fallback size is not set.
(From OE-Core rev: 058f720448a79ada94671bfd93e77a4f0003382c)
Signed-off-by: Teoh Jay Shen <jay.shen.teoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0569fa735458512d6e15aa3315218ecbdf8510a3)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The controller module loading code needs to be told what directories
to search for modules via the target_modules_path keyword argument, which
is set to BBPATH.
However, as the actual module loading is done via importlib this relies
on the paths being on sys.path, which it is as base.bbclass puts each
layer's lib/ in sys.path.
Simplify the code by removing this indirection, and simply search
sys.path directly.
(From OE-Core rev: f2736f9a1156e23efbb20ea44a4aa81775ccbeba)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 570a19581f582f77e04d6892adb647cd649a6943)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sstate has been sufficiently invalidated since this change was made, drop
the workaround now.
(From OE-Core rev: 81b602eaab686fa5a523c023285f8ffd7050888d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 7a497d8f30f21bafc78d0f22f3442a9cc99544cb)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the function a ThreadedWorker is executing raises an exception, don't
use print() as that mostly disappears. Instead, output it to the logger.
This is done using bb.mainlogger.debug directly instead of bb.debug() as
this allows us to pass the exception instance directly, which is then
incorporated into the log stream.
(From OE-Core rev: 3bad17c8cf58b2bfc36a1de754fa47fb344bd5d8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2f1ea25c222b344dd8b784b2bc73a6540ab30274)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Python 3.8 visit_Num(), visit_Str() and so on are all deprecated
and replaced with visit_Constant. We can't yet remove the deprecated
functions until we require 3.8, but we can implement visit_Constant to
silence the deprecation warnings.
(From OE-Core rev: d91fe6ecb9fbb410b3bab6ced66b7fe5f869cf83)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 875a944392a3c93f40081a14af357f70b6b8264f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Python 3.8 visit_Num(), visit_Str() and so on are all deprecated
and replaced with visit_Constant. We can't yet remove the deprecated
functions until we require 3.8, but we can implement visit_Constant to
silence the deprecation warnings.
(From OE-Core rev: 18f10a51387cdee7c7058a3cb3f7c8c24c57c36a)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit abc93390a3f19bc4cc159c5690a478b9e2270906)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If gpg is used, it will find the first gpg agent in the path, this
may lead to issues where gpg comes from the host, and the agent
comes from a gnupg-native due to package signing. The versions
being out of sync causes gpg to fail.
(From OE-Core rev: f9649bca64771e845a74475b621a3c91ca0a6901)
Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2b59fb9d52a405a32a1d069d4c5320b72fbd35ce)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Exceptions print more clearly using repr() instead of str(), fix
in fetch and unpack tasks.
Drop part of the test which no longer makes sense after this change.
(From OE-Core rev: 045124ce7ef7c53a1932848835f93abbe535f157)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c94816259cc1c09746353ad26ca0c811e0c962c2)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code pattern isn't very pythonic, improve it to drop the unneeded
len() calls.
(From OE-Core rev: e0cecf7b4a4b966f11527a88e632997bdf74af96)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 69376ac1a6147b26fe1abaa4cf68414024814d63)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If checkstatus returns an exception we should silently handle
FetchError, as this means the fetch failed for 'normal' reasons such as
file not found. However, other exceptions may be raised, and these
should be made visible.
(From OE-Core rev: f16803718a19ace7b582c5b73924fe3112502927)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 50d99faf88a1d82cbd939b9bd6e33ebed2b1ffd8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a recipe applies patches which are in machine-specific override
directories, devtool will fail to fetch the patches that don't match the
default configuration. For example where there are patches at
qemux86/x86.patch and qemuarm/arm.patch:
SRC_URI = "file://source"
SRC_URI_append_qemuarm = " file://arm.patch"
SRC_URI_append_qemux86 = " file://x86.patch"
The patch apply phase sets OVERRIDES but does not set FILESOVERRIDES, so
it cannot find the patch files as the search path isn't correct. Fix
this by setting FILESOVERRIDES too.
Also when iterating through the overrides we need to be sure that other
overrides that are used are not enabled, so extend no_overrides instead of
simply appending the current override.
Fixes most but not all of [ YOCTO #14060 ].
(From OE-Core rev: a372cdf8e175423c47faeecc98ad076ee26bbec8)
(From OE-Core rev: b20699229671ef37daac8b0ed1133aacb477f6a0)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4a35bcc9d1)
Signed-off-by: Enrico Jorns <ejo@pengutronix.de>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
