Currently, pseudo tracks all files referenced within its presence unless
they're listed in an exclusion list. The exclusion list has grown to be
fairly unwieldy.
This patch swaps PSEUDO_IGNORE_PATHS for PSEUDO_INCLUDE_PATHS which in
theory should be easier and more explicit to maintain.
This change does drop many directories from pseudo coverage including
/home and /tmp. There may be adapatations needed for recipes/classes
using pseudo in specific ways.
(From OE-Core rev: 2502da81709f25de499277b28d33c915638c45f6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As sstate_archive_package just calls tar, writing the function in shell
is actually more complex and opaque than the equivalent python.
Don't check for zstd vs pzstd, we have pzstd in HOSTTOOLS so it will
always be available.
(From OE-Core rev: c9ac5d9d94f254292cf3cafdf273dd6b61d3baa7)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code was changing the timestamps of the files in the do_package output,
particularly the files added for debug sources. This was to do two things:
a) make do_package sstate more reproducible
b) ensure better hash equivalence matching
Unfortuately the debug source files are hardlinks into the source tree for
efficiency so touching these, touches a lot of files in ${B} and ${S}. This
causes unpredictable effects if compile is run again for example, or could
cause compiling in the install task.
The hash equivalence matching is of key importance but we can mimic that
using clamping of the file timestamps in the depsig output used to generate
the hashes.
This patch drops the global timestamp clamping, instead allowing the files
to retain their creation timestamps into sstate. This makes do_package sstate
slightly less reproducibile. We could clamp the sstate timestamps but that
would lead to two different sets of timestamps depending on whether the
data came from sstate or not. I'd prefer to have consistent code behaviour,
rather than differing behavhour depending on whether data came from sstate
or not.
If we wanted to have reproducibiliy and fix the "corruption" of S/B and have
consistent codepaths, the only other option would be two copies of the
sources, which could end up huge and seems the least desireable option.
This patch therefore drops the timestamp clamping in the sstate files
and tweaks the depsig data generation to clamp the timestamps for do_package
instead since this seems the best compromise.
I validated that rpm/deb/ipk files still generate correctly as before.
(From OE-Core rev: 475759fdab7200488b2a568b2ba1aa31a456d113)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sstate functions currently pull in STAMP and SSTATE_PKG which
end up pulling in DEFAULTTUNE and other variables. The location on
disk encodes all the "architecture" information we need so clean up
the dependencies of these tasks and make them non-architecture specific.
(From OE-Core rev: 65df61ccff6781906449bfea386a8dd13112a51c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was deprecated with the introduction of postfunc support for tasks
in general and only used by buildhistory. Now that usage has been removed,
drop the code from sstate.bbclass. Any other users should be able to use
postfuncs too.
(From OE-Core rev: 74e08170a5584d83f5f03cd8a71978b5e0895c1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We planned to drop SSTATEPOSTINSTFUNC some time ago with the introduction of
postfuncs. Finally get around to doing that which should make the buildhistory
code a little more readable.
Unfortunately ordering the buildhistory function calls after the sstate ones is
difficult without coding that into the sstate class. This patch does that to
ensure everything functions as expected until we can find a better way. This is
still likely preferable than the generic sstate postfuncs support since the function
flow is much more readable.
(From OE-Core rev: c9e2a8fa2f0305ef1247ec405555612326f798f8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Unfortunately some recent patches caused non-deterministic output.
One input hash lead to both good and bad output and whilst that patch
has been fixed, the problematic hash 'cross' linkage remains. Bump to
a new sstate and hash equivalence version to avoid this and work from
a clean slate.
(From OE-Core rev: 639e42b9c14dff516688964dba4ab25bba7b8a55)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
buildhistory is showing issues where plaindirs installed files (such as package
listings) are not reliably being handled with installs from sstate.
The reason is that plaindirs is being handled after SSTATEPOSTINSTFUNCS
instead of before it, meaning the files visible in a non-sstate accelerated
code run are different to show from an accelerated run.
This can be observed by the missing files lists for packages in buildhistory, both
in from scratch builds and in builds from sstate. In builds where sstate is installed
over an existing build directory, the files are present though, so there is a
determinism problem.
Fix this by moving the code into sstate_install, this is the only call
site for the funciton.
Since the move needs prepdir, move that as well as it's call site,
being careful to handle the two different definitions of SSTATE_INSTDIR. The
version originally in the function was obsolete and was causing the postinstfuncs
to run in an incorrect directory. The only user is buildhistory and it wasn't
sensitive to cwd however so this happened not to cause a problem. Fix the
code to use the correct location.
(From OE-Core rev: 62ee349cf18532dac8736488752c00e89de78fcd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The only user was siteconfig which has been removed. The API was horrible
and we don't want to encourage this kind of usage.
(From OE-Core rev: cfbfd0b2e89eb71783c55a1be4a7e63e6cd82c2f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using tinfoil to control the build, multiple commands (serially) could
trigger an error such as:
When reparsing ....bb:do_package, the basehash value changed from ... to .... The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake esw-conf -cdo_package -Snone
ERROR: Then:
ERROR: $ bitbake esw-conf -cdo_package -Sprintdiff
However following these commands it was not able to be reproduced. Forcing
bitbake to dump the signatures and then running bitbake-diffsigs showed
that the value of _SSTATE_EXCLUDEDEPS_SYSROOT was being set in one run, but
was blank is a different version.
Upon inspecting the code in sstate.bbclass, one usage (without the _) is
already excludes, the leading _ version is used as a cache, only if set but
is not actually required to be defined. So ignoring the value should work
properly.
(From OE-Core rev: 4ec704ed6a1cfaf0a6c20f2038e7192e361ef590)
Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In testing websocket hashequivalence, corrupted sstate was injected into the
autobuilder extensively. With the new release/LTS, being able to clearly
differentiate between old and new sstate is probably desireable anyway
so bump the appropriate versions.
(From OE-Core rev: 7f107c180f592be29f57f580c60a6adbbebd7714)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The purpose of the change is to never sign a package not created by
the build itself.
sstate_create_package is refactored into Python and re-designed
to handle signing inside the function. Thus, the signing should never apply
to existing sstate packages. The function is therefore renamed into
sstate_create_and_sign_package.
The creation of the archive remains in a separate shellscript function.
Co-authored-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
(From OE-Core rev: ba223f8fff19ea59440d56cf3fe46200f3f71e22)
Signed-off-by: Tobias Hagelborn <tobias.hagelborn@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The vardepsexclude was subtly wrong in that it referenced STATE_MANMACH
when the actual variable name is SSTATE_MANMACH.
(From OE-Core rev: ec5054396f7fafea2a071d2695ae111fc585d6e6)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently use mkdir -p to create missing parent directories within SSTATE_DIR.
Reading the man page for mkdir mentions that parent directories are created with
the current umask, *not* the mode passed upon the commandline.
We could fix this by setting and resetting the umask but since we already have
decent python code able to do this, move to using that injecting a python function
into the chain of functions already present.
This should help fix the occasional sstate directory creation with the wrong
permissions.
[YOCTO #14385]
(From OE-Core rev: ae642a4b038c6946e6c8aa9778bf09099d938a31)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With recent changes to runqueue in bitbake, this horrible hack is no longer needed
and we can drop it.
(From OE-Core rev: f7f50b6013f7b636fe46448b1cbe1473de891527)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commands like "bitbake XXX -S printdiff" search for sstate files but don't download
them. This means that local files aren't touched as the download code would do, meaning
the sstate cleanup scripts can delete them. This can then lead to obtuse build failures.
Have the search code touch local files in the same way as the main code paths would to
avoid these files disappearing.
Move the function to a common touch() function in lib/oe instead of duplicating code.
(From OE-Core rev: a27fc0bd5706ab5b9c68a0271fcf57377a678cdf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When originally implemented, overlapping symlinks in DEPLOY_DIR were common. That
is no longer the case and these overlapping links are causing bugs in other areas
(e.g. bug 14123).
Therefore start showing errors for overlapping symlinks in shared areas. Whilst here,
fix a broken file reference in the grep command to match current file layouts and
update the message shown to users to match current times. Most of the message content
is obsolete now due to other advances and changes in the way the staging code
now works.
(From OE-Core rev: 15f98f72e1730fed5164fcd539737a7a7b96a1a2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently DEPLOY_DIR/licenses is added to SSTATE_ALLOW_OVERLAP_FILES. This
leads to bugs since when one MACHINE_ARCH recipes is cleaned, it removes the
files for another which then results in later build failures as license files
disappear.
The solution is to include SSTAGE_PKGARCH in the path names to the license files.
That does mean a search has to be used to find the correct license files for a
given PN but that can be done via SSTATE_ARCHS.
The implication for other tools is the layout has changed so tools will need to
adapt to the new paths. The benefit is no more strange build failures such as from
patterns like:
MACHINE=qemux86-64 bitbake core-image-minimal
MACHINE=genericx86-64 bitbake core-image-minimal
MACHINE=qemux86-64 bitbake linux-yocto -c clean
MACHINE=genericx86-64 bitbake core-image-minimal -C rootfs
[YOCTO #14123]
For anyone finding this commit, I'd question how much people should be relying on
this code for tooling and suggest the SPDX manifests should be the preferred data
format going forward anyway.
(From OE-Core rev: 1a4ab9fc26659507e678e87312b514e8ea515673)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When building multilibs, we need to inject the multilib sstate pkgarch
into SSTATE_ARCHS so the list forms a complete search path. Add a tweak
to do this.
PACKAGE_ARCH defaults to TUNE_PKGARCH so this is equivalent and just
guards against recipes changing the value which may have other unwanted
side effects.
(From OE-Core rev: 37126ffc7ccbd3df57ebbd8e581d158f03bb3b4c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PACKAGE_ARCH gets refined by each target so this value isn't valid in all contexts.
Tweak to use underlying variables to build it so that it remains valid in wider
contexts and works with new usage in sdpx and license classes.
(From OE-Core rev: ca90379147cb75755d205670781d8922161e9317)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that SRCPV isn't needed we can simplify things in a few places...
(From OE-Core rev: 843f82a246a535c353e08072f252d1dc78217872)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A recent package.py change has highlighted some corruption issues with -dev
pkgconfig package dependencies. Bump the output versions to trigger a rebuild
and ensure everything is consistent.
Take the opportunity to also drop all HASHEQUIV_HASH_VERSION entries since the
main version is changing.
(From OE-Core rev: f45ddfbf007de858327eef0ffefd5840ef4c69b8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add in the new 'check' calls which mean that the code can exit from longer
running code loops if requested by the user. This covers sstate checks and
sstate manifest cleanup code which currently couldn't be interrupted by the
user from the UI.
Increase the minimum bitbake version which brings in this API.
(From OE-Core rev: cb6c50689175668b66ffbe09ff8f250ba9a9034b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For the case of a global shared state cache and a local sstate cache
with a mix of signed and un-signed entries, there is a case where
a .sig is missing locally, but may exist in the global sstate cache.
For this case, do not just fail sstate fetch, but rather backfill
the .sig file from the global sstate cache.
If this case is detected, re-run the fetch operation to get the
global .sig file (if it exists).
(From OE-Core rev: d5ea1a5396bf9fd4303cae46bc0e042be8de8d67)
Signed-off-by: Tobias Hagelborn <tobias.hagelborn@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Transition to ThreadPoolExecutor (eb6a6820928472ef194b963b606454e731f9486f)
broke the
| Checking sstate mirror object availability: ...
progress bar because the removed 'thread_worker' was still referenced
in an asynchronous function. As the result of the future is never
read, the resulting backtrace is silently discarded.
Replace the information given to 'ProcessProgress' by a counter.
(From OE-Core rev: eacd52216815e0c75e5f257b6db792afd8b1b6df)
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_deploy_archives tasks don't need their dependencies so we can optimistion
this as we do for some other tasks.
(From OE-Core rev: 3dd9f6e398844380d3765c54d35afe0d2ccf82e7)
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After a recent fix in bitbake, the checksums may be checked for local file
fetches at an earlier code point. The underlying recipe checksums won't
match sstate values so ensure these aren't set via SRC_URI flags when
reconfiguring the fetch for sstate.
(From OE-Core rev: 315cfebe77a46c868f307919d3540046a6cb8587)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move classes to classes-global or classes-recipe as appropriate to take
advantage of new bitbake functionality to check class scope/usage.
(From OE-Core rev: f5c128008365e141082c129417eb72d2751e8045)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
