pseudo-native is a bit special. It conditionally compiles in support for
xattr, statx and statvfs amongst other options. If a pseudo-native binary is
used on a system where these functions are present but it wasn't compiled in
we see hard to debug permissions problems.
An example is the devtool.DevtoolExtractTests.test_devtool_deploy_target
oe-selftest which shows a cryptic error:
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/devtool.py", line 1388, in test_devtool_deploy_target
self.assertEqual(filelist1, filelist2)
File "/usr/lib64/python3.9/unittest/case.py", line 831, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib64/python3.9/unittest/case.py", line 1037, in assertListEqual
self.assertSequenceEqual(list1, list2, msg, seq_type=list)
File "/usr/lib64/python3.9/unittest/case.py", line 1019, in assertSequenceEqual
self.fail(msg)
File "/usr/lib64/python3.9/unittest/case.py", line 670, in fail
raise self.failureException(msg)
AssertionError: Lists differ: ['-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8'] != ['-rwxr-xr-x root root /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8']
First differing element 0:
'-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor'
'-rwxr-xr-x root root /etc/init.d/mdmonitor'
This is due to a version of pseudo without statx being used on a system where
ls uses statx, hence the files are displayed as 6000.6000 instead of root.root.
Avoid this by always building pseudo-native for the specific distro in question rather
than using a universal sstate feed.
This hopefully fixes one of the mysterious AB-INT issues.
(From OE-Core rev: 6e3785a3f1f3cf68f5fe101cd6bebe91db165973)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating directories in the sstate-cache, explicitly set the mode
passed to mkdir to 0775 so that the directories are group writable, as
otherwise they cannot be shared with other users.
(From OE-Core rev: 1b0624dffe1b8496533c86dfed873112c8c0a01b)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In contrast to 'touching' a file without parameters 'touch -a' can only be
performed by the file owner. In case of a shared sstate-cache served
locally (e.g. over NFS), where the user running bitbake is not the
owner of sstate-cache files, even if he has write access on group level,
the sstate_unpack_package task will fail.
Checking if the file is owned by the user before attempting to run
'touch -a' on it solves this.
(From OE-Core rev: f1554e8bd40c640fd58daac18ecf7b458e754109)
Signed-off-by: Norman Stetter <norman.stetter@garz-fricke.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prior to fetching, the system checks if the sstate file is present
either locally or on the mirror. If it is, then it goes to the fetch
stage. Up to three files can be fetched, sstate, sstate.siginfo and
sstate.sig (if signature validation is enabled).
The previous pstaging_fetch function would iterate over these, and if
a download error occurred would spew forth a great amount of fetcher
failure messages as well as stop fetching the next item in the set.
This was resolved by adding a fetcher.checkstatus() call prior to
the download. If the file isn't present, then the exception will
be triggered, and no fetcher failure messages will reach the user.
The exception handler is then modified to be a pass so that it will
loop and pull the rest of the files that that are requested.
Additionally, a check for the existance of the .sig file was added
to the sstate_installpkg to avoid an error trying to load the .sig
if it wasn't downloaded.
(From OE-Core rev: a9085140434e2d26c0bb75bb53fcb7f7c19ef86d)
Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It may be that a file is not allowed to be touched, e.g., if it is a
symbolic link into a global sstate cache served over NFS.
(From OE-Core rev: 462e3f505217c9b9e0d2c1cbdafb0c8910bb6a59)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now this function has a summary parameter we can drop this check.
It could well be why the mysterious "locked sigs" selftest fails
intermittently if this function were called with a single hash
to check.
[YOCTO #13605] (with luck)
(From OE-Core rev: 02aabe0e59f73bf206d9bada1e7089832ceed254)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When moving to python3, divison using '/' now returns float instead of
an integer. In upstream commit b8025e9720
sstate filenames longer than the limit are changed to just include
necessary info + 3 fields just for information. The space left over
after the necessary info is divided into 3 for each of the fields.
Using '//' instead to do the division to solve the following error
message:
avail = (254 - len(hash + "_" + taskname + extension) -
len(components[0]) - len(components[1]) - len(components[5]) -
len(components[6]) - 7) / 3
> components[2] = components[2][:avail]
components[3] = components[3][:avail]
TypeError: slice indices must be integers or None or have an __index__
method
(From OE-Core rev: 2acfee61a062c6520a413b2a797544d968bb0c76)
Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the introduction of the hash equivalence server, the progress bar
for "Checking sstate mirror object availability" is shown
repeatedly. Most of the times the number of objects scanned is very
low and the progress bar completes almost immediately. To avoid all
these unnecessary progress bars, set the minimum number of objects to
100 before the progress bar is shown.
(From OE-Core rev: 114340e8d393cf60d0a596cb0800cf1b7fd198df)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Depending on the kernel and coreutils version, mv operations test for existence
of files and can potentially race. It also leads to the file always changing which
leads to potential problems if using and NFS share and there are other readers.
Using ln instead means we don't overwrite the file if it already exists meaning
other readers aren't disrupted and should work more reliably on NFS which is used
for sstate on the autobuilder.
Since we're not overwriting files, touch the file to show activity as would have
been done it it were reused from sstate.
(From OE-Core rev: d8e9a22a4e23616ad01627c1e472296b1e26f13c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Try and avoid errors like:
Exception: bb.process.ExecutionError: Execution of 'build/tmp/work/x86_64-linux/libxslt-native/1.1.34-r0/temp/run.sstate_unpack_package.12343' failed with exit code 1:
touch: setting times of 'build/sstate_devtool/80/4d/sstate:libxslt-native::1.1.34:r0::3:804d33f3f8d032b01ae92207669dd0a8f95a84917d563d122a77df19e786d73c_populate_lic.tgz.siginfo': Stale file handle
WARNING: exit code 1 from a shell command.
by not overwriting existing files.
(From OE-Core rev: b2a389ad5111d587db3f95c6ce13fc2eafc22f27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Many filesystems can't cope with filenames longer that 255 characters.
Add code to detect this and truncate non-essential elements of the filename
to stay within the limit.
[YOCTO #13268]
(From OE-Core rev: 90cc3d1ed1a12294a2d3ac97c1ba528ab315605d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the task handling and extension handling into one common filename
construction function.
(From OE-Core rev: c0c158d38583648a801e959d91371f7b43a98da5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the task handling code into the SSTATE_PKGNAME variable using a temporary variable.
This makes the code more understandable as as well as allowing the length of the
final sstate filename to be more easily accesses for following patches.
(From OE-Core rev: be603dad6817948cfa09621d2311046631ec4ab2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're having speed issues on the autobuilder due to the numbers of files in sstate
directories. We previously split these by the first two characters of the hash.
This change extends this to split by the next two characters as well, creating
more layers of directories.
This should signifiantly speed up eSDK builds on the autobuilder as the current
sstate layout simply isn't scaling there but addresses a general complaint.
gen-lockedsig-cache needed to be updated for the new split level sstate.
Also update tests for new layout.
(From OE-Core rev: d05bde16bdad761ed8f4c0a48de60c649aa33e85)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids a directory not present error.
Fix a comment typo whilst here.
(From OE-Core rev: 1360d8d7d99b70a80c8cdbc1fc6d9e6752483139)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous patches meant the mkdir might no longer match the final target
directory. Fix this.
(From OE-Core rev: 0af4dae84099e8632a9ea6a4afdbea2f232bb170)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
STATE_PKG may have been changed by sstate_report_unihash so don't
cache the variable's value.
(From OE-Core rev: be29a25400c4ea285ab3f588c5831f00ba5d4f63)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its rather antisocial to overwrite SSTATE_PKG with an expanded form for the variable
and it stops the value of BB_UNIHASH being changed when the package is written out.
Instead of expanding the variable, append to it instead to avoid this rather
hard to figure out behaviour and allow the siggen code to behave as expected.
(From OE-Core rev: 62eca02024b4c44d618ab9bcf87a3166c886dadb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its confusing to keep seeing sstate summary messages when hash equivalency is
active. This adds an option to control it. A default value is given which
maintains compatibility with different bitbake versions.
(From OE-Core rev: 038004866ff6650bcff7bb1bde36de6c0f451d29)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removes all references to the SSTATE_HASHEQUIV_SERVER variable. This
variable is redundant now that BB_HASHSERVE is present.
(From OE-Core rev: 54b3adb6bc90a8e4b9e92952688772ee074d36e5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake has updated to a cleaned up form of BB_HASHCHECK_FUNCTION,
adapt to this cleanup. This is an API breaking change.
(From OE-Core rev: a4d413d8d809132b0e0a5dd673a36e2bd0e0be4c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we write to the sstate directory we try to do so atomically so
consumers either see one version or another but never an imcomplete file.
Unfortunately this is reliant on filesystem support and with some NFS
configurations a replaced file would be lost from memory even if users held
open descriptors.
It makes sense to try and avoid replacing existing files where we can.
(From OE-Core rev: 18cdc087fd5da30e2b31f3d4e81b153cd36ca844)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change ensures that the task signatures changes, and therefore
sstate tasks are rerun, when signing is enabled. This has the
positive outcome that if signing is enabled new signed shared state
objects will be produced, rather than just signing shared state
objects for tasks where no work has been performed yet.
The downside of this change is that enabling/disabling sstate object
signing alters the taskhash and results in rebuilding the world.
(From OE-Core rev: ef60c65e3856682b90b17ae06a1b675756fd4496)
Signed-off-by: Joshua Lock <jlock@vmware.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When signature verification of shared state objects is enabled
(SSTATE_VERIFY_SIG) use of an unsigned object, even though it produces a
warning, seems unexpected. Instead skip unsigned objects and force the
non-accelerated task to be run.
(From OE-Core rev: 571235978d98552e3734bf382454dd51272db782)
Signed-off-by: Joshua Lock <jlock@vmware.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Referring to the sstate object as a staging package is an artefact of the
code's origins. Switch to referring to an "Sstate package" in order to be more
accurate and consistent with the rest of the file.
(From OE-Core rev: 06559c1ed86dbce53505f9ed98111fe9d0b97ed7)
Signed-off-by: Joshua Lock <jlock@vmware.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the recent changes in bitbake to runqueue, we need to recheck sstate validity,
particularly in multiconfig builds where tasks have the same checksum.
Avoid printing summary messages in this case. Also avoid multiple events to toaster
which may not be expecting that at later points in the code.
(From OE-Core rev: 227125b96ad6fb0cf6e259e787d83415993db847)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When changing the SSTATE_SCAN_FILES variable in a recipe it doesn't cause a rebuild,
so if there's a sstate-cache available with "bad" sstate data in it that will still
be used even though the recipe is updated to address this.
[YOCTO #13144]
(From OE-Core rev: ea3526961920a229e0bb5fb459952be89fce2255)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updates the output hash calculation for determining if tasks are
equivalent. The new algorithm does the following based on feedback:
1) The output hash function was moved to the OE library.
2) All files are printed in a single line tabular format
3) Prints the file type and mode in a user-friendly ls-like format
4) Includes the file owner and group (by name, not ID). These are only
included if the task is run under pseudo since that is the only time
they can be consistently determined.
5) File size is included for regular files
(From OE-Core rev: 4bd297dfe92851f3b44f6b5560bac9d8f9ccf9f2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There might be do_cleansstate errors sometimes:
ERROR: When reparsing
/path/to/meta/recipes-devtools/quilt/quilt-native_0.65.bb.do_cleansstate,
the basehash value changed from
b4dbcb956a32ed4c3f58b7971717907bfc03bb21f3b140fa97f7765ee695f4d0 to
c8307418a671686349b73efbd51c5c82c897a88707a759ddb22fd95baa5df2ba. The metadata
is not deterministic and this needs to be fixed.
The stable reproducer is:
- Initial a fresh build, this is a must, otherwise we may can't reproduce it
$ . oe-init-build-env build
$ bitbake quilt-native -ccleansstate
This is because uninative.bbclass resets NATIVELSBSTRING from distro (e.g.,
ubuntu) to universal, remove dependencies of SSTATE_EXTRAPATHWILDCARD as
SSTATE_EXTRAPATH did can fix the problem.
(From OE-Core rev: 781117f9f02c0080dadc8797a8f8f9377a99b164)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A couple have still been missed in the past despite multiple
attempts at doing so (or simply have re-appeared?).
Search & replace made using the following command:
sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \
-i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \
| cut -d':' -f1 \
| sort -u)
(From OE-Core rev: 9f551d588693328e4d99d33be94f26684eafcaba)
Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Converts sstate so that it can use a hash equivalence server to
determine if a task really needs to be rebuilt, or if it can be restored
from a different (equivalent) sstate object.
The unique hashes are cached persistently using persist_data. This has
a number of advantages:
1) Unique hashes can be cached between invocations of bitbake to
prevent needing to contact the server every time (which is slow)
2) The value of each tasks unique hash can easily be synchronized
between different threads, which will be useful if bitbake is
updated to do on the fly task re-hashing.
[YOCTO #13030]
(From OE-Core rev: d889acb4f8f06f09cece80fa12661725e6e5f037)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Handles the argument that passes task unique hash in the hash check
function, as it is now required by bitbake
[YOCTO #13030]
(From OE-Core rev: 1c14b6969e58b51a325c1c1acf9c96e55675035c)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sstate bbclass uses workdir as a hardcoded string in path
manipulations. This means that the sstate caching mechanism does
not work for the work-shared directory which the kernel uses to
share its build configuration and source files for out of tree
kernel modules.
This commit modifies the path manipulation mechanism to use the
work-shared directory if detected in the paths when handling the
sstate cache packages.
(From OE-Core rev: 27642449f95e38598f9c83948ce109c5891e5877)
Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* in some rare cases the file format might be broken, e.g. when you run
out of disk space while writing to it (hint: make sure to use
BB_DISKMON_DIRS on _all_ the builds, to make sure that they can stop
gracefully before this happens).
* will show error like this:
ERROR: Invalid line '/OE/build/luneos-thud/webos-ports' in sstate manifest '/OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch'
when the file isn't complete, like in my case:
$ tail -n 2 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/index-allarch
/OE/build/luneos-thud/webos-ports/tmp-glibc/stamps/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0 /OE/build/luneos-thud/webos-ports/tmp-glibc/sstate-control/manifest-allarch-org.webosports.app.calculator /OE/build/luneos-thud/webos-ports/tmp-glibc/work/all-webos-linux/org.webosports.app.calculator/0.1.1+gitrAUTOINC+9e9eb67c28-r0
/OE/build/luneos-thud/webos-ports
* instead of much longer exception which doesn't really show what's
wrong to selectively fix that (other than removing while TMPDIR):
ERROR: Command execution failed: Traceback (most recent call last):
File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/command.py", line 113, in runAsyncCommand
self.cooker.updateCache()
File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/cooker.py", line 1562, in updateCache
bb.event.fire(event, self.databuilder.mcdata[mc])
File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 225, in fire
fire_class_handlers(event, d)
File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 134, in fire_class_handlers
execute_handler(name, handler, event, d)
File "/OE/build/luneos-thud/webos-ports/bitbake/lib/bb/event.py", line 106, in execute_handler
ret = handler(event)
File "/OE/build/luneos-thud/webos-ports/openembedded-core/meta/classes/sstate.bbclass", line 1083, in sstate_eventhandler2
(stamp, manifest, workdir) = l.split()
ValueError: not enough values to unpack (expected 3, got 1)
(From OE-Core rev: 870ba5d80e76e0f989971532fc2adc0ebe811ab6)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This can improve the performance a lot for "bitbake <recipe-native/cross/crosssdk>
-ccleansstate" when there are a lot of sstate files.
For example:
* Before
$ bitbake quilt-native -ccleansstate
- Check log.do_cleansstate:
Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*
Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_qa.tgz*
Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz*
Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_packagedata.tgz*
Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz*
Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
There are no package tasks for quilt-native, so the first 4 lines doesn't
make any sense, but the glob pattern "sstate-cache/*/*" is very time
consuming when there are no disk caches. E.g., I have more than 600,000
sstate files:
- Without disk caches
# echo 3 >/proc/sys/vm/drop_caches
$ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")'
real 4m32.583s
user 0m5.768s
sys 0m12.892s
- With disk caches (e.g., run it in the second time)
$ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")'
real 0m5.128s
user 0m2.772s
sys 0m2.308s
So the 4 removing *package* commands cost more than 20s or 272s in theory.
* After
$ bitbake quilt-native -ccleansstate
- Check log.do_cleansstate:
Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz*
Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
We can see that it saved 20s or 272s in theory.
(From OE-Core rev: bb2d6349ea87f090c58001f0d4348b24c2982cde)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The glob.glob("/sstate/*/*/") is very time consuming, set
SSTATE_EXTRAPATHWILDCARD explicity to avoid that. This can save a lot of time
when there are many sstate files.
For example, I have more than 600,000 sstate files:
* Before
- Without disk caches
$ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
real 4m32.583s
user 0m5.768s
sys 0m12.892s
- With disk caches
$ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
real 0m4.111s
user 0m2.348s
sys 0m1.756s
* After
$ time python3 -c 'import glob; glob.glob("/sstate-cache.bak/universal/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
- Without disk caches:
real 0m7.928s
user 0m0.172s
sys 0m0.124s
- With disk caches:
real 0m0.131s
user 0m0.088s
sys 0m0.044s
We can see that it saves about 3.8s with disk caches, and saves about 264s
without disk caches.
(From OE-Core rev: 8b31c919814b8bdf25b3381053656523c001ae0d)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update SSTATE_DUPWHITELIST in sstate.bbclass.
* remove ${DEPLOY_DIR_RPM}/noarch/ which is not overwritten any more
* add directories for package target-sdk-provides-dummy
(From OE-Core rev: 6d3ca476dbc2059f4b7fa3dfd73de6bbfed49198)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Define variable SSTATE_PRUNE_OBSOLETEWORKDIR with '?=' in sstate.bbclass,
then it could be overwritten by user configuration.
(From OE-Core rev: 22af59c9bfec31b31027ebd2a4da162f481aa6b5)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using re.compile() is around six times faster than recompiling the regexp
each time so maintain a cache.
(From OE-Core rev: 41eb382737706e245f2b7104e313c8dfaa370945)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying
the base-passwd|shadow-sysroot recipes to be excluded from a recipe sysroot.
(From OE-Core rev: 68e502e9063a88532fe0154f152ba408f0091900)
Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying
the *-initial recipes to be excluded from a recipe sysroot.
(From OE-Core rev: 6706bad52f9311ea79c534ee90014c3216992999)
Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, a dependency on any -native recipe will pull in
all dependencies of that -native recipe in the recipe
sysroot. This behaviour might not always be wanted, e.g.
when that -native recipe depends on build-tools that are
not relevant for the current recipe.
This change adds a SSTATE_EXCLUDEDEPS_SYSROOT variable,
which will be evaluated for such recursive dependencies to
be excluded. The idea is similar to
http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146324.html
except that the list is not hard-coded anymore.
SSTATE_EXCLUDEDEPS_SYSROOT is evaluated as two regular
expressions of recipe and dependency to ignore, e.g. in
the above flex-native / bison-native use-case, one would
specify
SSTATE_EXCLUDEDEPS_SYSROOT = ".*->(flex|bison)-native"
in layer.conf.
The existing special handling of "-initial" as well as
"base-passwd" and "shadow-sysroot" could also be
streamlined:
SSTATE_EXCLUDEDEPS_SYSROOT += "\
.*->.*-initial.* \
.*(base-passwd|shadow-sysroot)->.* \
"
Another anticipated user is meta-java, where certain newer
JDKs can only be bootstrapped (built) using older JDKs,
but it doesn't make much sense to copy all those older
JDKs and their own build tools (ant, etc.) into the
sysroot of recipes wanting to be built using the newer JDK
(only), e.g.:
SSTATE_EXCLUDEDEPS_SYSROOT += "\
openjdk-8-native->(ant-native|attr-native|coreutils-native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-native|unzip-native) \
"
(From OE-Core rev: 92c5131a2feae2036c71a36c18bb9175bb2856dc)
Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Files when restored from sstate were missing their sparseness. Fix up various
functions to preserve this and make things more deterministic.
(From OE-Core rev: 055402e5504f041c346571e243c7cf0894955cad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you build qemux86 and then generic86, the latter will
remove all of the former from deploy and workdir. This is because
qemux86 is i586, genericx86 is i686 and the architctures are compatible
therefore the sstate 'cleaup' code kicks in.
There was a valid reason for this to ensure i586 packages didn't get into
an i686 rootfs for example. With the rootfs creation being filtered now, this
is no longer necessary.
Instead, save out a list of stamps which a give machine has ever seen in
a given build and only clean up these things if they're no longer
"reachable".
In particular this means the autobuilder should no longer spend a load of time
deleting files when switching MACHINE, improving build times.
(From OE-Core rev: 5634f2fb1740732056d2c1a22717184ef94405bf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace the generic whitelist entry with entries for the three specific
'problem' cases in OE-Core. This means the general DEPLOY_DIR_IMAGE entry
doesn't mask problems for others as was recently encoutered by users
reported on irc. In the whitelisted cases they occur only in multilib builds
and the files are identical.
(From OE-Core rev: 05f6042a40bb772f7ce8d6819c5b2937d8c9808d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the user has no indication of how much sstate was already present
or that would be used by the build. This change adds some summary information
so that the user can see how much reuse is occurring. To fully work it
needs some extra information from a recent bitbake commit but this is
optional.
When combined with bitbake --dry-run this feature can be used to check
if sstate would be reused in a build.
[YOCTO #12749]
(From OE-Core rev: 596f76029ccb6f87c3b049552bd08f5034c41d9c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the compression of sstate objects is single threaded. In the case
of ltp, this takes around 33s. If we add pigz into the list of non-fatal
HOSTTOOLS and then use if it available when building the sstate object,
this time drops to around 6s.
Since pigz is now widely available this is an optimisation we should
utilise.
(From OE-Core rev: 2de56aa0792ec93445130d801936a8ea643fad27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
