Give this thread pool a name for clarity.
(From OE-Core rev: 57ea50fcfe81f47b93b9302d1aab2e81dcdd3105)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existence of missed files on the local sstate cache is the
condition to search for it on the sstate mirror.
The missed collection are populated when we search the files
on the local sstate cache.
- When we have sstate mirrors we don't need to iterate all files
again because we already know what's missing.
- When we found a file on the sstate mirror is because of they
are missing on the local sstate mirror.
So avoid checking if the hash exists on the missed collection.
- When we can't find the file on the sstate mirror we don't need
to add the hash again to the missed collection as it already exists.
(From OE-Core rev: 46e1f9a900e2be1abeda0dc540fc14040ecd0d7c)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We only have the progress bar when there are more than 100 objects.
So we can check for this and store the result to show the progress bar.
(From OE-Core rev: 18e7a77aa1626e11820804ad64b86a32f97da2ca)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch is a fixup for 676757f "sstate: fix touching files inside pseudo"
running the 'id' command inside the sstate_unpack_package
function shows that this funcion run inside the pseudo:
uid=0(root) gid=0(root) groups=0(root)
The check for [ -w ${SSTATE_PKG} ] and [ -O ${SSTATE_PKG}.siginfo ]
will always return true and the touch can fail when the real user
don't have permission or in readonly filesystem.
As the documentation refers:
- the file test operator "-w" check if the file has write permission
(for the user running the test).
- the file test operator "-O" check if you are owner of file
We can avoid this test running the touch and mask any return errors
that we have.
(From OE-Core rev: 29fc85997ade490ae46ffca37ef8e1a56957c876)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, when shortening sstate filenames, the reserved
characters for .siginfo were not considered, when siginfo=False,
resulting in differently shortend filenames for the sstate and siginfo
files. With this change, the filenames of the truncated sstate and
siginfo files have the same basename, just as is already the case for
untruncated filenames.
Making sure that the .siginfo files always have the filename of the
corresponding sstate file plus its .siginfo suffix, also when being
truncated, makes it easier to manage the sstate cache and an sstate
mirror outside of Bitbake/Yocto.
(From OE-Core rev: c2e0e43b7123cf5149833e0072c8edaea3629112)
Signed-off-by: Manuel Leonhardt <mleonhardt@arri.de>
Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
running the 'id' command inside the sstate_create_package
function shows that this funcion run inside the pseudo:
uid=0(root) gid=0(root) groups=0(root)
The check for touch files [ ! -w ${SSTATE_PKG} ]
will always return true and the touch can fail
when the real user don't have permission or
in readonly filesystem.
As the documentation refers, the file test operator "-w"
check if the file has write permission (for the user running the test).
We can avoid this test running the touch and mask any return errors
that we have.
(From OE-Core rev: f6e7445c94443544e92fda97a017ce93393c5f84)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't need extra python collections to count the found files
on the sstate cache and sstate mirrors.
The main found collections provides all the files that were found.
Then we only need to store the nunber of files that found on the
local sstate cache and the files that found on the sstate cache
mirror is derived from that.
(From OE-Core rev: 0fe71f31593e87b9188b388860d2ba94bf8266bc)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If checkstatus returns an exception we should silently handle
FetchError, as this means the fetch failed for 'normal' reasons such as
file not found. However, other exceptions may be raised, and these
should be made visible.
(From OE-Core rev: 50d99faf88a1d82cbd939b9bd6e33ebed2b1ffd8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow a user to validate sstate objects against a list of keys, instead
of just any known key in the user's keychain.
(From OE-Core rev: 52ba0c5e6e2e3d5d01dc3f01404f0ab1bb29b3b5)
Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch switches the compressor from Gzip to ZStandard for ssate cache
files.
Zstandard compression provides a significant improvement in
decompression speed as well as improvement in compression speed and disk
usage over the 'tgz' format in use. Furthermore, its configurable
compression level offers a trade-off between time spent compressing
sstate cache files and disk space used by those files. The reduced disk
usage also contributes to saving network traffic for those sharing their
sstate cache with others.
Zstandard should therefore be a good choice when:
* disk space is at a premium
* network speed / resources are limited
* the CI server can sstate packages can be created at high compression
* less CPU on the build server should be used for sstate decompression
(From OE-Core rev: 0710e98f40ec7394567ede1f88f7c7bae306b925)
Signed-off-by: Henry Kleynhans <hkleynhans@fb.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We hardlink some files into the build, such as licence files in
do_populate_lic tasks. Depending on the umask that the source tree
was checked out with, the group permissions would vary. This
results in inconsistent task outhashes.
Avoid this by ignoring the group/other bits unless we're under
pseudo context.
Bump the ABI numbers to ensure we don't see cache corruption from
earlier builds.
(From OE-Core rev: 2bd9c806de7e5197168360e3bd1d223a04a92291)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At this point the hash equivalence and sstate is 'junk' on the autobuilder
unforuntately due to the volume of fixes and also the volume of slighly
not quite right patches tested during the development of the fixes.
In order to try and help any remaining sanity I might have, bump the
version numbers to start with a clean slate so we're working from a known
good baseline rather than risk chasing phantom issues. For those
upgrading, there wouldn't be much reuse anyway after the changes.
(From OE-Core rev: be32692c627a14509de5eb3834e7321c3c5faf25)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some files in the populate_sysroot tasks have hardcoded paths in them,
particularly if they are postinst-useradd- files or crossscripts.
Add some filtering logic to remove these paths.
This means that the hashequiv "outhash" matches correcting in more
cases allowing for better build artefact reuse.
To make this work a new variable is added SSTATE_HASHEQUIV_FILEMAP
which maps file globbing to replacement patterns (paths or regex)
on a per sstate task basis. It is hoped this shouldn't be needed
in many cases. We are in the process to developing QA tests which
will better detect issues in this area to allow optimal sstate
reuse.
(From OE-Core rev: d9852ffbbe728dac33dc081538a08af98f52fd4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are several tasks which should only be triggered by direct
dependencies. Some are listed in the dep loop, some are not. They
should all be in both cases.
Fix this by making a list and using the list in both cases to fix
various inconsistencies.
(From OE-Core rev: 29d285bc07bd7732c7ea39c0fd03c0b3c0d93e52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This sstate task is only needed when depended upon, it can be skipped
if there are no tasks running that directly depend upon it.
This reduced the number of sstate tasks in something like an image
build.
(From OE-Core rev: 601cee016da5c7505915e26641a085714de175ce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an image is built from sstate, the cross toolchain was being pulled in.
This was due to the sstate dependencies for deploy tasks not being considered
in the "depvalid" logic.
do_deploy tasks do not need their populate_sysroot dependencies when installed
from sstate so skip these. This reduces the sstate pulled in by an image
build from sstate significantly.
(From OE-Core rev: 8047c184f8266fc047c0e5a5524586d414528626)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating packages we build them with --clamp-mtime and use
SOURCE_DATE_EPOCH as the maximum mtime. This makes the end packages
reproducible. The data stored in sstate for do_package and the package
task doesn't benefit from this though and have varying timestamps.
This means their outhash varies and means hash equivalance isn't
effective at all and doesn't work as intended/desired.
We could create the sstate archives with the same clamping however
that would lead to different results depending on whether a task was
installed from sstate or not. Making that differ is a path to madness.
It also wouldn't fix the outhash of the task to be determninistic
without clamping of the date in the hash calculation code.
Instead, iterate over the files in sstate output and clamp them at
the code level. This isn't ideal but does make the file timestamps
determnistic everywhere and means we don't have to change the hash
calculation code.
This issue can be clearly seen looking at the do_package outhash for
a recipe which you then re-run the package task for after adding
something like whitespace to the install task. The outhash shouldn't
change but currently does.
(From OE-Core rev: c3b3cc4745811b48b9193f83889946b2e1788932)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When fetching sstate, SRCPV can be expanded withuin PV which breaks the
sstate fetcher as other variables like SRC_URI are changed by the fetcher
code. Expand the variable before doing this to avoid those problems and
hence avoid setscene task failures with remote sstate storage in git
recipes.
(From OE-Core rev: 0d9d7acebe0be42d4cf8a30ecbbbbcb2dfd16a4b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- bitbake BB_NUMBER_THREADS uses cpu_count from oe utils that uses
the python os.sched_getaffinity and it is more acurrate.
grep -nH ^BB_NUMBER_THREADS meta/conf/bitbake.conf
meta/conf/bitbake.conf:806:BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
- multiprocessing.cpu_count() returns the number of CPUs on the host,
not the number of usable CPUs on the host. If the user is using
scheduler affinity then the number of usable CPUs may be less,
so when determining how many cores we can use check the affinity instead.
(From OE-Core rev: 45c52f08a289c6eb2329de50634a0406204d1d8e)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The commit dd555537fc
'sstate.bbclass: fix errors about read-only sstate mirrors'
adds an additional exception handler to silently mask read
only rootfs errors thrown during the touch.
The exception handler checks the error type with the python module errno
but this module needs to be imported as it don't exist.
Example of the error:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:sstate_task_postfunc(d)
0003:
File: '/home/builder/src/base/poky/meta/classes/sstate.bbclass', lineno: 778, function: sstate_task_postfunc
0774:
0775: omask = os.umask(0o002)
0776: if omask != 0o002:
0777: bb.note("Using umask 0o002 (not %0o) for sstate packaging" % omask)
*** 0778: sstate_package(shared_state, d)
0779: os.umask(omask)
0780:
0781: sstateinst = d.getVar("SSTATE_INSTDIR")
0782: d.setVar('SSTATE_FIXMEDIR', shared_state['fixmedir'])
File: '/home/builder/src/base/poky/meta/classes/sstate.bbclass', lineno: 708, function: sstate_package
0704: except PermissionError:
0705: pass
0706: except OSError as e:
0707: # Handle read-only file systems gracefully
*** 0708: if e.errno != errno.EROFS:
0709: raise e
0710:
0711: return
0712:
Exception: NameError: name 'errno' is not defined
(From OE-Core rev: 15f30ad144fbe25e9a5e71bc7e42e746d2039992)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When adding a layer which changed SSTATE_EXCLUDEDEPS_SYSROOT, the state
hashes were changing when they should not. This was caused by wider use
of setscene_depvalid which means the dependency on the variable was seen
when it was previously not.
Exclude the variable since this should be be included in the hashes.
(From OE-Core rev: 09725a29365c69ccbd603fe3a1de72189f26d5ac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that pseudo-native always includes statx support and we have sanity
checks on pseudo-native to ensure it always contains a minimum feature set,
we no longer need to mark pseudo-native as distro specific. This fixes
eSDK build problems.
[YOCTO #14428]
(From OE-Core rev: 3279e43fcb469edb63c7c4eb60fdc565d5751f9d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sstate code tries to be careful about racing around directory creation.
In particular, the copyhardlinktree code creates the directory tree first
allowing for "already exists" errors and ignoring them, then hardlinks the
files in.
Unfortunately the sstate removal code can race against this since it
will try and remove empty directories. If there is some bad timing,
a newly created directory can be removed before it was populated, leading
to build failures.
We could try and add locking but this would damage performance, we've been
there before. It is also unclear where to actually place locks just based on
the contents of a manifest file which may cover multiple sstate install
locations for a given task.
Instead, lets disable directory removal in the problematic "shared" core
path. This could result in a few more empty directories being left on disk
but those should be harmless and better than locking hurting performance
or rare build races.
[YOCTO #13999]
[YOCTO #14379]
(From OE-Core rev: 4f94d9296394bc7ce241439f00df86eb5912875f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a read-only sstate mirror is used in conjunction with hash equiv,
then OSError will be raised when an sstate-cache hit is achieved.
This is because sstate_task_postfunc will try to "touch" the symlinks
that point to the read-only sstate mirror when sstate_report_unihash
has changed SSTATE_PKG.
This commit adds an additional exception handler to silently mask read
only rootfs errors thrown during the touch.
The fix is also duplicated to sstate_eventhandler as the code is very
similar but it may not be needed there.
Example of the error:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:sstate_task_postfunc(d)
0003:
File: '/poky/meta/classes/sstate.bbclass', lineno: 774, function: sstate_task_postfunc
0770:
0771: omask = os.umask(0o002)
0772: if omask != 0o002:
0773: bb.note("Using umask 0o002 (not %0o) for sstate packaging" % omask)
*** 0774: sstate_package(shared_state, d)
0775: os.umask(omask)
0776:
0777: sstateinst = d.getVar("SSTATE_INSTDIR")
0778: d.setVar('SSTATE_FIXMEDIR', shared_state['fixmedir'])
File: '/poky/meta/classes/sstate.bbclass', lineno: 703, function: sstate_package
0699: if not os.path.exists(siginfo):
0700: bb.siggen.dump_this_task(siginfo, d)
0701: else:
0702: try:
*** 0703: os.utime(siginfo, None)
0704: except PermissionError:
0705: pass
0706:
0707: return
Exception: OSError: [Errno 30] Read-only file system
(From OE-Core rev: 244b3be0358a66e0cca4016fe26144e3d7323390)
Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
What key is used to sign sstate artefacts should not affect the hash of
the object, otherwise everyone would need to use the same signing key.
(From OE-Core rev: 57cc9429dba4f9bd23127633dbc1f57dc2d5dd16)
Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under certain build patterns, warnings about missing manifests can appear. These
are real issues where the manifest was removed and shouldn't have been.
Martin Jansa was able to find a reproducer of:
MACHINE=qemux86 bitbake zlib-native
echo 'PR = "r1"' >> meta/recipes-core/zlib/zlib_1.2.11.bb
MACHINE=qemux86-64 bitbake zlib-native
MACHINE=qemux86 bitbake zlib-native
<the zlib-native manifest is now removed along with the sysroot-components contents>
The code maintains a per machine list of stamps but a per PACAGE_ARCH list of
stamp/manifest/workdir mappings. The latter is only appended to for speed with
the assumption that once stamps are gone, the code wouldn't trigger.
The code only ever appends to the mapping list (for speed/efficency under lock)
meaning that multiple entries can result where the stamp/workdir differs due to
version changes but the manifest remains the same.
By switching MACHINE part way through the build, the older stamp is referenced
and the manifest is incorrectly removed as it matches an now obsolete entry in
the mapping file.
There are two possible fixes, one is to rewrite the mapping file every time
which means adding regexs, iterating and generally complicating that code. The
second option is to only use the last mapping entry in the file for a given
manifest and ignore any earlier ones. This patch implments the latter.
Also drop the stale entries if we are rewriting it.
(From OE-Core rev: 63da9a4f889c5b0e41bc8ec08abe0acea1546479)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Incremental build in Docker fails with:
OSError: [Errno 18] Invalid cross-device link
when source and destination are on different overlay filesystems.
Rather than adding fallback code to every call site, use a new wrapper
in bitbake which detects this case and falls back to shutil.move
which is slower but will handtle the overlay docker filesystems correctly.
[YOCTO #14301]
(From OE-Core rev: 656a65b2b84e7d529b89cf5de7eb838f902d84a2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It isn't immediately obvious what these do, add some better examples
to document them. Also improve the naming for one of the handlers.
This will help make it clear in any tracebacks which code is being
used.
(From OE-Core rev: 2b5c553744804f44a17b61c345e482027e45b71e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The split of util-linux-uuid out from util-linux caused some interesting
sstate file overlap errors on existing build directories. This is a
challenge to handle since util-linux depends on util-linux-uuid and has
overlapping files in package data and deploy/packages directories.
The util-linux build happens later and is what would clean up those files
but it happens too late for uuid.
Fixing this is hard as we don't know the taskhashes until the task
graph is calculated. Once that is ready, we can compare the hashes
with the existing hashes and know which sstate tasks are "stale".
This patch adds a handler which iterates the sstate manifests looking
for matching stamp paths and then removes the manifests along with the
associated stamp files.
(From OE-Core rev: 60e77b1777c6c304aa1d629c4cfdabe0daa22eb1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Modify sstate class to add extra output, differentiating
between artifacts found locally and those found over the network.
(From OE-Core rev: f017a2f677fa289c36b0e56ce480f69d7674ca11)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
pseudo-native is a bit special. It conditionally compiles in support for
xattr, statx and statvfs amongst other options. If a pseudo-native binary is
used on a system where these functions are present but it wasn't compiled in
we see hard to debug permissions problems.
An example is the devtool.DevtoolExtractTests.test_devtool_deploy_target
oe-selftest which shows a cryptic error:
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/devtool.py", line 1388, in test_devtool_deploy_target
self.assertEqual(filelist1, filelist2)
File "/usr/lib64/python3.9/unittest/case.py", line 831, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib64/python3.9/unittest/case.py", line 1037, in assertListEqual
self.assertSequenceEqual(list1, list2, msg, seq_type=list)
File "/usr/lib64/python3.9/unittest/case.py", line 1019, in assertSequenceEqual
self.fail(msg)
File "/usr/lib64/python3.9/unittest/case.py", line 670, in fail
raise self.failureException(msg)
AssertionError: Lists differ: ['-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8'] != ['-rwxr-xr-x root root /etc/init.d/mdmonitor', '-rw-r-[10124 chars]n.8']
First differing element 0:
'-rwxr-xr-x 6000 6000 /etc/init.d/mdmonitor'
'-rwxr-xr-x root root /etc/init.d/mdmonitor'
This is due to a version of pseudo without statx being used on a system where
ls uses statx, hence the files are displayed as 6000.6000 instead of root.root.
Avoid this by always building pseudo-native for the specific distro in question rather
than using a universal sstate feed.
This hopefully fixes one of the mysterious AB-INT issues.
(From OE-Core rev: 6e3785a3f1f3cf68f5fe101cd6bebe91db165973)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating directories in the sstate-cache, explicitly set the mode
passed to mkdir to 0775 so that the directories are group writable, as
otherwise they cannot be shared with other users.
(From OE-Core rev: 1b0624dffe1b8496533c86dfed873112c8c0a01b)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In contrast to 'touching' a file without parameters 'touch -a' can only be
performed by the file owner. In case of a shared sstate-cache served
locally (e.g. over NFS), where the user running bitbake is not the
owner of sstate-cache files, even if he has write access on group level,
the sstate_unpack_package task will fail.
Checking if the file is owned by the user before attempting to run
'touch -a' on it solves this.
(From OE-Core rev: f1554e8bd40c640fd58daac18ecf7b458e754109)
Signed-off-by: Norman Stetter <norman.stetter@garz-fricke.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prior to fetching, the system checks if the sstate file is present
either locally or on the mirror. If it is, then it goes to the fetch
stage. Up to three files can be fetched, sstate, sstate.siginfo and
sstate.sig (if signature validation is enabled).
The previous pstaging_fetch function would iterate over these, and if
a download error occurred would spew forth a great amount of fetcher
failure messages as well as stop fetching the next item in the set.
This was resolved by adding a fetcher.checkstatus() call prior to
the download. If the file isn't present, then the exception will
be triggered, and no fetcher failure messages will reach the user.
The exception handler is then modified to be a pass so that it will
loop and pull the rest of the files that that are requested.
Additionally, a check for the existance of the .sig file was added
to the sstate_installpkg to avoid an error trying to load the .sig
if it wasn't downloaded.
(From OE-Core rev: a9085140434e2d26c0bb75bb53fcb7f7c19ef86d)
Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It may be that a file is not allowed to be touched, e.g., if it is a
symbolic link into a global sstate cache served over NFS.
(From OE-Core rev: 462e3f505217c9b9e0d2c1cbdafb0c8910bb6a59)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now this function has a summary parameter we can drop this check.
It could well be why the mysterious "locked sigs" selftest fails
intermittently if this function were called with a single hash
to check.
[YOCTO #13605] (with luck)
(From OE-Core rev: 02aabe0e59f73bf206d9bada1e7089832ceed254)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When moving to python3, divison using '/' now returns float instead of
an integer. In upstream commit b8025e9720
sstate filenames longer than the limit are changed to just include
necessary info + 3 fields just for information. The space left over
after the necessary info is divided into 3 for each of the fields.
Using '//' instead to do the division to solve the following error
message:
avail = (254 - len(hash + "_" + taskname + extension) -
len(components[0]) - len(components[1]) - len(components[5]) -
len(components[6]) - 7) / 3
> components[2] = components[2][:avail]
components[3] = components[3][:avail]
TypeError: slice indices must be integers or None or have an __index__
method
(From OE-Core rev: 2acfee61a062c6520a413b2a797544d968bb0c76)
Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the introduction of the hash equivalence server, the progress bar
for "Checking sstate mirror object availability" is shown
repeatedly. Most of the times the number of objects scanned is very
low and the progress bar completes almost immediately. To avoid all
these unnecessary progress bars, set the minimum number of objects to
100 before the progress bar is shown.
(From OE-Core rev: 114340e8d393cf60d0a596cb0800cf1b7fd198df)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Depending on the kernel and coreutils version, mv operations test for existence
of files and can potentially race. It also leads to the file always changing which
leads to potential problems if using and NFS share and there are other readers.
Using ln instead means we don't overwrite the file if it already exists meaning
other readers aren't disrupted and should work more reliably on NFS which is used
for sstate on the autobuilder.
Since we're not overwriting files, touch the file to show activity as would have
been done it it were reused from sstate.
(From OE-Core rev: d8e9a22a4e23616ad01627c1e472296b1e26f13c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Try and avoid errors like:
Exception: bb.process.ExecutionError: Execution of 'build/tmp/work/x86_64-linux/libxslt-native/1.1.34-r0/temp/run.sstate_unpack_package.12343' failed with exit code 1:
touch: setting times of 'build/sstate_devtool/80/4d/sstate:libxslt-native::1.1.34:r0::3:804d33f3f8d032b01ae92207669dd0a8f95a84917d563d122a77df19e786d73c_populate_lic.tgz.siginfo': Stale file handle
WARNING: exit code 1 from a shell command.
by not overwriting existing files.
(From OE-Core rev: b2a389ad5111d587db3f95c6ce13fc2eafc22f27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Many filesystems can't cope with filenames longer that 255 characters.
Add code to detect this and truncate non-essential elements of the filename
to stay within the limit.
[YOCTO #13268]
(From OE-Core rev: 90cc3d1ed1a12294a2d3ac97c1ba528ab315605d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the task handling and extension handling into one common filename
construction function.
(From OE-Core rev: c0c158d38583648a801e959d91371f7b43a98da5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the task handling code into the SSTATE_PKGNAME variable using a temporary variable.
This makes the code more understandable as as well as allowing the length of the
final sstate filename to be more easily accesses for following patches.
(From OE-Core rev: be603dad6817948cfa09621d2311046631ec4ab2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're having speed issues on the autobuilder due to the numbers of files in sstate
directories. We previously split these by the first two characters of the hash.
This change extends this to split by the next two characters as well, creating
more layers of directories.
This should signifiantly speed up eSDK builds on the autobuilder as the current
sstate layout simply isn't scaling there but addresses a general complaint.
gen-lockedsig-cache needed to be updated for the new split level sstate.
Also update tests for new layout.
(From OE-Core rev: d05bde16bdad761ed8f4c0a48de60c649aa33e85)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids a directory not present error.
Fix a comment typo whilst here.
(From OE-Core rev: 1360d8d7d99b70a80c8cdbc1fc6d9e6752483139)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous patches meant the mkdir might no longer match the final target
directory. Fix this.
(From OE-Core rev: 0af4dae84099e8632a9ea6a4afdbea2f232bb170)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
STATE_PKG may have been changed by sstate_report_unihash so don't
cache the variable's value.
(From OE-Core rev: be29a25400c4ea285ab3f588c5831f00ba5d4f63)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
