We sometimes see exceptions from code seeing the hashserv DB files
being removed at directory cleanup time. Add a check to ensure the
hashserv has written the data base journal (and hence likely exited)
before cleaning up.
This will hopefully avoid errors like:
Traceback (most recent call last):
File "[...]/meta/lib/oeqa/sdk/buildtools-cases/build.py", line 30, in test_libc
delay = delay - 1
File "/usr/lib/python3.6/tempfile.py", line 948, in __exit__
self.cleanup()
File "/usr/lib/python3.6/tempfile.py", line 952, in cleanup
_rmtree(self.name)
File "/usr/lib/python3.6/shutil.py", line 486, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/lib/python3.6/shutil.py", line 444, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/usr/lib/python3.6/shutil.py", line 442, in _rmtree_safe_fd
os.unlink(name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'hashserv.db-wal'
(From OE-Core rev: 635833734b4c61e453ca9843a9fb5cecf3eb1c97)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0b07d9add687d78495176cda0f3011c10ffa4d4b)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are situations when the user have the 'patchdir' defined
as a parameter on SRC_URI. However he doesn't know that with this
the patch is applied relatively to the receipe source dir 'S'.
- When user have 'patchdir' defined check if this directory exist.
- If the patch fails show addition info to the user:
- Import: show the striplevel
- Resolver: show the expanded 'patchdir' to the user.
The next example is from opencv in meta-oe layer, here the
patch is applied on the target directory ${WORKDIR}/git/contrib.
S = "${WORKDIR}/git"
SRCREV_FORMAT = "opencv_contrib"
SRC_URI = "git://github.com/opencv/opencv.git;name=opencv \
git://github.com/opencv/opencv_contrib.git;destsuffix=contrib;name=contrib \
file://0001-sfm-link-with-Glog_LIBS.patch;patchdir=../contrib \
"
* When the patch fail there are no message that indicates the real reason.
patchdir=../no-found-on-file-system
ERROR: opencv-4.5.2-r0 do_patch: Command Error: 'quilt --quiltrc /build/tmp/work/core2-64-poky-linux/opencv/4.5.2-r0/recipe-sysroot-native/etc/quiltrc push' exited with 0 Output:
stdout: Applying patch 0001-sfm-link-with-Glog_LIBS.patch
can't find file to patch at input line 37
Perhaps you used the wrong -p or --strip option?
* The check of the patchdir will add a new fatal error
when the user specifies a wrong path than don't exist.
patchdir=../no-found-on-file-system
ERROR: opencv-4.5.2-r0 do_patch: Target directory '/build/tmp/work/core2-64-poky-linux/opencv/4.5.2-r0/git/../no-found-on-file-system' not found, patchdir '../no-found-on-file-system' is incorrect in patch file '0001-sfm-link-with-Glog_LIBS.patch'
* When we can't aplly the patch but the patchdir exist,
show the expanded patchdir on fatal error.
patchdir=../git
ERROR: opencv-4.5.2-r0 do_patch: Applying patch '0001-sfm-link-with-Glog_LIBS.patch' on target directory '/build/tmp/work/core2-64-poky-linux/opencv/4.5.2-r0/git/../git'
Command Error: 'quilt --quiltrc /build/tmp/work/core2-64-poky-linux/opencv/4.5.2-r0/recipe-sysroot-native/etc/quiltrc push' exited with 0 Output:
stdout: Applying patch 0001-sfm-link-with-Glog_LIBS.patch
can't find file to patch at input line 37
Perhaps you used the wrong -p or --strip option?
(From OE-Core rev: caf21ee38f7a96af6c10e80f9422611e317b29d6)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit c44bc7c0fb8b7c2e44dd93607a3bfd9733e1df80)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
DEPENDS here should match what 5.14 is using.
(From OE-Core rev: adc33c4bb8a0f5c542cb1da3b986e89ecea75714)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 899fd41723f41fe0a0cc24373c326b88cb385fe9)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5.14 changed how the GCC plugins are built, which means they now
depend on both GMP and MPC to be built. We already depend on gmp-native,
so add libmpc-native aswell.
(From OE-Core rev: 0c15ed141ea3b23140d3aa4e6ae17ddee0947f3f)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit f242a6db0757b31c0d4eba5c362f616e1ace14d6)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, installed packages are listed for images in image-info.txt, but
not for SDKs in sdk-info.txt. Add TOOLCHAIN_HOST_TASK and
TOOLCHAIN_TARGET_TASK to the output variables in sdk-info.txt.
Moreover, package output files for the SDK host are empty because
PKGDATA_DIR defaults to the target directory. Fix this bug and create a new
variable called PKGDATA_DIR_SDK which stores the correct path for the SDK
host package data.
(From OE-Core rev: af7b5c664649d2c0d1b23eb1d553080b9d2a7864)
Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit 82e6172c1df378dff4e503aa878501c08937b5bb)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing pthread being linked sometimes and not others leading to
non-reproducible target binaries. The reason is mixing the native python
config with the target one. We should use the target one.
(From OE-Core rev: 5d27faf68ff94519d6618351ce87a8b3818ba611)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3fe5101b335384ef83e96ccc58687fd631164075)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing pthread being linked sometimes and not others leading to
non-reproducible target binaries. The reason is mixing the native python
config with the target one. We should use the target one.
(From OE-Core rev: 0a390b5b36bbd1b2a3aefa74d03e8e40240c68fb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1bc5378db760963e2ad46542f2907dd6a592eb66)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Recently a number of CVEs have been logged against a nodejs project
called "node-tar". These appear as false positives against the GNU tar
being built by Yocto. Some of these have been manually excluded using
CVE_CHECK_WHITELIST.
To avoid this problem, use the vendor name (in addition to package name)
for filtering CVEs. The syntax for this is:
CVE_PRODUCT = "vendor:package"
When not specified, the vendor defaults to "%" which matches anything.
(From OE-Core rev: d11e970c6e2482ad0b21994e4ec85ddf2aea1ede)
Signed-off-by: Ralph Siemsen <ralph.siemsen@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit 45d1a0bea0c628f84a00d641a4d323491988106f)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We occasionally see bitbake-worker failing and from the logs, an unpickle error
occurs. Add more debug so we can further debug this next time it fails.
[YOCTO #14595]
(Bitbake rev: 692fa35f4c23722f3179502cb965960cc230e709)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit fe8105cc06beca8240b76ea366a1eff5aa9c5412)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen races where the socket may be gone but the server is still writing
out it's database. Handle that case too to avoid cleanup tracebacks.
[YOCTO #14440]
(Bitbake rev: 36b1b4c4fcee9dde628c7113203939730ab12ae5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b9e4fb843cb9d3a4d4404af093a781fab5520465)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It turns out that for Ubuntu, lz4 is the valid package name for newer
versions (e.g. 20.04 LTS) but not for older ones (e.g. 18.04 LTS, where
the correct package is liblz4-tool). In 20.04 the lz4 package includes
liblz4-tool in its "provides" so it's best to list that one for now.
(From yocto-docs rev: 222af72b9ee307d43a8463283e058c6ebb18fefc)
Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure we have a brief introductory section and tweak the general
migration considerations a little.
(From yocto-docs rev: c94aa8b9d828f9267a70deee05bdf483dc570101)
Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add migration instructions gathered by combing the commits in this
release.
(From yocto-docs rev: b864f8570271df4e6cb47d21cb658d13ffd1d8f5)
Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These are now required so update the corresponding distro-specific
lists used in the system requirements documentation.
(From yocto-docs rev: 1ddd56a98064015582a8c161a1b998c06ebcaf26)
Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was recently removed so remove the reference to it.
(From yocto-docs rev: 46bfdb0b4ae2cb834589ef09436b120715663a31)
Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have a choice of policy with hashequivalence - whether to reduce
sstate duplication in the sstate feed to a minimum or have maximal
sstate reuse from the user's perspective.
The challenge is that non-matching outhashes are generated due to
determinism issues, or due to differences in host gcc version,
architecture and so on and the question is how to reconcile then.
The approach before this patch is that any new match is added and
matches can update. This has the side effect that a queried value
from the server can change due to the replacement and you may not
always get the same value from the server. With the client side
caching bitbake has, this can be suboptimal and when using the
autobuilder sstate feed, it results in poor artefact reuse.
This patch switches to the other possible behaviour, once a hash is
assigned, it doesn't change. This means some sstate artefacts may be
duplicated but dependency chains aren't invalidated which I suspect
may give better overall performance.
Update the tests to match the new behaviour.
(Bitbake rev: 20d6ac753efa364349100cdc863e5eabec8e5b78)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes the hashequivalence server to resolve the diverging report race
error. This error occurs when the same task(hash) is run simultaneous on
two different builders, and then the results are reported back but the
hashes diverge (e.g. have different outhashes), and one outhash is
equivalent to a hash and another is not. If taskhash was not originally
in the database, the client will fallback to using the taskhash as the
suggested unihash and the server will see reports come in like:
taskhash: A
unihash: A
outhash: B
taskhash: C
unihash: C
outhash: B
taskhash: C
unihash: C
outhash: D
Note that the second and third reports are the same taskhash, with
diverging outhashes.
Taskhash C should be equivalent to taskhash (and unihash) A because they
share an outhash B, but the server would not do this when tasks were
reported in the order shown.
It became clear while trying to fix this that single large table to
store all reported hashes was going to make these updates difficult
since updating the unihash of all entries would be complex and time
consuming. Instead, it makes more sense to split apart the database into
two tables: One that maps taskhashes to unihashes and one that maps
outhashes to taskhashes. This should hopefully improve the parsing query
times as well since they only care about the taskhashes to unihashes
table, at the cost of more complex INNER JOIN queries on the lesser used
API.
Note this change does delete existing hash equivlance data and starts a
new database table rather than converting existing data.
(Bitbake rev: dff5a17558e2476064e85f35bad1fd65fec23600)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
