52891 Commits

Author SHA1 Message Date
Richard Purdie
6c5a847a45 bitbake: fetch/git: Handle github dropping git:// support
github is dropping support for git protocol in Git urls. Add code to remap
this to https in a way that could be used in older bitbake versions.

(Bitbake rev: 964958b8b11dc69fb289fc6c97c1dbc8d76ad0f8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 11:31:58 +00:00
Richard Purdie
2faacaf8f4 bitbake: fetch2: Fix os.errno references
os.errno used to happen to work but is invalid. Correct to use errno.

[YOCTO #13068]

(Bitbake rev: b3fc65289d33274cd5dace4d4ffe55be11c991f4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-31 11:28:39 +00:00
Khem Raj
e52122a3e6 qemu: Replace stime() API with clock_settime
(From OE-Core rev: 2cca75155baec8358939e2aae822e256bed4cfe0)

(From OE-Core rev: 1351f9be973cfbd043f9b10d218e3cecaa0ab372)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:10 +01:00
Jaewon Lee
e7e1b9a43d Adding back wrapper and using OEPYTHON3HOME variable for python3
Adding back the python wrapper and adding a patch to use OEPYTHON3HOME
instead of PYTHONHOME if set, for python3.

If we add back the wrapper as is, we would see the following error that
we also see in Thud:

ImportError: No module named site
OpenEmbedded requires 'python' to be python v2 (>= 2.7.3), not python
v3.
Please upgrade your python v2

This is because python3 would've set PYTHONHOME to use nativesdk
python3 libraries but when the oe-buildenv-internal script tries to call
python2 for the py_v27_check, there will be no python2 libraries in the
PYTHONHOME directory.
In other words, bitbake needs host python2 and the env variable set from
the wrapper contaminates the env and host python2 won't be able to find
its libraries

Creating another variable OEPYTHON3HOME and using this in the python3
wrapper to allow for a way to set a different paths for python3 and
python2

[YOCTO #13208]

(From OE-Core rev: 75d2a85e24ef9a2bf0e218521944523f0ff281e0)

(From OE-Core rev: b29e87376fdd49ce07749b87c3000033fa96e43f)

Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:10 +01:00
Hongxu Jia
904e05f467 iso-codes: switch upstream branch master -> main
(From OE-Core rev: a6e098e2e5932781b9c1012825bc86bc08382931)

(From OE-Core rev: 54690c51765d3071406ef1bfd81c9d9db9552108)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6e16ef0c2e0ec2bbb862231cd84e7650bd5789af)
Signed-off-by: Jeremy A. Puhlman <jpuhlman@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:10 +01:00
Konrad Weihmann
bc319fd044 cve-update: handle baseMetricV2 as optional
Currently in NVD DB an item popped up, which hasn't set baseMetricV2.
Let the parser handle it as an optional item.
In case use baseMetricV2 before baseMetricV3

(From OE-Core rev: 77f119baf6f4b85194a9b26d8442ddc7fb3bb97c)

(From OE-Core rev: 4cee5c4bc74edde48fe19ec11c78f6c598cf08b6)

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:10 +01:00
Richard Purdie
0d061fc545 selftest/signing: Ensure build path relocation is safe
Similarly to 04ee0e8b95cd8ed890374e0007f976684206b630, ensure only full
build paths are replaced in the environment to avoid breaking buildtools.

(From OE-Core rev: be07d93a4f59d4563f2d064be1997b39f05e9f0e)

(From OE-Core rev: 7a46226288179df565b7c21c3316672d2e2a1ac0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:10 +01:00
Richard Purdie
422d55320a maintainers: Add entry for buildtools-extended-tarball
(From OE-Core rev: 4281342a04078990bb0a110760ff2dc053eccc93)

(From OE-Core rev: 665ef4274e0261bb8351c8d4fd2c8496a2dc27e7)

(From OE-Core rev: 848af99b4e6afda0658db44128a6921217653e95)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 61d4d3d5a9f27e0fbf1d7ed6db818a779643b8f3)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
000ae7db61 python3-testtools: Avoid traceback2 module requirement
traceback2 adds traceback for python2. Rather than depend on traceback2, we're
python3 only so just use traceback.

This caused breakage in oe-selftest -j which uses testtools on the autobuilder
using buildtools-tarball.

[YOCTO #13652]

(From OE-Core rev: ee80a06c107375e3cf0d246ea17c09dda4536dab)

(From OE-Core rev: ee82e3c24fe5727ce81e972cadedca431d6086c5)

(From OE-Core rev: be4470c9590183b388d9ff176331d0c50984dec8)

(From OE-Core rev: e15ff4775aef99a13acb98501454d1b99c923969)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
e1499a0481 attr: Disable parallel make install
do_install fails on newer versions of make with interesting and hard to
debug errors. Disablle parallle make install as a workaround. Later verisons
of attr in newer releases don't have the issue.

(From OE-Core rev: 6043b9a2ea879f8960897b11eb947801508a94da)

(From OE-Core rev: f06861bbe402fff3f370687585e43c0270609d00)

(From OE-Core rev: 77bfdb505c8483416fbd4e78cf42ad09923c401b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Khem Raj
5592bfe79c uninative: Upgrade to 2.9
This supports glibc upto 2.32 which is now rolling into distributions

(From OE-Core rev: 622371678ddb013fc456eaf75def26fc4e142d15)

(From OE-Core rev: 4543eeacd65eebe74ff3a44182915a732ba26e47)

(From OE-Core rev: ab3c7e09c347a2c57d894ba5e04f38fc9adfad59)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Jeremy Puhlman
5158b599a9 buildtools-extended-tarball: add nativesdk-libxcrypt-dev
virtual/crypt-native is assume provided in bitbake.conf, so
buildtools-extended-tarball shoud provide crypt since it doesn't
use the host's headers/libraries.

[YOCTO #13714]

(From OE-Core rev: da948b25d5ef452fb35275d108e18d2a2829f4fb)

(From OE-Core rev: bc42406d83310398bc4d4db4244252411eff117d)

(From OE-Core rev: 6f6d7278358b042aca3e911aefd0d6128480f32d)

(From OE-Core rev: e1b5cab5cf65df4310b63826690a12ea7083e192)

Signed-off-by: Jeremy A. Puhlman <jpuhlman@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
47724b354a glibc: Update nativesdk locale relocation patch
The locale binary reported incorrect locale lists in relocated toolchains
as some path references were not relocated by this patch. Fix this missing
relocations so the locale binary correctly reports the locales.

(From OE-Core rev: f7a6a72880009380ae81bc7fc863921a26811c8c)

(From OE-Core rev: e4c4337e642f565e9988a4a2c50a995090d1f49e)

(From OE-Core rev: c9e8b7a40b2628331c7cb564aa3f3d9e1822fe36)

(From OE-Core rev: a41c008eb12004ec8938c03dbc495e07c77d45a6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Jeremy Puhlman
bdf9be3ebb buildtools-tarball: add nativesdk-python
(From OE-Core rev: 6467eb4461f3cab16cab2ba63154c92fc2adacef)

(From OE-Core rev: 848c61a07f691638fa529bbe0f0ff1dfded4a967)

(From OE-Core rev: afa4cacff186f28d6a4c4246d1e5caf0aa6938e9)

Signed-off-by: Jeremy Puhlman <jpuhlman@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Steve Sakoman
4f3b564ce4 buildtools-tarball: export OPENSSL_CONF in environment setup
The autobuilder has been experiencing SSL: CERTIFICATE_VERIFY_FAILED
errors during error report uploads when using buildtools due to looking
for certs in /opt/poky

(From OE-Core rev: 197f1d5d14b8e57295f5a81c03c86abba5328614)

(From OE-Core rev: 35c6ab2501672083cf8b974d8b9c3daa3202de36)

(From OE-Core rev: 0cb479a5e99289b75e89b2ed5058f33605f15936)

(From OE-Core rev: f96a3082a0822106dfed73d55117552ccff5734f)

Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Liwei Song
ceef407f00 buildtools-tarball: export OPENSSL_CONF for openssl
export OPENSSL_CONF to aviod SDK openssl can not find openssl.cnf.

(From OE-Core rev: 0aaf3dd17dcde959e9c0d62543cb91c9b33551b4)

(From OE-Core rev: 63d8569b2c9f66e8123e2672a7f8fb8e7cc1f0b4)

(From OE-Core rev: e733a5f3b0e3c3b8a830db5ae99b3fc6b7e56921)

(From OE-Core rev: 22dd23e3d6c4ee2066198fb91554bbe00a582db0)

Signed-off-by: Liwei Song <liwei.song@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Jeremy Puhlman
0e3bb6019d buildtools-extended-tarball: Add libstc++.a
Builds like native-openjdk, really wants a to link
some tools against the static version. Since when
using the extended tarball, its the only place to
get it, add the library.

(From OE-Core rev: 59c4a3fdbbfd5a6aaba7e0a1675dcd5866a7f3a4)

(From OE-Core rev: 152709dec03bbac582ca63b65f2efb835e0b33fb)

(From OE-Core rev: 5e3664e5f9a0dde07b0f8a56cdce1321456abaa5)

(From OE-Core rev: 2cbc936110f1a5d9532b47439b6da1b12caa307b)

Signed-off-by: Jeremy Puhlman <jpuhlman@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Tim Orling
9fbf615baa nativesdk-buildtools-perl-dummy: add dependencies for autoconf and automake
* For buildtools-extended-tarball, where we are adding all of build-essentials
  to the nativesdk, we need additional perl modules for autoconf and automake.

(From OE-Core rev: f0f766160663407ea7683d31bbf5f011accc9ba2)

(From OE-Core rev: e7ade58a7da52ebb40120020dd86dd3ae9b2148e)

(From OE-Core rev: ed9d60fb5d471b4ec472088cc9307fd8575b187a)

(From OE-Core rev: 1276b895008919f510f609d8da4a157d47f09c48)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
6b8c4f7cdf buildtools-extended-tarball: Add locale command
The eSDK installation code checks installed locales with the locale command which is
from glibc-utils. Add this so that we find the correct locales from the buildtools.

(From OE-Core rev: 7d35e4bc6ff94a2d03c48827d7d60a6855c9029d)

(From OE-Core rev: d99b6432decec0964ac0e08698abc782c9b114f5)

(From OE-Core rev: 3562a6848aa3e866ad8e2d3caed3211971817234)

(From OE-Core rev: 76227185faedc0946f2b69a8cfe4286f6e5355d9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
47ba4c7dec binutils: Install non-alternatives links for nativesdk
In the SDK we need the plain symlinks and don't use alternative providers.
When these are missing the toolchain can work incorrectly so fix this.

(From OE-Core rev: 0c06cfaa016d06cc56d80dc1c244a938f3d38a3c)

(From OE-Core rev: 0d299c5dc04407d2d54574157f4014f50f2d0468)

(From OE-Core rev: aa37b5fe0620122e47f36165f5c7a07d3328dba3)

(From OE-Core rev: 6540c5bb9241d5729a0e56f5cf24e1d1d1d4a4cc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
9055290590 buildtools-tarball: Add an ld.so.conf for nativesdk-binutils
We need to search our own libdirs, then fall back to the system ones as our
customised dynamic loader will. Have ld.so.conf reflect that.

This ensures that binutils finds libraries here when linking too.

(From OE-Core rev: ab729c362684474a8346e5256d636200826feb47)

(From OE-Core rev: 8de0aee6befc0541fa40563f63dfe1cc36f064fe)

(From OE-Core rev: d7894d3578d9e97185b4a326c346a3fbb6936ab6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Tim Orling
d6c572cd48 buildtools-extended-tarball: add recipe with build-essentials
* For some aging distros, such as CentOS 7, the native version
  of gcc is simply too ancient and is a constant source of
  headaches for moving forward.

* Add an extended version of buildtools-tarball which adds all
  of build-essential, so that the host is now modernized and
  capable of compiling the latest versions of components.

Fixes [YOCTO #13714]

(From OE-Core rev: f0377af2325613b63716b0bb4db1ab253d79f388)

(From OE-Core rev: bb4979f0e8367b475cc9a5274933a61bb0eb64b3)

(From OE-Core rev: f492e172e133a4b52dbe818d806cab783204e575)

(From OE-Core rev: 4b23c235bdf29cc45ab084e6fdce8cba3ce7fce2)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
c08f2dc635 binutils: Fix relocation of ld.so.conf in nativesdk builds
We need binutils to look at our ld.so.conf file within the SDK to ensure
we search the SDK's libdirs as well as those from the host system.

There add a patch which passes in the directory to the code using a define,
then add it to a section we relocate in a similar way to the way we relocate
the gcc internal paths. This ensures that ld works correctly in our buildtools
tarball.

Standard sysroot relocation doesn't work since we're not in a sysroot,
we want to use both the host system and SDK libs.

(From OE-Core rev: f6c1089642934ad93056ef19a0888965486ee030)

(From OE-Core rev: 09a2b16ac2bd1e3e415131e46315c851373aa7e0)

(From OE-Core rev: d0b7811b0e8654cf83d1b0f8256c7941fc3d9c41)

(From OE-Core rev: 669b73c9f469642085c6ad11b55a9065c889ddbd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Changqing Li
030b0223aa runqemu: add lockfile for port used when slirp enabled
There is race condition when multi qemu starting with slirp,
add lockfile for each port to avoid problem like:

runqemu - ERROR - Failed to run qemu: qemu-system-x86_64: Could not set up host forwarding rule 'tcp::2323-:23'

[YOCTO #13364]

(From OE-Core rev: ceb3555a40ba06e58914465376aaf41392c12a7c)

(From OE-Core rev: 9f9657683df90c18c1dfc7e65715b134a44a9d5a)

Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
8117b6b195 files/toolchain-shar-extract.sh: Rework PATH cleaning
Trying to create a clean PATH breaks cases where we install a buildtools tarball
on hosts to provide newer versions of gcc. Rework the fix for #8698 to clean up
directories in PATH which don't exist isntead. Do it with python as the shell
version was too fraught with corner cases.

(From OE-Core rev: 7674b63819aa7ca95ca5ca5477a5cce32e9691eb)

(From OE-Core rev: cd935db103312f6caec2832de80e49e3ed7d1ed8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
0e4f4f860d populate_sdk_ext: Fix to use python3, not python
We should be using python3 here, it was missed in the conversion. Spotted on
autobuilder tests failing on systems with python missing.

(From OE-Core rev: db07b09196022078346aadd565760240b7da6a71)

(From OE-Core rev: 2ce4dd53443e86c707280716bfe23572eff58abb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
08216625d0 populate_sdk_ext: We now require python3, not python
We no longer expect a "python" binary in PATH so update the eSDK's
expectations to match. This was the only failure on autobuilder test
systems with python missing.

(From OE-Core rev: 946ce21b10dcad506edcaadb4e4242c049e4c316)

(From OE-Core rev: 775336424bcc7c083e2ac6ccd3db0b16e87dc29a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
24f57d0379 oeqa/testsdk: Use original PATH
We want to test the SDK with PATH from the original host, not with our own
tools injected via HOSTTOOLS. It even uses some tools which aren't in
HOSTTOOLS.

This is necessary after changing the SDK to not reset PATH to the system
default which is bad for other reasons and brings the testing into sync
with that change.

(From OE-Core rev: 87c9602fd0dedc7bcf75b822aaf5f6ebfc17737c)

(From OE-Core rev: 2cb99a44c650db7fd6fbd269f5788e4ebfd523fc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
32d7c4b179 oeqa/selftest: Ensure buildtools in environment variables isn't replaced
This avoids the seeing broken replacements like:
oe-selftest-centos/build/build-st-926tools/sysroots/x86_64-pokysdk-linux/etc/ssl/certs/ca-certificates.crt
which understandably break builds.

(From OE-Core rev: 04ee0e8b95cd8ed890374e0007f976684206b630)

(Cherry-picked from f930e2cadb but adjusted for thud)
(From OE-Core rev: 611d3947054dad764aeded4c6a050415f7ca4991)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Michael Halstead
2e1f7a335a yocto-uninative.inc: version 2.8 updates glibc to 2.31
Allow sstate use in Tumbleweed and other distros as they update glibc.

(From OE-Core rev: ccb374c279b260b1fd3460f6bfd1567240816055)

(From OE-Core rev: 0e12f41848fd2fdbc0f70f568ce13baeb3263d03)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Charles-Antoine Couret
08c3882891 utils: fix gcc 10 version detection
Utils can not detect GCC 10 correctly due to wrong regex.
It generates this error "ERROR: Can't get compiler version from gcc  --version output"

Sub-version numbers should be 1 or more digits instead of 1 only.

(From OE-Core rev: 1d6f50a5e58f46f8af6e83c4e288d93a717187ea)

(From OE-Core rev: e73228e6b039bd972d36774bfb360a638a03d821)

Signed-off-by: Charles-Antoine Couret <charles-antoine.couret@mind.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 186fe4a3d390a52b87282c3e694ce3251e45ee78)
Signed-off-by: Jeremy A. Puhlman <jpuhlman@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-16 13:35:09 +01:00
Richard Purdie
7b515c6df5 bitbake: tests/fetch: Allow wget upgrade tests to run against a local server
Currently these tests rely upon multiple uptream webservers which may change
or be unavailable. Add local copies of the test data, copy the httpserver
from OE-Core (used for testing there) and run these tests against a local
server instead.

(Bitbake rev: a21671e8a483ba8a6986d961987eda2d36ec61ca)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-09 09:44:53 +01:00
Richard Purdie
04eaedb804 bitbake: fetch2: Change git fetcher not to destroy old references
It looks like we're about to see a lot of changes in branch names in repos. If
we have the prune option here, those old names are lost, the changes propagate
to our source mirrors and our old releases break.

We have the force option so any replaced references should be replaced, its only
orphaned branches which will now be preserved.

I believe this behaviour will cause us fewer problems given the changes that
look likely to happen.

(Bitbake rev: 12d8cc3fecd550c4aadf0519e80711d755ee75ba)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-08 14:46:13 +01:00
Richard Purdie
3315b9f4a5 bitbake: tests/fetch: Switch from git.infradead.org to a YP mirror
Upstream is unavailable, breaking tests. Switch to a YP mirror since
if we can't reach that there are bigger problems. This should remove
a source of intermittent failures on the autobuilder.

(Bitbake rev: f4e60b29df88393302957c5bbdbe24ca38c4633c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-10 21:10:54 +01:00
Jefro
958427e9d2 Adding memoriam to scottrif
Added a few comment lines in Makefile to commemorate Scott's
contributions

(From yocto-docs rev: 421a80308c36c3da98d5fb6f6100ee3fab6abd0e)

Signed-off-by: Jefro <jefro@jefro.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-31 08:03:28 +00:00
Armpit
f3dfec8aae linux-yocto/4.14: update Yocto Bsps to 4.14.154
(From meta-yocto rev: bf00cab7a55e2038e09a307378af5aec04c99380)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-02 13:33:33 +00:00
Armin Kuster
390f760d17 linux-yocto/4.14: update to 4.14.154
(From OE-Core rev: e68991ceb5933f7d03b96697e8a0ba0829feb320)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00
Ross Burton
3de2aeb687 glibc: finish incomplete fix for CVE-2016-10739
Somehow the patch for this CVE only included one of the four required patches.

(From OE-Core rev: e7ed139e48b683ebe3e6863886e712998aaa239c)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00
Ross Burton
962cbc60d7 cve-check: fetch CVE data once at a time instead of in a single call
This code used to construct a single SQL statement that fetched the NVD data for
every CVE requested.  For recipes such as the kernel where there are over 2000
CVEs to report this can hit the variable count limit and the query fails with
"sqlite3.OperationalError: too many SQL variables".  The default limit is 999
variables, but some distributions such as Debian set the default to 250000.

As the NVD table has an index on the ID column, whilst requesting the data
CVE-by-CVE is five times slower when working with 2000 CVEs the absolute time
different is insignificant: 0.05s verses 0.01s on my machine.

(From OE-Core rev: 53d0cc1e9b7190fa66d7ff1c59518f91b0128d99)

(From OE-Core rev: b52d6340acdad27d41caf057b78f181297a9a75e)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00
Ross Burton
51553d9da2 cve-check: neaten get_cve_info
Remove obsolete Python 2 code, and use convenience methods for neatness.

(From OE-Core rev: f19253cc9e70c974a8e21a142086c13d7cde04ff)

(From OE-Core rev: 1f3863bc31e03207856f55591cbf17543e188587)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00
Ross Burton
1a09e2a3cf cve-check: rewrite look to fix false negatives
A previous optimisation was premature and resulted in false-negatives in the report.

Rewrite the checking algorithm to first get the list of potential CVEs by
vendor:product, then iterate through every matching CPE for that CVE to
determine if the bounds match or not.  By doing this in two stages we can know
if we've checked every CPE, instead of accidentally breaking out of the scan too
early.

(From OE-Core rev: d61aff9e22704ad69df1f7ab0f8784f4e7cc0c69)

(From OE-Core rev: 541dc24d974d3e22c45a650c34298eebc45121e8)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00
Ross Burton
3091533130 cve-update-db-native: clean up proxy handling
urllib handles adding proxy handlers if the proxies are set in the environment,
so call bb.utils.export_proxies() to do that and remove the manual setup.

(From OE-Core rev: 6b73004668b3b71c9c38814b79fbb58c893ed434)

(From OE-Core rev: aa197b91e1770925ae1a31ee7334b593bfcdc9e3)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00
Ross Burton
7da85f3a88 cve-update-db-native: add an index on the CVE ID column
Create an index on the PRODUCTS table which contains a row for each CPE,
drastically increasing the performance of lookups for a specific CVE.

(From OE-Core rev: b4048b05b3a00d85c40d09961f846eadcebd812e)

(From OE-Core rev: 27ee95bd1ec2076509cfc2230eadb876fb35d6c2)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00
Ross Burton
bc70e97a88 cve-update-db-native: don't hardcode the database name
Don't hardcode the database filename, there's a variable for this in
cve-check.bbclass.

(From OE-Core rev: 0d188a9dc4ae64c64cd661e9d9c3841e86f226ab)

(From OE-Core rev: 29cc2b5cd4bcce1c9e93395a1640014877486d7a)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00
Ross Burton
d5ef4072e8 cve-update-db-native: don't refresh more than once an hour
We already fetch the yearly CVE metadata and check that for updates before
downloading the full data, but we can speed up CVE checking further by only
checking the CVE metadata once an hour.

(From OE-Core rev: 50d898fd360c58fe85460517d965f62b7654771a)

(From OE-Core rev: 091a35cfbd2f3e82a7783ba9c8fd5586433ba59f)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00
Ross Burton
e60099368b cve-check: we don't actually need to unpack to check
The patch scanner works with patch files in the layer, not in the workdir, so it
doesn't need to unpack.

(From OE-Core rev: 2cba6ada970deb5156e1ba0182f4f372851e3c17)

(From OE-Core rev: cbb5d26d88465c95a4a879f8635253259e8df0f0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00
Ross Burton
f9b5a31a65 cve-check: failure to parse versions should be more visible
(From OE-Core rev: 72f44bef3867295f73f8b91e17294b2876447c89)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00
Ross Burton
1675f9638a cve-check: ensure all known CVEs are in the report
CVEs that are whitelisted or were not vulnerable when there are version
comparisons were not included in the report, so alter the logic to ensure that
all relevant CVEs are in the report for completeness.

(From OE-Core rev: 98256ff05fcfe9d5ccad360582c36eafb577c264)

(From OE-Core rev: 9d01a64844998d98fcfcebbe8580422094cd2dde)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00
Ross Burton
593fe7e352 cve-check: backport rewrite from master
As detailed at [1] the XML feeds provided by NIST are being discontinued on
October 9th 2019.  As cve-check-tool uses these feeds, cve-check.bbclass will be
inoperable after this date.

To ensure that cve-check continues working, backport the following commits from
master to move away from the unmaintained cve-check-tool to our own Python code
that fetches the JSON:

546d14135c5 cve-update-db: New recipe to update CVE database
bc144b028f6 cve-check: Remove dependency to cve-check-tool-native
7f62a20b32a cve-check: Manage CVE_PRODUCT with more than one name
3bf63bc6084 cve-check: Consider CVE that affects versions with less than operator
c0eabd30d7b cve-update-db: Use std library instead of urllib3
27eb839ee65 cve-check: be idiomatic
09be21f4d17 cve-update-db: Manage proxy if needed.
975793e3825 cve-update-db: do_populate_cve_db depends on do_fetch
0325dd72714 cve-update-db: Catch request.urlopen errors.
4078da92b49 cve-check: Depends on cve-update-db-native
f7676e9a38d cve-update-db: Use NVD CPE data to populate PRODUCTS table
bc0195be1b1 cve-check: Update unpatched CVE matching
c807c2a6409 cve-update-db-native: Skip recipe when cve-check class is not loaded.
07bb8b25e17 cve-check: remove redundant readline CVE whitelisting
5388ed6d137 cve-check-tool: remove
270ac00cb43 cve-check.bbclass: initialize to_append
e6bf9000987 cve-check: allow comparison of Vendor as well as Product
91770338f76 cve-update-db-native: use SQL placeholders instead of format strings
7069302a4cc cve-check: Replace CVE_CHECK_CVE_WHITELIST by CVE_CHECK_WHITELIST
78de2cb39d7 cve-update-db-native: Remove hash column from database.
4b301030cf9 cve-update-db-native: use os.path.join instead of +
f0d822fad2a cve-update-db: actually inherit native
b309840b6aa cve-update-db-native: use executemany() to optimise CPE insertion
bb4e53af33d cve-update-db-native: improve metadata parsing
94227459792 cve-update-db-native: clean up JSON fetching
95438d52b73 cve-update-db-native: fix https proxy issues
1f9a963b9ff glibc: exclude child recipes from CVE scanning

[1] https://nvd.nist.gov/General/News/XML-Vulnerability-Feed-Retirement

(From OE-Core rev: 8c87e78547c598cada1bce92e7b25d85b994e2eb)

(From OE-Core rev: beeed02f9831e75c3f773e44d7efc726f1ff859c)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00
Dan Tran
53acd121ab sudo: Fix CVE-2019-14287
(From OE-Core rev: e21a8e3b2b2b035cf71883f72eeb665e3fa9c078)

Signed-off-by: Dan Tran <dantran@microsoft.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-16 23:11:10 +00:00