The way distutils.version.LooseVersion compare version are tricky, it treat
all these ( "1.0-beta2", "1.0-rc1", "1.0A", "1.0p2" and "1.0pre1") as greater
version than "1.0". This might be right for "1.0A" and "1.0p1" but not for
the rest, also these version could be confusing, the "p" in "1.0p1" can be
"pre" or "patched" version or even other meaning.
Replace Looseversion with custom class, it uses regex to capture common
version format like "1.1.1" or tag format using date like "2020-12-12" as
release section, check for following known string/tags ( beta, rc, pre, dev,
alpha, preview) as pre-release section, any other trailing characters
are difficult to understand/define so ignore them. Compare release
section and pre-release section saperately.
included selftest for the version class.
[YOCTO#14127]
(From OE-Core rev: 6ced85e9ddd3569240f1e8b82130d1ac0fffbc40)
Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output currently shows the remapped product and version fields,
which may not be the actual recipe name/version. As this report is about
recipes, use the real values.
(From OE-Core rev: 18827d7f40db4a4f92680bd59ca655cca373ad65)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Default behavior is not changed. To suppress patched CVEs, set:
CVE_CHECK_REPORT_PATCHED = ""
(From OE-Core rev: 05bd9f1f006cf94cf5324f96df29cd5862abaf45)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The addition of this variable also makes it possible to change the
output suffix of the check files, e.g. in local.conf:
CVE_CHECK_MANIFEST_append = ".txt"
CVE_CHECK_RECIPE_FILE_append = ".txt"
(From OE-Core rev: 0d40f1482c6d87785ae47c46c2305e1df46f459a)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of inventing a new task to fetch the CVE data, use the existing
fetch task.
(From OE-Core rev: f5f97d33a1703d75b9fd9760f2c7767081538e00)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For example, if you just run 'bitbake cve-update-db-native' in a clean
build system, |cve_tmp_file| won't exist yet.
(From OE-Core rev: dd4473f3d8e1c1a587b6de660775e4b46ddc5fad)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously CVE_CHECK_DB_FILE / CVE_CHECK_DB_DIR was the same across
multiconfigs which led to a race condition wherein multiple
cve-update-db-native:do_populate_cve_db tasks could attempt to write to
the same sqlite database. This led to the following task failure:
Error executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_populate_cve_db(d)
0003:
File: '/mnt/data/agent/work/74f119cccb44f133/yocto/sources/poky/meta/recipes-core/meta/cve-update-db-native.bb', lineno: 103, function: do_populate_cve_db
0099: if year == date.today().year:
0100: cve_f.write('CVE database update : %s\n\n' % date.today())
0101:
0102: cve_f.close()
*** 0103: conn.commit()
0104: conn.close()
0105:}
0106:
0107:def initialize_db(c):
Exception: sqlite3.OperationalError: disk I/O error
Use a lockfile to ensure multiple tasks don't step over each other.
(From OE-Core rev: 24e9380643a2ae3fcae193519cb64aedaf682153)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cve-check file should be saved always, it has good info.
Put a copy in the log dir as cve-summary with symlinks to latest run.
[Yocto #13974]
(From OE-Core rev: 859849c7b594d844819ad8c3f7d8325388d94b93)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the generated cve.log files, include the epoch in the product
version. This better matches how versions are displayed elsewhere,
in particular the bb.warn("Found unpatched CVE...") that appears
on the terminal when CVEs are found.
(From OE-Core rev: e1c3c0b6e5b01304e2127f5058986697e82adf93)
Signed-off-by: Ralph Siemsen <ralph.siemsen@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Certain recipes e.g. bash readline ( from meta-gplv2 ) download patches instead of having them in
metadata, this could fail cve_check
ERROR: readline-5.2-r9 do_cve_check: File Not found: qemuarm/build/../downloads/readline52-001
This patch ensures that download is done before running CVE scan, even
though these will be external patches and may not contain CVE tags as it
expects, but it will fix the run failures as seen above
(From OE-Core rev: e406fcb6c609a0d2456d7da0d2406d2d9fa52dd2)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CPE version could be '-' to mean no version info.
Current cve_check treat it as not valid and does not report these
CVE but some of these could be a valid vulnerabilities.
Since non-valid CVE can be whitelisted, so treat '-' as all version
and report all these CVE to capture possible vulnerabilities.
Non-valid CVE to be whitelisted separately.
[YOCTO #13617]
(From OE-Core rev: c69ee3594079589d27c10db32bc288566ebde9ef)
Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
change whitelisted CVE status from "Patched" to "Whitelisted".
[Yocto #13687]
(From OE-Core rev: 181bdd670492525f9488d52c3ebb9a1b142e35ea)
Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With out these changes, a traceback displayed when a file
is listed in the SRC_URI but the file does not exist.
raise FileNotFoundError and print the patch then mark the task as failed.
(From OE-Core rev: d4926c11a4ab9148bdb640a9367c9e1891491a5b)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switch to recently released version 1.1 of NVD CVE JSON feed, as in
https://nvd.nist.gov/General/News/JSON-1-1-Vulnerability-Feed-Release
it is mentioned that
Due to changes required to support CVSS v3.1 scoring, the JSON
vulnerability feeds must be modified. This will require the consumers
of this data to update their internal processes. We will be providing
the JSON 1.1 schema on the data feeds page and the information below
to prepare for this transition.
...
The JSON 1.1 data feeds will be available on September 9th, 2019. At
that time the current JSON 1.0 data feeds will no longer available.
This change was tested briefly by issuing 'bitbake core-image-minimal'
with 'cve-check.bbclass' inherited via local.conf, and then comparing
the content between the resulting two
'DEPLOY_DIR_IMAGE/core-image-minimal-qemux86.cve' files, which did not
seem to contain any other change, except total of 167 entries like
CVSS v3 BASE SCORE: 0.0
were replaced with similar 'CVSS v3 BASE SCORE:' entries which had
scores that were greater than '0.0' (up to '9.8').
(From OE-Core rev: cc20e4d8ff2f3aa52a2658404af9a0ff358cc323)
Signed-off-by: Niko Mauno <niko.mauno@iki.fi>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code used to construct a single SQL statement that fetched the NVD data for
every CVE requested. For recipes such as the kernel where there are over 2000
CVEs to report this can hit the variable count limit and the query fails with
"sqlite3.OperationalError: too many SQL variables". The default limit is 999
variables, but some distributions such as Debian set the default to 250000.
As the NVD table has an index on the ID column, whilst requesting the data
CVE-by-CVE is five times slower when working with 2000 CVEs the absolute time
different is insignificant: 0.05s verses 0.01s on my machine.
(From OE-Core rev: 53d0cc1e9b7190fa66d7ff1c59518f91b0128d99)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove obsolete Python 2 code, and use convenience methods for neatness.
(From OE-Core rev: f19253cc9e70c974a8e21a142086c13d7cde04ff)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A previous optimisation was premature and resulted in false-negatives in the report.
Rewrite the checking algorithm to first get the list of potential CVEs by
vendor:product, then iterate through every matching CPE for that CVE to
determine if the bounds match or not. By doing this in two stages we can know
if we've checked every CPE, instead of accidentally breaking out of the scan too
early.
(From OE-Core rev: d61aff9e22704ad69df1f7ab0f8784f4e7cc0c69)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The patch scanner works with patch files in the layer, not in the workdir, so it
doesn't need to unpack.
(From OE-Core rev: 2cba6ada970deb5156e1ba0182f4f372851e3c17)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CVEs that are whitelisted or were not vulnerable when there are version
comparisons were not included in the report, so alter the logic to ensure that
all relevant CVEs are in the report for completeness.
(From OE-Core rev: 98256ff05fcfe9d5ccad360582c36eafb577c264)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
djb2 hash algorithm was found to do collisions, so the database was
sometime missing data. Remove this hash mechanism, clear and populate
elements from scratch in PRODUCTS table if the current year needs an
update.
(From OE-Core rev: 78de2cb39d74b030cd4ec811bf6f9a6daa003d19)
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CVE_CHECK_WHITELIST does not contain version anymore, as it was not
used. This variable should be set per recipe.
(From OE-Core rev: 7069302a4ccbb5b72e1902f284cf078516fd7294)
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some product names are too vague to be searched without also matching the
vendor, for example Flex could be the parser compiler we ship, or Adobe Flex, or
Apache Flex, or IBM Flex.
If entries in CVE_PRODUCT contain a colon then split it as vendor:product to improve the search.
Also don't use .format() to construct SQL as that can lead to security
issues. Instead, use ? placeholders and lets sqlite3 handle the escaping.
(From OE-Core rev: e6bf90009877d00243417898700d2320fd87b39c)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes build failure with core-image-minimal:
Exception: UnboundLocalError: local variable 'to_append' referenced before assignment
(From OE-Core rev: 270ac00cb43d0614dfe1c95f960c76e9e5fa20d4)
Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As glibc will be scanned for CVEs, we don't need to scan glibc-locale,
glibc-mtrace, and glibc-scripts which are all separate recipes for technical
reasons.
Exclude the recipes by setting CVE_PRODUCT in the recipe, instead of using the
global whitelist.
(From OE-Core rev: 1f9a963b9ff7ebe052ba54b9fcbdf7d09478dd17)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CVE-2014-2524 is a readline CVE that was fixed in 6.3patch3 onwards, but the
tooling wasn't able to detect this version. As we now ship readline 8 we don't
need to manually whitelist it, and if we did then the whitelisting should be in
the readline recipe.
(From OE-Core rev: 07bb8b25e172aa5c8ae96b6e8eb4ac901b835219)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that cve-update-db added CPE information to NVD database. We can
check for unpatched versions with operators '<', '<=', '>', and '>='.
(From OE-Core rev: bc0195be1b15bcffe60127bc5e8b7011a853c2ed)
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the NVD url is not accessible, print a warning on top of the CVE
report, and continue. The database will not be fully updated, but
cve_check can still run on the previous database.
(From OE-Core rev: 0325dd72714f0b447558084f481b77f0ec850eed)
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of generating a series of indexes via range(len(list)), just iterate the
list.
(From OE-Core rev: 27eb839ee651c2d584db42d23bcf5dd764eb33f1)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the NVD json CVE feed, affected versions can be strictly matched to a
version, but they can also be matched with the operator '<='.
Add a new condition in the sqlite query to match affected versions that
are defined with the operator '<='. Then use LooseVersion to discard all
versions that are not relevant.
(From OE-Core rev: 3bf63bc60848d91e90c23f6d854d22b78832aa2d)
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are both "curl" and "libcurl" CPEs in NVD.
All "curl" CVEs are currently missing in the reports.
Hence, switch "CVE_PRODUCT" to a space separated list.
It is useful for recipes generating several packages,
that have different product names in NVD.
(From OE-Core rev: 404f75e026393ddc55da87f6f04fb1201cff4e11)
Signed-off-by: Grygorii Tertychnyi <gtertych@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Disable downloading of the vulnerability DB in do_check_cves() task.
When invoked in this task, cve-check-tool attempts re-download of the CVE DB
if the latter is older than certain threshold. While reasonable for a
stand-alone CVE checker, this behavior can cause errors in parallel builds
if the build time is longer than this threshold:
* Other tasks might be using the DB.
* Several packages can start the download of the same file at the same time.
This check is not really needed, as the DB has been downloaded by
cve_check_tool:do_populate_cve_db() which is a prerequisite of any do_build().
The DB will be at most (threshold + build_time) old.
(From OE-Core rev: 125789b6ee6d47ab84192230f63971c4e22418ba)
Signed-off-by: Konstantin Shemyak <konstantin.shemyak@ge.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some backported patches fix multiple CVEs and list the corresponding
identifiers on multiple lines, rather than on a single line.
cve-check.bbclass yields false positive warnings when CVE IDs are
presented on multiple lines because re.search() returns only
the first match.
An example of this behavior may be found when running do_cve_check() on
the wpa-supplicant recipe while in the rocko branch. Only CVE-2017-13077
is reported to be patched by commit de57fd8, despite the patch including
fixes for a total of 9 CVEs.
This is resolved by iterating over all regular expression matches,
rather than just the first.
(From OE-Core rev: 8fb70ce2df66fc8404395ecbe66a75d0038f22dd)
Signed-off-by: Jon Szymaniak <jon.szymaniak.foss@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The binary 'cve-check-update' downloads the CVE database from the Internet.
If the system is behind a web proxy, the download fails, as proxy-related
variables are not exported.
In turn, 'cve-check-tool' does not connect to the network and correspondingly
does not need exported proxies.
Exported all proxy-related environment variables to 'cve-check-update' and
removed the unneeded export from 'cve-check-tool'.
(From OE-Core rev: 17db210975c740aff12732c511cf4fb32b507365)
Signed-off-by: Konstantin Shemyak <konstantin.shemyak@ge.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some recipes is is meaningless to do a CVE check, for example packagegroups
or images. Check that CVE_PRODUCT is set and short-circuit the scan if it
isn't.
(From OE-Core rev: d1e7cb5c9e0d5d253b6bb5c968fa58944ea42d06)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For reasons which I don't understand, the Berkeley DB tarball is version 5.3.28
but in CVE reports the version is 11.2.5.3.28.
To handle this allow recipes to override their version as well as their name.
(From OE-Core rev: 36fbf96cf284acbc810ff3bf00702f1f82bc0da9)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is where the other task logs go, so it's a sensible place to put it.
(From OE-Core rev: 4bbb8cd5f3943231ab5be0448d1b0d4a08341249)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While poky master branch has been fixed so that all CVE patch files have
the:
CVE: CVE-2017-1234556
strings in the patch comments, many older versions of poky and other meta
layers are not, but the CVE patches quite often have the CVE id in the
patch file name.
If the CVE: string also found, there are no duplicates in the report.
(From OE-Core rev: 5ee5b0c66627c9e974c838b86e2e659c2f601f2a)
Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PV is the recipe major version number. cve-check tries to map that to
NVD database release versions of the component. If the recipe sources
are taken from git, the PV can be automatically modified to include
git details, but the syntax is like 233+gitAUTOINC+a1e2ef7ec9.
In CVE checks we want to remove the git details and just use the major
version information, in this example 233.
Thus use "+git" as the separator and use the first part before the separator
as SW product version number in CVE check.
Fixes version number for e.g. systemd recipe. If systemd PV is
233+gitAUTOINC+a1e2ef7ec9 there will be no matches from CVE database where
latest release mentioned is plain 233. If the filter is set to +git, then
CVE PV is 233 and issues like this are detected by do_cve_check:
https://nvd.nist.gov/vuln/detail/CVE-2017-1000082
(From OE-Core rev: db8815abe3db60b0510fb378bf6d82172c2f2768)
Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This way also bbclasses can override it. For example kernel.bbclass
could set CVE_PRODUCT to linux_kernel for all users of the class
which compile Linux kernels.
(From OE-Core rev: 74672a7de5ada45ab8e25b89cbdea3ec33b63b7f)
Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since do_rootfs depends on cve-check results of all recipes,
we need to recursively depend on recipe do_cve_check.
(From OE-Core rev: a1af526e43cb476472a6203882c12deef297f542)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes there are control messages in cve-check-tool printed to
stderr. These lead to parsing error and thus failed build.
This can happen for instance when cve database needs to be
refreshed during build.
(From OE-Core rev: 6d8a17f4c50be292990e37de65630a6b84466ee6)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is cve report for a recipe in previous build and there
is no result for current one, old cves are kept in CVE_CHECK_DIR.
This happens on version upgrade or when cve/recipe is whitelisted.
(From OE-Core rev: 85b4941c71a0e3c08a8c48d52a94dfe2897d2c92)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When warning users about unpatched CVE, we'd better put CVE IDs into
the warning message, so that it would be more straight forward for the
user to know which CVEs are not patched.
So instead of:
WARNING: gnutls-3.5.9-r0 do_cve_check: Found unpatched CVE, for more information check /path/to/workdir/cve/cve.log.
We should have:
WARNING: gnutls-3.5.9-r0 do_cve_check: Found unpatched CVE (CVE-2017-7869), for more information check /path/to/workdir/cve/cve.log.
(From OE-Core rev: ad46069e7b58f2fba373131716f28407816fa1a6)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With recipe-specific sysroots the cve_check task must depend on
cve-check-tool-native:do_populate_sysroot to get the cve-check-tool
binary into the recipe sysroot.
A normal DEPENDS isn't used to avoid cyclic dependencies.
(From OE-Core rev: bd60b1018bc0304bc928701e6d1090c8b1223616)
Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
