Ensure that begin_idx (i.e., beginline - 1) and end_idx (i.e.,
endline) are positive numbers in copy_license_files(). This makes sure
the same lines are copied as populate_lic_qa_checksum() uses when it
calculates the checksum. Before, beginline=0 would typically lead to
that no lines were copied at all.
(From OE-Core rev: 27cd074d747c5ef4b475c8a62a8ede2bbe58f996)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ab3cc3651d08d226675c461da760cda0bb6c0ce0)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Several tasks deactivate exiting on non-zero return codes via set +e because
they run subcommands that have legitimate non-zero return codes. However when
appending to those tasks, this behavior is not expected and can lead to builds
silently proceeding in case of an error. Therefore reset the default behavior
at the end of the respective tasks via set -e.
(From OE-Core rev: 7777cc853db7200b819095be6f6a3561738ac94d)
Signed-off-by: Stefan Wiehler <stefan.wiehler@nokia.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 83a6f28d2e464f00202090e998a63045adba9e4e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Until now the CVE checker was giving information about CVEs found for
a product (or more products) contained in a recipe. However, there was
no easy way to find out which products or recipes have no CVEs. Having
no reported CVEs might mean there are simply none, but can also mean
a product name (CPE) mismatch.
This patch adds CVE_CHECK_COVERAGE option enabling a new type of
statistics. Then we use the new JSON format to report the information.
The legacy text mode report does not contain it.
This option is expected to help with an identification of recipes with
mismatched CPEs, issues in the database and more.
This work is based on [1], but adding the JSON format makes it easier
to implement, without additional result files.
[1] https://lists.openembedded.org/g/openembedded-core/message/159873
(From OE-Core rev: c63d06becc340270573bdef2630749db1f5230d4)
Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit d1849a1facd64fa0bcf8336a0ed5fbf71b2e3cb5)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the cve-check text mode output, we didn't write fragment
files if there are no CVEs (if CVE_CHECK_REPORT_PATCHED is 1),
or no unpached CVEs otherwise.
However, in a system after multiple builds,
cve_check_write_rootfs_manifest might find older files and use
them as current, what leads to incorrect reporting.
Fix it by always writing a fragment file, even if empty.
(From OE-Core rev: 4c10ee956f21ea2f805403704ac3c54b7f1be78c)
Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f1b7877acd0f6e3626faa57d9f89809cfcdfd0f1)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running CVE checks in CI we're usually not interested in warnings on the
console for any CVEs present. Add a configuration option CVE_CHECK_SHOW_WARNINGS
to allow this to be disabled (it is left enabled by default).
(From OE-Core rev: d009233f36fb866f6bdaa12fb6deedf5e253e9c9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1054d3366ba528f2ad52585cf951e508958c5c68)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
(cherry picked from commit 8fd6a9f521ea6b1e10c80fe33968943db30991ba)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before this the rootfs manifest and the summary were identical.
We should separate the summary and rootfs manifest more clearly,
now the summary is for all CVEs and the rootfs manifest is only for
things in that image. This is even more useful if you build multiple
images.
(From OE-Core rev: 2bacd7cc67b2f624885ce9c9c9e48950b359387d)
Signed-off-by: Ernst Sjöstrand <ernstp@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3b8cc6fc45f0ea5677729ee2b1819bdc7a441ab1)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
(cherry picked from commit 65498411d73e8008d5550c2d0a1148f990717587)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As product, sdk should do cve check as well as rootfs.
(From OE-Core rev: df09cd71b4cd3f830fced9ce91aa202c1609bfc5)
Signed-off-by: Lei Maohui <leimaohui@fujitsu.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit cc17753935c5f9e08aaa6c5886f059303147c07b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The addition of summary output caused two issues: error when building
an image and the fact that JSON output was generated even when
CVE_CHECK_FORMAT_JSON.
When generating an image it caused an error like:
ERROR: core-image-minimal-1.0-r0 do_rootfs: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:cve_check_write_rootfs_manifest(d)
0003:
File: '/home/alexk/poky/meta/classes/cve-check.bbclass', lineno: 213, function: cve_check_write_rootfs_manifest
0209:
0210: link_path = os.path.join(deploy_dir, "%s.json" % link_name)
0211: manifest_path = d.getVar("CVE_CHECK_MANIFEST_JSON")
0212: bb.note("Generating JSON CVE manifest")
*** 0213: generate_json_report(json_summary_name, json_summary_link_name)
0214: bb.plain("Image CVE JSON report stored in: %s" % link_path)
0215:}
0216:
0217:ROOTFS_POSTPROCESS_COMMAND:prepend = "${@'cve_check_write_rootfs_manifest; ' if d.getVar('CVE_CHECK_CREATE_MANIFEST') == '1' else ''}"
Exception: NameError: name 'json_summary_name' is not defined
The fix is to pass the d variable to the pure python function generate_json_report
to get correct values of variables and add conditions for the JSON
output where needed.
In addition clarify the message presenting the summary JSON file,
which isn't related to an image.
Uses partial fixes from Alex Kiernan, Ernst Sjöstrand (ernstp),
and Davide Gardenal.
Fixes: f2987891d315 ("cve-check: add JSON format to summary output")
(From OE-Core rev: 665f981fccbb09d51349c4bd4cfe4ca91001e3bd)
Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9015dec93233c7d45fd0c9885ff5d4ec23ad377d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We'd like to intercept git calls but we don't want circular references
and HOSTTOOLS currently sets them up. Tweak to avoid them.
(From OE-Core rev: 1567b7cec5ccbe198bfd0cca9ee8a2b1cf6dbf42)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 52c37e133fa55846aca2248ffcf3a10648dbb8d7)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating the manifest and the testdata.json links, if the link
name is equal to the output name the link is not created, otherwise
it is. This prevents a link-to-self in the first case.
(From OE-Core rev: e3672b5ccd6e0f130b1657017802db130a859d20)
Signed-off-by: Davide Gardenal <davide.gardenal@huawei.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit bed63756c56f296ff3d5a7eef66e978bd19f1008)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
An if statement now checks if the link and output path are
the same, if they are then the link is not created,
otherwise it is.
(From OE-Core rev: 62965ca8ca7077c12d75dac37efe204d7159cddd)
Signed-off-by: Davide Gardenal <davide.gardenal@huawei.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit 2f024c0236c4806f0e59e4ce51a42f6b80fdf1b3)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create generate_json_report including all the code used to generate the JSON
manifest file.
Add to cve_save_summary_handler the ability to create the summary in JSON format.
(From OE-Core rev: d8ef964ffeb92684d01d71c983af9dbb1e1b0c4f)
Signed-off-by: Davide Gardenal <davide.gardenal@huawei.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit f2987891d315466b7ef180ecce81d15320ce8487)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The only part of the cve-check task which needs files is the patch
examination, and typically these patches are local so fetch isn't needed.
(From OE-Core rev: 72e5204bc7272414cc7bcfba18f52a177242ed79)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2c9b3186d3b7c18cbea239ab9b06e85b7c243b54)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
(cherry picked from commit 3dc8edd6611e7ad4abcece44ca4701eda7aeff94)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're going to use the environment approach for solving this issue.
(From OE-Core rev: a58a94e451bb62df70c60b5d7d27c0074b849e0e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0982977dc052ad4e65608f6853f930121d08837a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In a devshell, recent versions of git will complain if the repo is owned
by someone other than the current UID - consider this example:
------
bitbake -c devshell linux-yocto
[...]
kernel-source#git branch
fatal: unsafe repository ('/home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source' is owned by someone else)
To add an exception for this directory, call:
git config --global --add safe.directory /home/paul/poky/build-qemuarm64/tmp/work-shared/qemuarm64/kernel-source
kernel-source#
------
Of course the devshell has UID zero and the "real" UID is for "paul" in
this case. And so recent git versions complain.
As the whole purpose of the devshell is to invoke a shell where development
can take place, having a non-functional git is clearly unacceptable.
Richard suggested we could use PSEUDO_UNLOAD=1 to evade this issue, and I
suggested we probably will see other similar instances like this and should
make use of PATH to intercept via devshell wrappers - conveniently we already
have examples of this.
Here, we copy the existing "ar" example and tune it to the needs of git to
combine Richard's suggestion and mine.
As such we now also can store commit logs and use send-email with our user
specific settings, instead of "root", so in additon to fixing basic
commands like "git branch" it should also increase general usefulness.
RP: Tweaked the patch so the PATH change only applies to the devshell task
and is a generic git intercept rather than devshell specific.
RP: Also apply the PATH change to do_install tasks since that also runs under
fakeroot and several software projects inject "git describe" output into
their binaries (systemd, iputils, llvm, ipt-gpu-tools at least) causing
reproducibility issues from systems with different git versions.
(From OE-Core rev: 0ff2cff2c1eac3fe6304644db0fc27ba3130c237)
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3266c327dfa186791e0f1e2ad63c6f5d39714814)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Backport to dunfell from master df567de36ae5964bee433ebb97e8bf702034994a
Add an option to output the CVE check in a JSON-based format.
This format is easier to parse in software than the original
text-based one and allows post-processing by other tools.
Output formats are now handed by CVE_CHECK_FORMAT_TEXT and
CVE_CHECK_FORMAT_JSON. The text format is enabled by default
to maintain compatibility, while the JSON format is disabled
by default.
The JSON output format gets generated in a similar way to the
text format with the exception of the manifest: appending to
JSON arrays requires parsing the file. Because of that we
first write JSON fragments and then assemble them in one pass
at the end.
(From OE-Core rev: 92b6011ab25fd36e2f8900a4db6883cdebc3cd3d)
Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Define METADATA_BRANCH and METADATA_REVISION using immediate expansion.
This avoids running `git rev-parse HEAD` multiple times during recipe
parsing.
(From OE-Core rev: f76fc325520386c0d25c6d6b422c85e9fb87e663)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 34e1841ec14c545c73fbe03a9f946d43d65ab326)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The missing gitsm:// mappings looks like an oversight, add them.
(From OE-Core rev: f748360a854a5c80c8008ba6ac90546865651f10)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6600b9fca7888fb41647cd000b9efb7f0762dfde)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
nativesdk is a cross compiled target and therefore should use the target
config, not the native one. Copy the target entries accordingly.
(From OE-Core rev: e997487c0068bfe4017fc98c4fa5b51f660a1b4e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b1b5fec350b390fa7f2d26966df1411b032faf87)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When inside the threadedpool we make a copy of the localdata
to avoid some race condition, so we need to use this new
localdata2 and stop write the shared localdata.
(From OE-Core rev: 604146a242c3d5f5a9872bb756910f4bd1b58406)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 90fe6948a9df0b43c58120a9358adb3da1ceb5b9)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
uninative works via hashes and doesn't need the version in the tarball name but
it does make things easier to inspect in DL_DIR. There were reasons such as
ease of publication of the build tarballs but we can handle those differently
now and the signature issues from the early code aren't an issue now. From 3.4
onwards we can use a version'd name.
[YOCTO #12970]
(From OE-Core rev: 0ec0e49d0d2a7478efbf20bc3554f0ffba40afa0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit dadba70d6a24d8ebb5576598efffa973151c7218)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the BUILDHISTORY_RESET is enabled we need to move the
content from BUILDHISTORY_DIR to BUILDHISTORY_OLD_DIR but
when we start a clean build in the first run we don't have the
BUILDHISTORY_DIR so the move of files will fail.
| ERROR: Command execution failed: Traceback (most recent call last):
| File "/xxx/poky/bitbake/lib/bb/command.py", line 110, in runAsyncCommand
| commandmethod(self.cmds_async, self, options)
| File "/xxx/poky/bitbake/lib/bb/command.py", line 564, in buildTargets
| command.cooker.buildTargets(pkgs_to_build, task)
| File "/xxx/poky/bitbake/lib/bb/cooker.py", line 1481, in buildTargets
| bb.event.fire(bb.event.BuildStarted(buildname, ntargets), self.databuilder.mcdata[mc])
| File "/xxx/home/builder/src/base/poky/bitbake/lib/bb/event.py", line 214, in fire
| fire_class_handlers(event, d)
| File "/xxx/poky/bitbake/lib/bb/event.py", line 121, in fire_class_handlers
| execute_handler(name, handler, event, d)
| File "/xxx/poky/bitbake/lib/bb/event.py", line 93, in execute_handler
| ret = handler(event)
| File "/xxx/poky/meta/classes/buildhistory.bbclass", line 919, in buildhistory_eventhandler
| entries = [ x for x in os.listdir(rootdir) if not x.startswith('.') ]
| FileNotFoundError: [Errno 2] No such file or directory: '/xxx/buildhistory'
(From OE-Core rev: de89dc125758f828a7886012bd9b1c8a1017ef48)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 97bc2168da7dbacdfbf79cd70db674363ab84f6b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The linux kernel will by default use pkg-config to get ncurses(w) paths,
falling back to absolute path checks otherwise. If the build host does
not have ncurses installed this will fail as pkg-config will not search
the native sysroot for ncurses.
To more all kernel/kconfig sources, inject the equivalent native
pkg-config variables similar to what is done by the pkg-config-native
script. This only affects the menuconfig python task itself and the
oe_terminal call inside it.
(cherry picked from commit abb95c421bb67d452691819e3f63dabd02e2ba37)
(From OE-Core rev: dc6b20475a69c9fbab9a97a93119aeedf54deb23)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that all of the functions in cve-check open the database read-only,
we can remove this lockfile.
This means cve-check can run in parallal again, improving runtimes
massively.
This reverts commit d55fbf4779.
(From OE-Core rev: 1a30a8513ca47890470ee9d19a5ea36437e664bf)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e60d149b41d14d177df20dbecaef943696df1586)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All of the function in cve-check should open the database read-only, as
the only writer is the fetch task in cve-update-db. However,
get_cve_info() was failing to do this, which might be causing locking
issues with sqlite.
(From OE-Core rev: 2b3d13a451e99db669977d4d1172653b736ae6e1)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8de517238f1f418d9af1ce312d99de04ce2e26fc)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
example.com is proving unreliable at present so switch to our own connectivity
page instead. That page is very simple avoiding app overhead on our web server
which was an original reason for switching to example.com.
(From OE-Core rev: 0ed4759eb306596575e3e9859a023f44f814cf4c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit dc6b043cb75c5751b5a98afd2201aa31f9b4b9f6)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create directory of the CVE_CHECK_MANIFEST variable before copy to it,
so that the variable can use an arbitrary directory name.
(From OE-Core rev: ee05539fc6fadff7f14c8609641021fc497cbd8e)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9829c16301bf2dce39fa046401a984f112fa0322)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This continues where commit 676757f "sstate: fix touching files inside
pseudo" and commit 29fc8599 "sstate: another fix for touching files
inside pseudo" left off.
The previous changes switched from trying to check if the sstate file is
writable before touching it, to always touching the sstate file and
ignoring any errors. However, if the sstate file is actually a symbolic
link that links to nothing, this would actually result in an empty
sstate file being created. And this in turn leads to that future
setscene tasks will fail when they try to unpack the empty file.
Change the code so that if an sstate file linking to nothing already
exists, it is overwritten with the new sstate file. Also change it so
that the temporary file that is used is always removed, even if ln
fails to link the sstate file to it.
Change-Id: I3800f98d0f2a0dd076352df85fad7c81460e733d
(From OE-Core rev: f3cd092bf9f66d8d73075e5b777d89d8598691dd)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
this should prevent running into the very rare error
sqlite3.OperationalError: attempt to write a readonly database
As highlighted by https://www.sqlite.org/faq.html#q5
it is likely that the adapter won't allow use multiple exec calls
at the same time.
So it's best to prevent multiple accesses at a time, by reusing
the already in place CVE_CHECK_DB_FILE_LOCK
YOCTO #14110
(From OE-Core rev: 3c69e4eb08701516150c78cf8b48f3e90d197b2b)
Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 677f5741bd265be49d4a5bb933b3e8d8c4eec653)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of inventing a new task to fetch the CVE data, use the existing
fetch task.
(From OE-Core rev: 1ed53d5cfc2be40b2d57b5392ec4d30313209934)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f5f97d33a1703d75b9fd9760f2c7767081538e00)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code was assuming that the a recipe with only one srcrev wouldn't "name"
it. This isn't the case as the glibc or bzip2 recipes show, you can have
a single srcrev which is named.
We can pull the data from the fetcher and in fact we already have it, we just
need to handle the "default" case and make that code the default for all srcrev
regardless of length.
[YOCTO #14017]
(From OE-Core rev: 9a7ea10660d0efd87e8cadf866e6dbed112b7f94)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 45ae567932ba52b758eb41754453e9828d9533a1)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
kernel.org now has a mirror of the downloads.yoctoproject.org sources
archive so include this in our mirrors list.
(From OE-Core rev: dbbec36638f035aefbaf22f152a2ca95836d08bd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously if BUILD_REPRODUCIBLE_BINARIES was set to 0, the system would
fall back and select the default epoch (April 2011), but still perform
the reproducible build actions. This resulted in binaries that had an
unusually old date.
Simplify the functions and remove the anonymous python as no longer
necessary.
Also improve the documentation to better explain what the class is doing
and how a recipe can override the behavior if necessary.
(From OE-Core rev: 814bedacac9f5c343c9888c0b14649189a84f817)
Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1976013b026cfba94de32a13e994d92d7e9e39e5)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This sstate task is only needed when depended upon, it can be skipped
if there are no tasks running that directly depend upon it.
This reduced the number of sstate tasks in something like an image
build.
(From OE-Core rev: 884e44701ada57abe4d8ad9ece424435be25c6a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 601cee016da5c7505915e26641a085714de175ce)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating packages we build them with --clamp-mtime and use
SOURCE_DATE_EPOCH as the maximum mtime. This makes the end packages
reproducible. The data stored in sstate for do_package and the package
task doesn't benefit from this though and have varying timestamps.
This means their outhash varies and means hash equivalance isn't
effective at all and doesn't work as intended/desired.
We could create the sstate archives with the same clamping however
that would lead to different results depending on whether a task was
installed from sstate or not. Making that differ is a path to madness.
It also wouldn't fix the outhash of the task to be determninistic
without clamping of the date in the hash calculation code.
Instead, iterate over the files in sstate output and clamp them at
the code level. This isn't ideal but does make the file timestamps
determnistic everywhere and means we don't have to change the hash
calculation code.
This issue can be clearly seen looking at the do_package outhash for
a recipe which you then re-run the package task for after adding
something like whitespace to the install task. The outhash shouldn't
change but currently does.
(From OE-Core rev: 06b8f2a5a24be1a87f0eaf29fdba719ebe3bb06e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c3b3cc4745811b48b9193f83889946b2e1788932)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch is a fixup for 676757f "sstate: fix touching files inside pseudo"
running the 'id' command inside the sstate_unpack_package
function shows that this funcion run inside the pseudo:
uid=0(root) gid=0(root) groups=0(root)
The check for [ -w ${SSTATE_PKG} ] and [ -O ${SSTATE_PKG}.siginfo ]
will always return true and the touch can fail when the real user
don't have permission or in readonly filesystem.
As the documentation refers:
- the file test operator "-w" check if the file has write permission
(for the user running the test).
- the file test operator "-O" check if you are owner of file
We can avoid this test running the touch and mask any return errors
that we have.
(From OE-Core rev: 29fc85997ade490ae46ffca37ef8e1a56957c876)
(From OE-Core rev: e7d94a9cc5ab1b2c5d160fd06d643a4bc3409d26)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5b9210d66c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At the last nas outage, we realised that we don't have good mirrors of the
uninative tarball if our main system can't be accessed. kernel.org mirrors
some Yocto Project data so we've ensured uninative is there. Add the appropriate
mirror url to make use of that.
(From OE-Core rev: ebeb1458c7f24cd97978beb7cddf814cae43c6a2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This update was made with the convert-scruri.py script in scripts/contrib
This script handles two emerging issues:
1. There is uncertainty about the default branch name in git going forward.
To try and cover the different possible outcomes, add branch names to all
git:// and gitsm:// SRC_URI entries.
2. Github are dropping support for git:// protocol fetching, so remap github
urls as needed. For more details see:
https://github.blog/2021-09-01-improving-git-protocol-security-github/
(From OE-Core rev: 827a805349f9732b2a5fa9184dc7922af36de327)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sstate has been sufficiently invalidated since this change was made, drop
the workaround now.
(From OE-Core rev: 81b602eaab686fa5a523c023285f8ffd7050888d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 7a497d8f30f21bafc78d0f22f3442a9cc99544cb)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Python 3.8 visit_Num(), visit_Str() and so on are all deprecated
and replaced with visit_Constant. We can't yet remove the deprecated
functions until we require 3.8, but we can implement visit_Constant to
silence the deprecation warnings.
(From OE-Core rev: d91fe6ecb9fbb410b3bab6ced66b7fe5f869cf83)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 875a944392a3c93f40081a14af357f70b6b8264f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Exceptions print more clearly using repr() instead of str(), fix
in fetch and unpack tasks.
Drop part of the test which no longer makes sense after this change.
(From OE-Core rev: 045124ce7ef7c53a1932848835f93abbe535f157)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c94816259cc1c09746353ad26ca0c811e0c962c2)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code pattern isn't very pythonic, improve it to drop the unneeded
len() calls.
(From OE-Core rev: e0cecf7b4a4b966f11527a88e632997bdf74af96)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 69376ac1a6147b26fe1abaa4cf68414024814d63)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If checkstatus returns an exception we should silently handle
FetchError, as this means the fetch failed for 'normal' reasons such as
file not found. However, other exceptions may be raised, and these
should be made visible.
(From OE-Core rev: f16803718a19ace7b582c5b73924fe3112502927)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 50d99faf88a1d82cbd939b9bd6e33ebed2b1ffd8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
