By naming this test class git.py, any attempt to import GitPython (as
needed by oelib.buildhistory) failed.
As this class exercises the intercepts, rename it to intercept.py.
(From OE-Core rev: d557cbbf86767bc2ebf2beb3d70af3b3ca5e0529)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Historically there's been a split between /lib for early boot and
/usr/lib for everything else, but with modern systems this split is
meaningless and incomplete. If a minimal system for early boot is
needed, it should be a full minimal system in a initramfs.
[RP: Fixed up selftest to match]
(From OE-Core rev: 990073dfc167354b4af41db83ac46c18b1aa99d5)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some opkg commands print an error during cleanup when the tmp_dir
does not exist and an attempt is made to delete it. The error messages
are harmless and the opkg commands eventually succeed.
When these commands are run and stderr is piped to stdout, the error
messages may clobber the stdout and cause unexpected results while
parsing the output of the command. Therefore, when parsing the output
of a command, do not pipe stderr to stdout. Instead, capture stderr
and stdout separately, and upon success, send stderr to bb.note().
(From OE-Core rev: f2167ae80258253eb47a5b148546b265320284cc)
Signed-off-by: Shruthi Ravichandran <shruthi.ravichandran@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In bc, we found that there are files newer than Changelog (e.g. scan.l)
which means after install runs, the timestamp is clamped to SDE which means
"bitbake bc -C compile" would then try and regenerate generated files and
complain flex was missing.
Rather than assuming Changelog/NEWS are magic for dates, drop that scan
method and scan all unpacked files. This shouldn't be that slow as they'd
be in the disk cache already after an unpack.
For bc, this changes SDE by about two minutes and avoids the problems
rebuilding.
Add a version comment to the task definition as changes in python library
code won't trigger a rebuild and we need one here.
(From OE-Core rev: 32dda0ad91a9e7946351c897578b4c97ae142341)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is the absolute bare minimum for testing the RT patches, but it
does mean we if we build and boot a RT kernel we can verify that it is
what we expect.
(From OE-Core rev: 0301d5845115d09299f87683b3efa46f3b4c7be9)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
SDPX generation involves looking through BB_TASKDEPDATA for
dependencies, then linking to the generated documents for those
dependencies. These document links use a checksum to validate the
document, which means that if a upstream document changes, all
downstream documents must be regenerated to get the new checksum,
otherwise the compendium of documents produced by the build will have
broken links; therefore all dependent task should be included in the
signature (even from "ABI safe" recipes).
(From OE-Core rev: 5fe543b9ceec971cf0297ff0ae3b0ccc4703cece)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ThreadedPool in OE-core is mainly because python2
doesn't have threaded pools but python2 is dead for
some time now and python3 have a ThreadPoolExecutor.
The only local in OE-core where this ThreadedPool
is in use is on the sstate.bbclass that is ported
to the python3 ThreadPoolExecutor.
(From OE-Core rev: aa8fd5e7c2a1125895accfd55ce9320819a10959)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can't support vgem on RHEL derived distros so disable this test for
all almalinux hosts rather than specific versions.
(From OE-Core rev: e921f3c1b917072e4c5a110c7dfeeadd2e571bde)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes an end user might want to choose another kernel type argument
for uboot-mkimage other than "kernel", for instance: "kernel_noload".
Let's introduce a variable UBOOT_MKIMAGE_KERNEL_TYPE to support that,
and it could be used by BSP layers as well.
(From OE-Core rev: e288686e97de1265eeeaf452141e1473867efb1b)
(From OE-Core rev: 4eb7bbcc2f08b25387a15b7e4a89ef199783c973)
Signed-off-by: Ming Liu <liu.ming50@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake is dropping the DL_DIR fallback for local file urls and ensuring
local urls are fetchable. This test fails as it copies the meta directory
of COREBASE but not scripts and nativesdk-qemu-helper references runqemu
from there which doesn't exist in the copied data.
Tweak to symlink scripts into position in the copied metadata which
avoids the now fatal parsing error.
(From OE-Core rev: 5a5199943de5df9a4d44277d07f4313642c34b3a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before this commit the test expected a runtime error but the logic of
the fetcher changed so that files not found for checksumming cause a
fatal interruption of the parse, thuse never reaching the runtime error
this test expected. This commit aligns with the bitbake changes to the
fetcher.
(From OE-Core rev: 8b506bfbdd18dfdb411080f69ef5c0d416b3f2e0)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test that verifies that devtool modify + devtool finish do the
right thing on a recipe that fetches from git and sets S to point to
a subdirectory of the source tree. We have a few examples among the core
recipes, dos2unix is a convenient one so let's use that. (The test first
verifies that that is still true in case the recipe is changed in
future.)
(From OE-Core rev: a84d9ed14173b0bf467ea78dff4f0f7bae0bc082)
Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If devtool finish needs to create a patch and have it applied to the
sources for a recipe where S points to a subdirectory of the sources,
then the patch needs to be applied at the root of the repo i.e. we need
to add a patchdir= parameter to the SRC_URI entry.
(From OE-Core rev: ad3736d9ca14cac14a7da22c1cfdeda219665e6f)
Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If PATCHTOOL = "git", SRC_URI fetches from a git repo and S points to
a subdirectory of the checked out sources, then we were erroneously
initialising the subdirectory as its own git repo. Check if the returned
top-level repo directory is a subdirectory of WORKDIR and do not
run initialise the source directory if that is the case.
(This was a regression introduced with OE-Core revision
6184b56a7a0fc6f5d19fdfb81e7453667f7da940, however we didn't have a test
that verified the behaviour.)
(From OE-Core rev: 9cca53a2bcbf6809615ce5626c86c6ee481a7a76)
Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake ran but we incorrectly did not assert the exit status needs to
be non 0. Now all sysroot tests commands expected to fail are verified
to do so.
(From OE-Core rev: 5fe8c14f50d414e768588cef0675d8ef296ced77)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Useful to work around shebang relocation issues, where
shebangs are too long or have arguments in them, thus preventing them
from using the /usr/bin/env shebang.
(From OE-Core rev: 6edc1fffcbe1405d8c309a75643d7d6cd9a92848)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These files are checked by qa_check_staged but there was no
test cases for whether the tests actually worked. Now there
are.
(From OE-Core rev: 2a96719a201cb7b8db774718adec89dbd7e1aec3)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Substitute expressions or whitespace from python egg requires.txt when
generating PACKAGECONFIG
Pysetuptools sees the uvicorn.egg-info/requires.txt as extra requirements.
Recipetool parses this information to generate the PACKAGECONFIG.
These extra requirements contain expressions and whitespace, which are not allowed in PACKGAGECONFIG.
This patch substitute them by hyphens to make PACKAGECONFIG parsable and readable.
Also adding an oe-selftest for this.
[YOCTO #14446]
(From OE-Core rev: a854d95a79e64f3f82abfa4cc1daec750abf4249)
Signed-off-by: Thomas Roos <throos@amazon.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Buildperf was designed to skip all non-directories under buildstats, i.e. proc
log files. With the /proc/pressure stats being collected in a new directory,
the following error was seen:
meta/lib/oeqa/buildperf/base.py", line 392, in split_nevr
n_e_v, revision = nevr.rsplit('-', 1)
ValueError: not enough values to unpack (expected 2, got 1)
Add an additional check to skip the reduced_proc_pressure directory.
(From OE-Core rev: d3a25dd7aa63ab98c8bb931b4b3bc61807806ed3)
Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use python 'with' symantics to ensure that the /proc/pressure/cpu file
descriptor used in SystemStats init is closed. Previously, this would
lead to a single file descriptor being leaked. For example:
ResourceWarning: unclosed file <_io.BufferedReader name='/proc/pressure/cpu'>
(From OE-Core rev: 643653160cd77d346cdc9b9ec25c7212c7dfe176)
Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The last time of sampling would be updated within the SystemStats class
but not re-recorded into the datastore, leading to multiple samples being
collected in the same second in the sample function of buildstats.py.
Fix this to collect and store only one sample per second within a
certain tolerance to deal with variation in the arrival time.
This fix elimates the spikiness of sampled data, in cases where the difference
between the current and the last sample is taken. Previously, since many
samples per second were recorded, certain types of data would result in a
very small elapsed time and hence a small numerical difference. For example,
the CPU usage from /proc/stat is a running total of usage and taking the
difference between data collected 0.1 seconds apart would result in usage
appearing lower than it actually was.
(From OE-Core rev: 0e2df45ab066bb4ad2c4f8622ee9c1a8ecdea9cb)
Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Randy MacLeod <randy.macleod@windriver.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The Linux pressure monitoring system helps determine when system resources
are being overutilized by measuring how contended the CPU, IO and memory are.
This information can be found under /proc/pressure/ which contains 3 files -
cpu, memory and io. In each of the files, the format is as follows:
some avg10=70.24 avg60=68.52 avg300=69.91 total=3559632828
full avg10=57.59 avg60=58.06 avg300=60.38 total=3300487258
The "some" state of a given resource represents when one or more tasks are delayed
on that resource whereas the "full" state represents when all the tasks are
delayed. Currently, we only collect data from the "some" state but the
"full" data can simply be appended to the log files if neccessary.
The "avg10", "avg60" and "avg300" fields represent the average percentage
of time runnable tasks were delayed in the last 10, 60 or 300 seconds
respectively. The "total" field represents the total time, in microseconds,
that some runnable task was delayed on a resource.
More information can be found at:
https://www.kernel.org/doc/html/latest/accounting/psi.html
and in the source code under kernel/sched/psi.c
This commit adds functionality to collect and log the "some" CPU, memory and IO
pressure. The "avg10", "avg60" and "avg300" fields are logged without change.
In place of the "total" field, the difference between the current "total" and
the previous sample's "total" is logged, allowing the measurement of pressure
in between each polling interval, as was done for /proc/stat data. The log files
are stored in:
<build_name>/tmp/buildstats/<build_time>/reduced_proc_pressure/{cpu,io,memory}.log
mirroring the directory structure of /proc/pressure. If the /proc/pressure
directory does not exist or the resource files can't be read/opened, the
reduced_proc_pressure directory is not created.
(From OE-Core rev: 061931520b8baa7f3a03bf466aa9ec8bf995bc14)
Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Randy MacLeod <randy.macleod@windriver.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switch the default DEPENDS for ${PN}-dev to be a RRECOMMENDS instead. This
takes advantage of a change to complmentary package globbing to not follow
RRECOMMENDS and means and SDK for an image with both openssh and dropbear
compoments will now build successfully.
(From OE-Core rev: 6f28420ab0e8f2ab5eb06326024777a40aded0a6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've some long standing bugs where the RDEPENDS from -dev packages causes
problems, e.g. dropbear and openssh components on an image working fine together
but then the SDK failing to build as the main openssh and dropbear packages
conflict with each other (pulled in by openssh-dev and dropbear-dev).
We propose changing the behavour of complementary package installation to
ignore RRECOMMENDS. If we then change the ${PN}-dev dependency on ${PN}
to a RRECOMMENDS, we can avoid many of the issues people run into yet still
have the desired behaviour of ${PN}-dev pulling in ${PN}.
This therefore changes the package manager code so that it doesn't follow
RRECOMMENDS for completementary package globs.
[RP: Added deb support]
(From OE-Core rev: b44b0b9294675f89aa51ff84f532664f4c479677)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Helper module to:
- generate meta information from package.json content. This data has
a format as provided by https://registry.npmjs.org
- put this meta information and the corresponding tarball in the
nodejs cache. This uses an external, nodejs version specific helper
script (oe-npm-cache) shipped in oe-meta
To avoid further nodejs version dependencies, future versions of this
module might omit the caching completely and serve meta information
and tarball by an http server.
(From OE-Core rev: 6cd5886ad05fee704e8a5892bd370c360c8c3b54)
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fedora is switching to use sftp as the backend for scp. This means the
scp test fails on Fedora 36 hosts with a dropbear target as dropbear
doesn't support sftp. This change is in the upstream openssh code, other
distros have not yet changed the default but probably will follow.
The easiest way to resolve test failures in dropbear images is to stop
testing this against dropbear as it is no longer expected to work and will
likely spread as the change filters through other distros.
(From OE-Core rev: a71fc7d455400f406b0d607be712a1133fe91166)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python 2.x has been EOL for a while, and so this test
never runs.
(From OE-Core rev: b687627e9cffb8123c156413f55ea1929f1a7831)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When multilib enabled and add layers/meta-openembedded/meta-oe in
conf/bblayers.conf, it reports below error when run oe-selftest.
$ oe-selftest -r oescripts
[snip]
[20:36:33-0700] 2022-05-16 03:36:33,494 - oe-selftest - INFO - RESULTS - oescripts.OEListPackageconfigTests.test_packageconfig_flags_option_flags: FAILED (585.37s)
[snip]
It is because the output of "list-packageconfig-flags.py -f" as below:
$ ../scripts/contrib/list-packageconfig-flags.py -f
[snip]
qt lib32-pinentry lib32-wxwidgets nativesdk-pinentry pinentry pinentry-native wxwidgets wxwidgets-native
secret lib32-pinentry nativesdk-pinentry pinentry pinentry-native
[snip]
But the check logic as below:
class OEListPackageconfigTests(OEScriptTests):
#oe-core.scripts.List_all_the_PACKAGECONFIG's_flags
def check_endlines(self, results, expected_endlines):
for line in results.output.splitlines():
for el in expected_endlines:
if line.split() == el.split():
expected_endlines.remove(el)
break
def test_packageconfig_flags_option_flags(self):
results = runCmd('%s/contrib/list-packageconfig-flags.py -f' % self.scripts_dir)
expected_endlines = []
expected_endlines.append("PACKAGECONFIG FLAG RECIPE NAMES")
expected_endlines.append("qt nativesdk-pinentry pinentry pinentry-native")
expected_endlines.append("secret nativesdk-pinentry pinentry pinentry-native")
self.check_endlines(results, expected_endlines)
And the test will fail as line.split() doesn't equal el.split() as
line.split() is ['lib32-pinentry', 'lib32-wxwidgets', 'nativesdk-pinentry',
'pinentry', 'pinentry-native', 'wxwidgets', 'wxwidgets-native'] and
el.split() is ['nativesdk-pinentry', 'pinentry', 'pinentry-native'].
So change the compare logic to fix the gap.
(From OE-Core rev: 239f22847bcae0cb31769adb0a42b5440173a7c5)
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reduce code duplication to a common function in oe.buildcfg.
(From OE-Core rev: 0b191a1df4c3722defb09dde0c16b1d9e7fe7ef6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This isn't used anywhere and everyone is using git now, the functions
are now hardcoded as such too. Drop this function.
(From OE-Core rev: 09ba96a2d7fa02c7a82758bbf4109b04ffca2c55)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The comments even say this was copy/paste code. Move to a
shared library function.
(From OE-Core rev: ac3de2f850a418673b87e1c454970cb099e191b0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a load of duplicated git/layer/revision code which makes
most sesne as a python library, not bbclass code. Start to refactor as such.
(From OE-Core rev: 439cdf8a1e52fd2c4dc81dc40ce7e6af282ce7ac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With CONFIG_MODULE_COMPRESS_ZSTD enabled, kernel modules will have a
.ko.zst extension. This fixes depmod not being run.
Fixes: 1b696a45ddb ("rootfs.py: Add check for kernel modules before running depmod")
(From OE-Core rev: 425efac7110f0f42d70643e0a448e834d0f01a7a)
Signed-off-by: Sean Anderson <sean.anderson@seco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the function to a library, it could be useful in other places.
(From OE-Core rev: debd37abcdde8788761ebdb4a05bc61f7394cbb8)
Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We may remove lzo so switch the test case to zstd.
(From OE-Core rev: f749a8b462b915713912342444f981125938a990)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The grep is too wide, so it falsely fits additional lines that have
a UUID (i.e, `/`).
(From OE-Core rev: f72fdea1c890ddd793aa63bb9c1c0857962161cc)
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test to verify that the JSON reports are generated correctly for
both single recipe builds and image builds.
More tests are needed, but this is better than nothing.
(From OE-Core rev: df0f35555b09c4bc75470eb45ec9c74e6587d460)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using += unintentionally removes all other entries from DISTRO_FEATURES
if DISTRO_FEATURES was set by ?= such as by poky.conf. This reduces
sstate reusage on the autobuilder. Fix this to speed up builds.
(From OE-Core rev: 124b82c32c4545bb216a8249954817f692f9795a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As a centos 8 spinoff, it lacks the same vgem kernel module.
(From OE-Core rev: 451605aa40482516c18cd1534feacb796516a785)
(From OE-Core rev: 8f74418637fbcdd7c68636ec1b9adfffc245f9c1)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fix for CVE-2022-24765 in git[1] breaks any use of git inside
pseudo. Add a simple test case to oe-selftest to verify that at least
basic uses of git work fine under pseudo.
[1] 8959555cee
(From OE-Core rev: 46822268040a23dbb81f71fe35aee8c2663a31f6)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The overlayfs_qa_check checks if the current root file system has a
mount configured for each overlayfs, when the overlayfs class is used.
However there are certain instances where this mount point is created at
runtime and not static in a fstab entry or systemd mount unit.
One such case would be if overlayfs-etc is used, where the device is
mounted in the preinit script and not via a mount unit or fstab entry.
However there are other possibilities for this as well, like startup
scripts that support a dynamic partition layout. For instance when
systemd-repart is used.
This adds the `OVERLAYFS_QA_SKIP` variable, which allows to define QA
skips via its flags. In principle it supports multiple QA skip flags
separated by whitespace, but only one (`mount-configured`) is
implemented here. To skip this QA check simply add `mount-configured` to
the flag of `OVERLAYFS_QA_SKIP` with the same name. For instance if a
overlayfs is configured as:
OVERLAYFS_MOUNT_POINT[data] = "/data"
Skipping this QA check can be done by setting:
OVERLAYFS_QA_SKIP[data] = "mount-configured"
Also add a testcase and fix a typo (fstat -> fstab).
(From OE-Core rev: 2ce9173169a2a86392c4a85fe9be7fbbd7353b7f)
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
