Bitbake is moving to a minimum python version of 3.8, update OE-Core
to match to make things consistent.
(From OE-Core rev: cc05bc04156122fd6f918191a9cec7bc0392415e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Most of our older distros with python older than 3.8 already need buildtools
tarballs apart from Ubuntu 18.04. 3.8 allows us to fix a few things, tidy
code in places and is widely available or can be obtained with buildtools.
Therefore update our minimum version to 3.8.
(Bitbake rev: 744310f360d2288ac2ef07745abc86852126b5b9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_setscene was from a different era before our modern setscene
per task code. It hasn't been used for years so remove some old
obsolete references to it.
(Bitbake rev: ef72282298f7c4db74383c23bb0251dd06d3c6d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All siggens in common use should now support multiconfig, drop the
compatibility code.
(Bitbake rev: b36545b4df6d935ed312ff407d4e0474c3ed8d1a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're now used to using hashes as part of the task hashes and the sstate code
relies on this. The older OEBasic hash approach therefore wouldn't work and
can be removed.
(From OE-Core rev: 3667e589ba16eb261cfd72c2b11429f482c239f6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When obtaining latest upstream versions, the code needs
to check if the existing tarball is in a versioned directory
(e.g. component-name/x.y/component-name-x.y.z.tar.gz) and
if it is, it needs to first obtain the list of all
such versioned directories and then check all of them by going
one step up in the directory hierarchy.
Existing code was returning a correct match when the component
name did not have numbers, e.g. a check on 'source/epiphany/43/'
would return 43, but was stopping too soon when the component
name itself had numbers ('source/libxml2/2.10/' would return libxml2).
This change ensures the last match is taken instead of the first.
Also, adjust the fetcher tests to check that versioned directories
are correctly traversed in this case (e.g. the step to go one level
up is taken and a new tarball is discovered in a different versioned
directory).
(Bitbake rev: b6601be22c6d776327acdcd1fa931400f41ac786)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
largefile was made the default and the distro feature is no longer needed.
Drop the leftovers in poky.
(From meta-yocto rev: c5da2d28ed517fff4a32e3bbad4511a1a4bf19b0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every
single object in the remote repository. This works poorly with gitlab
and github, which use the remote git repository to track its metadata
like merge requests, CI pipelines and such.
Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/*
and refs/keep-around/* and they all contain massive amount of data that
are useless for the bitbake build purposes. The amount of useless data
can in fact be so massive (e.g. with FDO mesa.git repository) that some
proxies may outright terminate the 'git fetch' connection, and make it
appear as if bitbake got stuck on 'git fetch' with no output.
To avoid fetching all these useless metadata, tweak the git fetcher such
that it only fetches refs/heads/* and refs/tags/* . Avoid using negative
refspecs as those are only available in new git versions.
Per feedback on the ML, Gerrit may push commits outsides of branches or
tags during CI runs, which currently works with the 'nobranch=1' fetcher
parameter. To retain this functionality, keep fetching everything in case
the 'nobranch=1' is present. This still avoids fetching massive amount of
data in the common case, since 'nobranch=1' is rare. Update 'nobranch'
documentation.
Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
(Bitbake rev: d32e5b0ec2ab85ffad7e56ac5b3160860b732556)
Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In checkstatus() opener.open() is used to check if an artifact is available.
The check fails if the uri contains username password in the format:
"username:password@hostname..". Moreover, the checkstatus function already uses
the username from the "ud" object to craft a header, is username and password is
provided.
This fix ensure the uri in the Requests object used does not contain username as
password.
(Bitbake rev: 88350002d45e0aa85ecd5356da2c8d71e450641e)
Signed-off-by: Kasper Revsbech <kasper.revsbech.ext@siemensgamesa.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We keep seeing server issues where the lockfile is present but we can't
connect to it. Reuse the lockfile debugging code from the server to
dump better information to the console from the client side when we
run into this issue. Whilst not pretty, this might give us a chance
of being able to debug the problems further.
(Bitbake rev: 22685460b5ecb1aeb4ff3436088ecdacb43044d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have timestamps in the server logs but we don't know which
UI messages correlate to them. Add some timestamps to the messages
which doesn't make them pretty but which might make it possible to
debug problems.
(Bitbake rev: a1a86f8c311cb1fc4f5562f1c77f82aa95141eee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It appears that rngd is not needed as of linux-5.6 and later[1]
and should not be installed by default since the purpose of rngd
is to provide additional trusted sources of entropy.
We did some testing on real hardware, the result seems to support that
we no longer need rngd by default on kernel v5.6 and later.
Testing result as below:
1. observing the crng init stage.
the "random: crng init done" always available before fs being mounted.
2. generating random number without rngd.
testing command: dd if=/dev/random of=/dev/null status=progress
on Marvell CN96xx RDB board, speed almost 20.4 MB/s without block
on NXP i.mx6q board, speed almost 31.9 MB/s without block
on qemu x86-64, speed almost 2.6MB/s without block
3. using rngtest command without rngd
testing command: rngtest -c 1000 </dev/random
on Marvell CN96xx RDB board:
rngtest: input channel speed: (min=4.340; avg=135.364; max=146.719)Mibits/s
rngtest: FIPS tests speed: (min=8.197; avg=69.020; max=72.800)Mibits/s
rngtest: Program run time: 418771 microseconds
on NXP i.mx6q board:
rngtest: input channel speed: (min=96.820; avg=326.769; max=340.598)Mibits/s
rngtest: FIPS tests speed: (min=15.090; avg=37.543; max=40.324)Mibits/s
rngtest: Program run time: 570229 microseconds
on qemu x86-64:
rngtest: input channel speed: (min=37.769; avg=101.136; max=136.239)Mibits/s
rngtest: FIPS tests speed: (min=10.288; avg=30.682; max=40.155)Mibits/s
rngtest: Program run time: 836800 microseconds
4. observing sshd service.
using "systemctl disable rng-tools" disable service and reboot system.
system boot up normal, sshd service also start in normal time without
block.
Reference:
[1] 30c08efec8
(From OE-Core rev: 868dfb46d96a27ec9041cb902fb769330277257d)
Signed-off-by: Xiangyu Chen <xiangyu.chen@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that we have Python 3.11, hatchling will use the internal tomllib
instead of tomli.
(From OE-Core rev: f33168a18ddc9929c8e302937051fb6430047990)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since qemuboot is part of IMAGE_CLASSES via qemu.inc it is being
inherited before we set the QB_FOO variables.
Since our variables have conditional definitions and at that point
they've already been defined by qemuboot, we can no longer define
them in our class.
Move the IMAGE_CLASSES inherit to execute it after we set the
QB_FOO variables to fix booting via runqemu.
(From OE-Core rev: 8ed78ec262b2502dc3b673b24a868a3eec616a20)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandro@enedino.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This has been ported from setuptools to flit_core, and now contains a
hand-written parser instead of using pyparsing.
(From OE-Core rev: cf4c11fe2e13ec63c08ded790e4d7b64ce94e4f2)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For unknown reasons we've never seemingly run the check layer script
against OE-Core itself. This isn't entirely straightforward as the core
layer is a bit of a special case, we can't for example compare signatures
against ourselve and we can't remove core from bblayers.conf.
Core does have distro, machine and software components too, in the case
of distro, our fallback default settings. Whilst the qemu machines could
be split into a seperate layer directory, core wouldn't then parse at all
standalone due to the lack of any machine so it seems a bit pointless to
do that.
These changes tweak the script to handle core's special cases, specifically
to allow distro and machine directories and to account for the README placed
a directory level higher than other layers.
(From OE-Core rev: ba312ed228507d05f280aeb96819d671b01400b8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Patch so ppoll is properly redirected to it's 64bit time variant even
when source fortification is active.
(From OE-Core rev: 8cb7008090e98d990e7249b4ef723f65825915ae)
Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Opkg 0.6.1 Changes:
- Opkg will no longer complain when trying to clean up the temporary
directory, if the directory does not exist.
- Fixed a SEGFAULT when parsing package indexes with invalid `Size` or
`Installed-Size` fields. These indexes will now produce a
comprehensible error.
- Fixed an inconsistecy in .list generation where files would sometimes
be entered with/without a trailing slash. The trailng slash should now
always be removed.
- Fixed [a bug](https://bugzilla.yoctoproject.org/show_bug.cgi?id=10461)
in package removal, where empty common directories would be left on
disk, even after all owning packages were removed.
(From OE-Core rev: 037ff235fa8e369c0eac9f84cb82c9eaffba85f3)
Signed-off-by: Alex Stewart <alex.stewart@ni.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake additions for the addpylib API mean we need to update the parsing
function call to be clear we're parsing in configuration context.
(From OE-Core rev: ef7677dc90fac089f8b9f6da301cca022ed7284c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It seems some layers want to subvert the intent of LAYERSERIES_COMPAT
so bitbake is going to have to become stricter about the values there.
To work with this, use LAYERSERIES_CORENAMES to generate the entries in
LAYERSERIES_COMPAT instead of the current magic LAYERSERIES_COMPAT_core
value which may not continue to work.
The downside to this is when migating between releases, people would
need to update devtool workspace layer.conf files. I guess you could
argue this is a feature!
(From OE-Core rev: 96ff9baa8ead57504f40f362ed3a4aaa776d1b58)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since bitbake now supports an official method to inject python modules,
switch to it.
Anyone using OE_EXTRA_IMPORTS will need to adjust their code accordingly,
probably switching to their own module namespace.
Also switch to using BB_GLOBAL_PYMODULES to list the global modules
to import.
(From OE-Core rev: 1f56155e91da2030ee0a5e93037c62e1349ba89f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some layers think they're going to be 'clever' and copy the values from
another layer, e.g. using ${LAYERSERIES_COMPAT_core}. The whole point of
this mechanism is to make it clear which releases a layer supports and
show when a layer master branch is bitrotting and is unmaintained.
Therefore add some code to avoid people doing this. I wish we didn't have
to but...
(Bitbake rev: 6709aedccbb2e7ddbb1b2e7e4893481a7b536436)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For many years OE-Core has injected it's own python modules into the
python namespace using an immediate expansion of a variable in base.bbclass.
It also added all entries from BBPATH to sys.path.
We really need this to become a first class citizen of the langauge, this new
addpylib directive allows that. Usage is of the form:
addpylib <directory> <namespace>
The namespace is imported and if there is an attribute BBIMPORT, that
list of names is iterated and imported too.
This mirrors what OE-Core has done for a long time with one difference in
implmentation, sys.path is only appended to. This means later imported
namespaces can't overwrite an earlier one and can't overwrite the main
python module space. In practice we've never done that and it isn't
something we should encourage or support.
The new directive is only applied for .conf files and not other filetypes
as it only makes sense in that context. It is also only allowed in the
"base configuration" context of cookerdata since adding it at the recipe
level wouldn't work properly due to the way it changes the global namespace.
At the same time, move the list of modules to place in the global namespace
into a BB_GLOBAL_PYMODULES variable. It is intended that only the core layer
should touch this and it is meant to be a very small list, usually os and sys.
BB_GLOBAL_PYMODULES is expected to be set before the first addpylib directive.
Layers adding a lib directory will now need to use this directive as BBPATH
is not going to be added automatically by OE-Core in future. The directives are
immediate operations so it does make modules available sooner than the current
OE-Core approach.
The new code appends to sys.path rather than prepends as core did, as
overwriting python standard library modules would be a bad idea and naturally
encouraging people to collaborate around our own core modules is desireable.
(Bitbake rev: afb8478d3853f6edf3669b93588314627d617d6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change ptest away from using the upstream Makefiles to manually running
the tests: they're not actually integrated with automake anyway so this
didn't gain us anything apart from patches we can't send upstream. Drops
the following patches:
- 0001-Make-ptest-run-the-python-tests-if-python-is-enabled.patch
- 0001-Port-gentest.py-to-Python-3.patch
- fix-execution-of-ptests.patch
- remove-fuzz-from-ptests.patch
- runtest.patch
Add a simple patch to install the test binaries via the Makefile:
- install-tests.patch
The Python module is built differently and a number of patches no longer
apply and appear to be redundant, remove:
- python-sitepackages-dir.patch
libxml-m4-use-pkgconfig.patch has been sent upstream now, mark as
backport.
Remove obsolete --without-docbook option.
Remove obsolete xml2Conf.sh packaging.
(From OE-Core rev: ec5f380a14246e31b2a1a12dda9fe2178b1e5f83)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When starting to use combo-layer, or if someone else is using it too,
the local last_revision may be incorrect.
This command will forcibly update the last_revision config values to the
latest SHA on the remote branch that is tracked.
(From OE-Core rev: 2bb5d12ecd1b0273983f7c05699f34dd64b11c25)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake may not be configured, and bb isn't imported anyway.
Instead just use os.rename(), and take the filename from the file object
instead of duplicating logic.
(From OE-Core rev: 528f4fb3683d048537604e4562ea758968060d62)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Get rid of the duplicate code and add extra check that the
locale en_US.UTF-8 is available on the system. This new helper
method is now located right above the method filter_environment()
which sets LC_ALL environment variable to 'en_US.UTF-8'.
[YOCTO #10165]
(Bitbake rev: a4ce040a6fd540a1cac52f808f909f9fcf8c961c)
Signed-off-by: Frank de Brabander <debrabander@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Starts the sync thread slightly earlier to give it some extra time to
dump out the caches while the main process tears down the parsing
processes
(Bitbake rev: 105f2897b0618713b036fc0f7a6e0f3e78d1707a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Uses an event to terminate the parser threads instead of a queue. This
is not only simpler, but saves about 500ms on shutdown time
(Bitbake rev: 2aed34e1d4bf24bba6263f168ff31b55b5fbe982)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to track siggen hash construction data can be useful for cache
debugging. For now, add an extra cache class which contains this information.
It can be enabled in the same way as the hob data cache through a feature flag
to cooker. This allows us to experiment with the data without carrying larger
patches around and ultimately may allow use to have a hash mismatch debugging
mode that is more easily enabled.
(Bitbake rev: 0736a8a03da8b774fafbd28f746bef4705378049)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The basehash data is really internal bitbake data and passing an object
directly is more efficient than creating and then extracting variables.
This will match the format of other data we may optionally wish to
store in the cache so more to the more efficient method. Nothing I
can see is using this data today (and nothing should be).
(Bitbake rev: e621093a1bf37cd75ede3fb77ab6845556870fc7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make a small change in the layout of the code in build_dependencies
which makes subsequent patches easier to read. No functionality change,
just moving the function definitions to the start of the function block.
(Bitbake rev: fff13b1e5e8397130b4378e0ba2301336ec651a2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit 0533edac27 broke
bitbake parsing when bitbake is executed from directory with existing .gitmodules
and the recipe in externalsrc does not have .gitmodules
The check needs to search for .gitmodules in sources path, not cwd.
iParsing recipes...ERROR: ExpansionError during parsing <path to recipe>
...
bb.data_smart.ExpansionError: Failure expanding variable do_compile[file-checksums], expression was ${@srctree_hash_files(d)} which triggered exception CalledProcessError: Command '['git', 'config', '--file', '.gitmodules', '--get-regexp', 'path']' returned non-zero exit status 1.
(From OE-Core rev: 66ff3d1f65cd2e7f5319e98fa41f47a59b714c72)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The /proc/pressure support in buildstats is creating directories in the
buildstats tree called reduced_proc_pressure, which confuses the parsing
logic as that cannot be parsed as a name-epoc-version-revision tuple.
Explicitly skip this directory to solve the problem.
(From OE-Core rev: 24f0331f0b7e51161b1fa43d4592b491d2037fe9)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, allyesconfig test runs for x86_64 fail with:
ERROR: linux-yocto-5.19.17+gitAUTOINC+0cba9aa404_aaf4490d18-r0 do_package:
QA Issue: linux-yocto: Files/directories were installed but not shipped in any
package:
/lib/modules/5.19.17/kernel/drivers/nvdimm
With CONFIG_NVDIMM_TEST_BUILD=m, an empty nvdimm directory is created during
modules_install, which triggers the QA issue.
Extend kernel_do_install() to also remove inner empty directories that might
get created by modules_install.
(From OE-Core rev: 7120b09a33af4c9a18063c0f2e51fb598697e39c)
Signed-off-by: Ovidiu Panait <ovidiu.panait@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For now, if we use rm_work and `bitbake core-image-minimal', some
recipes' WORKDIRs are not cleaned up, e.g., makedevs-native.
Adjust the dependency to make do_rm_work_all depend on do_rm_work
to solve this problem.
Below are the detailed explanation of why this would work.
Without this patch, the dependency chain is like:
[other deps] -> [do_rm_work] -+-> [do_build]
|
[do_rm_work_all] -------------+
With this patch, the depedency chain is like:
[other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build]
Such dependency chain adjustment fixes the issue because do_rm_work_all
now depends on [other deps] and thus the [depends] of these [other deps].
Take core-image-minimal as an example. Before this adjustment,
do_rm_work_all does not have any relationship with do_rootfs, and we have
do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..."
This essentially prevents 'recrdeptask' setting of do_rm_work_all extend
to makedevs-native. With this patch, the do_rm_work_all now depends
on do_rm_work which in turn depends on do_rootfs, and so do_rm_work_all's
recrdeptask could have effect on makedevs-native.
With this patch, all built recipes WORKDIR will be cleaned up with
a few expected exceptions such as kernel and qemu-helper-native.
(From OE-Core rev: b25cc45c9b39f79ba0a03c4556cb2e2431677b4e)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
texrels are generated while compiling on x86/clang because pic/pie flags
that are passed via bitbake do not make it everywhere evenly.
--enable-pie is default way of getting it enabled in bluez5
PIE is enabled by default which ensures that textrels are not generated
(From OE-Core rev: 804aa082e4c4fdd4881ff668f52790bea818eabb)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
