Commit Graph

7025 Commits

Author SHA1 Message Date
Samantha Jalabert
4edf3bf2c6 bitbake: Fix disk space monitoring on cephfs
Error occured while running bitbake on cephfs:

WARNING: The free inode of path is running low (-0.001K left)
ERROR: Immediately halt since the disk space monitor action is "HALT"!

(Bitbake rev: c7e094ec3beccef0bbbf67c100147c449d9c6836)

Signed-off-by: Samantha Jalabert <samantha.jalabert@syslinbit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2023-11-09 04:44:46 -10:00
Marta Rybczynska
0ac25662c7 bitbake: SECURITY.md: add file
Add a SECURITY.md file with hints for security researchers and other
parties who might report potential security vulnerabilities.

(Bitbake rev: 973aa73b9f1f542f4a636cf8eca733b4df5a778f)

Signed-off-by: Marta Rybczynska <marta.rybczynska@syslinbit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-10-24 12:52:46 +01:00
Richard Purdie
5e72da9780 bitbake: runqueue: Fix deferred task/multiconfig race issue
If there are several multiconfigs in play for example a non-multiconfig with
a task with one hash and then three multiconfigs for the same task, different
architectures but the same hash (different to the non-mc), the three mcs
will be deferred until after the non-mc task but then will all run together
and race against each other.

Change the code to re-enable deferred tasks one at a time. This way, if they do
race, they won't run in parallel against each other.

(Bitbake rev: 08033b63ae442c774bd3fce62844eac23e6882d7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9523e28658ad7fb446645b590608dfac2812afd3)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2023-07-07 04:30:25 -10:00
Paul Eggleton
801734bc6c bitbake: bitbake-user-manual: add BB_HASH_CODEPARSER_VALS
New variable for 2.4.

(Bitbake rev: d97d62e2cbe4bae17f0886f3b4759e8f9ba6d38c)

Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-22 10:30:42 +01:00
Paul Eggleton
a91fb4ff74 bitbake: bitbake-user-manual: add addpylib and BB_GLOBAL_PYMODULES
New directive and variable for 2.4.

(Bitbake rev: 1b74a7b71e6c8a034163b57ef4e13b4a58c1f844)

Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-22 10:30:42 +01:00
Paul Eggleton
54f3339f38 bitbake: bitbake-user-manual: document BB_CACHEDIR
New variable for 2.4.

(Bitbake rev: fa91e1a02240be1f7b78ef70953ff4a0e0106cc2)

Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-22 10:30:42 +01:00
Enrico Scholz
50c5035dc8 bitbake: cooker: do not abort on single ctrl-c
After b7ed7e9a815c4e10447fd499508be3dbb47f06e8 bitbake aborts immediately
when a single ctrl-c is pressed.

Patch restores the previous behavior where a single ctrl-c waits for
active tasks to finish.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=15094

(Bitbake rev: 66131fa6a3e12c28710d09e1dbf3c03f2981280d)

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-22 10:30:42 +01:00
Svend Meyland Nicolaisen
c078df73b9 bitbake: npmsw fetcher: Avoid instantiating Fetch class if url list is empty
Recipes containing both git and npmsw sources in the SRC_URI fail
during fetch from the shrinkwrap. It seems that when the fetcher is
fetching from the shrinkwrap, the SRCREV variable has been deleted but it
till ends up fetching from the git source resulting in an error because
SRCREV is undefined. The root cause of this is that the Fetch class defaults
to urls from the SRC_URI when the urls parameter contains an empty list. This
patch will ensure that Fetch is not instantiated if the urls list is empty.

(Bitbake rev: e602963dfd505eef08702366383358d29ee20c4d)

Signed-off-by: Svend Meyland Nicolaisen <public@smn.dk>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-22 10:30:42 +01:00
Chen Qi
c570cf1733 bitbake: runqueue: fix PSI check calculation
The current PSI check calculation does not take into consideration
the possibility of the time interval between last check and current
check being much larger than 1s. In fact, the current behavior does
not match what the manual says about BB_PRESSURE_MAX_XXX, even if
the value is set to upper limit, 1000000, we still get many blocks
on new task launch. The difference between 'total' should be divided
by the time interval if it's larger than 1s.

(Bitbake rev: b4763c2c93e7494e0a27f5970c19c1aac66c228b)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-22 10:30:42 +01:00
Kai Kang
39428da6b6 bitbake: bitbake: ConfHandler: Allow variable flag name with a single character
Update regex pattern to allow variable flag name with a single character.

Regression tests have also been updated in `bb.parse` and
`bin/bitbake-selftest -k ParseTest` has been successfully executed.

Eliminate a trailing space as well.

(Bitbake rev: bb9e523291a3cad6e1596ee6a1e715b5e5feba8f)

Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-22 10:30:42 +01:00
Mingli Yu
acf268757f bitbake: event: add bb.event.ParseError
Add bb.event.ParseError to let error-report can catch this kind of error.

(Bitbake rev: 316524ab59a5e738c25e062923ee5717d88ae5c7)

Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-22 10:30:42 +01:00
Enrico Scholz
b92918eb0e bitbake: fetch2/crate: create versioned 'name' entries
It is common for rust packages to depend on different versions of the
same crate.  E.g.

|    crate://crates.io/windows_x86_64_msvc/0.42.2 \
|    crate://crates.io/windows_x86_64_msvc/0.48.0 \

Identification only by the plain crate name makes the sha256sum
ambiguous

| SRC_URI[windows_x86_64_msvc.sha256sum] = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0"
| SRC_URI[windows_x86_64_msvc.sha256sum] = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a"

and requires lot of manual work to fix the SRC_URI list.

Making the 'name' property unique by appending the version allows
direct copy & paste of reported sha256sum errors to complete the
crates list.

(Bitbake rev: ae2efb05196f9e29ef56ad9a84e2eae5fbdd8030)

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-06 14:28:59 +01:00
Richard Purdie
fd0de1368e bitbake: bitbake: Bump to version 2.4.0
(Bitbake rev: 46e1ea2e9a203992bb4de48ea21a8e736419ada2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-05 17:28:40 +01:00
Frederic Martinsons
bbf2e79f88 bitbake: fetch2: Display all missing checksum at once
Instead of only display the first and stop.
For recipe (crago based) that can contains several artifacts
to fetch with their checksum, it will particularly handy
to display all of missing one.

An example of error message would be

NOTE: Executing Tasks
ERROR: zvariant-3.12.0-r0 do_fetch: Missing SRC_URI checksum, please add those to the recipe:
SRC_URI[anyhow-1.0.70.sha256sum] = "7de8ce5e0f9f8d88245311066a578d72b7af3e7088f32783804676302df237e4"
SRC_URI[chrono-0.4.24.sha256sum] = "4e3c5919066adf22df73762e50cffcde3a758f2a848b113b586d1f86728b673b"
SRC_URI[serde-1.0.158.sha256sum] = "771d4d9c4163ee138805e12c710dd365e4f44be8be0503cb1bb9eb989425d9c9"
SRC_URI[system-deps-1.3.2.sha256sum] = "0f3ecc17269a19353b3558b313bba738b25d82993e30d62a18406a24aba4649b"
ERROR: zvariant-3.12.0-r0 do_fetch: Bitbake Fetcher Error: BBFetchException('There was some missing checksums in the recipe')
ERROR: Logfile of failure stored in: /home/jenkins/yocto-poky-master/poky/build/tmp/work/core2-64-poky-linux/zvariant/3.12.0-r0/temp/log.do_fetch.1025
ERROR: Task (/home/jenkins/yocto-poky-master/poky/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb:do_fetch) failed with exit code '1'

(Bitbake rev: dafa07c080e05975b6319b5adf78a9691c6b6643)

Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-04 13:12:23 +01:00
Robert Yang
e8693364c5 bitbake: bitbake: bitbake-user-manual: Update EXCLUDE_FROM_WORLD
(Bitbake rev: ccb2878516d6ac2ee348ad67d0db6dda40e12018)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-30 10:51:54 +01:00
Robert Yang
178c6e0de8 bitbake: cache: Make EXCLUDE_FROM_WORLD boolean
Fixed:
Set EXCLUDE_FROM_WORLD = "0" in recipe_A:
$ bitbake world -g

Check pn-buildlist, the recipe_A won't be built, this patch fixes the problem:
EXCLUDE_FROM_WORLD = "1": Not build in world
EXCLUDE_FROM_WORLD = "0": Add back to world build

(Bitbake rev: 3f4ede2d67a2d75d3faf8887f90371bd7554d492)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-30 10:51:54 +01:00
Richard Purdie
c28a9a5799 bitbake: fetch2/local: Mention the value of localpath in failure message
We have an obtuse rare and intermittent bitbake fetch failure. Understanding
the value of ud.localpath at the time of failure would be helpful to debug it
further. Tweak the debugging to provide this in the output.

(Bitbake rev: 935ab36f64d25c09f83209cd369714c2bc89aa7f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-30 10:51:54 +01:00
Robert Yang
ac3eb2418a bitbake: fetch/git: Fix local clone url to make it work with repo
The "git clone /path/to/git/objects_symlink" couldn't work after the following
change:

6f054f9fb3

But repo command manages the git repo as symlinks, so check whether the objects
is an symlink to fix the problem:

* Nothing is changed if git/objects is not a symlink
* Use "git clone file://" when git/objects is a symlink

(Bitbake rev: a0d8108eba8d542707740d00c66c1c5f5b963f18)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-30 10:51:54 +01:00
Frederic Martinsons
4d9886e143 bitbake: crate.py: make checksum verification mandatory
(Bitbake rev: 9c07e377a21fe1d9660d084da93873acd289b581)

Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-30 10:51:54 +01:00
Frederic Martinsons
2b12c58724 bitbake: fetch2: Add checksum capability for crate fetcher
This change brings checksum verification of each crate
in a recipe, e.g

| SRC_URI += " \
|     crate://crates.io/aho-corasick/0.7.20 \
|     crate://crates.io/atomic-waker/1.1.0 \
|     crate://crates.io/cc/1.0.79 \
| "
|
| SRC_URI[aho-corasick.sha256sum] = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac"
| SRC_URI[atomic-waker.sha256sum] = "debc29dde2e69f9e47506b525f639ed42300fc014a3e007832592448fa8e4599"
| SRC_URI[cc.sha256sum] = "50d30906286121d95be3d479533b458f87493b30a4b5f79a607db8f5d11aa91f"

That will require to move the checksum initialization
after the possible call to urldata_init method in order
for the crate fetcher to parse the url.

Another way of doing could have been implementing a decodeurl
method that would have been specific for each fetcher class.

(Bitbake rev: 4920686c13dd66f9bfa4f7dd38d6e955f153eeec)

Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-30 10:51:54 +01:00
Daniel Ammann
92993b4f77 bitbake: fetch2/sftp: Fix fetching URIs with spaces
Spaces can be used in URIs with %20, but the URI needs to be quoted.

(Bitbake rev: 0f887589c279ce07692b4d8e118954a9d214ca45)

Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-23 22:41:32 +00:00
James R T
f14328c22f bitbake: ConfHandler: Allow the '@' character in variable flag names
This patch enables the usage of the '@' character in variable flag
names.

One use case of variable flags is to assign the network namespaces of
some systemd services/targets to configure other parts of the build
process of some system. The filenames of systemd services/targets might
contain the '@' character if they are template unit files that can take
in a single parameter/argument and be instanced multiple times, as
indicated by systemd's official manual page.

The '@' character is disallowed as the first character in a variable
flag name. Imposing more restrictions on the first character is a
compromise to make parsing easier and to allow for more options in the
future to extend the syntax.

This patch is successfully verified by creating a custom BitBake recipe
that sets and unsets the value of a variable flag with the '@' character
in its name and ensuring that no ParseError is being thrown. Regression
tests have also been added to `bb.parse`.

`bin/bitbake-selftest` has also been successfully executed and all tests
passed.

(Bitbake rev: 00f9ab2cacfbd2a63b6b4959cf5401babae7e32a)

Signed-off-by: James Raphael Tiovalen <jamestiotio@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-23 22:41:32 +00:00
Michael Opdenacker
f98f1fe24d bitbake: bitbake-user-manual: fix links to supported release manuals
Links to manuals for supported releases were all broken.
Only the pre-sphinx ones are still valid.

(Bitbake rev: 887804de05d704644328a105e2add6861adb4cab)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-23 22:41:32 +00:00
Richard Purdie
04c27888d0 bitbake: fetch2: Add autorev warning when it is set too late
Bitbake expects a consistent metadata environment when it parses. There
are plenty of ways you can set a recipe to autorev at a point where parts
of the fetcher have already been triggered leading to obsure bugs which
I struggled to debug, let alone anyone not familar with the code.

If anyone is running into the message from the commit, the issue is
likely one of timing. Keep in mind that the anonymous python code
in base.bbclass will expand variables like FILESPATH, WORKDIR and others
which contain PV. The recipe needs to be set to AUTOREV before that
anonymous python runs.

In particular, that means you can't set SRCREV = "${AUTOREV}" in other
anonymous python, it needs to happen earlier.

(Bitbake rev: 4d9ec332d5bfc8b60b54f8ec2a17d34e35aa903a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-22 13:57:04 +00:00
Richard Purdie
2b8761e5fc bitbake: fetch2: Rename __BBSEENSRCREV -> __BBSRCREV_SEEN
For readability of following patches, rename this internal variable to
allow for others in a similar format.

(Bitbake rev: 450b4a00895ed1f9396a8ff859dc1cc0eccc838f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-22 13:57:04 +00:00
Tim Orling
614b2a8379 bitbake: toaster: update fixtures for mickledore
After updating gen_fixtures.py for 'mickledore' 4.2,
run './gen_fixutures.py --all' to update oe-core.xml
and poky.xml.

Drop langdale as support is ending soon.

(Bitbake rev: 282a3d618b565a60b8734ca0fb710db93023b5c8)

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-21 22:44:00 +00:00
Tim Orling
44d1001a68 bitbake: toaster: update gen_fixtures.py for mickledore
Update for 'mickledore' 4.2.0 release and bitbake 2.4 branching.
Drop 'langdale' as support will be ending soon.
Update stable branches to latest patch release.

(Bitbake rev: b01d369f6d6cccf77b23017ff343820d800d5621)

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-21 22:44:00 +00:00
Mark Asselstine
312db493ee bitbake: build: Make python output print to stdout when running with -v (verbose)
When tasks are run with -v (verbose) on the bitbake commandline, shell
tasks print their stdout, python tasks do not.

This change redirects the python task's print output to an in memory
buffer. After the task is executed the output is printed to stdout via
the logger. This makes the python task behavior match the shell task
behavior when running with -v. The contents of the task's log files
remain unchanged after this change.

This approach should keep the correct order in most cases, however, if
the python task accesses the logger directly, that content will appear
before other output. On the other hand, this change should negate the
need for python tasks to access the logger directly.

Special care is taken to save/restore the existing stdout and stderr
and preventing sending output directly to the logger when there are
"recursive" calls, for instance when a python function calls a shell
function, avoiding printing things potentially out of order and/or
multiple times.

The logging-test.bb in meta-selftest can be used to review this
change. This has been tested with the full bblogging oeqa tests.

[Yocto #14544]

(Bitbake rev: 81a58647b2f4fc0a2589b2978fc9d81b2bfe6aec)

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-21 22:44:00 +00:00
Frederic Martinsons
ee5e6a86fd bitbake: crate.py: authorize crate url with parameters
This allow to have classic fetch parameters
(like destsuffix, sha256, name ...) not being
considered by crate fetcher itself (and so mess
up its download)

Moreover, it allow to overload the name of the downloaded
crate (maybe usefull if there is a naming clash between
two crates coming from different repositories)

(Bitbake rev: 278bd2f1758b8af97552af8d23d16ffb5127a131)

Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-17 17:20:05 +00:00
Mark Hatle
f29a1c2966 bitbake: wget.py: Combine urlopener exceptions
No reason to have three identical exception handles, refactor to catch any
of the exceptions with the same block of code.

(Bitbake rev: b29f6e04091b6bfe697dc41c76880de466736fc3)

Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-11 20:49:22 +00:00
Mark Hatle
f9b422152b bitbake: wget.py: Add catch TimeoutError exception
We've observed TimeoutError exceptions during the sstate-cache mirror fetch,
it appears that due to the number of (invalid) files requested the remote
side is eventually dropping the connection (not closing it) which can result
in a TimeoutError exception being sent, while rate it is different from the
urllib.error.URLError or ConnectionResetError.

(Bitbake rev: 6041b34740deee09ea65d705702555456a5e05d8)

Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-11 20:49:22 +00:00
Fawzi KHABER
e0a6ba834e bitbake: bitbake-user-manual: update Hello World example
Fix broken link "Obtaining bitbake".
Update documentation for the bitbake hello world example, the output was
outdated.
Fix LAYERSERIES_COMPAT warning by adding dunfell as default compatible release.
Add proper formating for base.bbclass command.

Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Ever ATILANO <ever.atilano@smile.fr>
Reviewed-by: Yoann CONGAL <yoann.congal@smile.fr>
(Bitbake rev: e859ce61a7422425960342a47785468cef49caa1)

Signed-off-by: Fawzi KHABER <fawzi.khaber@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-07 21:24:44 +00:00
Fawzi KHABER
28e334e8a4 bitbake: doc: ref-variables: add LAYERSERIES_COMPAT to term glossary
Bitbake throws a warning if the layer compatibility is not defined since
cca81e33b58c390dcf5cc3a31555a43b79177166. The description of this variable
comes from the Yocto Project manual.

Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Ever ATILANO <ever.atilano@smile.fr>
Reviewed-by: Yoann CONGAL <yoann.congal@smile.fr>
(Bitbake rev: cbad8cb28a8134d6b2053ef5c2e04bb455fc798b)

Signed-off-by: Fawzi KHABER <fawzi.khaber@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-07 21:24:44 +00:00
Richard Purdie
082e46f9ca bitbake: server/xmlrpc: Fix after currentAsyncCommand locking changes
After changes in bitbake b5215887d2f8ea3f28f1ebda721bd5b8f93ec7f3,
"process/cooker/command: Fix currentAsyncCommand locking/races", command.py
assumes it has access to the process server but the xmlrpc backend was
passing in the xmlrpc server object leading to errors like:

xmlrpc.client.Fault: <Fault 1: "<class 'AttributeError'>:'BitBakeXMLRPCServer' object has no attribute 'set_async_cmd'">

Fixing to pass the process server to command.py resolves this issue.

[YOCTO #15008]

(Bitbake rev: ce5b65d5fada474ef21ac28440af6ad45287650a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-07 12:52:10 +00:00
Richard Elberger
d86cbd3d3d bitbake: documentation: bitbake: add file-checksums to varflags section
Fixes [YOCTO #11605] by:

- Adding definition of file-checksums to Variable Flags section.
- Describe data to add to list which adds external file dependencies.
- Write example on usage to prepend a value to file-checksums list.

(Bitbake rev: 6bc65e6402a74c9a65e29333a7e0f1f85dcbcf58)

Signed-off-by: Richard Elberger <rich@richelberger.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:49:41 +00:00
Mark Asselstine
67d5541f65 bitbake: fetch/npmsw: add more short forms for git operations
>From the npm-install documentation [1] the CLI provides a set of
short forms when the install fetches from git. These include

"github:"
example: npm install github:mygithubuser/myproject

"gist:"
example: npm install gist:101a11beef

"gitlab:"
example: npm install gitlab:mygitlabuser/myproject

"bitbucket:"
example: npm install bitbucket:mybitbucketuser/myproject

Commit 1d8af6aed0a9 [fetch2: npmsw: Add support for github prefix in
npm shrinkwrap version] by Stefan Herbrechtsmeier added support for
the "github:" but the others would marked as 'Unsupported dependency'.

The other prefixes are added in this commit, along with extending the
tests to cover some of these.

However, there is one more short form for github which npm-install
allows which forgoes the prefix altogether.

example: npm install mygithubuser/myproject

Unfortunately this format is a bit problematic as it lacks any easily
identifiable 'marker' to match against, and it could be either the
github short form or install from folder format. Experimentation shows
that the folder format requires a leading './' or '/', so we use this
to rule out the ambiguity.

If this approach to folder and github formats disambiguation is
incorrect it won't matter anyways as the folder format is unrecognized
by the code as-is and thus with this change or without, things would
fail.

Since we have to be less strict in the check for git operations we
move it to be the last install format which we check, such that the
less ambiguous formats can be sorted out first.

[1] https://docs.npmjs.com/cli/v9/commands/npm-install

[Yocto #14236]

(Bitbake rev: 0ac6f6cb5d807919ed13a8b7bb3fb551b79c5a71)

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:49:41 +00:00
Mike Crowe
c1f1422a95 bitbake: gitsm: Fix path construction for relative submodule URI
The submodule repository URI contains a path to something not
necessarily on the local filesystem. This means that we can't use
realpath to normalise it without risking getting bad results if the path
happens to match something on the local filesystem. This situation can
cause very confusing errors if that matching local path happens to be a
symlink to somewhere else.

Using normpath rather than realpath means that the path simplification
follows simple rules on the string rather than looking at the local
filesystem and avoids problems.

(Bitbake rev: 47b271e6c8d96960ebe70f80e58f30cc4cbf42e1)

Signed-off-by: Mike Crowe <mac@mcrowe.com>
Co-authored-by: Dave Craig
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:49:41 +00:00
Richard Purdie
c08e2c2e20 bitbake: utils: Use internal fetcher function to avoid duplication
We currently have two lists of "proxy" or "fetcher" environment exports.
Make the one in utils match the one on the fetcher which has a more complete
list of variables now.

(Bitbake rev: f9165a798a307a6f0fee120d5c3de660d3a44ae8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:49:41 +00:00
Richard Purdie
9345140f09 bitbake: fetch2/wget: Drop unused import
This import is no longer used anywhere so can be removed.

(Bitbake rev: 956128e394581855bf0d03b32a975dc91c2a7e0c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:49:41 +00:00
Caner Altinbasak
24e29f2d08 bitbake: fetch2: Add GIT_CACHE_PATH and SSL_CERT_DIR into FETCH_EXPORT_VARS
These environment variables are needed by gclient and needed to be
passed into fetcher.

(Bitbake rev: 9414ba62454c6b911addf6b0bc02af2afc69b926)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:49:41 +00:00
Richard Purdie
584a913d75 bitbake: cooker: Fix memory resident cache invalidation issue
We've been seeing weird PRServ failures on the autobuilder. These had
one failure always followed by a second. Whilst I can't reproduce the first,
if I made that test fail, I could reproduce the second with memory resident
bitbake. This was with the tests:

prservice.BitbakePrTests.test_import_export_replace_db
and then
prservice.BitbakePrTests.test_pr_service_deb_arch_dep

which was giving a strange looking error:

NOTE: Running task 1053 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm)
NOTE: Running task 1054 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_qa)
ERROR: No such task: do_package_write_rpm
ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) failed with exit code '1'

where the issue is that selftest.inc written by the test framework
and containing PACKAGE_CLASSES = "package_deb" was being ignored.

The issue is the cached_statements{} within BBHandler() is not being
invalidated at the right time.

This patch changes the code to ensure base configuration is not parsed
until inotify updates have been processed.

(Bitbake rev: cada37c6b9e5862ca2c5a54ad6fd1e1f1939cd9c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-24 15:38:24 +00:00
Ross Burton
9b0277c3c3 bitbake: lib/bb/siggen: fix debug() call
Bitbake f68682 changed the logger's debug() method to be compatible with
logging.debug(), but this caller was still using the old API where you
can pass an integer as the first argument:

WARNING: Invalid arguments in bbdebug: (1, 1, 'Found unihash[...]')

Instead, call bbdebug() which has the priority argument.

(Bitbake rev: 18d4f9e8387f7994cf6d46300e25dda1c3a593b2)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-24 15:38:24 +00:00
Richard Purdie
77be687a37 bitbake: bitbake: Bump to version 2.3.1
So that OE-Core can depend on bb.event.check_for_interrupts(), bump our
verison number to a development series version.

(Bitbake rev: dea1b2f3fc31f28daed5da16e62da8895d6e5716)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-23 12:12:18 +00:00
Richard Purdie
42c92cbc83 bitbake: event/cooker/runqueue: Add ability to interrupt longer running code
Bitbake is now able to understand when a UI wants it to stop the current
processing. There are some long running functions which currently have no
mechanism to interrupt them however.

This patch adds a call, bb.event.check_for_interrupts(d) which can be
placed in long running code and allows an internal state flag within
bitbake to be checked. If set, that flag will trigger an exit.

This means that Ctrl+C can be made to work in a variety of places where
it currently would not.

Long running event handlers in OE-Core can also then benefit from this
new approach with the addition of the function call as well.

(Bitbake rev: b7ed7e9a815c4e10447fd499508be3dbb47f06e8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-23 12:12:18 +00:00
Paulo Neves
97b258985a bitbake: fetch/git: Replace mkdtemp with TemporaryDirectory and avoid exception masking
Due to using mkdtemp instead of TemporaryDirectory we needed to
manually cleanup the directory in a try finally block.
With tempfile.TemporaryDirectory we can handle the cleanup with
a "with" statement and not need to manually clean up oursevels.

(Bitbake rev: 36a231d2a45e7c87172196538d18ded81a234774)

Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22 12:03:39 +00:00
Paulo Neves
d085dcb449 bitbake: fetch/git: Removed unused variables in _contains_lfs
branchname was set but never used in the context of _contains_lfs
method.

(Bitbake rev: a64ec64c2faf1ac6ce995cdd4e311d8be6046155)

Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22 12:03:39 +00:00
Paulo Neves
ffcc20770d bitbake: tests/fetch: Add real git lfs tests and decorator
Added tests that verify that git-lfs works with an actual
real git-lfs server. This was not previously the case because
the repo in the test was a simulation of git-lfs but not
a real git lfs repo.

The 2 added tests are almost the same but test that the
git lfs file checkout is successfult with or without the
lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers
2 different code paths for git lfs.

lfs=1, when used on git lfs repositories triggers the git lfs
downloading at the fetch bare stage.

lfs query parameter unset triggers the git lfs downloading only
on checkout as an implicit behavior of git. This leads to possible
network access on the unpack stage and outside the DL_DIR.

lfs=0 actually disables git-lfs functionality even if supported.

(Bitbake rev: d2be7f7f652360f13cd66d0850f3e19ffe2afb0a)

Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22 12:03:39 +00:00
Paulo Neves
41c86ad30e bitbake: tests/fetch: git-lfs restore _find_git_lfs
Not restoring the mocked _find_git_lfs leads to other tests
failing.

(Bitbake rev: 70f848631450bd723c223227c21c60e815ee033d)

Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22 12:03:39 +00:00
Richard Purdie
76db0baba2 bitbake: server/process: Improve idle thread exception handling
If the inotifier code has an exception, bitbake currently hangs. Catch any
exception and exit if seen. Also check the idle thread is alive and exit
if it disappears. This should stop bitbake hanging if such a situation arises
in future such as this example:

3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS']
Exception in thread Thread-1 (idle_thread):
Traceback (most recent call last):
  File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread
    self.cooker.process_inotify_updates()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates
    n.read_events()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events
    if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1:
OSError: [Errno 9] Bad file descriptor
3323260 21:48:32.206995 Command Completed (socket: True)

(Bitbake rev: 358b5b02d5de1ab0f98104c4ec4953e46999b9a5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-20 15:18:53 +00:00
Richard Purdie
7244bf2457 bitbake: cooker: Ensure lock is held with changing notifier
We've seen a couple of cases which bitbake hangs due to an inotifer exception
such as:

3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS']
Exception in thread Thread-1 (idle_thread):
Traceback (most recent call last):
  File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread
    self.cooker.process_inotify_updates()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates
    n.read_events()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events
    if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1:
OSError: [Errno 9] Bad file descriptor
3323260 21:48:32.206995 Command Completed (socket: True)

Ensure we don't destory the inotifier when the idle thread is reading is
by holding the lock during setup/teardown.

(Bitbake rev: 8fc5c50c2e23017833f93bcd514d708a14fa4266)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-20 15:18:53 +00:00