Commit Graph

7383 Commits

Author SHA1 Message Date
Richard Purdie
6bd8367aa9 bitbake: BBHandler: Handle unclosed functions correctly
A function accidentally defined as:

somefunction() {
	:
 }

which is unclosed due to the space at the end, would currently silently
cause breakage. Have the parser throw and error for this.

[YOCTO #15470]

(Bitbake rev: a7dce72da6be626734486808f1b731247697e638)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Michael Opdenacker
9889a0ff1a bitbake: prserv: remove unnecessary code
In db.py, the ifnull() statement guarantees that the SQL request will
return a value. It's therefore unnecessary to test the case when no
value is found.

(Bitbake rev: e4ae5177861c9a27e93e5a2d3a6c393baecd6416)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Michael Opdenacker
f2a83b50b3 bitbake: prserv: correct error message
according to db.py, prserv.NotFoundError is returned here when
adding a new value to the database failed

(Bitbake rev: 4cc4069987edd14f51715dfaf0c6e1a3aa307106)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Michael Opdenacker
d19e32dcd7 bitbake: prserv: remove redundant exception handler
This exception handler is already present in db.py's get_value() code.

(Bitbake rev: 2fd38b1bb685ec441f0eb0f28f3d84ba252ba90b)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Michael Opdenacker
112a37e6a9 bitbake: prserv: add extra requests
Useful for connecting a PR server to an upstream one

- "test-package" checks whether the specified package
  version and arch is known in the database.

- "test-pr" checks a specified output hash is found in the database.
  Otherwise it returns 'None' instead of a new value.

- "max-package-pr" returns the highest PR number for
  (version, arch) entries in the database, and None if not found

Add new DB functions supporting the above, plus test_value()
which tells whether a given value is available for the specified
package and architecture.

(Bitbake rev: 0f1474a30f741b760ca81c19dd1d8f3bd5647251)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Michael Opdenacker
62a3a7172a bitbake: prserv: capitalization and spacing improvements
Choosing only one style of capitalization
Add extra space after some commas too
Remove idle spaces

(Bitbake rev: daad17bccec8cb98ef2fca4262641167500bd46e)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Michael Opdenacker
d133dc7e77 bitbake: asyncrpc: include parse_address from hashserv
Moving the code and related definitions from
hashserv/__init__.py to asyncrpc/client.py,
allowing this function to be used in other asyncrpc clients.

(Bitbake rev: b67bb05e431414866b8e8c6a4c88d20b9cdb44a3)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Suggested-by: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Michael Opdenacker
9a2d08995e bitbake: prserv: use self.logger instead of logger directly
In both the PRServerClient and PRClient objects.

This aligns with what is done in hashserv/server.py and makes it
possible to benefit from possible specializations of the logger
in the corresponding super classes, instead of using
always the global logger.

(Bitbake rev: 5fc6d2b1a5db617e16c1eb9fbd25e821237611d8)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Michael Opdenacker
d57e17425e bitbake: bitbake-prserv: replace deprecated optparse by argparse
optparse is deprecated since Python 2.7

Note that this is neither supposed to change the options
supported by bitbake-prserv nor the way they are interpreted.

Note that in the "--help" output, long options are now reported
for example as "--host HOST" instead of "--host=HOST" but
both are equivalent anyway, as they already were with optparse.

(Bitbake rev: 434cd00a9e5a8ef32088f1a587005adf910a92eb)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Michael Opdenacker
4b1ef692a9 bitbake: prserv: use double quotes by default
To aligh with the hashserv code

(Bitbake rev: 7a6999750791659eaffe49aabfbfba9f37f51913)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Michael Opdenacker
8d78b5f9c5 bitbake: prserv: simplify the PRServerClient() interface
serv.py: simplify the PRServerClient() interface by passing the
server object instead of multiple arguments, and then retrieving
the data through this object.

This replicates what is done for ServerClient() in hashserv/server.py

(Bitbake rev: d3be073218feb4d6e68a751832da4936da485dbc)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Richard Purdie
e1365064be bitbake: doc/user-manual: Add BB_LOADFACTOR_MAX
Document BB_LOADFACTOR_MAX which was recently added.

(Bitbake rev: 833b76e9333e317cab5f17d6f7daaecc89c69547)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-12 17:23:27 +01:00
Simone Weiß
793b31dbf4 bitbake: doc: Add section for variable context
This is inspired by the same section in the yocto-docs. It aims to provide
information in what contexts(recipes, .conf, bbclass,...) a variable is usually
used. For that I tried to group similar variables, so that a short overview is
given. This was inspired by [YOCTO #14072], but of course does not implement a
warning if a variable is used in an unintended context.

(Bitbake rev: 5ced476685376b1a32b7fdd9546f9b61c5962aa0)

Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-11 14:53:15 +01:00
Rob Woolley
c0cd7a6d3b bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftps
Fedora 40 introduces wget2 as a drop-in replacement for wget.  This
rewrite does not currently have support for FTP.  This causes
the wget fetcher to fail complaining about an unrecognized option.

Making --passive-ftp conditional based on the protocol used in
the SRC_URI limits the scope of the problem.  It also gives us
an opportunity to build the older wget as a host tool.

(Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a)

Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-11 08:47:07 +01:00
Joshua Watt
f2ff622a4c bitbake: bitbake-hashclient: Warn on bad .netrc
If there is an error parsing .netrc, warn the user on stderr

(Bitbake rev: 6366ea8d9c284d10bb8f4129004b55239d9022c0)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-26 17:11:15 +00:00
Joshua Watt
978206fed4 bitbake: siggen: Add support for hashserve credentials
Adds support for hashserver credentials to be specified in the
SignatureGenerator

(Bitbake rev: 741bef3755fde7bae1386aad575ea704d9fe0969)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-23 10:19:00 +00:00
Derek Erdmann
825055e83e bitbake: fetch2/git: Install Git LFS in local repository config
Git uses a lock file to prevent concurrent modifications to the global
config, so if unpack tasks for different recipes try to run "git lfs
install" simultaneously the operation can fail:

    error: could not lock config file /home/build/.gitconfig: File exists exit status 255
    Run `git lfs install --force` to reset Git configuration.

Adding "--local" sets the smudge and clean filters in the local
repository's config instead of modifying the user's global config.

(Bitbake rev: 328ca4de8422be514fa0d0c9e3cfd36bb9d3e9a7)

Signed-off-by: Derek Erdmann <derek.erdmann@sonos.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:46 +00:00
Yang Xu
64057e6b15 bitbake: bitbake-worker: Fix silent hang issue caused by unexpected stdout content
This patch addresses an issue in bitbake-worker where stdout,
reserved for status reporting, is improperly accessed by child processes.

The problem occurs during the execution of parseRecipe,
which calls anonymous functions. If these functions use print-like operations,
they can inadvertently output data to stdout. This unexpected data can cause
the runqueue to hang silently, if the stdout buffer is flushed
before exec_task is executed.

To prevent this, the patch redirects stdout to /dev/null and ensures it is
flushed prior to the execution of exec_task.

(Bitbake rev: 08f3e677d6af27a41a918aaa9da9c1c9b20a0b95)

Signed-off-by: Yang Xu <yang.xu@mediatek.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:46 +00:00
Felix Moessbauer
9504df41f9 bitbake: utils: better estimate number of available cpus
When running in a cgroup which is limited to a subset of cpus (via
cpuset.cpus), cpu_count() should return the number of cpus that can be
used instead of the number of cpus the system has.

This also aligns the semantics with nproc.

(Bitbake rev: a029bfe96c6542f178720c72a772b7ede9898118)

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Viswanath Kraleti
3a97792820 bitbake: fetch2: Fix misleading "no output" msg
When a command is run with a non-null log, errors are only output to the
log and are not returned in the exception. In that case direct users to
that logfile instead of telling the command had no output.

(Bitbake rev: 944fe0a77932a5559e01ae6035c4bffa5185ea6a)

Signed-off-by: Viswanath Kraleti <quic_vkraleti@quicinc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Ross Burton
e20ee877ed bitbake: fetch2: handle URIs with single-valued query parameters
Whilst typically the URI query is a list of key-value pairs, that's not
actually required by the URI specification.

For example:  http://example.com/foo?bar is a valid query, but this will
result in the fetcher raising an exception:

  File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__
    self.query = self._param_str_split(urlp.query, "&")
  File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split
    for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]:
ValueError: not enough values to unpack (expected 2, got 1)

In this case the query is just "bar", but the fetcher is trying to split
it into a key-value pair.

The URI object exposes the parsed query explicitly as a dictionary of
key-value pairs, so we have to be a little creative here: if a value is
None then it isn't a key-value pair, but a bare key.

Fix this by handling elements without the deliminator in _param_str_split()
(by assigning the value to None), and handle a None value when formatting
the query in _param_str_join().

(Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Peter A. Bigot
adb0ea98d1 bitbake: lib/bb: support NO_COLOR
Red text on a black background can make it difficult for people with
visual impairments to read the text of error messages.  Respect the
presence of a non-empty NO_COLOR environment variable as an indication
the user doesn't want colorization to be automatically enabled.

See: https://no-color.org/
(Bitbake rev: d9986c54cd3d67ed1f7cb636b17696c8d0d4db85)

Signed-off-by: Peter A. Bigot <pab@pabigot.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Philippe Rivest
2eeef2800b bitbake: bitbake: fetch2/git: Escape parentheses in git src name
FIXES [YOCTO #15404]

When using git fetcher on a repo with parentheses in its URL, the
invocation of the git clone command will fail. The clone directory
is not quoted thus the shell will return an error and won't execute
the command.

(Bitbake rev: b5624ee564)

Cc: Philippe Rivest <privest@genetec.com>

(Bitbake rev: 12f9738577934ad7c99f0770f1392a9d6050e7d6)

Signed-off-by: Philippe Rivest <technophil98@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
david d zuhn
ecb1248914 bitbake: bitbake-worker: allow '=' in environment variable values
Limit the split to key & value (2 items) instead of the n items one
can get if there are '=' characters in the value.

Fixes [YOCTO #15447]

(Bitbake rev: 86315961829ab1d137a0265cc246c44d3929e1fb)

Signed-off-by: david d zuhn <david.zuhn@sonos.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Sava Jakovljev
0acdb81ca6 bitbake: bitbake-worker: Fix bug where umask 0 was not being applied to a task
* In the current implementation, "umask" variable is initially set to
  None and overwritten with user-specified value. However, in the worker
  implementation, a faulty if clause would only check whether the
  variable contains a value that evaluates to True, and not whether
  the variable is defined, so the value of 0 would lead to umask not
  being changed.
  This bug makes it impossible to have a task set its umask to value 0,
  for any possible reason it may want to.
  Fix this bug by extending the condition checked in the worker
  implementation.

(Bitbake rev: 19f9df6c750c592316a0fa18165b68636281fe3e)

Signed-off-by: Sava Jakovljev <sjakovljev@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Alexander Kanavin
98d09d41fa bitbake: bitbake: improve descriptions of '-S printdiff'
(Bitbake rev: becf88c2250a47102c8d36ad8b40839e0bfa9137)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Michael Opdenacker
be10de3423 bitbake: utils: remove BB_ENV_PASSTHROUGH from preserved_envvars()
preserved_envvars() is used when the BB_ENV_PASSTHROUGH
environment variable is not set. Therefore, its code shouldn't
return this variable.

(Bitbake rev: 0a33b560233b983456178541603ab96fea22238b)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Robert P. J. Day <rpjday@crashcourse.ca>
Tested-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-05 12:25:08 +00:00
Yoann Congal
ef88dee4f7 bitbake: prserv/serv: Fix a PID file removal race on prserv stop
A race condition has happened where the exiting server removed the PID
file between the existence check and the removal, resulting in a
FileNotFoundError exception.

The fix is to ignore the FileNotFoundError exception, the existence
check is now redundant so remove it to simplify.

Fixes [YOCTO #14341]

(Bitbake rev: 40d00bf9308e0bf73a00134a99a012a292daa1c5)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-03 16:26:17 +00:00
Richard Purdie
2689d8cf22 bitbake: fetch/git: Avoid clean upon failure
Currently when git fetches fail, it destroys all the existing local clone data.
For large repositories this can introduce long build delays when for example,
you just typo'd the git revision hash.

The git fetcher should be able to recover most directories so when the fetch is
for a git repo, avoid removing things unless clean is explicitly called
(e.g. a -c cleanall task).

(Bitbake rev: 1b3cd039fe19b24bd4be9a0202a98cdcbb0e9443)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-03 16:26:17 +00:00
Ulrich Ölmann
a32b45289b bitbake: taskexp_ncurses: fix execution example in introductory comment
Drop the ".py" file extension from the "-u" option's argument that has been
overlooked while applying the original patch (see [1]) to make the example work.
While at it sort the recipes' names consistently with respect to what is found
in the self-test examples below.

[1] https://lore.kernel.org/bitbake-devel/6f2645a7c4db2ae149d387544d2b94209cfed3f4.camel@linuxfoundation.org/

(Bitbake rev: 1f4d517b7a0389f78d1f791135f8dc9120e9912b)

Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-28 09:21:48 +00:00
Joshua Watt
be57fda542 bitbake: asyncrpc: Add support for server headers
Adds support for asyncrpc servers to send connection headers to clients
on connection. Since this is a breaking protocol change, clients must
opt-in to expect headers from the server, corresponding to a version
bump in the client protocol.

(Bitbake rev: 1cb2b8be6cc5269553f549285592e47b7d29db03)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-27 11:36:38 +00:00
Tim Orling
2841449527 bitbake: layerindexlib: fix missing layer branch backtrace
When a LayerBranch (a specific layer at a specific release) does not
exist in the layerindex database ("Layerindex Metadata"), the dependency
would throw a backtrace. Instead fail early and provide an error message.

Since layerindexlib will also check the local layers, inform the user that
a local checkout might resolve the situation. Recommend that they reach
out to the layer maintainers and layer index admins to properly fix it for
everyone.

While we are here, remove some trailing whitespace.

[YOCTO #15365]
[YOCTO #13954]

(Bitbake rev: 96cbe8f87209a927c157ebcf469f8b9d54fcf92e)

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-27 11:36:38 +00:00
Philip Lorenz
0e3bcc5103 bitbake: fetch2: Ensure that git LFS objects are available
The current implementation only performs a git lfs fetch alongside of a
regular git fetch. This causes issues when the downloaded revision is
already part of the fetched repository (e.g. because of moving back in
history or the updated revision already being part of the repository at
the time of the initial clone).

Fix this by explicitly checking whether the required LFS objects are
available in the downloade directory before confirming that a downloaded
repository is up-to-date.

This issue previously went unnoticed as git lfs would silently fetch the
missing objects during the `unpack` task. With network isolation turned
on, this no longer works, and unpacking fails.

(Bitbake rev: cfae1556bf671acec119a6c8bbc4b667a856b9ae)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-23 14:34:05 +00:00
Enguerrand de Ribaucourt
fff242b5d2 bitbake: bitbake: progressbar: accept value over initial maxval
There is a very rare case where the maxval is improperly computed
initially for cache loading progress, and the value will go over.

Explanation from bitbake/lib/bb/cache.py:736 in MulticonfigCache:__init__:progress()
    # we might have calculated incorrect total size because a file
    # might've been written out just after we checked its size

In that case, progressbar will receive a value over the initial maxval.
This results in a ValueError stack trace as well as bitbake returning 1.
    Traceback (most recent call last):
    File ".../poky/bitbake/lib/bb/ui/knotty.py", line 736, in main
        cacheprogress.update(event.current)
    File ".../poky/bitbake/lib/progressbar/progressbar.py", line 256, in update
        raise ValueError('Value out of range')
    ValueError: Value out of range

This fix mirrors the behavior of MulticonfigCache and accepts the new
value as the new maxval. This is also what the percentage printout
is doing in bitbake/lib/progressbar/progressbar.py:191 in ProgressBar:percentage()

I encountered this issue randomly while working on a project with
VSCode saving files while commands where fired.

Note: This file is a fork from python-progressbar. It hasn't been
refreshed in 8 years. We did only two commits, 5 years ago with minor
modifications. This new change is also not how the upstream project is
behaving.

(Bitbake rev: 7cea7f7a87da041fc1ad370c5c3d15aabad3a0d4)

Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-23 14:34:05 +00:00
Richard Purdie
08aa69a1fd bitbake: runqueue: Add support for BB_LOADFACTOR_MAX
Some ditros don't enable /proc/pressure and it tends to be those which we
see bitbake timeout issues on, seemingly as load gets too high and the bitbake
processes don't get scheduled in for minutes at a time.

Add support for stopping running extra tasks if the system load average goes
above a certain threshold by setting BB_LOADFACTOR_MAX.

The value used is scaled by CPU number, so a value of 1 would be when
the load average equals the number of cpu cores of the system, under one
only starts tasks when the load average is below the number of cores.

This means you can centrally set a value such as 1.5 which will then
scale correctly to different sized machines with differing numbers
of CPUs.

The pressure regulation is probably more accurate and responsive, however
our graphs do show singificant load spikes on some workers and this
patch is aimed at trying to avoid those.

Pressure regulation is used where available in preference to this load
factor regulation when both are set.

(Bitbake rev: 14a27306f6dceb4999c2804ccae5a09cc3d8dd49)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-23 14:34:05 +00:00
Tobias Hagelborn
42242fb9ef bitbake: hashserv: Re-enable connection pooling with psycopg 3 driver
Re-enable connection pooling in case `postgresql+psygopg` driver
is used. Async connection pooling is supported in psycopg 3 [psycopg]
driver in SQLAlchemy. Allow the connection pool to grow to
arbitrary size.

(Bitbake rev: 4fe05513b5314c201725e3f8ad54f58d70c56258)

Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-23 14:34:05 +00:00
Alexander Kanavin
730bd999d6 bitbake: Revert "bitbake: wget.py: always use the custom user agent"
This reverts commit 987ab2a446.

There's been a report that this breaks downloads from Jfrog Artifactory
as self.user_agent is set to 'Mozilla Firefox', and when Artifactory
sees that, it sends a response tailored for showing in an interactive browser
(which in my opinion it has every right to).

If we're using wget, we should say so via wget's default; handling uncooperative
servers should be done on per-recipe basis, and ideally with tickets
to admins of those servers.

(Bitbake rev: feef5cd12e877f42ffcace168d44b0e6eb80a907)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-20 12:56:40 +00:00
Peter Kjellerstedt
d25f6b970f bitbake: fetch2/git: Make latest_versionstring extract tags with slashes correctly
Before, everything up to the last slash was removed when extracting the
names of the tags. This would lead to that a tag such as "agent/11.0.0"
would be incorrectly identified as "11.0.0", which would then be treated
as a correct version matching "^(?P<pver>\d+(\.\d+)+)".

(Bitbake rev: 8b21024b9966d5158ac4a77e87ffb935c2a57764)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Peter Kjellerstedt
9694fe1d68 bitbake: fetch2/git: A bit of clean-up of latest_versionstring()
This is mostly preparations for the next commit.

(Bitbake rev: dcd2abfde55cc59d9869a7c97620b6fc30a52047)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Peter Kjellerstedt
21603031df bitbake: tests/fetch: Make test_git_latest_versionstring support a max version
In some cases, the version found by latest_versionstring() may be higher
than the real version. Make it possible to specify a maximum version so
that this case can be detected.

(Bitbake rev: 9134d4777109bc78410c3e641420d9a78b485e33)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Chen Qi
272d5ae098 bitbake: fetch2/git.py: add comment in try_premirrors
The purpose of ensuring 'incremental fetch' is not easy to see from
the codes. So add comments to explain this.

(Bitbake rev: 8b890b87e30cd05ec92ed71ee3691a47b4d77253)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Chen Qi
09ecbcd307 bitbake: tests/fetch.py: add test case for using premirror in restricted network
We had issue when BB_ALLOWED_NETWORKS is set and `bitbake grpc-native -c fetch'
failed even with all contents available in PREMIRRORS.

Add a test case to ensure no regression in the future.

(Bitbake rev: 80c91ceb81b1cae203067af58d3f1fe9c619ae83)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Chen Qi
c984b03f02 bitbake: fetch2/git.py: fix a corner case in try_premirror
For gitsm recipes, it's possible that some URL is used more than once.
e.g.,
A -> B:rev1 (B is a submodule of A)
A -> C (C is a submodule of A)
C -> B:rev2 (B is a submodule of C)
A anc C are both using B as submodules, but on different revs.
Now if we have:
B:rev1 -> D
B:rev2 -> E
Then, the mirror will not be fully used.
Say we have all repo mirrors for A, B, C, D, E, then in theory it's not
necessary to reach out to any network for downloading. But it's not the
case. After downloading B(rev1) and its submodule D from mirrors, the fetch
process continues to download C, thus B(rev2) and E. Now it finds that B
needs an update because its submodule E needs an update. Of course this is
true because E is not downloaded yet. Now the problem comes to whether to
use mirror or not. The git.py defines try_premirror to return 'False' when
the ud.clonedir exists. As B has been cloned, the ud.clonedir exists and
try_mirror returns False, resulting in not using mirror and going to upstream
directly.

We can see that the mirrors are not fully used. This is usually not problem,
as the cost is only some network download. But in case the following two
settings are there, we get errors.
BB_NO_NETWORK = "0"
BB_ALLOWED_NETWORKS = "*.some.allowed.domain"
In such case, the gitsm recipe A will fail to fetch. Note that all contents
that A needs are in mirrors and now it's failing to fetch. This is unexpected.

Note that the different revs of the same repo in gitsm recipe is not the only
way to reveal this problem. For example, there might be a recipe call B that
uses B:rev3. Check the protobuf and grpc recipes as an example.

For now, we can use the following steps to reproduce this issue. To be clear,
the grpc recipe in meta-oe is now 1.60.0.
1. Add in local.conf:
   DL_DIR = "${TOPDIR}/downloads-premirror"
   bitbake grpc -c fetch
2. Comment out the DL_DIR setting in local.conf and add the following lines:
   PREMIRRORS:append = " \
     git://.*/.* git://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
     gitsm://.*/.* gitsm://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
   "
3. Set BB_NO_NETWORK = "1" and then 'bitbake grpc -c fetch'.
   This command succeeds and this shows that the premirror holds everything we need.
4. Add the following lines and then 'bitbake grpc -c fetch'.
   BB_NO_NETWORK = "0"
   BB_ALLOWED_NETWORKS = "*.some.domain"

After step 4, the error message is as below:
ERROR: grpc-1.60.0-r0 do_fetch: The URL: 'gitsm://github.com/protocolbuffers/protobuf.git;protocol=https;name=third_party/protobuf;subpath=third_party/protobuf;nobranch=1;lfs=True;bareclone=1;nobranch=1' is not trusted and cannot be used

This patch fixes this problem by handling this corner case, that is, if the URL is
not trusted from the settings of BB_NO_NETWORK and BB_ALLOWED_NETWORKS, then we should
try premirrors because trying to reach upstream is destined to fail.

(Bitbake rev: e1be272ad105b47d3131b77168d9172386993fcb)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Pavel Zhukov
de27ecd228 bitbake: tests/fetch.py: add multiple fetches test
Fetch from premirror few times to emulate multiple machines sharing same
clonedir or few rebuilds of the package from (pre)mirror
Regression test for [Yocto #15369]

(Bitbake rev: 7fcbac574c68f16b95ab7abb2874931d168d3c9e)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 12:52:30 +00:00
Pavel Zhukov
b2d0f31d24 bitbake: fetch2/git.py: Fetch mirror into HEAD
Fix the issue with using of (pre)mirror in case if clonedir exists but
outdated.
Previous version of the code fetched new mirror content into FETCH_HEAD
instead of branch which caused refetch from the upstream. Add new remote
add fetch from it instead so the ref can be found by "_contains_ref"

Fixes [Yocto #15369]

(Bitbake rev: 69588e2a5c7c200e47b02b2391498dcb72388bd2)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 12:52:30 +00:00
André Draszik
e6892bc47a bitbake: git-make-shallow: support git's safe.bareRepository
When git is configured with safe.bareRepository=explicit [1], the
git-make-shallow fails miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].

It also seems that it is being rolled out in some environments as a
default for users.

In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.

The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.

Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: 7c63989db4590564516ed150930f4e2fa503e98f)

Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 12:00:59 +00:00
André Draszik
577e73606a bitbake: tests/fetch: support git's safe.bareRepository
When git is configured with safe.bareRepository=explicit [1], the
bitbake selftests fail miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].

It also seems that it is being rolled out in some environments as a
default for users.

In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.

The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.

Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: a45e14a7343e36101e45639931322e5649587f57)

Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 12:00:59 +00:00
André Draszik
b3d9663817 bitbake: fetch/git2: support git's safe.bareRepository
When git is configured with safe.bareRepository=explicit [1], the
bitbake git fetcher fails miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].

It also seems that it is being rolled out in some environments as a
default for users.

In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.

The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.

While at it, fix one open-coded invocation of git that wasn't using
ud.basecmd

Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: 5f3b1d8dc9ee70e707536bd75ee845b547440c97)

Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 12:00:59 +00:00
Richard Purdie
eca5708b87 bitbake: bitbake: Bump version to 2.7.3 for hashserv changes
(Bitbake rev: c1e0a0b6ddc9667c9d62319bd9ccd4eb8c64c2a6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:58:12 +00:00
Tobias Hagelborn
4e673ccb00 bitbake: bitbake: hashserv: Postgres adaptations for ignoring duplicate inserts
Hash Equivalence server performs unconditional insert also of duplicate
hash entries. This causes excessive error log entries in Postgres.
Rather ignore the duplicate inserts.

The alternate behavior should be isolated to the postgres
engine type.

(Bitbake rev: e8d2d178d0fe96f9d6031c97328e8be17d752716)

Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:58:12 +00:00