Commit Graph

5000 Commits

Author SHA1 Message Date
Richard Purdie
125ca0ff2f bitbake: tests/fetch: Tweak test to match upstream repo url change
Upstream changed their urls, update our test to match.

(Bitbake rev: 0791e66a47185d6f202af6be0d39e36a8e41850a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit dc391b86540ec5e0a0f1d811c776a22d443b1c06)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-06-14 05:19:21 -07:00
Joshua Watt
83793ccd86 bitbake: siggen: Enable batching of unihash queries
Uses the batching API of the client to reduce the effect of latency when
making multiple queries to the server

(Bitbake rev: fdac31655fb5441139f70e50292dbb5774cfdafe)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-06-06 06:53:49 -07:00
Joshua Watt
d1811356b7 bitbake: bb: Use namedtuple for Task data
Task dependency data is becoming unwieldy with the number of indices it
contains. Convert it to use a named tuple instead, which allows members
to be indexed by a named property or an index (which allows it to retain
backward compatibility).

(Bitbake rev: bc7c44affe8e3fa94e92c3bcb8ad85bf11963779)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-06-06 06:53:49 -07:00
Richard Purdie
bcb569e698 bitbake: runqueue: Improve rehash get_unihash parallelism
Improve the rehash code to query unihashes in parallel since this is more
efficient on slower links.

(Bitbake rev: 682fb42420eebf18f8a98bd3992baf9034dac5ac)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-06-06 06:53:49 -07:00
Richard Purdie
cc341e44bf bitbake: runqueue: Process unihashes in parallel at init
Improve the runqueue init code to call unihash queries in parallel since
this is faster and more efficient, particularly on slower links with longer
round trip times.

The call to the function from cooker is unneeded since that function calls
prepare() and hence this functionality will already have run, so drop
that obsolete call.

(Bitbake rev: e0486054c7a4c637446c18608e9983cc8dc4d7fe)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-06-06 06:53:49 -07:00
joshua Watt
fa23d0fc6d bitbake: siggen/runqueue: Report which dependencies affect the taskhash
Report which task dependencies in BB_TASKDEPDATA are included in the
taskhash. This allows tasks to identify which tasks dependencies may
change without the task re-running. Knowing this information is
important for tasks that want to transfer information from dependencies
(such as SPDX)

(Bitbake rev: 853423661779023763a87462b623b6e9ff2798b2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-06-06 06:53:49 -07:00
Richard Purdie
fb60467f0e bitbake: runqueue: Allow rehash loop to exit in case of interrupts
The initial hash serve loop exits in the case where interrupts are present
but probably checks a bit too often. Tweak that and also allow the slow
rehash loop to break on interrupt, improving bitbake Ctrl+C response.

(Bitbake rev: ca2f63d84b90cbd99936c9d7815d912d9019a594)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-06-06 06:53:49 -07:00
Richard Purdie
bb93b6d1fb bitbake: runqueue: Add timing warnings around slow loops
With hashserve enabled, there are two slow paths/loops, one at initial runqueue
generation and also during the rehash process when new outhashes are found.

Add timing information at the hashserve log level for when these loops
take longer than 30s or 60s overall. This will leave evidence in the logs when
things are running particularly slowly.

(Bitbake rev: 9ee503c79936b13f1d45f9e43211f77a528cdbfa)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-06-06 06:53:49 -07:00
Sven Schwermer
c5df9c829a bitbake: fetch2/gcp: Add missing runfetchcmd import
This adds the missing import. This bug was introduced with 1ab1d36c.

(Bitbake rev: b5159c0373e2e7d403aed16e096ad655f38b1fa7)

Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-05-29 07:50:00 -07:00
Richard Purdie
74ea960113 bitbake: parse: Improve/fix cache invalidation via mtime
We have been seeing obscure failures in devtool, particularly on newer
autobuilder workers where it appears the cache is assumed to be valid
when it shouldn't be.

We're using the 'seconds' granulation mtime field which is not really
a good way of telling if a file has changed. We can switch to the "ns"
version which is better however also add in inode number and size as
precautions. We already have all this data and tuples are fast so there
isn't really any cost to do so.

This hopefully fixes [YOCTO #15318].

(Bitbake rev: 2aa519fec30ecf0e55202d6421fbba2f0320e211)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d9e5d313c79500e3c70ab9c3239b6b2180194f67)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-05-29 07:50:00 -07:00
joshua Watt
107a6cec75 bitbake: cooker: Handle ImportError for websockets
Handles ImportError when creating a hash equivalence to ping the server.
This notifies user earlier with a more precise error if websockets can't
be used, and also prevents passing a known bad upstream value to the
local server

(Bitbake rev: 93190565fd1251e8f47d9a6291739f8b8ed5ec87)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit aa80b3cfc5d16dfba13ca7fb9b78bae179ce3b74)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-05-29 07:50:00 -07:00
joshua Watt
6c9d0bddbc bitbake: cooker: Use hash client to ping upstream server
The cooker attempts to connect to the upstream hash equivalent server to
warn the user early if it is misconfigured. However, this was making the
assumption that it was a raw TCP connection and failed when attempting
to use a websocket upstream server. Fix this by creating an hash client
and using the ping API to check the server instead of using a raw
socket.

(Bitbake rev: 8f90d10f9efc9a32e13f6bd031992aece79fe7cc)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-02 16:07:25 +01:00
Richard Purdie
3bbe0a45b4 bitbake: bitbake: Bump to version 2.8.0
(Bitbake rev: c86466d51e8ff14e57a734c1eec5bb651fdc73ef)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-16 07:38:22 +01:00
Joshua Watt
2ecd97fa59 bitbake: siggen: Capture SSL environment for hashserver
Now that the bitbake hash server supports SSL connections, we need to
capture a few environment variables which can affect the ability to
connect via SSL. Note that the variables are only put in place to affect
the environment while actually invoking the server

[RP: Tweak to use BB_ORIGENV as well]
[RP: Tweak to handle os.environ restore correctly]
(Bitbake rev: 0bacf6551821beb8915513b120ae672ae8eb1612)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-16 07:33:19 +01:00
Richard Purdie
6bd8367aa9 bitbake: BBHandler: Handle unclosed functions correctly
A function accidentally defined as:

somefunction() {
	:
 }

which is unclosed due to the space at the end, would currently silently
cause breakage. Have the parser throw and error for this.

[YOCTO #15470]

(Bitbake rev: a7dce72da6be626734486808f1b731247697e638)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Michael Opdenacker
d133dc7e77 bitbake: asyncrpc: include parse_address from hashserv
Moving the code and related definitions from
hashserv/__init__.py to asyncrpc/client.py,
allowing this function to be used in other asyncrpc clients.

(Bitbake rev: b67bb05e431414866b8e8c6a4c88d20b9cdb44a3)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Suggested-by: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Rob Woolley
c0cd7a6d3b bitbake: wget: Make wget --passive-ftp option conditional on ftp/ftps
Fedora 40 introduces wget2 as a drop-in replacement for wget.  This
rewrite does not currently have support for FTP.  This causes
the wget fetcher to fail complaining about an unrecognized option.

Making --passive-ftp conditional based on the protocol used in
the SRC_URI limits the scope of the problem.  It also gives us
an opportunity to build the older wget as a host tool.

(Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a)

Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-11 08:47:07 +01:00
Joshua Watt
978206fed4 bitbake: siggen: Add support for hashserve credentials
Adds support for hashserver credentials to be specified in the
SignatureGenerator

(Bitbake rev: 741bef3755fde7bae1386aad575ea704d9fe0969)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-23 10:19:00 +00:00
Derek Erdmann
825055e83e bitbake: fetch2/git: Install Git LFS in local repository config
Git uses a lock file to prevent concurrent modifications to the global
config, so if unpack tasks for different recipes try to run "git lfs
install" simultaneously the operation can fail:

    error: could not lock config file /home/build/.gitconfig: File exists exit status 255
    Run `git lfs install --force` to reset Git configuration.

Adding "--local" sets the smudge and clean filters in the local
repository's config instead of modifying the user's global config.

(Bitbake rev: 328ca4de8422be514fa0d0c9e3cfd36bb9d3e9a7)

Signed-off-by: Derek Erdmann <derek.erdmann@sonos.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:46 +00:00
Felix Moessbauer
9504df41f9 bitbake: utils: better estimate number of available cpus
When running in a cgroup which is limited to a subset of cpus (via
cpuset.cpus), cpu_count() should return the number of cpus that can be
used instead of the number of cpus the system has.

This also aligns the semantics with nproc.

(Bitbake rev: a029bfe96c6542f178720c72a772b7ede9898118)

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Viswanath Kraleti
3a97792820 bitbake: fetch2: Fix misleading "no output" msg
When a command is run with a non-null log, errors are only output to the
log and are not returned in the exception. In that case direct users to
that logfile instead of telling the command had no output.

(Bitbake rev: 944fe0a77932a5559e01ae6035c4bffa5185ea6a)

Signed-off-by: Viswanath Kraleti <quic_vkraleti@quicinc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Ross Burton
e20ee877ed bitbake: fetch2: handle URIs with single-valued query parameters
Whilst typically the URI query is a list of key-value pairs, that's not
actually required by the URI specification.

For example:  http://example.com/foo?bar is a valid query, but this will
result in the fetcher raising an exception:

  File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__
    self.query = self._param_str_split(urlp.query, "&")
  File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split
    for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]:
ValueError: not enough values to unpack (expected 2, got 1)

In this case the query is just "bar", but the fetcher is trying to split
it into a key-value pair.

The URI object exposes the parsed query explicitly as a dictionary of
key-value pairs, so we have to be a little creative here: if a value is
None then it isn't a key-value pair, but a bare key.

Fix this by handling elements without the deliminator in _param_str_split()
(by assigning the value to None), and handle a None value when formatting
the query in _param_str_join().

(Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Peter A. Bigot
adb0ea98d1 bitbake: lib/bb: support NO_COLOR
Red text on a black background can make it difficult for people with
visual impairments to read the text of error messages.  Respect the
presence of a non-empty NO_COLOR environment variable as an indication
the user doesn't want colorization to be automatically enabled.

See: https://no-color.org/
(Bitbake rev: d9986c54cd3d67ed1f7cb636b17696c8d0d4db85)

Signed-off-by: Peter A. Bigot <pab@pabigot.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Philippe Rivest
2eeef2800b bitbake: bitbake: fetch2/git: Escape parentheses in git src name
FIXES [YOCTO #15404]

When using git fetcher on a repo with parentheses in its URL, the
invocation of the git clone command will fail. The clone directory
is not quoted thus the shell will return an error and won't execute
the command.

(Bitbake rev: b5624ee564)

Cc: Philippe Rivest <privest@genetec.com>

(Bitbake rev: 12f9738577934ad7c99f0770f1392a9d6050e7d6)

Signed-off-by: Philippe Rivest <technophil98@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
david d zuhn
ecb1248914 bitbake: bitbake-worker: allow '=' in environment variable values
Limit the split to key & value (2 items) instead of the n items one
can get if there are '=' characters in the value.

Fixes [YOCTO #15447]

(Bitbake rev: 86315961829ab1d137a0265cc246c44d3929e1fb)

Signed-off-by: david d zuhn <david.zuhn@sonos.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Alexander Kanavin
98d09d41fa bitbake: bitbake: improve descriptions of '-S printdiff'
(Bitbake rev: becf88c2250a47102c8d36ad8b40839e0bfa9137)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Michael Opdenacker
be10de3423 bitbake: utils: remove BB_ENV_PASSTHROUGH from preserved_envvars()
preserved_envvars() is used when the BB_ENV_PASSTHROUGH
environment variable is not set. Therefore, its code shouldn't
return this variable.

(Bitbake rev: 0a33b560233b983456178541603ab96fea22238b)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Robert P. J. Day <rpjday@crashcourse.ca>
Tested-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-05 12:25:08 +00:00
Richard Purdie
2689d8cf22 bitbake: fetch/git: Avoid clean upon failure
Currently when git fetches fail, it destroys all the existing local clone data.
For large repositories this can introduce long build delays when for example,
you just typo'd the git revision hash.

The git fetcher should be able to recover most directories so when the fetch is
for a git repo, avoid removing things unless clean is explicitly called
(e.g. a -c cleanall task).

(Bitbake rev: 1b3cd039fe19b24bd4be9a0202a98cdcbb0e9443)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-03 16:26:17 +00:00
Ulrich Ölmann
a32b45289b bitbake: taskexp_ncurses: fix execution example in introductory comment
Drop the ".py" file extension from the "-u" option's argument that has been
overlooked while applying the original patch (see [1]) to make the example work.
While at it sort the recipes' names consistently with respect to what is found
in the self-test examples below.

[1] https://lore.kernel.org/bitbake-devel/6f2645a7c4db2ae149d387544d2b94209cfed3f4.camel@linuxfoundation.org/

(Bitbake rev: 1f4d517b7a0389f78d1f791135f8dc9120e9912b)

Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-28 09:21:48 +00:00
Joshua Watt
be57fda542 bitbake: asyncrpc: Add support for server headers
Adds support for asyncrpc servers to send connection headers to clients
on connection. Since this is a breaking protocol change, clients must
opt-in to expect headers from the server, corresponding to a version
bump in the client protocol.

(Bitbake rev: 1cb2b8be6cc5269553f549285592e47b7d29db03)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-27 11:36:38 +00:00
Philip Lorenz
0e3bcc5103 bitbake: fetch2: Ensure that git LFS objects are available
The current implementation only performs a git lfs fetch alongside of a
regular git fetch. This causes issues when the downloaded revision is
already part of the fetched repository (e.g. because of moving back in
history or the updated revision already being part of the repository at
the time of the initial clone).

Fix this by explicitly checking whether the required LFS objects are
available in the downloade directory before confirming that a downloaded
repository is up-to-date.

This issue previously went unnoticed as git lfs would silently fetch the
missing objects during the `unpack` task. With network isolation turned
on, this no longer works, and unpacking fails.

(Bitbake rev: cfae1556bf671acec119a6c8bbc4b667a856b9ae)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-23 14:34:05 +00:00
Richard Purdie
08aa69a1fd bitbake: runqueue: Add support for BB_LOADFACTOR_MAX
Some ditros don't enable /proc/pressure and it tends to be those which we
see bitbake timeout issues on, seemingly as load gets too high and the bitbake
processes don't get scheduled in for minutes at a time.

Add support for stopping running extra tasks if the system load average goes
above a certain threshold by setting BB_LOADFACTOR_MAX.

The value used is scaled by CPU number, so a value of 1 would be when
the load average equals the number of cpu cores of the system, under one
only starts tasks when the load average is below the number of cores.

This means you can centrally set a value such as 1.5 which will then
scale correctly to different sized machines with differing numbers
of CPUs.

The pressure regulation is probably more accurate and responsive, however
our graphs do show singificant load spikes on some workers and this
patch is aimed at trying to avoid those.

Pressure regulation is used where available in preference to this load
factor regulation when both are set.

(Bitbake rev: 14a27306f6dceb4999c2804ccae5a09cc3d8dd49)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-23 14:34:05 +00:00
Alexander Kanavin
730bd999d6 bitbake: Revert "bitbake: wget.py: always use the custom user agent"
This reverts commit 987ab2a446.

There's been a report that this breaks downloads from Jfrog Artifactory
as self.user_agent is set to 'Mozilla Firefox', and when Artifactory
sees that, it sends a response tailored for showing in an interactive browser
(which in my opinion it has every right to).

If we're using wget, we should say so via wget's default; handling uncooperative
servers should be done on per-recipe basis, and ideally with tickets
to admins of those servers.

(Bitbake rev: feef5cd12e877f42ffcace168d44b0e6eb80a907)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-20 12:56:40 +00:00
Peter Kjellerstedt
d25f6b970f bitbake: fetch2/git: Make latest_versionstring extract tags with slashes correctly
Before, everything up to the last slash was removed when extracting the
names of the tags. This would lead to that a tag such as "agent/11.0.0"
would be incorrectly identified as "11.0.0", which would then be treated
as a correct version matching "^(?P<pver>\d+(\.\d+)+)".

(Bitbake rev: 8b21024b9966d5158ac4a77e87ffb935c2a57764)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Peter Kjellerstedt
9694fe1d68 bitbake: fetch2/git: A bit of clean-up of latest_versionstring()
This is mostly preparations for the next commit.

(Bitbake rev: dcd2abfde55cc59d9869a7c97620b6fc30a52047)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Peter Kjellerstedt
21603031df bitbake: tests/fetch: Make test_git_latest_versionstring support a max version
In some cases, the version found by latest_versionstring() may be higher
than the real version. Make it possible to specify a maximum version so
that this case can be detected.

(Bitbake rev: 9134d4777109bc78410c3e641420d9a78b485e33)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Chen Qi
272d5ae098 bitbake: fetch2/git.py: add comment in try_premirrors
The purpose of ensuring 'incremental fetch' is not easy to see from
the codes. So add comments to explain this.

(Bitbake rev: 8b890b87e30cd05ec92ed71ee3691a47b4d77253)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Chen Qi
09ecbcd307 bitbake: tests/fetch.py: add test case for using premirror in restricted network
We had issue when BB_ALLOWED_NETWORKS is set and `bitbake grpc-native -c fetch'
failed even with all contents available in PREMIRRORS.

Add a test case to ensure no regression in the future.

(Bitbake rev: 80c91ceb81b1cae203067af58d3f1fe9c619ae83)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Chen Qi
c984b03f02 bitbake: fetch2/git.py: fix a corner case in try_premirror
For gitsm recipes, it's possible that some URL is used more than once.
e.g.,
A -> B:rev1 (B is a submodule of A)
A -> C (C is a submodule of A)
C -> B:rev2 (B is a submodule of C)
A anc C are both using B as submodules, but on different revs.
Now if we have:
B:rev1 -> D
B:rev2 -> E
Then, the mirror will not be fully used.
Say we have all repo mirrors for A, B, C, D, E, then in theory it's not
necessary to reach out to any network for downloading. But it's not the
case. After downloading B(rev1) and its submodule D from mirrors, the fetch
process continues to download C, thus B(rev2) and E. Now it finds that B
needs an update because its submodule E needs an update. Of course this is
true because E is not downloaded yet. Now the problem comes to whether to
use mirror or not. The git.py defines try_premirror to return 'False' when
the ud.clonedir exists. As B has been cloned, the ud.clonedir exists and
try_mirror returns False, resulting in not using mirror and going to upstream
directly.

We can see that the mirrors are not fully used. This is usually not problem,
as the cost is only some network download. But in case the following two
settings are there, we get errors.
BB_NO_NETWORK = "0"
BB_ALLOWED_NETWORKS = "*.some.allowed.domain"
In such case, the gitsm recipe A will fail to fetch. Note that all contents
that A needs are in mirrors and now it's failing to fetch. This is unexpected.

Note that the different revs of the same repo in gitsm recipe is not the only
way to reveal this problem. For example, there might be a recipe call B that
uses B:rev3. Check the protobuf and grpc recipes as an example.

For now, we can use the following steps to reproduce this issue. To be clear,
the grpc recipe in meta-oe is now 1.60.0.
1. Add in local.conf:
   DL_DIR = "${TOPDIR}/downloads-premirror"
   bitbake grpc -c fetch
2. Comment out the DL_DIR setting in local.conf and add the following lines:
   PREMIRRORS:append = " \
     git://.*/.* git://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
     gitsm://.*/.* gitsm://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
   "
3. Set BB_NO_NETWORK = "1" and then 'bitbake grpc -c fetch'.
   This command succeeds and this shows that the premirror holds everything we need.
4. Add the following lines and then 'bitbake grpc -c fetch'.
   BB_NO_NETWORK = "0"
   BB_ALLOWED_NETWORKS = "*.some.domain"

After step 4, the error message is as below:
ERROR: grpc-1.60.0-r0 do_fetch: The URL: 'gitsm://github.com/protocolbuffers/protobuf.git;protocol=https;name=third_party/protobuf;subpath=third_party/protobuf;nobranch=1;lfs=True;bareclone=1;nobranch=1' is not trusted and cannot be used

This patch fixes this problem by handling this corner case, that is, if the URL is
not trusted from the settings of BB_NO_NETWORK and BB_ALLOWED_NETWORKS, then we should
try premirrors because trying to reach upstream is destined to fail.

(Bitbake rev: e1be272ad105b47d3131b77168d9172386993fcb)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Pavel Zhukov
de27ecd228 bitbake: tests/fetch.py: add multiple fetches test
Fetch from premirror few times to emulate multiple machines sharing same
clonedir or few rebuilds of the package from (pre)mirror
Regression test for [Yocto #15369]

(Bitbake rev: 7fcbac574c68f16b95ab7abb2874931d168d3c9e)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 12:52:30 +00:00
Pavel Zhukov
b2d0f31d24 bitbake: fetch2/git.py: Fetch mirror into HEAD
Fix the issue with using of (pre)mirror in case if clonedir exists but
outdated.
Previous version of the code fetched new mirror content into FETCH_HEAD
instead of branch which caused refetch from the upstream. Add new remote
add fetch from it instead so the ref can be found by "_contains_ref"

Fixes [Yocto #15369]

(Bitbake rev: 69588e2a5c7c200e47b02b2391498dcb72388bd2)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 12:52:30 +00:00
André Draszik
577e73606a bitbake: tests/fetch: support git's safe.bareRepository
When git is configured with safe.bareRepository=explicit [1], the
bitbake selftests fail miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].

It also seems that it is being rolled out in some environments as a
default for users.

In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.

The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.

Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: a45e14a7343e36101e45639931322e5649587f57)

Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 12:00:59 +00:00
André Draszik
b3d9663817 bitbake: fetch/git2: support git's safe.bareRepository
When git is configured with safe.bareRepository=explicit [1], the
bitbake git fetcher fails miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].

It also seems that it is being rolled out in some environments as a
default for users.

In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.

The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.

While at it, fix one open-coded invocation of git that wasn't using
ud.basecmd

Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: 5f3b1d8dc9ee70e707536bd75ee845b547440c97)

Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 12:00:59 +00:00
Richard Purdie
eca5708b87 bitbake: bitbake: Bump version to 2.7.3 for hashserv changes
(Bitbake rev: c1e0a0b6ddc9667c9d62319bd9ccd4eb8c64c2a6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:58:12 +00:00
Joshua Watt
61e184b3ed bitbake: siggen: Add parallel unihash exist API
Adds API to query if unihashes are known to the server in parallel

(Bitbake rev: 7e2479109b40ce82507f73b4f935903f7f79fb06)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:58:12 +00:00
Joshua Watt
e5056394e0 bitbake: siggen: Add parallel query API
Implements a new API called get_unihashes() that allows for querying
multiple unihashes in parallel.

The API is also reworked to make it easier for derived classes to
interface with the new API in a consistent manner. Instead of overriding
get_unihash() to add custom handling for local hash calculating (e.g.
caches) derived classes should now override get_cached_unihash(), and
return the local unihash or None if there isn't one.

(Bitbake rev: 6faf48c09a4003a31b32e450779fb8ac9cc5e946)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:58:12 +00:00
Joshua Watt
2406bd1055 bitbake: asyncrpc: Add Client Pool object
Adds an abstract base class that can be used to implement a pool of
client connections. The class implements a thread that runs an async
event loop, and allows derived classes to schedule work on the loop and
wait for the work to be finished.

(Bitbake rev: f113456417f9ac0a4b44b291a6e22ea8219c3a5f)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:58:12 +00:00
David Reyna
f62e4d81ec bitbake: taskexp_ncurses: ncurses version of taskexp.py
* Create an ncurses version of the GTK app "taskexp.py".
* Add these additional features:
  - Sort tasks in recipes by their dependency order
  - Print individual and/or recipe-wide dependencies to a file
  - Add a wild card filter
  - Show the target recipes on BOLD
* Provide a GUI self test
* Provide a non-ncurses self test for ptest

(Bitbake rev: f49bec66ad51c8cddeceafbbe2445c46e396ee8b)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-18 17:38:47 +00:00
Richard Purdie
cdb1cb2ed8 bitbake: runqueue: Improve setcene performance when encoutering many 'hard' dependencies
"bitbake world -n --setscene-only" shows poor performance as the numbers of tasks
increases. Analysys shows this is due to the "deferred" hard dependencies being
repeatedly processed which doesn't allow much forward progress in overall task
execution.

To avoid this, mark when it has been done and don't reprocess until dependencies
are updated. We have to be careful as we've seen bugs where these dependencies
aren't processed at the right time. They also have to be reproceseed during task
migration/rehashing so the code has to "self heal" the data structures.

(Bitbake rev: e5609bac06c17dabcf6286b47b1a3f19f5a1160f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-13 13:52:10 +00:00
Richard Purdie
71f55957f0 bitbake: runqueue: Optimise taskname lookups in next_buildable_task
A quick profile of bitbake world showed 147 million calls to taskname_from_tid().
The next_buildable_task function is performance senstive so move the call
inside the if block to reduce the number of calls and speed the code up.

(Bitbake rev: 8b332c16a7b6b85c5cbe1919dd8cae45fda6adf9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-13 13:52:10 +00:00